The Model Was 99% Confident and Completely Wrong

The model had been trained, exported to ONNX, and was returning predictions in about seven milliseconds.

Then I gave it this:

"nataka kuuza magunia 20 ya viazi gikomba"

"I want to sell 20 bags of potatoes at Gikomba."

It predicted:

pledge

Confidence:

99%.

The correct intent was:

sale_listing

That was the moment the project became interesting.

I am relatively new to machine learning. I built this classifier with AI coding agents doing most of the implementation. They wrote much of the training pipeline, synthetic data generators, serving stack, cloud training configuration, and parts of the analysis tooling.

Code arrived much faster than understanding.

That changed what I spent my time doing.

Instead of spending most of my time asking:

How do I implement this?

I increasingly found myself asking:

What would convince me that this is actually working?

That question uncovered a broken quantized model, a dataset shortcut, a leaky evaluation split, a taxonomy mismatch between the model and the application, and thousands of supposedly "real" messages that turned out to be load-test traffic.

The classifier is the thing I built.

Verification is the story.

I didn't need a better model

The system I was working on has a WhatsApp assistant that talks to food traders.

When a trader sends a free-form message, the application currently sends that message to Gemini 2.5 Flash. Gemini determines what the trader is trying to do and extracts the relevant information.

It works.

The problem was architectural rather than functional.

Every message that needed intent routing depended on a relatively large general-purpose model. That introduces latency and another dependency into a path where the answer is usually one of a finite set of intents.

So I started exploring a smaller encoder model that could run locally on CPU.

The idea was simple:

Use a small classifier for the common case. Fall back to Gemini when it isn't confident enough.

This wasn't primarily a cost optimization. The attraction was latency and resilience.

If the classifier could answer in single-digit milliseconds, we could potentially avoid paying the latency of a general-purpose model for every message while retaining Gemini as the fallback.

That sounds straightforward.

It wasn't.

First, I tried not training anything

Before generating a dataset, I tried zero-shot classification.

The idea was to give an NLI model the trader's message and a list of possible intent descriptions, then ask it to choose the most likely one.

It was useful as a baseline.

It was also clearly not good enough.

Some predictions were reasonable. Others were not. On a small benchmark, several messages landed on the wrong intent.

Then I tried something more interesting.

I tested a quantized model from a model hub.

It returned valid JSON.

It looked like a working classifier.

It wasn't.

Its predictions were effectively neutral: the same class kept winning with almost no meaningful separation between the logits.

This was a useful lesson because nothing about the output looked obviously broken.

The API worked.

The JSON was valid.

The model loaded.

The prediction looked plausible.

The model was wrong.

So I started putting acceptance tests around the model rather than trusting the existence of an output.

For model exports, I wanted to check things like:

Does the exported model produce the same predictions?
How different are the logits?
Does accuracy remain intact?
What happens to latency?
Does quantization actually preserve the behavior I care about?

My own int8 export improved substantially through several attempts:

72% → 79% → 82% → 94.4% → 95.6% agreement

But my acceptance bar was 98%.

So I didn't ship it.

The fp32 ONNX model matched the PyTorch reference and ran at roughly 7 ms on CPU.

That was good enough.

The lesson wasn't "ONNX is great."

It was:

An optimization isn't an optimization until you measure what it preserved.

Then I had to create a dataset

The next problem was data.

I didn't have a clean labelled dataset of trader conversations ready to train on.

So I generated one.

Using Gemini.

Which meant I was asking the same general family of model I was trying to reduce dependency on to invent examples for the smaller model.

There is something slightly uncomfortable about that.

The first dataset had 1,386 examples across 33 intents and seven language/register categories.

The first model was mediocre.

Then I changed the dataset.

Instead of asking the generator to produce arbitrary examples for each class, I started giving it things that should make classification harder:

similar intents,
the same products,
the same quantities,
realistic vocabulary,
previous bot messages,
deliberately damaged WhatsApp-style text,
and examples from languages and registers used by the application.

The second dataset grew to 9,751 examples.

The held-out score improved from roughly 0.70 to 0.86, and then to about 0.916 in the later 33-label experiment.

That looked like progress.

But there was a problem hiding inside the improvement.

Synthetic data doesn't just create examples

Consider the potato message.

The generator understands that potatoes can appear in many contexts.

A trader might be buying them.

Selling them.

Pledging them.

Asking about their price.

Asking about availability.

Asking about a market.

The model needs to learn the difference.

So I deliberately introduced contrastive examples.

If the target intent was sale_listing, I asked the generator to produce examples for that intent alongside examples for several confusable intents.

Same product.

Similar quantity.

Similar language.

Different meaning.

That helped.

But it also exposed something deeper.

When you generate your training data, you aren't simply producing more observations.

You're defining the world the model gets to see.

The model isn't learning "how traders talk."

It's learning patterns in the examples I gave it.

And those examples came from another model.

So synthetic data does not only give you more examples.

It gives you more examples of the generator's worldview.

The 99% prediction

That brings me back to the potatoes.

After improving the dataset, I tested a message equivalent to:

"I want to sell 20 bags of potatoes at Gikomba."

The classifier predicted:

pledge

with probability:

0.99

This was particularly interesting because the previous version had actually classified the message differently.

After the dataset changes, the model became more confident.

It also became more wrong.

The correct sale_listing prediction had been around 0.50 in the comparison I was running.

This is the kind of result that makes a model dangerous to trust casually.

A prediction of 51% gives you a reason to hesitate.

A prediction of 99% feels authoritative.

But the number doesn't mean what you instinctively want it to mean.

A softmax probability isn't a guarantee that the model is correct.

And in this case, the model had found a shortcut.

It wasn't confused.

The dataset had taught it the wrong shortcut.

That distinction matters.

If I had only looked at aggregate accuracy, I might have concluded that the new dataset was better.

The model had improved on the benchmark.

The benchmark was telling me something real.

It just wasn't telling me everything I needed to know.

Then I found another shortcut

The classifier doesn't always have enough information in the trader's message alone.

Consider:

"yes"

Or:

"20 bags"

Those messages are nearly impossible to classify without knowing what the bot asked immediately before them.

So I changed the input from:

trader message

to:

previous bot message + trader message

The model now receives a sentence pair.

For example:

Bot: "Do you want to sell potatoes?"

Trader: "Yes."

The word "yes" isn't the intent.

The conversation is.

This made the architecture more useful.

It also created a new data problem.

One of the intents, follow_up, was disproportionately associated with a particular kind of previous message.

About 94% of its examples had question-shaped previous turns.

Other intents had that pattern much less consistently.

The model could therefore learn an extremely cheap rule:

If the previous message looks like a question, this is probably follow_up.

It didn't need to understand the conversation.

It just needed to recognize the shape of the previous turn.

So I changed the generation rules in the next dataset version.

The fix wasn't "train harder."

It was:

change what the model is allowed to learn from.

That is a recurring theme in this project.

Then I discovered the classifier wasn't connected to most of the application

This was probably my favorite failure.

At some point I stopped looking at the model and started reading the application code.

The original intent enum had grown from 33 labels to 35.

I audited the actual routing code.

There were roughly 45 destinations or branches that the existing intent taxonomy couldn't cleanly represent.

Worse, around 11 of the 35 intent kinds weren't handled by the main switch that consumed the classifier output.

There was also a followUpNeeded value that was being produced but not actually consumed in the relevant routing path.

And there was another three-tier detector sitting in the codebase that, as far as the current application was concerned, had no importers.

The model was getting better.

The application didn't necessarily have anywhere useful to send the result.

So the taxonomy eventually grew to 74 labels.

This changed how I thought about the project.

I had originally thought I was building a classifier.

I was actually changing a piece of a larger decision system.

The model's output only matters if:

the taxonomy represents the application's real decisions,
the application actually consumes those decisions,
the routing is correct,
and the model can distinguish the decisions reliably.

A model can be technically excellent and operationally irrelevant.

The dataset had another problem

There was another number I initially liked:

0.916

It looked good.

The later evaluation used 8,307 training examples and 1,444 evaluation examples.

Across epochs, the score climbed roughly:

0.787 → 0.876 → 0.904 → 0.917 → 0.916

That looks like a healthy training curve.

But I eventually inspected the split itself.

A substantial number of evaluation examples had near-duplicates in the training set.

In one audit, 435 of the 1,444 evaluation rows had a near-duplicate on the training side.

About 30%.

That makes the headline score much less interesting.

If the evaluation set contains variations of things the model has effectively already seen, the benchmark is answering a weaker question than I thought.

So I changed the process.

The correct order is closer to:

split first → augment only the training side → evaluate on untouched examples

rather than:

generate → augment → split

That sounds like a small implementation detail.

It isn't.

The order in which you construct an experiment can determine whether the experiment means anything.

There were 25,327 "real" messages

At one point I found a local chat_messages table containing:

25,327 rows.

That sounded promising.

I thought I might have a real corpus.

I investigated it.

There were only 123 distinct idle-stage messages, spread across about 720 sessions from a load-test day.

The messages came from scripted personas.

They weren't a representative sample of actual trader conversations.

So:

25,327 rows of data

became:

no real trader messages used for training or evaluation.

This was another case where the first number was technically true and practically misleading.

And it reinforced something I was beginning to understand:

Data provenance matters more than data volume.

A million synthetic examples are not automatically more valuable than a thousand carefully labelled production examples.

Training was the easy part

My laptop is a MacBook with 16 GB of memory.

The model I used, AfroXLMR, is around 270 million parameters.

Training locally was possible up to a point.

Then memory became the bottleneck.

I reduced the batch size and used gradient accumulation.

It initially worked.

Then the machine started thrashing.

A training step that had taken around 1.2 seconds eventually approached 10 seconds.

So I stopped trying to make the laptop do a job it wasn't suited for.

I moved training to Vertex AI.

That introduced an entirely different class of problems.

The first cloud training attempt failed because the selected PyTorch image wasn't compatible with the Transformers version I was using.

Some gcloud flags I assumed existed didn't exist in the form I expected.

One duplicated training script couldn't import the project correctly.

At one point a hardcoded label list in the cloud job drifted from the actual taxonomy.

These weren't sophisticated machine-learning problems.

They were engineering problems.

And AI coding agents were involved in several of them.

That's worth emphasizing because AI-assisted development can create a strange illusion of progress.

You can have a repository full of code very quickly.

You can have Docker files.

Training scripts.

Cloud configuration.

FastAPI endpoints.

Export scripts.

Benchmarking tools.

Documentation.

Everything can look finished.

And still be wrong.

AI can write the cloud plumbing. It cannot remove the need to read the error message.

The final training run completed in about 259 seconds on an L4 GPU.

The infrastructure worked.

But getting there required reading failures rather than asking the agent to simply try again.

What the AI actually did

The AI coding agents were extremely useful.

They accelerated almost every mechanical part of the project.

They helped scaffold:

the training pipeline,
dataset generators,
cloud training jobs,
model export,
FastAPI serving,
benchmarking,
documentation,
and parts of the application routing audit.

They also made mistakes.

Among them:

incompatible cloud images,
nonexistent CLI flags,
stale hardcoded labels,
shell quoting bugs,
an incorrect Python environment,
and a generator that hung without a timeout and lost thousands of generated rows held only in memory.

None of that makes the agents "bad."

It changes what I expect from them.

I don't think the valuable distinction is:

AI writes code vs. humans write code.

The more useful distinction is:

Who is responsible for deciding whether the code is correct?

The agent can generate ten implementations.

It can explain why each one should work.

It can confidently tell you that the cloud configuration is valid.

That doesn't make any of those things true.

The human's role moves upward.

Less typing.

More auditing.

Less implementation trivia.

More experimental design.

Less:

"How do I build this?"

"What evidence would make me change my mind?"

So is the model good?

I don't know yet.

And that's the honest answer.

The 0.916 held-out accuracy is a useful measurement.

It is not a production accuracy claim.

The current evaluation has several limitations.

The training data is synthetic.

The evaluation set had leakage problems that I had to fix in the data-generation pipeline.

There is no permanently untouched real-world test set.

The classes are imbalanced. One label had 573 generated examples while another had 170.

The weakest per-label recall was around 0.69.

The softmax confidence has not been properly calibrated on real labelled traffic.

And the application recognizes language and register patterns, including Kikuyu and Sheng, that aren't fully represented in the generated training data.

I also need better evaluation metrics than a single accuracy number.

Macro-F1.

Per-intent precision and recall.

Confusion matrices.

Calibration.

Performance on real production messages.

Most importantly, I need to know what happens when the model sees language it wasn't trained to imitate.

That is the experiment that matters.

What I would do next

The next version of this project is much less glamorous.

I want to export a representative sample of production messages.

Then I want humans to label them.

Not thousands at first.

A few hundred carefully selected examples would already tell me far more than another 10,000 synthetic messages.

I'd want people who actually understand the language and conversational patterns involved, particularly for Swahili and Hausa.

I'd keep a real test set permanently outside training.

Then:

Fix the split and augmentation pipeline.
Train without contaminating evaluation data.
Measure macro-F1 and per-intent confusion.
Test on real trader messages.
Calibrate the confidence threshold.
Finish routing the expanded 74-label taxonomy.
Run the classifier beside Gemini in shadow mode.
Compare predictions before allowing the classifier to make production decisions.

Only then would I decide whether the cascade is actually useful.

And I wouldn't be surprised if the answer is no.

That would still make the project successful.

The thing I was actually learning

When I started this, I thought I was learning machine learning.

I was.

But that wasn't the most important thing.

I learned what supervised classification is.

I learned about transfer learning, embeddings, sentence-pair classification, stratified splits, class imbalance, quantization, inference, calibration, and model serving.

Those things matter.

But the more important lesson was about engineering judgment.

When implementation becomes cheap, verification becomes scarce.

AI made it possible for me to go from:

"I want to build a classifier"

to:

"Here is a trained model, an evaluation script, a cloud job, an ONNX export, and a FastAPI endpoint"

very quickly.

That was useful.

It was also dangerous.

Because every stage produced artifacts that looked like progress.

The model had a score.

The API returned JSON.

The cloud job completed.

The ONNX model ran in seven milliseconds.

The dataset had thousands of rows.

The repository had documentation.

And yet:

the quantized model was broken,
the model learned a shortcut,
the evaluation split leaked,
the taxonomy didn't match the application,
and the "real" dataset wasn't real.

None of those failures were caused by a lack of code.

They were caused by asking the wrong questions, or asking them too late.

The most valuable skill I practiced wasn't training a model.

It was learning to ask:

What would I expect to see if this were actually working?

Then checking.

The classifier may eventually replace some of the Gemini calls.

It may not.

I still don't know whether a model trained entirely on synthetic trader language will hold up when real traders start using it.

That's the next experiment.

And this time, I know what I need to measure.