Machine Learning has a number that everyone loves to see: accuracy.
You train your model, run the evaluation, and suddenly you see:
Accuracy: 99% 🎉
It feels like you built an amazing AI system.
But here’s the problem:
A 99% accurate model can still be practically useless.
The Accuracy Trap
Imagine we are building a Machine Learning model to detect fraudulent transactions.
Suppose we have 10,000 transactions:
9,900 are normal
100 are fraudulent
Now imagine our model simply predicts:
“Every transaction is normal.”
The model correctly predicts 9,900 transactions.
That gives us:
99% accuracy.
Sounds impressive, right?
But the model detected:
0 fraudulent transactions.
For a fraud detection system, that model is basically useless.
This is why accuracy should never be the only metric we look at.
So What Should We Measure?
Depending on the problem, we should also consider:
Precision
Precision answers:
“When the model predicts positive, how often is it actually correct?”
Recall
Recall answers:
“Of all the actual positive cases, how many did the model find?”
F1 Score
F1-score combines precision and recall into a single metric.
It becomes especially useful when dealing with imbalanced datasets.
The Bigger Problem: Real-World Data
Even if your model performs extremely well on your test dataset, it can still fail after deployment.
Why?
Because real-world data is rarely identical to training data.
For example, imagine training a model using data collected in 2024.
Then you deploy it in 2026.
User behavior may have changed.
New products may exist.
New attack patterns may appear.
The environment may be different.
This is often related to what we call distribution shift or data drift.
Your model hasn't necessarily become “stupid.”
The world around the model changed.
Another Silent Killer: Data Leakage
Data leakage happens when information that should not be available during training accidentally enters the training process.
This can make your model look incredibly powerful during evaluation.
You might see:
Training Accuracy: 99.8%
Validation Accuracy: 99.2%
Everything looks perfect.
Then production arrives...
Real-world performance: 71%
😐
The model wasn't necessarily production-ready.
Your evaluation process was giving you a misleading picture.
What I Now Check Before Trusting an ML Model
Instead of asking only:
“How accurate is my model?”
I try to ask:
Is my dataset representative?
Is the dataset balanced?
Could there be data leakage?
What are precision and recall?
What does the confusion matrix look like?
How does the model perform on unseen data?
What happens when real-world data changes?
Does the model actually solve the problem it was designed for?
These questions are often more important than the accuracy number itself.
The Lesson
Machine Learning isn't about creating a model that produces an impressive number.
It's about creating a system that continues to produce useful results when the data, environment, and users are different from what you saw during training.
A model with 99% accuracy isn't automatically a great model.
And a model with 90% accuracy isn't automatically a bad model.
The right metric depends on the problem.
So the next time you see:
99% Accuracy 🚀
Don't celebrate immediately.
Ask:
“99% of what?”
That question might be more important than the accuracy itself.
What do you think?
Have you ever built an ML model that performed great during testing but failed when you used real-world data?
I'd love to hear what happened and what you learned from it. 👇
Top comments (9)
The “99% of what?” question applies beyond ML metrics too.
A system can look extremely reliable when evaluated against the context it was given, but still fail when the environment around it changes.
I think AI coding has a similar problem. An agent can produce technically correct code and even pass its tests, while still making the wrong change because it doesn't have the project's current constraints, previous decisions, or the reason behind an existing pattern.
In both cases, the difficult part isn't only improving the model's accuracy — it's making sure the evaluation happens against the right and current context.
That makes me wonder: should AI systems be evaluated more on how they behave when the surrounding context changes, rather than only how well they perform on a fixed benchmark?
I completely agree. I think this points to a broader issue: benchmark accuracy measures performance under a defined context, while real-world reliability depends on how well a system adapts when that context changes.
For AI coding agents, this becomes especially important because correctness is multi-dimensional. Passing tests only proves that the generated code satisfies the tested behavior; it doesn't necessarily prove that the change respects the project's architecture, historical decisions, security constraints, performance requirements, or intended design.
A useful way to think about it is:
Model correctness ≠ Contextual correctness ≠ System correctness
For example, an agent might generate a perfectly valid implementation and pass every existing test, but still introduce a duplicated abstraction, violate an architectural convention, or break an implicit dependency that isn't represented in the test suite.
This suggests that AI evaluation should include context-shift testing: evaluate the same system under changing requirements, codebase state, dependencies, architectural constraints, and previously established decisions.
Instead of asking only:
we could also ask:
That could lead to metrics around context sensitivity, regression under context shift, constraint adherence, and decision consistency.
I think this is particularly important as AI systems move from generating isolated code snippets to making autonomous changes inside long-lived software systems. At that point, understanding the why behind existing code becomes just as important as generating technically correct code.
Exactly. I really like the idea of context-shift testing here.
It makes me think the evaluation shouldn't only ask whether an agent produced the correct output, but whether it continues to make the correct decision after the environment changes.
For example, if an architectural decision changes halfway through a project, an agent might still produce a technically valid solution based on the old decision. The output can pass the existing tests and still be wrong for the current state of the system.
So maybe one useful evaluation scenario would be:
same task + same model + different project context/state → measure whether the decision changes appropriately.
That could make things like constraint adherence and context sensitivity much more measurable rather than treating them as vague “AI quality” problems.
Curious how you would design a practical benchmark for this without making the evaluation itself too subjective?
I’d make it measurable through controlled context perturbation: keep the task and model fixed, change one architectural constraint at a time, and measure whether the agent’s decision changes when—and only when—the relevant context changes.
A practical benchmark could score:
Task correctness + constraint adherence + context sensitivity + unnecessary decision changes.
That turns “understands context” into something experimentally measurable rather than subjective. The key is to treat the context itself as the independent variable.
Thanks for reading! 🙌
I wrote this because I’ve noticed how easy it is to get excited when an ML model shows a high accuracy score, while overlooking what that number actually means.
I’m currently learning more about AI/ML and experimenting with different models and datasets. I’d love to hear from other developers:
Have you ever had a model that looked great during testing but performed poorly with real-world data?
Share your experience below — I’d love to learn from it! 🤖💻
This kind of hypothetical doesn't really hold up. A model needs proper training data and correct configuration — without that, you can "prove" anything, including 100% failure. A well-configured model trained on representative, balanced data will actually learn to spot fraud patterns. The "always predict normal" example isn't a real model, it's a strawman. The real discussion should be about how to configure and train properly, not about a model that was never built right in the first place.
I agree that proper training data and configuration are essential. However, my point was not that a properly trained model will always predict “normal.” The example was intended to illustrate how class imbalance and evaluation metrics can create misleadingly high accuracy, even when the model performs poorly on the minority fraud class.
Even with representative data and a well-configured model, fraud detection still requires careful handling of class imbalance, precision/recall, F1-score, ROC-AUC/PR-AUC, threshold tuning, and false-negative costs.
So I agree that a strawman model isn't useful—but the underlying issue is very real: a model can appear highly accurate while still failing to detect a significant portion of actual fraud cases.
The model must also have clear criteria for classifying transactions as fraud or not. An AI model isn’t a silver bullet for everything; you can’t just tell it, “Yes, detect fraud, and make sure there are zero errors.” You don’t have to tell it exactly what constitutes fraud—for example, 10 transactions within 10 minutes totaling more than 100,000 euros—and then the AI, i.e., a regular program, should find exactly the data you want and filter it out from among many other transactions.
Absolutely agree. I think the key distinction here is between defining the detection objective and learning the decision boundary.
An AI system cannot simply be given “detect fraud with zero errors” as an objective. The system needs a well-defined target, relevant features, an appropriate loss function, and an operational definition of what constitutes fraud.
Your example of:
10 transactions within 10 minutes + total value > €100,000is a deterministic rule and can be implemented directly with conventional logic. There is no reason to use ML for something where the business rule is already explicitly known.
Where ML becomes more useful is when the fraud patterns are not completely known in advance. For example, the model might learn that an unusual combination of transaction velocity, merchant behavior, account history, device characteristics, location changes, and spending patterns is associated with fraudulent activity.
In practice, I think the strongest architecture is often hybrid:
Explicit rules → known/high-confidence patterns
ML model → unknown or complex patterns
Human/risk layer → ambiguous cases and high-impact decisions
And the evaluation should reflect the real cost of errors. A false positive that blocks a legitimate transaction and a false negative that allows fraud are not necessarily equally expensive.
So instead of optimizing only for accuracy, we should consider metrics such as precision, recall, false-positive rate, false-negative rate, expected financial loss, and calibration.
The important question isn't simply:
It's:
That's where defining the problem correctly becomes just as important as choosing the model.