Classic Machine Learning Through the Eyes of an SRE — Part 1
For years, every Monday morning started the same way.
Delivery managers would sit down, review a portfolio of projects, and instinctively rank which ones needed attention first.
Nobody called it machine learning.
They looked at dozens of signals: velocity slowing down, customer escalations increasing, risks piling up, and that usually proactive project manager suddenly going quiet.
Somehow all of those signals became a single mental estimate: "This project feels risky."
Looking back, they were doing something surprisingly similar to logistic regression. Not mathematically. Conceptually. They were combining signals, estimating the probability of something going wrong, and then deciding where to act.
The important part is that they made the decision—not the probability.
The bet the algorithm makes
Obviously, logistic regression is far more rigorous than human intuition. But the mental model is surprisingly similar: combine signals, estimate a probability, then decide what to do next.
Underneath, it's the same weighted-sum core as linear regression, squashed through a sigmoid so the output lands between 0 and 1. Its bet about your world is that risk rises smoothly with the signals — no cliffs, no chunky exceptions. For delivery risk, that's mostly right: a project with slightly worse velocity is slightly riskier.
One mechanical detail worth knowing: unlike linear regression, there's no closed-form solution here. You iterate with gradient descent until it converges. This is the first algorithm where "training" literally means guess, measure, adjust, repeat.
The threshold was never in the model
A logistic regression model might tell you there's a 73% chance of an SLA breach.
It never tells you to escalate.
That decision belongs to the business. The threshold isn't part of the algorithm. It's a product decision. An operational decision. Sometimes even a financial decision.
This is one of the most common misunderstandings I see. Teams accept the default 0.5 cutoff as if it were a property of mathematics. It isn't. If a missed breach costs ten times what a false alarm costs, 0.5 is the wrong line — and no amount of model tuning will tell you that.
Log loss and the cost of confidence
Log loss also changed how I think about confidence.
It doesn't just penalize mistakes. It penalizes being confident and wrong.
That reminded me of production systems. A noisy alert is frustrating. A dashboard showing green while customers are down? That's the kind of mistake nobody forgets.
Whether it's monitoring or machine learning, confidence attached to the wrong answer is usually more dangerous than uncertainty.
Two mistakes I made while learning this
One mistake I caught myself making was explaining logistic regression using mean squared error. That's linear regression. Logistic regression uses log loss.
I also used to say a train/test split prevents overfitting. It doesn't. It helps detect overfitting. Regularization is what helps reduce it.
Tiny wording differences. Completely different mental models.
When the bet is simply wrong
If the model is consistently wrong for one segment of your data, retraining alone often won't help — that's the smooth-line assumption failing against a boxy reality. If the relationship is inherently nonlinear, you may need different features, or a different class of model altogether.
That's a design decision, not a tuning problem.
The SRE lens
After spending a decade in SRE and DevOps, I've started looking at machine learning models the same way I look at monitoring systems.
The prediction isn't the product. The operational decision is.
A probability without a response plan is just another number on a dashboard.
The model computes. People—and increasingly AI systems—still have to decide what happens next.
That's where the real engineering begins.
Production takeaway
Probabilities don't create business value. Operational decisions do. Define what happens at high scores, at low scores, and in the uncertain middle before you ship the model.
Common interview mistake
Confusing probability with classification, or assuming a 0.5 threshold is always appropriate.
Where I'd use this in a real production system
SLA breach prediction · incident escalation risk · ticket priority prediction · customer churn probability · fraud risk scoring.
Part 1 of "Classic Machine Learning Through the Eyes of an SRE" — a decade in production, now moving into AI platform engineering, documenting it in public. Next: decision trees, the first algorithm that isn't trained at all.
Top comments (0)