Write Down the Number Before You Know the Answer

You make predictions all day. "This refactor will cut the p99." "The bottleneck is the N+1, not the index." "This'll take two days." "That library will bite us in six months."

Some of those are right. You remember the ones that were spectacularly right, and the ones that were embarrassingly wrong. What you almost certainly don't have is the aggregate: of all the calls you were about 80% sure about, how many actually landed? Not anecdotally — measured.

I work in growth, where the same gap shows up as "this test will win" instead of "this index will help," and I've come to think it's the most underrated skill in any job that involves judgment under uncertainty. There's twenty years of research on it, it came out of an intelligence-community forecasting tournament, and the practical version costs about two minutes per prediction.

Here's what the research found and how to run it on yourself.

Measuring judgment instead of theorizing about it

In the early 2010s, IARPA — the intelligence community's advanced-research arm — ran a multi-year forecasting tournament pitting research teams against each other on real geopolitical and economic questions. Will a specific country default within six months? Will a particular ceasefire hold? Will a named leader remain in power through year-end?

Philip Tetlock's team, the Good Judgment Project, recruited thousands of ordinary volunteers with no special access to classified information and had them submit probability forecasts on hundreds of these questions over multiple years.

The result that made the project famous: a small subset of these ordinary volunteers — "superforecasters" — didn't just do well. They substantially and consistently outperformed both random chance and, in the published results, trained intelligence analysts with access to classified information on the same questions.

That's the finding that matters here. Good judgment under uncertainty isn't primarily about domain expertise or access to better information. It's a measurable, learnable discipline, separable from subject-matter knowledge — and the tournament identified exactly what the discipline consists of.

Which is good news if you've ever suspected the most confident person in the architecture review isn't reliably the most correct one.

The three habits that separated them

Breaking big questions into sub-questions with knowable base rates. Superforecasters rarely answered the headline question directly. Asked whether a leader would remain in power, they'd decompose it: how often do leaders in structurally similar situations get removed within a year? What's specifically different about this case, and how much should that shift the estimate?

The engineering translation is nearly one-to-one. "Will this migration go smoothly" invites a gut read. "How many of our last eight migrations of this size slipped, and what's different about this one" invites an actual calculation. You almost always have the base rate. It's in the repo history.

Updating in small, frequent increments — not big dramatic reversals. The best forecasters revised their estimates often, usually by a few points at a time, as new information arrived. They didn't sit on a confident call and then flip entirely when disconfirming evidence became undeniable.

This is the business-legible version of Bayesian updating: confidence should move a little with each new piece of evidence, continuously, rather than staying frozen until the evidence forces a reversal you can no longer avoid. If you've ever watched a team defend an architecture decision for two quarters and then abandon it in a single meeting, you've seen the failure mode.

Tracking their own calibration, explicitly, over time. This is the one with the most direct application and the one almost nobody outside forecasting research has adopted. Superforecasters didn't just make predictions — they scored themselves against outcomes using a Brier score, a standard measure of forecast accuracy that penalizes both overconfidence and underconfidence.

Someone who says "90% chance" should be right roughly nine times out of ten on similar calls — not seven, not ten. A Brier score, tracked over enough predictions, tells you whether your gut runs hot, cold, or well-calibrated. It's the only way to actually know rather than assume.

Why the absence of this log matters more than it looks like it should

Confidence and accuracy are two different things, and without a calibration log you only ever get feedback on the first one.

A person who is consistently, wrongly, 90% confident sounds exactly as convincing in the room as someone genuinely well-calibrated at 90% — right up until somebody tracks the record. Most organizations never track the record. So conviction gets rewarded as a proxy for judgment, with no actual check on whether the two are the same thing for this particular person on this particular type of call.

Individual wins and losses get remembered anecdotally. The aggregate pattern — was I actually right 80% of the time when I said I was 80% sure, or closer to 50% — never gets computed, because the log that would let you compute it was never kept.

The lightweight practice

You don't need a research program. You need one recurring habit cheap enough to stick.

A plain text file with one line per prediction is enough. Date, the claim, the number. That's the whole tool.

The payoff connects to how you size a bet in the first place: a hypothesis you logged at 55% confidence and one you logged at 90% shouldn't get the same resourcing commitment, and a running calibration score is what tells you whether your stated 90% has historically earned that trust.

The uncomfortable part

The likely result of running this for a year, for most people, is discovering their gut runs more overconfident than they'd assumed — particularly on the calls that felt most obvious going in.

That's not a flattering finding, which is exactly why almost nobody generates the data that would reveal it. Tetlock's forecasters weren't more naturally gifted than intelligence professionals with more information and more experience. They were more willing to keep score on themselves and adjust based on what the score showed, rather than what their conviction in the moment suggested.

That's a discipline, not a talent. It's available to anyone willing to write the number down before they know the answer.

FAQ

How is a Brier score actually calculated? It's the squared difference between your stated probability and the outcome (scored as 1 for "happened," 0 for "didn't"), averaged across all predictions — lower is better, and 0 means perfect calibration. You don't need to compute it precisely to get the benefit; even an informal quarterly "how often did my 70%-confidence calls hit" captures most of the value.

Isn't this just journaling with extra steps? The difference is the number and the review discipline. Journaling captures reasoning; a calibration log captures a falsifiable claim that gets checked against reality on a schedule. Most journaling never gets scored against outcomes, which is exactly the step that produces the learning.

What if I don't have enough major decisions to build a sample? Log smaller calls too — estimates, bug-cause guesses, minor predictions about team output. Superforecasters built calibration on volume across many modest questions, not a handful of huge ones. The skill transfers.

Does this replace statistical rigor? No — it operates one level up. Statistical testing tells you whether a specific result is likely real. Calibration tracking tells you whether your own pre-test intuition about which hypotheses will pan out is trustworthy, which affects how much weight to give your gut when deciding what to try in the first place.

How long before it's useful? Meaningful patterns usually need a few dozen logged predictions before the aggregate signal beats noise — roughly a couple of quarters of consistent logging. The point isn't a fast verdict; it's building the first real dataset you've ever had on your own judgment.

Bottom line

Tetlock's tournament didn't find that some people are naturally gifted forecasters and everyone else isn't. It found that good judgment is a trackable, improvable discipline, and the single habit that improves it fastest is one almost nobody practices: writing down a real confidence number before you know the answer, and checking it later.

It costs a few minutes per call and a quarterly review. What it buys is the only honest answer to the question every confident estimate implicitly asks: how much should anyone actually trust this gut, based on its track record rather than its tone.

---

If you start a calibration log this week, the first entry can be a prediction about whether you'll still be keeping it in three months. Write down the number.

I write about experimentation, judgment under uncertainty, and growth decisions at atticusli.com.