LLM temperature is not a creativity dial. Here's what it actually does.

The single most common mistake I see with LLM APIs is treating temperature like a creativity slider — crank it up for "creative" tasks, turn it down for "serious" ones. That mental model is wrong, and it quietly costs people correctness in production.

Temperature does exactly one thing: it reshapes the probability distribution the model samples the next token from. That's it. Understanding that one mechanic tells you exactly when to change it and — more often — when to leave it alone.

What actually happens under the hood

At each step, the model produces a raw score (a logit) for every token in its vocabulary. To turn those scores into probabilities, the API runs them through a softmax. Temperature divides the logits before the softmax:

p_i = softmax(z_i / T)

• T = 1.0 — the distribution is used as-is.
• T < 1.0 — dividing by a number less than 1 magnifies the gaps between logits. The high-probability tokens get more probable, the tail gets crushed. The distribution gets sharper.
• T > 1.0 — the gaps shrink. Unlikely tokens become relatively more likely. The distribution gets flatter.
• T → 0 — the sharpest token dominates completely. Sampling collapses to "always pick the most likely token" (greedy decoding / argmax).

A quick worked example. Say three candidate tokens have logits [2.0, 1.0, 0.1]:

Same model, same logits, same prompt. Low temperature concentrates the mass on the top candidate; high temperature spreads it out and gives the long-tail tokens a real chance of being picked. "Creativity" is a side effect of that spread — the model isn't having better ideas, it's just being allowed to wander further down its own ranked list.

Why "creativity dial" is the wrong model

Two reasons it misleads you:

1. High temperature doesn't add anything the model doesn't already believe. It can only redistribute probability across tokens the model already scored. If the good next word isn't in the top of the distribution, raising temperature won't summon it — it just makes the mediocre candidates more likely to get picked. Past a point, higher temperature doesn't read as "creative," it reads as incoherent.

2. Low temperature isn't "less capable." A model at T = 0 is not dumber. It's giving you its single most-confident continuation every time. For anything with a correct answer, that's exactly what you want.

The rule I actually use

Default your agent tasks to temperature = 0. Turn it up only when variance is the point.

Use 0 for:

• Classification and routing ("which of these buckets?")
• Extraction and structured output (JSON, field parsing)
• Anything with tool calls or function selection
• Retrieval-augmented answers where the model should stick to the context
• Any step whose output another step depends on

Use 0.7–1.0 for:

• Open-ended generation — marketing copy, brainstorming, naming
• Anything where you'll sample several candidates and pick the best
• Cases where identical inputs producing identical outputs is a downside (e.g. you don't want the same three ideas every time)

I almost never go above 1.0 in production. Beyond that the failure mode isn't "more creative," it's "starts producing grammatically fine but semantically broken text."

The gotcha nobody warns you about

temperature = 0 is not a guarantee of identical output. People assume "temperature 0 = deterministic" and then file bug reports when two identical calls diverge. Temperature 0 removes sampling randomness, but other sources remain:

• Floating-point non-associativity across different batch sizes on the server
• Mixture-of-experts routing that depends on what else is in the batch
• Backend and model-version changes between calls

So T = 0 gets you stable and most-likely, not bit-for-bit reproducible. If you truly need reproducibility, pin the model version and use a fixed seed if the provider exposes one — and even then, treat it as best-effort.

One more: temperature and top_p are not the same knob

top_p (nucleus sampling) is a different mechanism — it truncates the distribution to the smallest set of tokens whose cumulative probability exceeds p, then samples from just those. Temperature reshapes; top_p clips. Tuning both at once makes their effects hard to reason about. Pick one to vary and leave the other at its default. In practice I move temperature and leave top_p at 1.0.

The takeaway

Temperature is a knob on the shape of a probability distribution, not a dial for how clever the model is. Once you see it that way, the decision gets simple: if the task has a right answer, run it at 0 and stop rolling dice on your own correctness. Turn it up only when you actually want the model to give you different answers to the same question.

Most agent pipelines are a chain of tasks that each have a right answer. Most of them should be running at 0.