Recently, I've added a bunch of hype-monsters to my AI Werewolf:
Kimi K3
Qwen 3.8 Max, Qwen 3.7 Plus, Qwen 3.7 Flash
MiniMax M3
Plus the ones I...
For further actions, you may consider blocking this person and/or reporting abuse
The Mistral row is the real headline. Seven runs from 2.9 to 35 seconds means any single measurement is a coin flip, and most benchmarks are just lucky coin flips. The same trap applies to evaluating people. One interview answer is one sample from a noisy distribution. That is why I prefer accumulating many small observations over one big snapshot. In Opportunity Skill the agent records impressions one at a time from real interactions and prunes the stale ones, because a moving average beats a single measurement. Letting the data kill your thesis is the whole craft.
After this little research, I started collecting metadata from each request-response. I'll have a good statistics over time for all models. It just haven't came up into my mind before.
Great honest benchmark — the latency gap on official Chinese APIs is real and rarely measured. The angle I'd push: latency is its own routing axis, not just cost. For your werewolf votes (tight turn, someone waiting) frontier wins; for async/batch the cheap Chinese models win big. Also worth isolating "model slow" from "provider-infra slow" — a gateway that streams and keeps connections warm can shave a lot off those 29s numbers without changing the model.
Yeah, this is fair, I'm not measuring just the models. Network latency and whatever the provider's servers are doing that hour is in every number there. Mistral Medium ran 2.9s to 35.0s on the same prompt with the same 57 to 109 output tokens every time, and that's nothing but their infrastructure having a day.
I do have some isolation though. DeepSeek V4 Flash over three runs wrote 990, then 1,415, then 2,275 output tokens, and the rate never moved off 73-76 tokens/s. The connection was steady the whole time, the model just decided to write twice as much on the third run. Haiku is the same story from the other side: it types faster than everything in my roster except Qwen 3.7 Flash, and it's still the slowest model I have, because it spends four thousand tokens to say four sentences.
Which doesn't leave a gateway much to shave, yeah. A warm connection saves a handshake against a 29-second call, and streaming moves the first token, not the last one. My votes come back as validated JSON, so the last one is the one I'm waiting for.
Appreciate you writing this up honestly — the latency gap on official Chinese APIs is exactly the kind of thing that gets buried under "just use a cheaper model" advice. What we kept seeing too: warm connections remove the handshake tax, streaming moves time-to-first-token not time-to-last, and a verified-JSON voting step is the only thing that makes the slow tail acceptable. None of it shows up in a headline benchmark. It's most of why we treat "landing AI in production" as an ops problem, not a model-picking problem.
Yeah, exactly. It's possible to tune the models up for a better speed, but the default settings lead to significant slowness. I also didn't expect to see what US models do to reply faster. Nobody is talking about this adaptive reasoning, but this is basically a suppressing. So their casual performance must be lower then in benchmarks where the reasoning in on max.
Thanks for pushing on this — the latency gap on the official China endpoints is real and almost never shows up in the headline numbers people quote. The parts that bit us hardest: the handshake tax disappears once connections are warm (connection pooling / keep-alive matters more than people admit), streaming helps time-to-first-token but not time-to-last so the tail stays ugly, and a verified-JSON voting step is what makes the slow tail tolerable in prod. None of that is visible in a benchmark screenshot. It's also why we treat "landing an LLM in production" as an ops problem first — routing, retries, and cache stability decide the real number, not the model pick.
Curious: did you see the gap shrink at all on the newer Qwen / DeepSeek revisions, or is it mostly stable across versions?
I see the gap widening because default settings for Kimi K3 and Qwen Max cause huge amount of reasoning and slowness. There must be a reason for that - maybe low thinking shows poor results for these models. Somehow Claude is doing ok with 200-300 output tokens. DeepSeek is a bit better - they put a lot of speed optimizations into their v4. And hype around them has calmed down - APIs are not overloaded anymore as it was with R1. Minimax is probably the fastest but the least stable - I'm struggling to get a valid JSON from it.
I expect everybody switch to adaptive reasoning in nearest future. And DeepSeek optimizations for attention will be copied by everybody. So things will get better, that's for sure.
I ran your exact test shape this evening (36k-char vote prompt, JSON vote, wall clock to last token, n=3 per config) against mostly the same models. I don't disagree with the conclusion — but I'd sharpen one line: it's not the token count doing the damage, it's the reasoning budget specifically. Same model, same prompt, only the thinking knob changed:
Wow, great data. Yes, you are right - this budget limit is the key control knob for most models. Chinese models like spending reasoning right up to the max budget, while GPT and Claude are more scrupulous about that. Interesting that Opus shows 0 reasoning in both cases - do you use
{type:'adaptive'}?Good question — it sent me down exactly the right rabbit hole, so this answer got longer than planned.
Short version: no, and then yes. The Opus arm originally went through OpenRouter (I had no Anthropic key on that machine at first), using OpenRouter's normalized reasoning field — enabled:false / effort:high. After your comment I also passed the native fields straight through: thinking: {type: "adaptive"} and a forced {type: "enabled", budget_tokens: 8000}. Everything came back 200 with zero reasoning tokens and completion_tokens in the 138–284 range on the 36k prompt — identical to a no-thinking profile. Since Anthropic bills thinking as output tokens, real thinking can't hide from completion_tokens even when a proxy hides the text. So through OpenRouter the knob never verifiably reaches the model. Useful instrument if you ever bench through a proxy: check completion_tokens before believing the reasoning knob.
Then I added an Anthropic key and did it properly. Two findings:
The Claude 5 surface is thinking.type.adaptive + output_config.effort — the API rejects {type: "enabled"} for Opus 5 outright. (Which retroactively proves OpenRouter transformed or dropped the config it happily accepted.) Measured first-hand across five configs — default, adaptive, adaptive+effort low, adaptive+effort high, disabled — Opus 5 emitted zero thinking blocks in all 15 runs: 4.1–5.1s medians, 107–226 output tokens, correct vote every time. So your "GPT and Claude are more scrupulous" is measurably right, and stronger than you put it: even explicitly offered high effort, Opus looks at this task and spends nothing.
Your Haiku 4.5 number also reproduces — but only one way. Default Haiku answers in ~2.4s with zero thinking, correct every time. With old-style forced thinking (enabled, 8k budget) it takes 18.3s median at 1,183–2,113 output tokens — squarely your 19.7–42s / 1,769–4,095 range. So the "too old to adapt" datapoint looks like forced thinking in the config rather than the model's age.
At this point every latency figure in the test — the Chinese arms, Opus, and Haiku — traces back to thinking configuration rather than serving speed or model vintage.
I agree. By "too old" I meant that Anthropic hasn't switched Haiku to this new adaptive thinking. They did it to Opus and Fable first, then to Sonnet in the latest update. Haiku is sitting without updates all forgotten for quite some time. This supposed to be the conclusion of the article, but maybe I messed this idea up and made it not so clear.
When I first added Qwen and MiniMax, then started playing with them, I was shocked how slow things are. I started writing about it and then I discovered this dependency on the reasoning tokens. And that Kimi and Qwen defaults are quite high. For Kimi, it's not that easy to disable thinking - it can only be dove via an undocumented argument. For Qwen the default cap is set to 4096. I have 1024 tokens max now, and I'll probably go down to 512 on all models.
And yeah, looks like my
budget_tokens: 1024for Haiku is not honored. I found this in docs: "The budget is a target rather than a strict cap. Actual token usage varies with the task, and Claude may stop reasoning well before the budget is exhausted; max_tokens remains the hard ceiling on total output." Interesting.that useful management of my time thanks you i learn a bunch worth reading
Thank you for reading