Databricks vs Snowflake (4/7): The 2021 Benchmark War — What the TPC-DS Fight Actually Proved

TL;DR — In November 2021 Databricks announced a genuinely TPC-audited 100TB TPC-DS world record, then in the same blog post attached an unaudited comparison claiming it was "2.7x faster and 12x better in terms of price performance" than Snowflake. Snowflake's co-founders answered ten days later with their own numbers and a refusal to keep playing; Databricks counter-rebutted three days after that. Nobody's competitive multiplier was ever adjudicated by a neutral party, and both sides quietly walked away from benchmark marketing afterward.

Three rounds in, the scoreboard is level — and every number this series has cited so far came out of a filing, a funding announcement, or an earnings deck. Today's came out of marketing departments, and that difference is the entire story.

The Nov 2021 benchmark war lasted thirteen days, produced three blog posts, and is still quoted by people who have merged two different claims into one. Untangling them is the useful part.

The concepts you need

TPC-DS. A standardized benchmark from the Transaction Processing Performance Council, a vendor-neutral industry body. It models a retail data warehouse and fixes 99 queries, a data generator, and rules for loading and concurrency — so two systems running it are at least answering the same questions.

Official vs derived runs. An official TPC-DS result is a full submission: a disclosure report, verification by a TPC-certified independent auditor, publication on TPC's leaderboard where rivals can attack the methodology. A derived benchmark borrows the schema and queries but skips the submission and the audit. Derived runs are legitimate engineering — just not comparable to official results, and TPC says so. Nearly every "we beat competitor X" chart in a vendor deck is derived.

QphDS and price-performance. The official metric, QphDS at a stated scale, is a composite throughput score — roughly queries per hour at that data size, folding in load time plus single- and multi-user runs. Higher is better. Results also carry a price-performance figure: cost divided by throughput. That's why performance and pricing marketing are the same argument.

What an audit proves. That a specific system, on a specific config, at a specific scale, on a specific date, produced the claimed number under the rules. It does not prove the config resembles anything you'd deploy, that the result generalizes, or anything about a product that wasn't in the run.

Day 0: the record that was real

On November 2, 2021, Databricks announced an official TPC-DS 100TB result: 32,941,245 QphDS, beating the prior record — Alibaba's custom system at 14,861,137 QphDS — by 2.2x. This part is not in dispute. It was, in Databricks' words, "formally audited and reviewed by the TPC council." A real benchmark body, a real audit, a real published number.

Snowflake does not appear in it at all. An official result is a system submitting against a spec, not a bake-off. To beat Alibaba you never have to touch a competitor's product.

Day 0, second half: the claim that wasn't audited

In the same post, Databricks cited a separate study run by the Barcelona Supercomputing Center (BSC) — worth stating precisely, because secondary coverage routinely mis-describes BSC as the auditor of the record. It wasn't. BSC ran the head-to-head, and the head-to-head was the unaudited half. Its finding: Databricks was "2.7x faster and 12x better in terms of price performance" than "a similarly sized Snowflake setup." A derived run, wearing the record's clothes.

Databricks said so itself, in its own footnote:

"The TPC does not audit or validate results of benchmarks derived from the TPC-DS and does not consider results of derived benchmarks to be comparable to published TPC-DS results."

Two claims, one post, one audited. The syndicated press release muddied it further: "2.7X faster and more than an order of magnitude cheaper on the same workload" — the "12x" lives on the databricks.com blog, not the wire copy. When a company's own two channels can't agree whether it's 12x or "more than 10x," that tells you something about precision.

Day 10: the founders answer

On November 12, 2021, Snowflake co-founders Benoît Dageville (President, Product) and Thierry Cruanes (CTO) published "Industry Benchmarks and Competing with Integrity." Two of the three people who designed the system wrote the rebuttal personally — not a routine competitive response.

Their argument had two halves. The principled half: this kind of comparison is "simply inconsistent with our core value of putting customers first." The numeric half: they ran it themselves and got 99 queries in 3,760 seconds on a 4XL warehouse at roughly $267 per run, plus a 5XL run at 2,597 seconds — more than two times faster, they said, than the performance Databricks had attributed to Snowflake. They also disputed the $1,791 Snowflake price/performance figure in Databricks' post.

Then they declined to keep going — the part most people forget.

Day 13: the counter-rebuttal

November 15, 2021: "Snowflake Claims Similar Price/Performance to Databricks, But Not So Fast!" Databricks pointedly did not accuse Snowflake of bad faith; it asked Snowflake to verify its results with the official TPC council instead. Its substantive objection was the data: Snowflake's dataset, it said, had been recreated two days after the November 2 announcement, and was "pre-baked" rather than the official TPC-DS dataset. On the official dataset, Databricks reported reproducing Snowflake at 7,276 seconds (best of three) to 10,085 seconds (first cold run) — roughly 1.9x slower than Snowflake's claim. It put Databricks SQL cost at $242 on the most expensive tier, or $146 with spot instances, and noted BSC's own re-measurement of Snowflake landed at 8,397 seconds.

Read that carefully: the disagreement was never about who ran faster. It's about which copy of the data was legitimate and who configures the other guy's system — where every vendor benchmark fight ends up.

The dueling claims, side by side

One row in five was audited, and it's the row that says nothing about Snowflake.

Why this fight was even legal: the DeWitt clause

Concept — why such a clause exists. A benchmark is only as honest as whoever configured the losing system. Give me your product, my hardware budget, my query selection and my tuning time, and I can publish a defensible-looking number that makes you look slow. The cheapest defense is to forbid publication — which kills the fair comparisons too.

For about four decades, database contracts carried what the industry calls a DeWitt clause — a term barring customers from publishing competitive benchmarks without vendor consent. Per press reports it traces to Larry Ellison's objection to a 1982 benchmark study by David DeWitt, and it is why "we can't publish those numbers" has been a normal sentence in database procurement. Both vendors relaxed or removed DeWitt-style restrictions around late 2021 — Databricks published a companion post, "Eliminating the DeWitt Clause for Database Benchmarking," on November 8, 2021, mid-fight.

The war was partly a demonstration of a freedom both had just granted — and within thirteen days, of why the clause survived forty years.

How it ended: it didn't

No neutral party ever adjudicated the multipliers. TPC audited the record and nothing else. The thing simply stopped. By August 2022, Dageville told an interviewer: "We've said from day one, we would never again participate in this really stupid benchmark war because it's not in the interest of customers." Snowflake never returned with follow-up numbers, and competitive TPC-DS marketing receded from both companies' messaging.

Note where the calendar lands: Snowflake's all-time-high close of $401.89 came on November 16, 2021, the day after the counter-rebuttal. The war was fought at the exact top of the market, and moved nothing.

Why benchmark numbers rarely survive contact with your bill

Concept — benchmark configs vs production. TPC-DS is 99 heavy analytical queries against a clean, freshly generated, statistics-perfect dataset on hardware sized for exactly that batch. Production is skewed data, late files, stale statistics, small repetitive dashboard queries, schedulers and humans colliding at 9am, warehouses idling at 3pm. Benchmarks reward sustained throughput; invoices are driven by concurrency, idle time and data layout.

Concept — consumption pricing turns every speed claim into a cost claim. Snowflake meters compute credits per second (60-second minimum), plus storage per compressed TB and egress. Databricks meters DBUs per second — and the cloud provider separately bills the underlying compute, storage and networking, which reporting puts at roughly 50–100% on top of the DBU line. So a cost figure is a function of tier, spot vs. on-demand, region, edition and idle time. Change one input and the multiplier moves more than the engine does.

Two measurement stacks, pros and cons

The deeper split is two theories of what counts as proof.

Databricks: benchmark-forward. Publish numbers, submit to TPC, remove the DeWitt clause, invite the fight.

• Pro — falsifiable. A published run has a config, a dataset, a metric; Snowflake attacked it within ten days. Claims you can attack beat claims you can't.
• Pro — a genuinely audited record exists. 32,941,245 QphDS at 100TB: real engineering, checked by an outside body.
• Pro — it made benchmarking legal again. Killing DeWitt-style terms lets customers publish, not just vendors.
• Con — the winner picks the config. "A similarly sized Snowflake setup" does enormous unexamined work, and the Nov 15 re-run was still Databricks configuring Snowflake.
• Con — marketing overreach on top of real work. The line everyone remembers, 2.7x/12x, is the half its own footnote disclaims.

Snowflake: real-workload. Decline the bake-off, point customers at a proof-of-concept on their own data.

• Pro — it matches how the product is bought. Spend is set by your dbt runs and BI concurrency, not 99 synthetic queries at 100TB, and a POC measures exactly that.
• Pro — the principle is defensible, and two co-founders signing it personally raised the cost of saying it.
• Con — unfalsifiable. "Our customers see great performance" can't be checked or reproduced from outside.
• Con — conveniently opaque. The vendor that declines measurement never publishes a number it can lose on.
• Con — the principle arrived with an asterisk. Snowflake did publish self-run figures when stung (3,760s, ~$267/run), on a dataset Databricks says was regenerated two days after the announcement, then declined to re-engage. Refusing benchmarks after posting your own number is weaker than refusing outright.

So benchmark it yourself. Take 10–20 of your heaviest queries — not TPC-DS. Use your data volume and skew, your concurrency including the 9am spike; report cold and warm runs separately; price it at your negotiated discount, plus the cloud bill under the DBUs. Measure the invoice, not the seconds — it's the only number indexed to you.

What to Learn From This

1. Separate the audited claim from the claim attached to it. One post carried a TPC-audited record and an unaudited multiplier — different things, only one checked. Read the footnote before quoting the headline.
2. Ask who configured the losing system. Every number here where a vendor tuned its rival's product — in both directions — is marketing, not measurement. That question disqualifies most competitive charts.
3. Interrogate the dataset, not just the runtime. Thirteen days ended arguing over which copy of the data was legitimate — generation, layout and freshness move results more than engines do.
4. Report cold and warm separately; never trust a lone best-of-N. 10,085s versus 7,276s is the same team, same system, same day. If a claim doesn't say which run it is, you don't know what it means.
5. Convert seconds into dollars at your own contract, including the cloud infrastructure billed on top of DBUs at roughly 50–100% more. Under consumption pricing, price-performance is the metric that matters.
6. Notice when a vendor stops publishing numbers. Both retired competitive benchmark marketing after 2021. Silence is positioning, not evidence — treat it and its opposite as sales strategy, and go run your own test.

Scorecard

Round 4: DRAW. Databricks won the only audited fact in the fight — the 100TB record was real, TPC-reviewed, 2.2x the prior mark — but the number it wanted you to remember, the 2.7x/12x over Snowflake, was unaudited by its own footnote and shipped with two inconsistent versions of the cost figure. Snowflake landed the integrity argument, then undercut it by publishing self-run numbers on a dataset Databricks showed had been regenerated after the fact. Databricks' reproduction attempt was equally compromised: one vendor configuring a rival's system is not evidence, whichever direction it points. Nothing survived scrutiny, no third party ruled, and the fight simply expired — a negative-sum thirteen days whose only durable output was the lesson.

Running tally: Databricks 1 — Snowflake 1 (2 draws)

Tomorrow

Part 5: the format war — Delta vs Iceberg, and the acquisition Databricks timed to land in the middle of Snowflake's own conference.

Sources

• Databricks Sets Official Data Warehousing Performance Record (Nov 2, 2021)
• PRNewswire: Databricks Lakehouse Sets the New World Record for Data Warehouse Performance
• Snowflake: Industry Benchmarks and Competing with Integrity (Nov 12, 2021)
• Databricks: Snowflake Claims Similar Price/Performance to Databricks, But Not So Fast! (Nov 15, 2021)
• Databricks: Eliminating the DeWitt Clause for Database Benchmarking (Nov 8, 2021)
• SiliconANGLE: The complicated rivalry between Snowflake and Databricks (Aug 8, 2022)