Is Exactly-Once Processing in Spark Actually Real?

The "Exactly-Once" guarantee in Spark Structured Streaming is the most dangerous myth in modern data engineering. We are taught to believe that if we just wrap our Kafka source and our Parquet sink in the right configuration, our data counts will magically remain perfect despite network partitions, executor crashes, or node evictions.

I’ve spent the last six years cleaning up the mess left by engineers who took that claim at face value. Exactly-once is not a mathematical certainty inherent to your code; it is a delicate, fragile contract between your storage layer, your checkpoint directory, and your retry logic. If any of those three pillars is misconfigured, "exactly-once" quickly devolves into "at-least-once" or, worse, "data-loss-forever."

How it actually works

At the heart of the guarantee is the Write-Ahead Log (WAL) mechanism managed via the checkpointLocation. When you start a Spark Structured Streaming job, it initializes a state store—usually RocksDB or an in-memory map—and records the progress of every micro-batch in the checkpoint directory.

The magic happens during the commit protocol. When a micro-batch finishes, Spark writes a metadata file to the _spark_metadata folder in your sink. This file tells the driver: "These are the files I just wrote, and here is the offset I just processed."

If an executor dies mid-batch, Spark doesn't try to magically recover the partial writes. Instead, it relies on the fact that your sink is idempotent. If you are writing to S3, Spark uses a temporary staging directory. It writes the files there first, then performs an atomic rename or a move operation to the final destination only after the batch succeeds.

If you are using Kafka as a sink, Spark relies on the transaction coordinator. It starts a Kafka transaction, writes the records, and then commits the transaction. If the Spark driver crashes before the commit, the transaction expires, Kafka aborts the writes, and the Spark re-run simply attempts to write that same batch again. The consumer sees no duplicates because it is configured with isolation.level = read_committed.

The code looks deceptively simple:

val query = df.writeStream
  .format("parquet")
  .option("checkpointLocation", "s3a://my-bucket/checkpoint/job-name/")
  .start("s3a://my-bucket/output-path/")

But notice what is missing: error handling. The "exactly-once" logic is baked into the checkpointLocation. If that directory is deleted, corrupted, or moved to a different S3 bucket, your exactly-once guarantee evaporates instantly.

Photo by Carrie Borden on Unsplash

The tradeoffs nobody mentions

The primary cost of exactly-once is latency. You are constantly flushing state to durable storage. Every micro-batch has to wait for the metadata log to be updated in S3 or HDFS. In a high-throughput environment, the fsync overhead on your metadata storage will eventually become your bottleneck.

Then there is the issue of schema evolution. If you change your schema and your sink is Parquet, you might break the append operation. If the task fails to write a new file due to a schema mismatch, Spark will retry. If your retry logic isn't perfectly tuned, you end up with infinite retry loops that crash the driver, forcing you to manually clean the checkpoint directory—which almost always results in duplicate data or missed records.

Another massive pain point is state store growth. If you are doing windowed aggregations with withWatermark, Spark keeps that state in the checkpointLocation forever unless you trigger manual compaction or set the watermark correctly. I’ve seen production jobs grind to a halt because the RocksDB state store grew to several terabytes. When the driver tried to read that metadata on restart, it timed out, and the job entered a boot-loop.

Furthermore, exactly-once only applies if the entire pipeline is idempotent. If your Spark job calls an external REST API for enrichment, Spark has no way of knowing if that API call succeeded during a failed micro-batch. If the job retries, it hits the API again. Suddenly, you’ve hit your rate limit or, worse, you’ve charged a customer twice for the same event. Exactly-once in Spark is a local guarantee; it does not extend to the outside world.

Photo by Claudio Guglieri on Unsplash

When to reach for it (and when not to)

You reach for exactly-once when your business logic involves financial accounting, inventory reconciliation, or any case where a missing or duplicate record triggers an audit flag. If you are building a real-time dashboard or a recommendation engine, you probably don't need the overhead.

If you are dealing with external systems that are not transactional, stop pretending that Spark is doing the heavy lifting for you. You need to implement your own idempotency keys. Write your output to a staging table keyed by (event_id, batch_id), then use a MERGE INTO statement with a WHERE clause to ensure you never overwrite or duplicate.

Never rely on the default behavior if your sink is anything other than Parquet or Delta Lake. If you’re writing to a legacy database using JDBC, Spark’s exactly-once guarantee is practically non-existent because JDBC drivers rarely support the two-phase commit protocol required to roll back an insert if the Spark task crashes after the write but before the metadata update.

If your data volume is massive, consider trading exactly-once for at-least-once. If you can handle duplicates via a downstream deduplication layer (like a SELECT DISTINCT or a row-number window function in your warehouse), you will save yourself a massive amount of stress. You can increase the micro-batch interval, reduce the frequency of checkpoint writes, and drastically improve your cluster’s stability.

Conclusion

Exactly-once in Spark Structured Streaming is a tool, not a religion. It is a highly optimized, state-aware commit protocol that works perfectly as long as your storage is fast, your checkpoint directory is immutable, and your sinks are transactional.

The moment you step outside those bounds—by hitting external APIs, using non-transactional databases, or mismanaging your checkpoint metadata—the guarantee dies. Stop chasing the perfect pipeline. Instead, focus on building idempotent sinks and robust observability. If you can’t verify your data at the sink, it doesn’t matter how many "exactly-once" boxes you checked in the Spark UI.

Tags: #spark #streaming #data #engineering

Cover photo by Albert Stoynov on Unsplash.