Exactly-Once Is a Promise About Your Consumer, Not Your Queue

Kafka's famous "exactly-once" guarantees don’t mean what you think they do. While Kafka can make sure a message is delivered from one Kafka topic to another—without any duplications or losses—even if the system crashes, this promise stops at the Kafka broker. Even transactions in Kafka Streams or Spring Kafka are not an end-to-end guarantee for your entire application. Once you cross the boundary from Kafka to your business logic or external systems, you're back in at least-once territory: duplicates can still happen, making consumer-side idempotency critical.

Broker Guarantees Are Narrower Than the Slogan

When Kafka introduces "exactly-once delivery", it's actually talking about transferring messages between Kafka topics atomically. This means if a single message is published or moved to another topic, the operation happens in full or not at all; there are no partial deliveries. In Apache Kafka's delivery guarantees framework, exactly once semantics are realized as a READ -> WRITE sequence, as opposed to the READ -> PROCESS -> WRITE found in other message brokers.

Additionally, on systems that allow exactly once, each message is intended to be delivered once and only once, meaning no losses or duplications. If the system crashes, the message still lands in the intended destination exactly once after a restart. However, these strong Kafka delivery semantics don't translate to the rest of your architecture.

Read, Process, and Write Aren't Atomic

Kafka pioneered the use of transactions to achieve exactly-once guarantees, but the efficacy of these depend on what they're applied to. In Kafka Streams, transactions are scoped to a Read -> Write operation when moving messages between Kafka topic streams, usually within the same Kafka cluster. For other messaging brokers, Spring Kafka highlights how at-least-once guarantees still apply to the Read -> Process part of a Read -> Process -> Write data pipeline, even with Kafka transactions. Here, the exact atomicity stops at the write to Kafka.

The process part of the pipeline isn't protected by Kafka's exactly-once guarantees, so the onus is on the message consumer to make sure processing each message doesn't result in unintended side effects. If that same consumer retries the process step, transactions alone won't stop it from applying that business logic more than once.

Duplicates Still Come From Everywhere

Even the cleanest Kafka transactional setups are vulnerable to duplicate side effects when publishing to external systems like databases or file systems. This sensitivity to duplicates is why AWS prescribes idempotent message processing for consumers, making them track which messages have already been consumed and applied.

Problems arise when operational factors like long-running processes and message re-broadcasting add up to the same message triggering consumer-side effects more than once. Unreliable network connections, service restarts, and process crashes can lead to double writes, lost state, and other credibilty issues.

Transactional Outbox Fills the Gaps

The popularity of microservices as the de facto architecture for modern applications has led to a resurgence in the Transactional Outbox pattern. This pattern serves as the bridge that ensures consistency between an application’s state and the events it publishes to downstream services, which are often other microservices with their own data stores.

At its core, the outbox is a per-application, transactionally consistent log of domain events or messages. A sending service writes an outbox record containing all the details of an event, alongside the application state changes, atomically. Only after the transaction commits does a background "outbox processor" move those events onwards.

This pattern provides an atomic commitment of an event and a corresponding state change. It also reduces the complexity of the above-mentioned retry patterns that mediate between multiple microservices. However, it mostly guards against a specific class of ineffectively acknowledged message retries. Protecting against the especial variety of partial updates and replay combinations distributed systems exhibit in actual implementations requires a different approach altogether.

Transactions Can't Make Everything Exactly-Once

These transactional outbox and other retry mechanisms help prevent some kinds of message errors, but they come with their own pitfalls. Making an outbox pursuit system resilient to partial failures is a complex problem, one that may come down to tradeoffs between eventual consistency and the exact ordering of events. While Kafka has developed some powerful tools to help, it can't completely solve these problems for all types of applications.

Spring Kafka acknowledges that creating exactly-once systems while "guaranteeing exactly-once sequencing for flows within Kafka can be a challenge. Particularly in distributed, multi-node clusters, the risks of partial failures become significant enough to warrant their own retry patterns." This makes it clear the "exactly-once" segment of Kafka's guarantees only applies within the cluster; consumer side effects need more.

What Teams Get Wrong

Kafka's exactly-once delivery guarantees are a powerful tool for preventing data duplication and losses in Kafka-to-Kafka message flows. However, too many teams treat them as the end of their migration to event-driven architectures, and assume they can skip building idempotent consumers..

The exact Kafka delivery guarantees within Kafka do not propagate to external systems or consumer side effects. They ensure messages get processed once, but not that the action has the same exactly-once characteristics. Implementation of these guarantees can be complex, and systems containing these mistakes can lead to transactional anomalies and costly mistakes.

Conclusion

Kafka's transactions provide powerful guarantees on Kafka-to-Kafka message delivery, but they don't translate to consumer-side side effects. Even within Kafka pipelines, the Read and Process parts of a Read-process-write flow still have at-least-once guarantees. For multi-system event-driven architectures, the consumer must still handle scenarios that result in Kafka duplicates. This makes designs like the Transactional Outbox a practical tool for coordinating state changes with downstream events, but they can't provide Guaranteed-lasting Exactly-Once as a single system.