Q43 — What is the difference between at-most-once, at-least-once, and exactly-once delivery?
Why do delivery guarantees exist as a concept at all?
In a direct function call, you get the result immediately and know exactly whether it happened. Either the function returned, or it threw an exception. No ambiguity.
In a distributed system, the producer sends a message and the consumer processes it — but these are two separate operations on separate machines, connected by a network that can drop, delay, or duplicate messages. A crash can happen between the send and the acknowledgment. What does the producer know? What does the consumer know? The delivery guarantee is the contract that answers these questions.
Every messaging system must make a choice about what it promises when failure occurs. There are exactly three meaningful positions: never deliver twice (at-most-once), never lose a message (at-least-once), and never lose or duplicate (exactly-once). Each requires different infrastructure and has different costs.
At-most-once — fire and forget
The simplest guarantee: send the message and move on. If the message is lost, it's lost. No retry, no acknowledgment required.
How it's implemented: the producer writes the message to the queue and does not wait for confirmation. The consumer reads the message, marks it as consumed immediately (before processing), and then processes it. If the consumer crashes after marking consumed but before processing, the message is gone — it won't be redelivered.
Why accept this? For some data, loss is better than duplication. Consider metrics or log aggregation: you're collecting millions of data points per second. Losing 0.01% of them has negligible impact on analytics. But if you retry every lost message, you double your infrastructure load and add complexity for marginal benefit. For high-throughput, loss-tolerant data pipelines, at-most-once is the right trade.
The failure scenario: Producer sends message → network drops it → consumer never receives it. Or: consumer receives message → consumer marks it as consumed → consumer crashes before executing the processing logic → message is gone.
At-most-once delivery is what happens when you have no retry logic and delete messages before processing them. It's usually not a deliberate choice — it's what you get when you don't think carefully about failure.
At-least-once — retry until acknowledged
The most common guarantee for business-critical messaging: every message will be delivered and processed at least once. Messages are never lost, but they might be processed more than once.
How it's implemented: the producer sends a message and waits for an acknowledgment (ACK) from the broker. If no ACK arrives within a timeout, the producer retries. The consumer reads a message, processes it, and only then sends an ACK. The broker delivers the message again if no ACK is received.
The duplication scenario: the consumer processes the message successfully, sends the ACK — but the ACK is lost in transit. The broker never received the ACK, so it redelivers the message. The consumer processes the same message a second time.
Or: the consumer processes the message, then crashes before sending the ACK. On restart, the broker redelivers the message. The consumer processes it again.
Why is this safe for most use cases? If the consumer's processing is idempotent — producing the same result regardless of how many times it's executed — duplicates are harmless. Updating a user's profile to the same value twice has no extra effect. Setting a value (not incrementing) is idempotent. Inserting a record with an idempotency key (see Q26) is idempotent.
At-least-once with idempotent consumers is the practical standard for most production messaging systems. It's simpler to implement than exactly-once, delivers the durability guarantee that at-most-once lacks, and idempotency is achievable for most business operations.
Exactly-once — the hardest guarantee
Exactly-once means every message is delivered and processed precisely once: never lost, never duplicated. It's the strongest guarantee and the hardest to achieve.
The fundamental difficulty: exactly-once requires coordination between the producer, the broker, and the consumer — all three — across network failures. Consider:
-
Producer sends a message. Network drops it. At-least-once says retry. But if the broker did receive it (the drop was on the ACK), the retry causes a duplicate. To prevent this, the producer needs an idempotent identity for each message — a sequence number the broker uses to deduplicate retries.
-
Consumer processes the message and commits its state — but the commit to the downstream system and the ACK to the broker are two separate operations. If the consumer commits its state but crashes before ACKing, the broker redelivers. The consumer processes again. To prevent this, the consumer's state commit and the ACK must be atomic.
Kafka's exactly-once implementation:
Kafka achieves exactly-once within Kafka via two mechanisms:
Idempotent producers: Each producer is assigned a producer ID. Each message gets a monotonically increasing sequence number. The Kafka broker deduplicates retried messages by (producer ID, sequence number) — if the same sequence number arrives twice, the second is discarded. This handles the producer retry duplication problem.
Transactional messaging: Kafka allows a producer to write to multiple partitions atomically — either all writes commit or none do. Combined with transactional consumers that commit their offset (position in the log) in the same transaction as their downstream write, you get atomic read-process-write cycles. Kafka Streams uses this internally.
The real-world complexity: Exactly-once within Kafka (message → Kafka broker) is achievable. Exactly-once from Kafka to an external system (a database, an API, another queue) requires that external system to participate in Kafka's transaction protocol or be idempotent itself. Most external systems cannot participate in Kafka's transactions, so in practice you get exactly-once within Kafka and at-least-once at the boundary with external systems — relying on idempotency to handle the rare duplicate.
The idempotency bridge
The practical takeaway that most engineers arrive at: exactly-once semantics in a distributed system is extremely complex to implement at the infrastructure level. The pragmatic alternative is at-least-once delivery + idempotent consumers.
Design your consumer so that processing the same message twice produces the same result as processing it once:
- Use idempotency keys to deduplicate in the database (Q26)
- Use conditional writes (update where version = N)
- Check before acting (if record doesn't exist, create it; otherwise, ignore)
With this design, the messaging infrastructure guarantees at-least-once (messages are never lost), and idempotency at the consumer guarantees that duplicates are harmless. The emergent behavior is effectively exactly-once from the user's perspective, without the complexity of a transactional messaging protocol.
The mental model
At-most-once:
Guarantee: message might be lost, never delivered twice
Mechanism: send, don't wait for ACK; mark consumed before processing
Failure: producer retry disabled → network drops → message lost
consumer crashes after mark but before process → message gone
Use when: loss is acceptable (metrics, logs, high-throughput analytics)
At-least-once:
Guarantee: message never lost, might be delivered multiple times
Mechanism: producer retries until ACK; consumer ACKs only after processing
Failure: ACK lost → broker redelivers → consumer processes twice
Use when: loss is unacceptable, consumer can be made idempotent (most business cases)
Exactly-once:
Guarantee: delivered and processed exactly once, always
Mechanism: idempotent producer (sequence numbers) + transactional consumer
Failure: complex, requires broker support + external system participation or idempotency
Kafka implementation: idempotent producers + transactional batches
Use when: duplicates cause real business harm AND you can pay the complexity cost
Pragmatic standard:
At-least-once + idempotent consumer = effectively exactly-once behavior
Simpler than transactional exactly-once, handles the common case
Ordering note:
At-most-once and at-least-once say nothing about ordering
Exactly-once within a Kafka partition preserves order (partitions are ordered logs)
Delivery guarantees are a contract about what happens when things go wrong — network failures, producer crashes, consumer crashes. Understanding what each guarantee costs and what it requires of your consumer is the foundation for building reliable event-driven systems.