Delivery Semantics & Schema Strategy

By the end of this lesson you will be able to:

• say which delivery mode a given Confluent.Kafka consumer loop and config actually implement — including the ones that look safer than they are;
• explain exactly what Kafka transactions make atomic, and what they leave to your own code;
• pick the schema-registry compatibility mode that matches a rollout order, and predict whether a schema change will be accepted;
• design duplicate handling (identity field + durable record) for an at-least-once consumer;
• spot five common misconfigurations from their code and config.

Prerequisites: Core Kafka Mental Model (offsets, retention, commit = position) and Consumer & Manual Commits (Commit vs StoreOffset). Each major section includes a short exercise — try it before opening the answer.

Why Delivery Semantics and Schema Design Are Inseparable

Imagine you've just deployed a Kafka consumer that processes payment confirmation events. Your producer and consumer are configured for at-least-once delivery — if the network hiccups, the send is retried, and the consumer commits only after processing. You feel confident. Then a teammate adds a new field to the event payload: currency_code, a string that your existing consumer never knew to expect. The message arrives, your consumer silently deserializes it, drops the unknown field, and proceeds. The funds transfer completes. The currency is wrong. Nobody noticed at deserialization time because nothing failed — the payload was still structurally readable, but its meaning had changed, and nothing in the contract said so.

This is the core tension in distributed messaging, and it's the reason delivery semantics and schema strategy must be designed together. A Kafka message is only as useful as the contract it carries. Configuring delivery guarantees is well-understood engineering; guaranteeing meaning at delivery is the harder part. Note that a schema registry (a service that stores each topic's schema versions and refuses a new version that breaks the configured compatibility rule, or mode) alone would not necessarily have caught this change — in Avro, adding a field with a default passes every compatibility mode. What was missing was a contract saying an amount is meaningless without its currency, and a versioning rule for changes that alter meaning. Tooling enforces the rules you write down; it cannot invent them.

The Two-Dimensional Producer-Consumer Contract

When a producer and consumer interact over Kafka, they're making an implicit contract along two distinct axes:

┌─────────────────────────────────────────────────────────┐
│           The Producer-Consumer Contract                │
├──────────────────────────┬──────────────────────────────┤
│  AXIS 1: DELIVERY        │  AXIS 2: MEANING             │
│  "Will this message      │  "Will it mean the same      │
│   arrive?"               │   thing when it does?"       │
├──────────────────────────┼──────────────────────────────┤
│  Controlled by:          │  Controlled by:              │
│  • acks configuration    │  • Schema definition         │
│  • Offset commit timing  │  • Schema registry           │
│  • Retry settings        │  • Compatibility rules       │
│  • Idempotent producers  │  • Serialization format      │
│  • Transactions          │  • Versioning discipline     │
└──────────────────────────┴──────────────────────────────┘

Most teams spend the majority of their Kafka configuration effort on the left column. The right column gets treated as an implementation detail — pick JSON, move fast, fix it later. The cost of that asymmetry shows up months into production, not in the initial build.

🎯 Key Principle: Both axes must be answered together — because the choice on one axis constrains what's practical on the other. A delivery guarantee that produces duplicates requires a schema that can carry a stable identity field. A schema that silently drops unknown fields undermines the guarantee that messages were processed correctly.

Three Delivery Modes, Three Different Schema Requirements

The three delivery modes — at-most-once, at-least-once, and effectively-once — are not just configuration knobs. Each mode makes a different implicit demand on your message schema.

At-most-once delivery commits offsets before processing completes. If the consumer crashes mid-handler, the group never reads that message again. You'll never see the same message twice, so you don't need duplicate detection in your payload — but nothing re-reads a lost message automatically. (The record is still in the log until retention removes it, so you can seek back and replay — but only if you notice the gap.) That is why schemas for these streams benefit from a sequence number or timestamp: it is what lets anyone notice a gap at all.

At-least-once delivery is where schema design gets load-bearing. When a consumer re-reads a message after a crash or rebalance, your consumer will occasionally process the same logical event more than once. A plain re-read comes back at the same topic, partition and offset, so those coordinates can catch it. But when the same event is published again, it arrives at a new offset, and a schema that carries no stable identity — no event_id, no idempotency_key, no natural key (a business ID already unique per event, such as the order number of an OrderPlaced) — gives the consumer no way to detect that it has already acted on this event. The delivery guarantee hands the consumer a problem that the schema is best placed to solve.

Effectively-once delivery uses idempotent producers (whose retried sends the broker writes only once) and Kafka transactions so that a pipeline which reads from Kafka and writes to Kafka publishes each output, and advances its input offsets, exactly once. The scope matters: a transaction covers Kafka writes and Kafka offsets — not a database row or an HTTP call your handler makes — so a consumer with external side effects still needs its own duplicate handling. It also changes when readers see data: a consumer reading with IsolationLevel.ReadCommitted (the Confluent.Kafka default) receives nothing past an open transaction on that partition until it commits or aborts. Order within the partition is preserved; delivery just waits.

The specific mechanics of each mode are covered in depth in the next section. The principle to hold here is that each mode opens a different set of failure scenarios, and schema design is what determines whether your consumer can handle those scenarios gracefully.

📋 Delivery Mode × Schema Requirement

| Delivery Mode    | Schema Must Carry              | Consumer Responsibility         |
|------------------|--------------------------------|---------------------------------|
| At-most-once     | Sequence/timestamp (loss       | Detect gaps, not dupes          |
|                  | detection only)                |                                 |
| At-least-once    | Stable EventId (dupe           | Check EventId before processing |
|                  | detection)                     |                                 |
| Effectively-once | Nothing extra for Kafka→Kafka; | None for Kafka→Kafka outputs;   |
|                  | EventId if outputs leave Kafka | dedupe any external side effect |

Choosing Raw JSON Is a Decision, Not a Default

There's a temptation — especially early in a project — to reach for System.Text.Json, serialize your C# objects to UTF-8 bytes, and push them onto the wire. It works immediately. Here's a minimal example:

// Producer side — no schema registry, no contract
var message = new OrderPlaced
{
    EventId = Guid.NewGuid().ToString(),
    OrderId = "ORD-001",
    CustomerId = "CUST-42",
    PlacedAtUtc = DateTime.UtcNow
};

var bytes = JsonSerializer.SerializeToUtf8Bytes(message);

await producer.ProduceAsync("orders", new Message<string, byte[]>
{
    Key = message.OrderId,
    Value = bytes
});

This code works. The problem is what it doesn't do: it establishes no shared contract between the producer and any consumer. The OrderPlaced class on the producer and the OrderPlaced class on the consumer are just two C# files that happen to share a name. When they diverge — and they will — the consumer has no mechanism to detect the mismatch.

Compare this to a producer that publishes a schema to a registry before producing:

// Producer side — schema registry enforces the contract
var config = new ProducerConfig { BootstrapServers = "localhost:9092" };
var schemaRegistryConfig = new SchemaRegistryConfig { Url = "http://localhost:8081" };

// OrderPlaced here is a class generated by avrogen from an Avro schema (.avsc; Avro is a
// compact binary format whose schemas are written in JSON), so it implements
// ISpecificRecord. AvroSerializer rejects a hand-written POCO like the one above.
using var schemaRegistry = new CachedSchemaRegistryClient(schemaRegistryConfig);
using var producer = new ProducerBuilder<string, OrderPlaced>(config)
    .SetValueSerializer(
        new AvroSerializer<OrderPlaced>(schemaRegistry) // registers + checks the schema on first use
    )
    .Build();

await producer.ProduceAsync("orders", new Message<string, OrderPlaced>
{
    Key = orderPlaced.OrderId,
    Value = orderPlaced
});

On the first ProduceAsync, the serializer registers the OrderPlaced schema (auto-registration is on by default) under the topic's subject — the registry's version history for the topic's values, named orders-value by default — and the registry checks it against the latest version registered there (or all of them, in a transitive mode) under the subject's compatibility mode. If the change violates those rules, registration is refused and ProduceAsync throws a ProduceException (error code Local_ValueSerialization, with the registry's SchemaRegistryException as its inner exception) — before a single record is written, so no consumer is affected. Teams that want the failure even earlier register schemas from CI and turn auto-registration off.

⚠️ Common Mistake: Treating the absence of a schema registry as "no schema" — in reality, the schema still exists; it just lives exclusively in your C# class files, unversioned and unenforced. Every deserialization is a silent hope that both sides have the same definition in mind.

Schema Drift Is a Delivery Semantics Problem

Schema drift — the gradual divergence between what producers write and what consumers expect — is not just a data quality problem. It interacts directly with delivery semantics. Consider at-least-once delivery, where replaying old messages is part of the normal failure-recovery path. Without a registry enforcing backward compatibility — meaning new consumers can read old messages — replay becomes a deserialization minefield. So does a producer-first rollout, which mixes versions the other way round — old consumers reading new messages:

Time →

Week 1:  Producer emits OrderPlaced { order_id, customer_id, total }
         Consumer commits offsets cleanly

Week 3:  Schema changes — 'total' renamed to 'total_amount'
         Producer deploys first
         Consumer redeploys second (rolling deploy, 10-minute gap)

         During the gap:
         ┌─────────────────────────────────────────────────────┐
         │ Old consumer reads new messages with 'total_amount' │
         │ 'total' is missing — POCO reads 0 (C# default)      │
         │ Orders processed with $0.00 total                   │
         └─────────────────────────────────────────────────────┘

A schema registry would have rejected this rename when the new version was registered. Look at the rollout direction, though: the producer upgraded first, so old readers met new data. The compatibility mode that guarantees exactly that direction is forward compatibility. The registry's default, backward, guarantees the opposite rollout — consumers upgraded first, reading old data. This rename (no default) happens to break both directions, so either mode rejects it; a rename that gave total_amount a default would pass BACKWARD and still break these old consumers. With auto-registration the rejection arrives at the first produce call after the deploy; registering schemas from CI moves it before the deploy. Either way the mismatch is caught at write time, not at consumer read time.

Schema compatibility rules are not bureaucratic overhead — they are the mechanism by which replay safety (a delivery semantics concern) and schema evolution (a data modeling concern) are kept in sync. With that framing established, the next section examines the three delivery modes in full mechanical detail.

✍️ Exercise: which axis failed?

For each incident, say whether the delivery axis, the meaning axis, both, or neither failed.

1. A payment consumer crashed after charging a card but before committing; after restart it charged the same card again.
2. A producer renamed total to total_amount; old consumers booked orders at $0.00.
3. A new consumer group started with auto.offset.reset=latest and never processed the 10,000 orders already in the topic.

Check your answer

1. Neither — the handler. Re-reading after a crash is at-least-once delivery working as designed, and nothing was misread. The handler had no duplicate check: for this re-read the record's topic, partition and offset would have identified the repeat (a payload ID is what it needs when an event is re-published at a new offset), and passing that key to the card provider as its idempotency key would have stopped the second charge.
2. Meaning. Every record was delivered exactly as written; it was interpreted with the wrong contract.
3. Delivery. The data was valid and reachable — the consumer's start position decided never to read it.

---

The Three Delivery Modes: Guarantees, Trade-offs, and What They Actually Cost

Every message you send through Kafka is meant to be delivered under one of three behavioral contracts: at most once, at least once, or effectively once — though a misconfigured consumer can end up in none of them. These are not just philosophical stances — each mode maps to specific configuration knobs, specific failure scenarios, and specific costs that show up in latency, throughput, or code complexity. Understanding which mode you are actually operating in (rather than which mode you assume you are in) is the difference between a system that behaves predictably under failure and one that silently corrupts state.

At-Most-Once: When Loss Is Cheaper Than Duplication

At-most-once delivery means a message is processed zero or one times — never more than once, but possibly never at all. This guarantee is achieved by committing the offset before processing the message. If your consumer crashes after the commit but before processing completes, the group's committed position is already past that message, so it is never read again.

In Confluent's .NET client, the deterministic way is to call Commit(result) before your processing logic runs:

// At-most-once pattern: commit offset before processing
using var consumer = new ConsumerBuilder<string, string>(consumerConfig)
    .Build();

consumer.Subscribe("telemetry-events");

while (!cancellationToken.IsCancellationRequested)
{
    var result = consumer.Consume(cancellationToken);

    // Offset committed BEFORE the handler runs (consumer built with EnableAutoCommit = false).
    // A crash after this line = the group never reads this message again.
    consumer.Commit(result);

    // Any exception here means this message is gone.
    await HandleTelemetryEventAsync(result.Message.Value);
}

The practical use case for at-most-once is genuinely narrow: high-throughput telemetry streams, real-time metrics, or any scenario where losing occasional data points is acceptable and processing the same reading twice would be harmful (such as triggering duplicate alerts). For most business-critical data flows, this is not the mode you want.

⚠️ Common Mistake: Assuming that EnableAutoCommit = true gives you at-least-once. EnableAutoOffsetStore defaults to true (whatever auto-commit is set to), so the .NET client stores each record's offset as soon as Consume() hands it to you — before your handler has done anything — and the background timer commits whatever is stored. A crash while a handler is running can lose that record (the timer may already have committed past it) and replay the records after the last commit: no clean guarantee in either direction. The fix is the EnableAutoOffsetStore = false pairing shown next, or explicit Commit calls. The async variant of this trap is covered in detail in Common Misconfiguration Patterns and How to Catch Them.

At-Least-Once: The Practical Default and Its Hidden Contract

At-least-once delivery means every message will be processed one or more times — you will never lose a message, but you might handle it more than once. This is the mode most Kafka consumers operate in, and it is the right default for the majority of use cases. The guarantee is achieved by committing offsets after successful processing:

// At-least-once pattern: commit after successful processing
var consumerConfig = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "order-processor",
    AutoOffsetReset = AutoOffsetReset.Earliest,
    EnableAutoCommit = true,        // the client commits stored offsets on a timer...
    EnableAutoOffsetStore = false   // ...but only offsets WE store, after success
};

using var consumer = new ConsumerBuilder<string, OrderEvent>(consumerConfig)
    .SetValueDeserializer(registeredDeserializer) // Schema-aware deserializer
    .Build();

consumer.Subscribe("order-events");

while (!cancellationToken.IsCancellationRequested)
{
    var result = consumer.Consume(cancellationToken);

    try
    {
        await ProcessOrderAsync(result.Message.Value);

        // Only store offset after success.
        // A crash between ProcessOrder and here = redelivery on restart.
        consumer.StoreOffset(result);
    }
    catch (Exception ex)
    {
        logger.LogError(ex, "Processing failed for offset {Offset}", result.Offset);
        throw; // or apply a retry/dead-letter strategy
    }
}

The failure scenario that produces duplicates is straightforward: your consumer processes a message successfully, then crashes before its offset is committed. On restart — or after a consumer group rebalance, where partition ownership is reassigned — the group resumes from the last committed offset, which is the next offset to read, and everything from there is read again.

At-least-once failure scenario:

Partition:  ... [43] [44] [45] [46] [47] [48] ...
                      ^
       committed offset = 44  (the NEXT offset to read)

Consumer processes 44, 45, 46 and stores their offsets;
  the auto-commit timer has not fired yet
Consumer applies 47 to the database...
CRASH before StoreOffset for 47 runs, before any further commit

On restart:
  Group resumes at the committed offset: 44
  Consumer re-reads: 44, 45, 46, 47, 48, ...
                     ^^^^^^^^^^^^^^
                     all four processed again!

🎯 Key Principle: At-least-once is not a weakness to tolerate — it is a contract that makes the consumer responsible for idempotency. If your handler is naturally idempotent (writing to a database with an upsert keyed on event ID, for example), you get the safety for free. If it is not, you need to build it. As we will examine in the next section, including a stable event ID in your message schema is the most practical mechanism for detecting and skipping duplicates.

Effectively-Once: What It Actually Means and What It Costs

Kafka itself calls this exactly-once semantics (EOS, introduced by KIP-98); many engineers say effectively-once because it is the more honest name. No system can prevent a message from arriving at a consumer more than once across all possible failure modes. What Kafka can guarantee, through a combination of idempotent producers and transactions, is that inside a read-from-Kafka, write-to-Kafka pipeline the observable effect happens once: the output records and the consumed offsets are committed together or not at all. Not "the message was delivered once" but "the write resulting from that message appears exactly once in the output topic." Writes that leave Kafka are not covered.

Idempotent Producers: Sequence Numbers and Broker-Side Deduplication

When you set EnableIdempotence = true on a .NET producer, the broker assigns each producer instance a Producer ID (PID) and tracks a monotonically increasing sequence number per partition. If a network failure causes the producer to retry a send, the broker recognizes the duplicate sequence number and silently discards it rather than writing a second copy.

var producerConfig = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    EnableIdempotence = true
    // Setting EnableIdempotence = true automatically configures:
    //   Acks = All             (all in-sync replicas must acknowledge)
    //   MessageSendMaxRetries = int.MaxValue
    //   MaxInFlight = 5        (up to 5 in-flight requests per connection)
    //
    // Do NOT manually set conflicting values — e.g., Acks = Leader (acks=1)
    // or MaxInFlight > 5 makes Build() throw an InvalidOperationException.
};

using var producer = new ProducerBuilder<string, OrderEvent>(producerConfig)
    .SetValueSerializer(registeredSerializer)
    .Build();

The automatic defaults that EnableIdempotence = true sets deserve careful attention. Acks = All means the leader waits for all in-sync replicas to acknowledge before confirming. MaxInFlight = 5 caps the number of unacknowledged requests per connection: the broker keeps sequence state for only the last five batches per producer and partition, so with more in flight a retried batch could no longer be recognised as a duplicate or kept in order. librdkafka enforces the ceiling on the client side — if you set MaxInFlight above 5 with idempotence enabled, ProducerBuilder.Build() throws an InvalidOperationException carrying librdkafka's configuration error.

Transactions: Extending the Guarantee Across Partitions

Idempotent producers deduplicate retries on a single partition. Transactional writes extend this to atomic, all-or-nothing commits across multiple partitions — the critical pattern for stream processing (read from input, write to output, commit input offset, atomically). All of those are Kafka operations; a transaction cannot include a database write.

var producerConfig = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    EnableIdempotence = true,
    TransactionalId = "order-processor-instance-1" // Must be unique per producer instance
};

using var producer = new ProducerBuilder<string, ProcessedOrder>(producerConfig)
    .SetValueSerializer(registeredSerializer)
    .Build();

producer.InitTransactions(TimeSpan.FromSeconds(10));

try
{
    producer.BeginTransaction();

    await producer.ProduceAsync("processed-orders", new Message<string, ProcessedOrder>
    {
        Key = order.OrderId,
        Value = processedOrder
    });

    // Atomically commit the input consumer's offsets as part of this transaction. That consumer
    // must run with EnableAutoCommit = false, and consumedOffsets must hold, per partition, the
    // NEXT offset to read (last processed + 1), as consumer.Position(partition) returns it.
    producer.SendOffsetsToTransaction(
        consumedOffsets,
        consumer.ConsumerGroupMetadata,
        TimeSpan.FromSeconds(5)
    );

    producer.CommitTransaction(TimeSpan.FromSeconds(10));
}
catch (KafkaTxnRequiresAbortException)
{
    producer.AbortTransaction(TimeSpan.FromSeconds(5));
    // Nothing from this transaction becomes visible and the group's committed offsets
    // did not move. This process has already read past those inputs, though: rewind the
    // consumer to its committed offsets (or let it restart) to process them again.
}
catch (KafkaException ex) when (ex.Error.IsFatal)
{
    // e.g. fenced by a newer instance with the same TransactionalId. This producer can
    // never be used again — dispose it and create a new one. Do not call Abort here.
    throw;
}
// Not shown: a non-fatal ProduceException from ProduceAsync also calls for AbortTransaction
// and a rewind; a KafkaRetriableException from CommitTransaction calls for a retry (below).

This pattern guarantees that the output write and the offset advance either both happen or neither happens. If the producer crashes after ProduceAsync but before CommitTransaction, the transaction is aborted (when it times out, or when a restarted instance with the same TransactionalId fences it, so the broker rejects the old instance from then on), the output messages are invisible to read_committed readers, and the input offset is not advanced — so after the restart the input is read again. One more failure type is worth knowing: a KafkaRetriableException from CommitTransaction (for example a timeout) means the outcome is not yet known, and the right response is to call CommitTransaction again.

The Consumer Side: ReadCommitted

Transactional guarantees on the producer side are only meaningful if consumers respect them. In Confluent.Kafka the default is already read_committed (librdkafka's isolation.level), but the Java client defaults to read_uncommitted — and a consumer on either stack set to read uncommitted sees messages from in-progress transactions before they commit, including messages from transactions that are later aborted. Setting the level explicitly documents the intent and survives a shared config that overrides it:

var consumerConfig = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "downstream-consumer",
    IsolationLevel = IsolationLevel.ReadCommitted
};

⚠️ Watch out: Running a ReadCommitted consumer against a topic that has slow-committing transactional producers can make the consumer appear to stall. A read_committed consumer reads only up to the last stable offset (LSO) — the first offset of the oldest still-open transaction — while the high watermark (how far a read_uncommitted consumer could read) keeps advancing, so its lag grows. It will not advance past an open transaction boundary, even if non-transactional messages arrive after it. This is correct behavior — not a bug — but it can look like a stuck consumer in monitoring dashboards.

The Real Cost of Effectively-Once

📋 Delivery Mode Trade-offs

+-----------------+-------------------+-------------------+----------------------+
| Mode            | Offset Commit     | Throughput        | Extra Config         |
+-----------------+-------------------+-------------------+----------------------+
| At-most-once    | Before processing | Highest           | Manual commit        |
| At-least-once   | After processing  | High              | Manual store/commit, |
|                 |                   |                   | idempotent handlers  |
| Effectively-once| Transactional     | Lower             | TransactionalId,     |
|                 |                   |                   | ReadCommitted        |
+-----------------+-------------------+-------------------+----------------------+

Each transactional commit requires the broker to write to the transaction log (the internal __transaction_state topic that tracks each transaction's state) and coordinate with all participating partitions — the round-trip cost is real. TransactionalId must be unique per producer instance and stable across restarts; if two instances share a TransactionalId, the broker fences the older one. And every downstream consumer of a transactionally-produced topic must read with read_committed. That is the default in Confluent.Kafka but not in the Java client — a single consumer on another stack left at its default, or one explicitly set to ReadUncommitted, will silently see aborted messages.

💡 Mental Model: Think of effectively-once as a database transaction projected onto a distributed log. The strength of the guarantee comes from the same source as its cost: coordination across brokers and partitions is never free.

The right mode is not the strongest one — it is the one whose costs match the value of the guarantee. At-most-once is a deliberate choice for loss-tolerant streams; at-least-once is the practical default for most business data; Kafka transactions are the right tool when a pipeline reads from Kafka, writes to Kafka, and can accept neither loss nor duplication. When the effect lands in a database — a financial ledger, an inventory table — the same requirement is met by at-least-once delivery plus an idempotent write inside that database. How these modes pair with a schema strategy is where we turn next.

✍️ Exercise: trace the crash

A consumer uses EnableAutoCommit = false and this loop, writing each order to SQL:

var result = consumer.Consume(ct);
await SaveOrderAsync(result.Message.Value);   // commits its own DB transaction
consumer.Commit(result);

The group's committed offset for the partition is 44. The consumer handles 44, 45 and 46 fully, then saves 47 to the database and crashes before Commit runs.

1. What is the committed offset at the moment of the crash, and which offset does the restarted consumer read first?
2. Which orders end up saved twice?
3. Now swap the last two lines (commit, then save) and replay the same crash — this time it happens after Commit for 47 but before its save finishes. What changes?

Check your answer

1. Commit(result) for offset 46 committed 47 — the next offset to read. The restarted consumer reads 47 first.
2. Only 47: it was saved, but its offset was never committed, so it is read and saved again. That is at-least-once, and why SaveOrderAsync should be an upsert keyed on the order or event ID.
3. Now the committed offset is 48 when the crash happens, so 47 is never read again and its save never completed: order 47 is lost. Committing before the work turns the same loop into at-most-once.

---

Pairing Delivery Semantics with a Schema Strategy

Delivery guarantees and schema design answer different questions, but they impose constraints on each other that become visible the moment something goes wrong. At-least-once delivery — the practical default — means your consumer will occasionally process the same message twice. A schema that carries no stable identity field leaves that consumer nothing in the payload to recognize a duplicate by. The schema is not decoration on top of the delivery guarantee; it is what makes the guarantee actionable.

Why Delivery Mode Shapes Schema Requirements

Detecting a duplicate takes two things: an identity to compare, and a durable record of which identities have already been handled — usually a table in the same database the handler writes to, updated in the same transaction. The identity can come from the message itself (an EventId or natural key) or from its Kafka coordinates (topic, partition, offset). The schema's job is the first half: coordinates only identify a record, so if the same event is ever published twice — a producer retrying at the application level, a replay tool — only an identity carried in the payload recognises it.

🎯 Key Principle: An EventId field in your schema is not a nice-to-have. Under at-least-once delivery, it (or a natural key) is the minimum information a consumer needs to recognise the same event again, even if it arrives at a different offset. An in-memory deduplication cache is not enough: the crash that causes the redelivery wipes it, and a rebalance can hand the partition to an instance that never saw the event — so the record of handled IDs must be durable. The identity field is necessary but not sufficient on its own; it just ensures the schema contract supports idempotent handling at all.

At-most-once delivery inverts this. Because offsets are committed before processing, duplicates are not a concern — but loss is. Schemas for at-most-once streams benefit from fields that support downstream reconstruction if a gap is detected, such as sequence numbers or timestamps, rather than identity fields for deduplication.

Effectively-once via Kafka transactions removes duplicates only for Kafka-to-Kafka pipelines: output records and consumed offsets are committed together. A consumer whose handler writes to a database or calls an API is outside that guarantee and still needs duplicate handling — which is why an event ID stays useful there, not merely for auditing.

Schema Evolution and Message Replay

Schema evolution is the process of changing a schema over time — adding fields, removing fields, or changing types — while keeping producers and consumers deployable independently. The three standard compatibility modes are backward compatibility (new schema can read old messages), forward compatibility (old schema can read new messages), and full compatibility (both directions). Translated into rollout order: BACKWARD means upgrade consumers first, FORWARD means upgrade producers first, FULL allows either order. Confluent Schema Registry defaults to BACKWARD, and each mode has a _TRANSITIVE variant that checks a new version against every earlier version rather than only the latest.

This intersects directly with delivery semantics: when old messages can be replayed — a natural consequence of at-least-once delivery and Kafka's durable log — a redeployed consumer will encounter messages produced under a previous schema version. If your schema evolution strategy is not enforced, that consumer may silently misinterpret those older messages.

💡 Real-World Example: A consumer group is redeployed after a schema change that adds a CustomerId field. Kafka replays messages from the last committed offset — some produced before the deployment. If the new consumer cannot read messages without CustomerId, it either throws, or silently reads a default value of null and processes incorrect data. Backward-compatible evolution makes that explicit instead of accidental: CustomerId is added as optional with a declared default, the registry accepts it because new readers can read old records, and the consumer code treats the default as "customer unknown" rather than as a real value. Compatibility guarantees the bytes can be read; only your handler can decide what a missing value means.

This is where a schema registry earns its cost. A registry enforces compatibility when a schema version is registered — at the first produce call when serializers auto-register (their default), or earlier if CI registers schemas — so a bad version never reaches the log. The records themselves never pass through the registry:

Producer registers
new schema version
        │
        ▼
┌───────────────────┐    PASS     ┌───────────────────────────┐
│  Schema Registry  │────────────►│ Producer sends records    │
│  (compatibility   │             │ straight to Kafka, each   │
│   check)          │             │ tagged with the schema ID │
└───────────────────┘             └───────────────────────────┘
        │
        │ FAIL (incompatible change)
        ▼
  Registration error
  returned to producer
  (message never sent)

Without a registry, compatibility is enforced only by convention and code review — which means it is enforced inconsistently.

Registered Serializer vs. Raw JSON: Where Enforcement Enters the Call Chain

The most concrete way to see the difference is in the producer code itself.

Producer with a Registered Serializer

var schemaRegistryConfig = new SchemaRegistryConfig
{
    Url = "http://localhost:8081"
};

var producerConfig = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    EnableIdempotence = true
};

using var schemaRegistry = new CachedSchemaRegistryClient(schemaRegistryConfig);

// OrderPlaced is an avrogen-generated class (ISpecificRecord); AvroSerializer does not
// accept hand-written POCOs. With AutoRegisterSchemas (default true) the first ProduceAsync
// registers the schema; if the registry rejects it, ProduceAsync throws
// ProduceException (Local_ValueSerialization) with the SchemaRegistryException as
// InnerException — before any message is sent.
using var producer = new ProducerBuilder<string, OrderPlaced>(producerConfig)
    .SetValueSerializer(new AvroSerializer<OrderPlaced>(schemaRegistry))
    .Build();

var orderPlaced = new OrderPlaced
{
    EventId = Guid.NewGuid().ToString(),  // create once per business event; reuse on any retry
    OrderId = "ORD-001",
    CustomerId = "CUST-42",
    PlacedAtUtc = DateTime.UtcNow
};

await producer.ProduceAsync(
    "orders.placed",
    new Message<string, OrderPlaced>
    {
        Key = orderPlaced.OrderId,
        Value = orderPlaced
    });

Schema enforcement here is part of the serialization step. An incompatible schema change — say, under the default BACKWARD mode, a new field added without a default — causes the produce call to throw before any message reaches the broker.

Producer with Raw JSON (No Registry)

var producerConfig = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    EnableIdempotence = true
};

using var producer = new ProducerBuilder<string, byte[]>(producerConfig).Build();

var orderPlaced = new OrderPlaced
{
    EventId = Guid.NewGuid().ToString(),
    OrderId = "ORD-001",
    CustomerId = "CUST-42",
    PlacedAtUtc = DateTime.UtcNow
};

// No compatibility check occurs. If OrderPlaced gains a new field tomorrow,
// old consumers deserializing with System.Text.Json will silently ignore it
// (unknown fields are dropped by default unless JsonUnmappedMemberHandling
// is set to Disallow).
var bytes = JsonSerializer.SerializeToUtf8Bytes(orderPlaced);

await producer.ProduceAsync(
    "orders.placed",
    new Message<string, byte[]>
    {
        Key = orderPlaced.OrderId,
        Value = bytes
    });

The raw JSON path is not wrong by definition — it is a deliberate trade-off. The problem is the failure mode: when the schema drifts, the consumer does not know. Under at-least-once delivery, where old messages can be replayed, this silent data loss compounds across every replayed message.

⚠️ Common Mistake: Assuming that [JsonRequired] or JsonUnmappedMemberHandling.Disallow on the POCO solves the schema drift problem. These shift the failure to deserialization time — better than silent corruption — but every rollout order still has to be worked out by hand, with nothing checking that it is safe, and they provide no compatibility guarantees across message versions already in the log.

Serialization Format vs. Schema Registry: Two Separate Decisions

A common conflation is treating the wire format and the registry as a single choice. They are separable.

Serialization format (Avro, Protobuf, or JSON described by a JSON Schema) determines how bytes are laid out on the wire, what native types are supported, and how evolution is expressed structurally.

Schema registry determines whether compatibility rules are enforced and how schemas are versioned, and it lets a consumer look up, by ID, the exact schema a given message was written with.

You can keep JSON on the wire, described by a registered JSON Schema, and still enforce compatibility through a registry. Conversely, you can use Avro's binary encoding without a registry — though then every reader must already know the exact writer schema (the schema the record was serialized with, as opposed to the reader schema it is decoded into), which is what makes independent producer/consumer deployment unsafe.

📋 Quick Reference Card: Wire Format vs. Schema Registry

🤔 Did you know? By default, Confluent's serializers prepend 5 bytes to each payload — a magic byte (0x00) and a 4-byte big-endian schema ID (Protobuf adds a short message-index list after it). Since Confluent.SchemaRegistry 2.10.1 (Confluent documents 2.13.0 and later as supported), the serializer can instead put a 16-byte schema GUID in a record header (SchemaIdSerializerStrategy.Header); the prefix stays the default. Either way, a consumer can fetch the exact schema a message was written with, however old, and resolve it locally against its own reader schema. That local resolution — made safe in advance by the registry's compatibility check — is what makes replay safe in practice, provided the check covered the versions involved. Plain BACKWARD compares a new version only with the latest, so reading records several versions old is guaranteed only under BACKWARD_TRANSITIVE (or FULL_TRANSITIVE).

✍️ Exercise: will the registry accept it?

Version 1 of an Avro value schema, registered under BACKWARD (the default), is { order_id: string, total: double } with no defaults. For each proposed version 2, say whether registration succeeds, and whether a producer-first rollout would be safe for consumers still on version 1.

1. Add currency: string with default "EUR".
2. Add currency: string with no default.
3. Remove total.
4. Rename total to total_amount, giving total_amount a default of 0.

Check your answer

1. Accepted. A new reader fills currency from the default when reading old data. Producer-first is also safe: old readers ignore the extra field. This change is fully compatible.
2. Rejected. A version-2 reader could not read version-1 records, which have no currency and no default to fall back on.
3. Accepted under BACKWARD — a new reader simply ignores total in old data. But producer-first is unsafe: version-1 readers need total, the new records lack it, and there is no default. BACKWARD assumes consumers upgrade first.
4. Accepted under BACKWARD, and it is the trap: new readers will read every old record's total as 0, because Avro matches fields by name and total_amount is missing from old data. Producer-first also breaks old readers, which lose total. The registry proves the bytes can be read, not that the numbers are right — a rename needs a migration plan (add the new field, dual-write, then retire the old one), not just a passing check.

The choice of format matters for .NET specifically because the code-generation story differs. Avro typically requires a build-time code generation step or a generic record approach. Protobuf has mature .NET tooling through protoc and the Grpc.Tools package. JSON Schema is the most familiar but sacrifices the binary efficiency and stricter typing of the other two. None of these choices is universally correct — the registry enforces compatibility rules regardless of which you pick.

---

Common Misconfiguration Patterns and How to Catch Them

Configuration mistakes with Kafka tend to fail silently or fail late. Unlike a compilation error, a misconfigured client will often start successfully, process thousands of messages, and only reveal the problem under specific runtime conditions — a rebalance, a first deployment, a topic with compaction enabled. The five patterns below are worth examining because each one looks reasonable at a glance and only reveals its flaw when you know what to look for.

Mistake 1: Idempotence + Too Many In-Flight Requests

When you set EnableIdempotence = true on a Confluent .NET producer, the client enforces a hard limit: MaxInFlight must not exceed 5. It matches the broker, which remembers sequence numbers for only the last five batches per producer and partition — with more in flight, a retried batch could fall outside that window and could no longer be recognised as a duplicate or kept in order.

// ⚠️ This configuration will throw at runtime, not at compile time
var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    EnableIdempotence = true,
    MaxInFlight = 10  // Build() throws InvalidOperationException — incompatible with idempotence
};

using var producer = new ProducerBuilder<string, string>(config).Build();
// Exception raised here, during producer construction

The failure happens during ProducerBuilder.Build(). In a hosted service this surfaces as a startup exception — traceable, but confusing if you don't already know the constraint exists. The corrected form is to let EnableIdempotence control the in-flight limit, or set MaxInFlight explicitly to any value ≤ 5.

The underlying history matters: documentation examples written before the idempotence defaults stabilized often showed higher in-flight values as performance tuning advice. If your team's producer configuration originated from copied snippets of uncertain age, this is worth auditing directly.

Mistake 2: The Fire-and-Forget Offset Commit

This is the most operationally dangerous pattern in this section because it produces at-most-once delivery semantics while every surface indicator suggests at-least-once. The code looks like responsible async consumer code. It compiles. It passes basic testing. Under normal operation it even works.

// Consumer config: EnableAutoCommit = true, EnableAutoOffsetStore = false
// (StoreOffset throws a KafkaException if EnableAutoOffsetStore is left at its default, true)

// ⚠️ This is at-most-once, not at-least-once
while (!cancellationToken.IsCancellationRequested)
{
    var result = consumer.Consume(cancellationToken);

    _ = ProcessMessageAsync(result.Message); // NOT awaited

    // StoreOffset runs immediately, before ProcessMessageAsync completes
    consumer.StoreOffset(result);
}

StoreOffset records the offset in the client's local store, and the next commit (manual or auto) will include it. Because ProcessMessageAsync is not awaited, the commit can occur before the handler has finished or even started meaningful work.

Timeline of the bug:

  t=0   Consume message at offset 47
  t=1   Launch ProcessMessageAsync (unawaited)
  t=2   StoreOffset(result) stores 48 (the next offset) immediately
  t=3   Auto-commit fires: offset 48 is committed to broker
  t=4   ProcessMessageAsync throws / process crashes
  t=5   Consumer restarts, reads from offset 48
        Message 47 is permanently skipped

The corrected pattern awaits the handler before storing the offset:

// ✅ Correct: offset is stored only after handler confirms success
while (!cancellationToken.IsCancellationRequested)
{
    var result = consumer.Consume(cancellationToken);

    try
    {
        await ProcessMessageAsync(result.Message);
        consumer.StoreOffset(result);
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, "Handler failed for offset {Offset}", result.Offset);
        throw; // or route to DLQ before rethrowing
    }
}

⚠️ Watch out: Code review tools and static analyzers will not catch this by default. The discard _ = someTask pattern is valid C# for intentional fire-and-forget scenarios. The Kafka consumer loop is one of the places where it causes silent data loss.

🎯 Key Principle: If your offset commit is not causally downstream of your handler completing successfully, you do not have at-least-once delivery — regardless of what your EnableAutoCommit or EnableAutoOffsetStore settings say.

Mistake 3: auto.offset.reset=latest on a New Consumer Group

auto.offset.reset controls what a consumer does when it joins a partition with no committed offset on record. The latest setting means: start reading from the end of the log as of the moment the consumer first connects. For a consumer group that has never committed, that is every partition.

Topic: order-events  (10,000 messages accumulated before deployment)

Consumer group: order-processor-v1  (first run, no committed offsets)
auto.offset.reset = latest

Result:
  Partitions all reset to their current end offset
  All 10,000 historical messages are silently skipped
  Consumer begins reading only messages produced after it started

No error is logged. No exception is raised. A schema registry does not help here — the data was structurally valid, correctly serialized, and entirely reachable. The consumer chose not to read it.

latest is a reasonable choice for a log-shipping or metrics consumer where historical data is irrelevant. It is also the default in Confluent.Kafka (librdkafka's auto.offset.reset), so the trap needs no copied config at all: a new event-processing group that never sets AutoOffsetReset skips its backlog. For services where processing historical messages on first run is required, set AutoOffsetReset = AutoOffsetReset.Earliest explicitly. For services where it genuinely should not matter, set Latest explicitly anyway, so the next developer can see it was a decision rather than an accident.

💡 Real-World Example: This pattern commonly surfaces after blue-green deployments where a new consumer group name is introduced for the green deployment. The green group has no committed offsets. With latest, the green group starts clean and silently misses messages produced while the deployment was in progress.

Mistake 4: Keys That Don't Carry Identity in Compacted Topics

Kafka log compaction retains only the most recent message for each unique key within a partition. This means the key is the identity of the entity being represented — not merely a routing label.

❌ Key used purely for partitioning:
   Key: "order"          <- all orders go to same partition, no per-entity identity
   Value: { orderId: 123, status: "shipped" }

Result after compaction:
   Only ONE message with key "order" is retained
   All other order events are compacted away

✅ Key carries entity identity:
   Key: "order-123"      <- unique per order, compaction is meaningful
   Value: { orderId: 123, status: "shipped" }

Result after compaction:
   Latest state for each orderId is retained
   The topic behaves as an order-state changelog

The schema design implication: your key schema must be stable and sufficiently granular for compaction semantics to hold. If you are using Avro or Protobuf with a schema registry, the key schema should be registered separately from the value schema. Changing the key schema after a compacted topic is in production requires careful coordination: with registry serializers in their default prefix mode the key bytes begin with the magic byte and schema ID, so any new key-schema version gives the same logical identity different bytes. Compaction then never supersedes its old records, and the default partitioner, which hashes those bytes, can route the entity to a different partition, breaking its per-key ordering.

🤔 Did you know? Kafka's compaction process runs asynchronously and does not compact the "head" of the log (recent, uncompacted segments). A key identity mistake in a new compacted topic will not be obvious until the log ages enough for compaction to run — often hours or days after deployment.

Mistake 5: Local POCO Deserialization Without Schema Registry Resolution

Deserializing Kafka messages into a locally defined C# class using raw bytes is the path of least resistance when starting out. It becomes a coupling mechanism at scale. The issue is not that local deserialization is wrong in isolation — it is what it eliminates: independent deployability.

When your consumer deserializes by matching bytes against a locally compiled POCO, the POCO definition and the message bytes must agree at the moment of deserialization. There is no intermediary that can negotiate compatibility between what the producer wrote and what the consumer expects.

Local POCO approach — tight coupling:

  Producer (v1 schema) ──writes──▶ Kafka ──reads──▶ Consumer (must have v1 POCO)

  Changing the schema:
    → Nothing checks the change: added fields are silently ignored,
      renamed or retyped fields become defaults or throw at runtime
    → A safe rollout order must be worked out by hand, every time

Schema-registry approach — loose coupling:

  Producer (v2 schema) ──writes──▶ Kafka ──reads──▶ Consumer (v1 reader schema)
                                                          │
                                          deserializer fetches v2 by its ID
                                          and resolves it against v1 locally;
                                          the registry checked v1/v2 compat
                                          when v2 was registered

With registry-backed serializers and the compatibility mode that matches your rollout — FORWARD (or FULL) when producers upgrade first — a consumer can keep running the old reader schema while the producer writes the new one. The registry has already refused any version that would break that. At read time the deserializer fetches the writer schema by ID, and Avro schema resolution ignores fields the old reader does not know. (Defaults apply in the other direction: when the reader has a field the writer lacked.) No coordinated deployment required.

📋 Quick Reference Card: Local POCO vs. Registry-Resolved Deserialization

A practical first step toward registry resolution without a full migration is to begin registering your existing schemas as documentation. Even before switching deserializers, a registry entry establishes a compatibility baseline that makes future changes auditable.

Catching These in Practice

Each of these five patterns shares a common property: they are invisible until they matter. The idempotence conflict throws at startup — catch it in CI with a configuration validation step that constructs the producer in a test context. The fire-and-forget commit passes all unit tests and fails only under handler failure — a targeted code review rule or Roslyn analyzer flagging _ = someTask patterns inside consumer loop methods helps here. The offset reset issue is catchable by including first-run integration tests that seed a topic before starting the consumer group and asserting that historical messages were processed. The key-identity and POCO-coupling issues surface during schema review — most effective when schema registration is part of your PR process rather than a post-deployment activity.

✍️ Exercise: review this consumer

A new billing service is about to be deployed against a topic that already holds three weeks of PaymentCaptured events:

var config = new ConsumerConfig
{
    BootstrapServers = "kafka:9092",
    GroupId = "billing-v2",
    EnableAutoCommit = true,
    EnableAutoOffsetStore = false
};
// ...
var result = consumer.Consume(ct);
_ = PostToLedgerAsync(result.Message.Value);
consumer.StoreOffset(result);

Find two defects, say what each one does in production, and fix them.

Check your answer

1. No AutoOffsetReset. billing-v2 is a new group, so it starts at the librdkafka default, latest, and silently skips three weeks of payments. Set AutoOffsetReset = AutoOffsetReset.Earliest.
2. Unawaited handler. StoreOffset runs before PostToLedgerAsync finishes, so the timer can commit past a payment whose ledger write later fails or never completes — at-most-once. await the handler and store the offset only after it succeeds; make the ledger write idempotent (for example keyed on the payment ID), because awaiting turns this into at-least-once.

The EnableAutoCommit = true / EnableAutoOffsetStore = false pairing itself is correct — it is what makes StoreOffset meaningful.

---

Key Takeaways and What Comes Next

By this point you've moved from the surface-level question — "how do I get messages from A to B?" — to the deeper question that actually governs production system design: "what does it mean for a message to arrive, and what must be true about its shape for that arrival to matter?" Those two questions are not independent, and this section organizes what you now understand into a structure you can carry into real decisions.

The Decision Matrix

Every Kafka producer and consumer configuration you write reflects a position on two axes: delivery mode and schema strategy. Neither axis has a universally correct setting. What matters is that the two choices are made together, deliberately, with awareness of the behavioral consequences each combination produces.

📋 Quick Reference Card: Delivery Mode × Schema Strategy

Every column needs the same schema-evolution discipline: a record read with the wrong contract is misread whether it arrives once or twice.

Minimum Viable Configurations

The three code blocks below show the minimum viable configuration for each delivery mode using the Confluent .NET client — for at-most-once the producer side; its consumer side is the commit-before-handler loop shown earlier. Consumer & Manual Commits covers offset management in depth; what matters here is seeing where each decision surfaces in actual configuration code.

At-Most-Once Producer

// At-most-once: fire and forget
// Acks.Leader means the leader acknowledges but does not wait for follower replication.
// Appropriate when throughput matters more than durability — e.g., high-volume metrics.
var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    Acks = Acks.Leader,
    MessageSendMaxRetries = 0 // no retries — loss is acceptable, duplicates are not
};

using var producer = new ProducerBuilder<string, string>(config).Build();

producer.Produce("metrics-topic", new Message<string, string>
{
    Key = "cpu-utilization",
    Value = "{\"host\":\"web-01\",\"value\":0.73}"
});

// On shutdown: producer.Flush(TimeSpan.FromSeconds(5)) — disposing without a flush
// drops whatever is still queued.

At-Least-Once Consumer with Idempotent Handler

// At-least-once: commit after processing, handle duplicates via event ID.
// The schema for OrderPlacedEvent must include a stable EventId field.
var config = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "order-processor",
    AutoOffsetReset = AutoOffsetReset.Earliest,
    EnableAutoCommit = false
};

using var consumer = new ConsumerBuilder<string, OrderPlacedEvent>(config)
    // Registry-backed, not raw JSON. The consumer takes only sync deserializers, e.g.
    // new AvroDeserializer<OrderPlacedEvent>(schemaRegistry).AsSyncOverAsync()
    .SetValueDeserializer(schemaRegistryDeserializer)
    .Build();

consumer.Subscribe("orders-topic");

while (!cancellationToken.IsCancellationRequested)
{
    var result = consumer.Consume(cancellationToken);

    // Record the EventId and apply the change in ONE database transaction, so a crash
    // can never leave the effect without the marker (or the marker without the effect).
    await using var tx = await db.BeginTransactionAsync(cancellationToken);
    if (await idempotencyStore.TryMarkProcessedAsync(result.Message.Value.EventId, tx)) // false = seen before
        await orderService.HandleAsync(result.Message.Value, tx);
    await tx.CommitAsync(cancellationToken);

    consumer.Commit(result);
}

The schemaRegistryDeserializer is not a detail to defer — it is what makes EventId reliable across schema versions. If a future version renames the field without a default, the registry rejects it under BACKWARD; a rename with a default passes, and the new consumer then reads that default as the ID of every older record. Treat the identity field as frozen: never rename it.

Effectively-Once: Transactional Producer Sketch

// Effectively-once: transactional producer with a unique TransactionalId.
// EnableIdempotence is automatically set to true by TransactionalId.
var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    TransactionalId = "inventory-adjustment-producer-01"
};

using var producer = new ProducerBuilder<string, InventoryAdjustedEvent>(config)
    .SetValueSerializer(schemaRegistrySerializer)
    .Build();

producer.InitTransactions(timeout: TimeSpan.FromSeconds(30));

try
{
    producer.BeginTransaction();

    await producer.ProduceAsync("inventory-events", new Message<string, InventoryAdjustedEvent>
    {
        Key = adjustment.SkuId,
        Value = adjustment
    });

    // In a read-process-write pattern, SendOffsetsToTransaction goes here
    // to atomically commit the consumed offset alongside the produced message
    // (shown in full in "Transactions: Extending the Guarantee Across Partitions").

    producer.CommitTransaction();
}
catch (KafkaTxnRequiresAbortException)
{
    producer.AbortTransaction();
    throw; // or rewind the consumer to its committed offsets and retry the batch
}
// Fatal errors (e.g. fenced by a newer instance) propagate without an abort attempt:
// that producer must be disposed and recreated.

Summary

Delivery mode is a configuration decision with behavioral consequences. Calling StoreOffset right after an async handler you did not await isn't a performance optimization — it's a silent adoption of at-most-once semantics inside an at-least-once configuration.

Schema strategy is what makes delivery guarantees load-bearing. A guarantee that a message arrives is only as useful as the guarantee that it means the same thing it meant when it was produced. A schema registry with compatibility rules is the enforcement point for that second guarantee.

The three delivery modes change what the schema must carry, not how carefully it must evolve. At-most-once streams benefit from sequence numbers or timestamps so gaps can be noticed. At-least-once needs a stable identity so duplicates can be recognised. Effectively-once inside Kafka needs neither for correctness — but its outputs usually end up somewhere that does. Every mode needs the same evolution discipline.

The decision matrix is a checklist, not a preference. If the business consequence of duplication is a double charge or an incorrect inventory count, duplicates must be made harmless — with Kafka transactions if the pipeline stays inside Kafka, with an idempotent write if it ends in a database. If the business consequence of loss is a missing data point in a dashboard that refreshes every second, at-most-once is appropriately calibrated.

⚠️ The single most common error in applying this material: choosing at-least-once delivery without designing the schema to support it. At-least-once without a stable event ID field and an idempotency store is still at-least-once — nothing is lost — but every redelivery repeats its side effects: a double charge, a double shipment. You pay for redelivery without making it safe.

🧠 Mnemonic — DID: Delivery mode, Identity field in schema, Deduplication handling in consumer. If all three align, at-least-once delivery gives you effectively-once processing: duplicates still arrive, but they change nothing.

✍️ Exercise: where do Kafka transactions reach?

For each pipeline, say whether Kafka transactions alone can make it effectively-once end to end, and if not, what you would add.

1. Consume orders, enrich each record, produce to orders-enriched.
2. Consume orders, insert each order into PostgreSQL.
3. Consume orders, call a payment provider's HTTP API, produce payments with the result.

Check your answer

1. Yes. Read → process → write stays inside Kafka: produce to orders-enriched and SendOffsetsToTransaction in one transaction, with downstream readers on read_committed.
2. No. The insert is outside Kafka. Use at-least-once with an idempotent insert — an upsert on the order ID, or a processed-events table written in the same database transaction — or store the consumed offset in PostgreSQL in that same transaction.
3. No, and this is the hardest case. The transaction can make the payments record and the offset atomic, but not the HTTP call: after an abort or crash the call runs again. Pass the order or event ID as the provider's idempotency key so a repeated call does not charge twice.

Where to Go Next

Consumer & Manual Commits is the place to revisit offset management in detail: manual commit patterns, StoreOffset vs. Commit, and the EnableAutoOffsetStore interaction this lesson relied on.

Akka.Streams.Kafka (next) expresses the same at-least-once contract as committable sources and batched commits, and Production Patterns & Observability covers running these consumers in production. Standing up a Schema Registry and generating Avro or Protobuf types for .NET is not covered by a lesson in this roadmap yet; the documentation of the Confluent.SchemaRegistry.Serdes.* packages is the place to start.