Consumer Groups & Offsets
One partition is consumed by exactly one group member at a time. Offsets are your real checkpoints — not message acknowledgements.
Why Consumer Groups Exist: Scaling Reads Beyond a Single Consumer
Imagine an order-processing service reading from a Kafka topic that receives a steady stream of new orders. At launch, one consumer instance handles the load comfortably. Then a promotion goes live, order volume triples, and the same consumer starts falling further behind with every poll — the backlog grows, downstream fulfillment slows, and nobody touched the code. The topic didn't change shape; the read side simply couldn't keep pace. Why does adding more consumers fix this when the data lives in one place? And why does Kafka split the work by partition instead of handing out individual messages the way a traditional task queue would? Those two questions are what this section answers, by introducing the consumer group — the mechanism Kafka uses to let multiple consumers cooperatively read a topic without duplicating or losing work.
The Single-Consumer Bottleneck
A Kafka topic is not one continuous stream — it is split into partitions, each an independently ordered, append-only log. A single consumer instance that subscribes to a topic will, by default, be handed every partition and must read them all itself. Internally, the client still processes work sequentially relative to what your application code can handle: it polls for records, your handler runs, and only then does the loop ask for more. If message volume outpaces how fast that loop can process each record, the consumer's read position falls further and further behind the producer's write position. That gap is consumer lag, and it is the direct symptom of a throughput mismatch between one reader and a topic being written to faster than it's being drained.
The order-processing example makes this concrete. Suppose the orders topic has eight partitions and, during a peak sales event, producers write far more orders per second than the single consumer instance can validate, persist, and forward to a fulfillment queue. Vertical scaling — giving that one process more CPU or memory — helps only if the bottleneck is raw compute, and even then it hits a ceiling: a single thread pumping through a poll loop can only do so much sequential work per second. The structural fix is to have more than one consumer instance sharing the workload, each responsible for a subset of the partitions. That's the problem consumer groups exist to solve.
Defining the Consumer Group
A consumer group is a named set of consumer instances that coordinate to divide the partitions of one or more topics among themselves, so that the group as a whole reads the full topic while each member reads only a slice of it. Instances declare their membership by configuring the same GroupId; Kafka uses that shared identifier to know which consumers should split ownership together.
🎯 Key Principle: A consumer group parallelizes reads by dividing partitions, not by dividing individual messages. This is the single most important structural fact to internalize before anything else about groups makes sense.
That distinction matters because it's easy to picture a group the way you'd picture a thread pool pulling tasks off a shared queue — any idle worker grabs the next available item. Kafka consumer groups don't work that way. Ownership is assigned at the partition level: a given partition is handed to exactly one consumer instance in the group, and that instance reads every message in that partition in order, for as long as it holds that assignment. The exact rule governing assignment, and what happens when the number of consumers doesn't match the number of partitions, is the focus of the next section — this one only needs you to understand that partitions, not messages, are the unit that gets distributed.
💡 Mental Model: Think of a topic's partitions as a stack of ordered logbooks, and a consumer group as a small team assigned to read the whole stack. Instead of the team fighting over which page to read next, each person is handed entire logbooks to read cover to cover. Scaling the team means handing out logbooks differently, not handing out pages.
Scaling the Order-Processing Example
Returning to the scenario: with eight partitions on the orders topic, a single consumer instance owns all eight. If that team places three more instances into the same consumer group (same GroupId, subscribed to the same topic), Kafka's group coordination redistributes the eight partitions across the four instances — two apiece. Each instance now keeps pace with a quarter of the traffic a single reader previously had to absorb, which is why adding instances to a group is the standard scaling lever for read throughput.
Here is a minimal illustration in .NET using the Confluent.Kafka client. Building a full poll loop is the subject of "Building a Minimal Consumer with Confluent.Kafka" later in this lesson; this snippet shows only the piece relevant here — the shared GroupId that makes multiple instances cooperate as one group:
using Confluent.Kafka;
// Every instance of the order-processing service uses this same GroupId.
// That shared value is what tells Kafka these instances belong to one group
// and should split partition ownership rather than each reading everything.
var config = new ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "order-processing-service",
AutoOffsetReset = AutoOffsetReset.Earliest
};
using var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe("orders");
// Each running instance of this same code, pointed at the same GroupId,
// receives a subset of the "orders" topic's partitions.
Deploy four copies of that process and you have a four-member consumer group. Kafka's group coordination (detailed in the Consumer Group Mechanics lesson) assigns partitions across those four members automatically — your application code doesn't choose which partitions it gets. What matters here is the shape of the outcome: eight partitions split evenly across four readers rather than one reader draining all eight serially.
🤔 Did you know? A GroupId is just a string, and Kafka treats every distinct string as a separate, independent group. Two services can subscribe to the same topic under different GroupId values, and each group independently receives a full copy of every message — the groups don't compete with or affect each other at all. That's what lets an order-processing group and a fraud-detection group both read the same orders topic in full, each at its own pace.
What This Lesson Owns, and What Comes Next
This lesson establishes the mental model you need before touching deeper mechanics: the partition-ownership rule that governs how work is divided, the distinction between an offset as a checkpoint versus a per-message acknowledgement, a minimal working .NET consumer, and the mistakes that commonly break it.
What it does not cover in depth — because each deserves its own space — is the internal group protocol Kafka's broker-side coordinator runs to assign partitions, the mechanics of where committed offsets are stored, and the triggers and consequences of a rebalance. Those are the Consumer Group Mechanics, Offset Management, and Rebalancing & Backpressure lessons. Absorbing all of that alongside the basic ownership model tends to blur the distinction that makes consumer groups tractable: first understand what gets divided and why it scales reads, then learn how Kafka performs that division.
💡 Real-World Example: A payment-notification service reading from a single-partition topic gains nothing from adding a second consumer instance to its group — with one partition, one instance holds it and the second sits idle. This previews the ownership rule the next section covers in full, but it's worth flagging now: scaling a consumer group only helps up to the number of partitions the topic has, so partition count is a capacity decision made well before you write any consumer code.
Why Partitions, Not Messages
It's worth pausing on why Kafka divides by partition instead of by message, since the alternative feels more intuitive if you've worked with queues where any free worker grabs the next message. A partition is an ordered log, and Kafka guarantees ordering only within a partition. If individual messages were handed to whichever consumer happened to be free, two messages written next to each other in the same partition could be processed by two instances in an unpredictable order — one might finish before the other started. For many use cases that's tolerable, but for anything where sequence matters within a logical unit (all events for one order ID, keyed to the same partition), losing that ordering would silently change your application's correctness. Assigning whole partitions to single consumers preserves in-partition ordering as a hard guarantee even while parallelizing across the group.
⚠️ Common Mistake: Assuming that adding consumer instances always increases throughput. A team running eight instances against an eight-partition topic doubles the fleet to sixteen under load and sees no improvement at all — the eight new instances join the group, get zero partitions, and idle. The ceiling is set by partition count, a trade-off examined in depth in the next section.
This section's job was narrow: show why a single consumer stalls against real throughput, define the consumer group as a named set of consumers splitting partition ownership, and make clear that the thing being split is partitions rather than messages. Holding onto that picture — a fixed number of ordered logs handed out whole to group members — is what makes every subsequent detail about assignment rules, rebalances, and offset commits click into place instead of feeling like arbitrary broker behavior.
The Partition Ownership Rule: One Partition, One Consumer
Once a group exists, something has to decide who reads what. Kafka's answer is a single, unbending rule that shapes every capacity decision you'll make around a consumer group.
The Core Rule
🎯 Key Principle: Within a single consumer group, each partition is assigned to exactly one consumer instance at a time. Two members of the same group will never be handed the same partition simultaneously.
This is an ownership assignment, not a load-balancing suggestion. If topic orders has 6 partitions and your group order-processor has 3 running instances, the coordinator hands out those 6 partitions across the 3 consumers — commonly 2 apiece, though the exact split depends on the assignment strategy. What never happens is two instances both pulling from partition 3 at the same time. That guarantee is what lets you reason about ordering: since a partition is an ordered log and only one consumer reads it at once, message order within that partition is preserved from the consumer's point of view. Break that guarantee and you'd lose the one thing partitions promise you.
Topic: orders (6 partitions)
Group: order-processor (3 consumer instances)
Consumer A ← partition 0
Consumer A ← partition 1
Consumer B ← partition 2
Consumer B ← partition 3
Consumer C ← partition 4
Consumer C ← partition 5
Each arrow is an exclusive ownership link for as long as group membership stays stable. Exactly how the coordinator decides which consumer gets which partitions, and what happens the moment membership changes, is the internal protocol covered in "Consumer Group Mechanics" — for this section, treat the assignment as a given and focus on what it implies for scaling.
Over-Provisioning: Idle Consumers
The rule cuts both ways, and the first consequence surprises teams who assume that adding instances always adds throughput. Scale order-processor from 3 instances to 8 against a 6-partition topic and two of those 8 consumers get zero partitions. They're fully connected, fully healthy, sitting in the group — and doing nothing.
Topic: orders (6 partitions)
Group: order-processor (8 consumer instances)
Consumer A ← partition 0
Consumer B ← partition 1
Consumer C ← partition 2
Consumer D ← partition 3
Consumer E ← partition 4
Consumer F ← partition 5
Consumer G ← (idle, no partitions)
Consumer H ← (idle, no partitions)
⚠️ Common Mistake: Assuming more pods means more throughput. An order-processing service behind a Kubernetes horizontal pod autoscaler might scale from 6 replicas to 12 under load, expecting proportional speedup. With 6 partitions, the extra 6 replicas idle in the group, burning compute and connection overhead while contributing nothing. The fix isn't more consumers — it's more partitions, a topic-level change with its own tradeoffs (remapping cost, ordering scope) covered in the Topics, Partitions & Keys lesson.
Those idle consumers aren't harmless bystanders either — they still participate in group membership, still send heartbeats, and still factor into rebalances when they join or leave. You're paying coordination overhead for zero consumption capacity.
Under-Provisioning: One Consumer, Many Partitions
The opposite imbalance is more forgiving. Run order-processor with 2 instances against those same 6 partitions and each consumer simply owns more than one: Consumer A might get partitions 0, 1, and 2, while Consumer B gets 3, 4, and 5.
A single instance handling multiple partitions doesn't need multiple threads or any special code — the poll loop handles it transparently. Each call to Consume() can return a message from any partition that consumer owns, interleaved in whatever order they arrive. (Confluent.Kafka's Consume is synchronous and blocking; there is no ConsumeAsync overload, which is a common assumption for developers arriving from async-first .NET libraries.)
// A single consumer instance can own several partitions at once.
// The poll loop doesn't change — Consume() transparently returns
// messages from whichever owned partition has data ready.
while (!cancellationToken.IsCancellationRequested)
{
var result = consumer.Consume(cancellationToken);
// result.Partition tells you which of this consumer's
// owned partitions the message came from.
Console.WriteLine(
$"Partition {result.Partition.Value}, " +
$"Offset {result.Offset.Value}: {result.Message.Value}");
}
The tradeoff is throughput per partition, not correctness: Consumer A is serially working through three partitions' worth of traffic instead of one. If message handling is CPU- or I/O-bound and slow, a consumer owning three partitions processes each more slowly in wall-clock terms than it would owning just one, because it's dividing its attention. That's a real capacity constraint, but it's controlled degradation — nothing is lost or duplicated, throughput just narrows.
Groups Are Independent: Full-Topic Fan-Out
Every example so far has stayed inside one group, where partitions are divided so each message is handled once. That's easy to confuse with a traditional competing-consumers queue, where any worker can dequeue any message and the pool competes for a shared backlog. Kafka consumer groups look similar until you add a second group.
❌ Wrong thinking: "If I add a second consumer group reading the same topic, the two groups will split the partitions between them, so each group processes half the messages."
✅ Correct thinking: Each consumer group maintains its own independent set of partition assignments and its own committed offsets. A second group reading orders doesn't compete with the first for partitions at all — it gets its own full ownership assignment across its own members, and reads every message in the topic from its own offset position.
Group: order-processor (3 instances) — reads all 6 partitions
Consumer A ← partitions 0, 1
Consumer B ← partitions 2, 3
Consumer C ← partitions 4, 5
Group: fraud-detector (2 instances) — reads all 6 partitions, independently
Consumer X ← partitions 0, 1, 2
Consumer Y ← partitions 3, 4, 5
Every message published to orders is delivered once to order-processor's collective assignment and once, separately, to fraud-detector's. Neither group knows the other exists. This is what makes Kafka useful for fan-out patterns — feeding both an order-fulfillment pipeline and a fraud-analysis pipeline from the same stream, without duplicating the topic or the producer.
💡 Mental Model: A topic is a broadcast, and each consumer group an independent tuned-in receiver. Within a receiver, the six partitions of programming are split among however many radios that receiver owns — no two radios in the same receiver play the same channel. But a second receiver across the room hears the entire broadcast too, on its own radios, unaware of the first.
⚠️ Common Mistake: Mistaking group isolation for load-sharing across groups. It's tempting to assume adding a second group to a busy topic relieves load on the first — it won't, because the two groups share no offsets, assignments, or partition pool. This is different from a mistyped GroupId accidentally splitting one intended group into several unintentional full-topic readers, which is covered in "Common Pitfalls When Consuming with Confluent.Kafka."
The Capacity-Planning Trade-off
Put over- and under-provisioning together and you get the practical rule: partition count sets the hard ceiling on parallelism achievable within a single consumer group. No matter how many instances you deploy, a group can never have more actively working consumers than the topic has partitions.
This has a direct implication for topic design. If you anticipate orders eventually needing 20 parallel consumer instances at peak but provision the topic with 4 partitions, you've capped your own future scaling regardless of how much compute you're willing to throw at it. Partition count can be increased later, but it is not a transparent operation — it changes the mapping of keys to partitions, which breaks ordering guarantees for anything relying on key-based affinity, and it doesn't redistribute already-written data. It also cannot be decreased; shrinking means creating a new topic and migrating. That's a topic-design concern upstream of consumer behavior, but it's this ownership rule that makes the ceiling real.
<table> <tr><th>🔧 Scenario</th><th>🎯 Partition:Consumer Ratio</th><th>📚 Outcome</th></tr> <tr><td>🔒 Balanced</td><td>6 partitions, 6 consumers</td><td>Each consumer owns exactly 1 partition — maximum parallelism reached</td></tr> <tr><td>🔒 Under-provisioned</td><td>6 partitions, 2 consumers</td><td>Each consumer owns 3 partitions — correctness preserved, throughput narrowed</td></tr> <tr><td>🔒 Over-provisioned</td><td>6 partitions, 8 consumers</td><td>2 consumers sit idle — wasted capacity, no throughput gain</td></tr> </table>
💡 Pro Tip: When sizing a consumer group, treat partition count as the true capacity number and instance count as a variable you tune up to that ceiling — not past it. Provision enough partitions for the parallelism you expect at peak, then scale instances within that range as load changes, rather than repeatedly resizing the topic.
Offsets Are Checkpoints, Not Per-Message Acknowledgements
It's tempting to picture a Kafka consumer working like a message queue with acknowledgements: fetch a message, process it, "ack" it, move on. That model is wrong in a way with real consequences the first time your consumer crashes mid-batch. Kafka doesn't track the status of individual messages at all. It tracks a single number per partition, per consumer group, and understanding exactly what that number means — and when it moves — is the difference between predicting your system's failure behavior and being surprised by it.
Two Different Numbers Called "Offset"
The word "offset" gets used for two related but distinct things, and conflating them is where confusion starts.
The log offset is a message's permanent address inside a partition. Every record appended gets the next integer in sequence — 0, 1, 2, 3. This number is immutable and belongs to the message; it never changes no matter which consumer reads it or how often.
The committed offset belongs not to a message but to a consumer group. It's a single integer, stored per partition, answering one question: "where should this group resume reading if it starts fresh right now?" A committed offset of 47 means "this group has finished with everything through offset 46; hand it offset 47 next."
Partition 0 log:
offset: 0 1 2 3 4 5 6 7
message: [A] [B] [C] [D] [E] [F] [G] [H]
^
committed offset = 5
(group resumes at message F)
The critical thing to notice: the committed offset is a bookmark, not a ledger of which individual messages succeeded or failed. Nothing anywhere says "message D was processed correctly, message C threw an exception." There's only the bookmark at position 5, silently implying that everything before it — A, B, C, D, and E — is considered done, whether or not that's actually true.
🎯 Key Principle: One Bookmark, Not Many Receipts
A traditional message queue often gives you per-message acknowledgement: you ack message 17 specifically, independent of whether 16 was acked. Kafka's commit model doesn't work that way. Committing an offset advances one shared checkpoint, and that checkpoint only moves forward. You cannot commit "message 16 succeeded, message 17 failed" — you can only say "advance the bookmark to 18," which implicitly claims both succeeded, or not advance it at all, which implicitly claims nothing past the last checkpoint succeeded.
This matters because a batch of messages rarely fails atomically. Suppose your consumer pulls ten messages, successfully writes eight to a database, then throws on the ninth. There is no Kafka mechanism to say "commit the first eight, retry the ninth." Your code decides, at the granularity of whatever offset value it chooses to commit, where the bookmark goes — and Kafka trusts that decision completely. It doesn't inspect your business logic or verify your database writes; it stores the number you gave it.
💡 Mental Model: The committed offset is a bookmark in a paperback, not a set of highlighted sentences. You can't mark "I definitely read page 40 but skipped page 41." You place the bookmark after the last page you're confident you finished, and next time you start from there.
Why Commit Timing Is the Whole Game
Because the commit is just a number and Kafka enforces no relationship between it and actual processing outcomes, when your code advances it determines your delivery guarantees.
Commit the offset before processing and you're telling Kafka "I'm done with this" before you know that. If the consumer crashes while doing the real work — writing to a database, calling an API — that message is never processed, but on restart the consumer resumes from the committed offset, which already skipped past it. The message is silently lost from this group's perspective even though it's still sitting untouched in the log. That's at-most-once: each message is delivered zero or one times, never more, occasionally zero.
Commit the offset after processing completes and you avoid that loss but introduce the opposite risk. If the consumer crashes after finishing the work but before the commit lands, the next consumer to take that partition resumes from the older offset and reprocesses messages already handled. That's at-least-once: every message is delivered one or more times, never zero, occasionally more.
Commit BEFORE processing:
commit(offset) → process(message) → [crash here] → message lost
Commit AFTER processing:
process(message) → [crash here] → commit(offset) never runs → message reprocessed on restart
No timing choice eliminates both risks using offsets alone — that would require processing and committing to be one atomic operation spanning two independent systems (your business logic's side effects and Kafka's offset store), which is why exactly-once semantics require additional machinery rather than clever commit placement.
⚠️ The .NET Default Leans at-Most-Once, Not at-Least-Once
Most Kafka material tells you the defaults give you at-least-once. That is true of the Java client, whose auto-commit commits the offsets returned by the previous poll on the next poll — so a crash mid-batch leaves them uncommitted and they replay.
Confluent.Kafka behaves differently, because librdkafka splits the operation into two steps with two separate settings:
<table> <thead><tr><th>⚙️ Setting</th><th>📦 Default</th><th>🎯 What it does</th></tr></thead> <tbody> <tr><td><code>EnableAutoOffsetStore</code></td><td><code>true</code></td><td><strong>Stores</strong> a message's offset the moment <code>Consume()</code> hands it to your code — before your handler runs</td></tr> <tr><td><code>EnableAutoCommit</code></td><td><code>true</code></td><td><strong>Commits</strong> whatever is currently stored, on a timer</td></tr> <tr><td><code>AutoCommitIntervalMs</code></td><td><code>5000</code></td><td>How often that timer fires</td></tr> </tbody> </table>
Read those together and the consequence is stark: with defaults, the bookmark can move past a message while your handler is still working on it, or before the handler has run at all. A crash in that window loses the message with no error and no trace. The .NET defaults give you the commit-before-processing shape — at-most-once — even though the code reads as though processing comes first.
Store vs Commit: The Two Correct Pairings
There are two configurations that give you the at-least-once behavior most business logic wants, and one combination that silently records nothing at all.
Pairing A — take full manual control. Turn auto-commit off and commit explicitly after the handler succeeds. Simple to reason about; costs a synchronous broker round trip per commit, so commit per batch rather than per record in throughput-sensitive services.
var config = new ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "order-processing-group",
EnableAutoCommit = false // nothing moves unless we say so
};
// ...in the poll loop:
var result = consumer.Consume(cts.Token);
Process(result.Message.Value); // work FIRST
consumer.Commit(result); // then the bookmark moves
Pairing B — keep the background committer, control what it's allowed to commit. Leave auto-commit on but disable automatic offset storing, then call StoreOffset yourself after the handler succeeds. You get librdkafka's efficient batched background commits, and nothing is ever committed for a record your handler didn't finish.
var config = new ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "order-processing-group",
EnableAutoCommit = true, // keep the background committer...
EnableAutoOffsetStore = false // ...but only let it commit what WE store
};
// ...in the poll loop:
var result = consumer.Consume(cts.Token);
Process(result.Message.Value); // work FIRST
consumer.StoreOffset(result); // marks this offset eligible to be committed
⚠️ Common Mistake: Combining EnableAutoCommit = false with StoreOffset. Stored offsets are drained by the background committer, and with auto-commit off there is no committer — so nothing is ever committed, and every restart replays from the group's last real checkpoint. The code looks correct, runs without error, and records no progress at all. Pick one pairing; never mix them.
Worked Scenario: Crash Mid-Batch
Walk through a concrete case. A consumer in group order-processing is assigned partition 0, sitting at committed offset 100, configured with Pairing A (EnableAutoCommit = false). It calls Consume() in a loop and pulls messages 100 through 104, handling each then committing.
Starting state: committed offset = 100
Step 1: process offset 100 → success → commit(101) [committed = 101]
Step 2: process offset 101 → success → commit(102) [committed = 102]
Step 3: process offset 102 → success → CRASH before commit(103) runs
The process dies right after finishing the work for offset 102 — say it wrote a database row — but before the commit for 103 reaches the broker. The committed offset is frozen at 102, even though offset 102's message was fully handled.
When the consumer (or a replacement in the same group) restarts and rejoins, it asks the broker for the last committed offset on partition 0, gets 102, and resumes fetching from there. It processes message 102 again — the very message it finished just before crashing — followed by 103 and 104, which it never reached.
Restart: fetch resumes at committed offset = 102
Step 1 (again): process offset 102 → reprocessed (duplicate)
Step 2: process offset 103 → first time
Step 3: process offset 104 → first time
Notice what did not happen: Kafka lost no messages, and it did not need to know that offset 102 had already succeeded. It replayed everything from the last known bookmark forward. The duplicate is the direct consequence of the commit happening after processing but the crash landing in the gap between finishing the work and recording that fact. If this consumer's downstream effect — charging a payment, sending an email — isn't safe to run twice, this replay is where real damage happens, which is why idempotent processing is a standard companion to at-least-once consumption rather than an optional nicety.
⚠️ Common Mistake: Assuming "the message was consumed" means "the message was committed." A message can be fetched, handed to your code, fully processed, and used to produce side effects — and still count as unconsumed from Kafka's perspective if the commit never happened. Consume() returning a message and the group's checkpoint advancing are two separate events, and the gap between them is exactly where crashes cause reprocessing.
Seeing the Two Offsets in Code
The distinction between a message's own offset and the group's committed offset is visible directly in the API:
// consumeResult.Offset is the LOG offset: this message's fixed position
// in the partition. It never changes and belongs to the message.
ConsumeResult<string, string> consumeResult = consumer.Consume(cancellationToken);
Console.WriteLine($"Log offset of this message: {consumeResult.Offset}");
// Process the message first...
ProcessOrder(consumeResult.Message.Value);
// ...then advance the GROUP's committed offset. This updates the bookmark,
// not a per-message flag. Everything up to and including this message is
// now implicitly "done" as far as the group is concerned.
// (Requires EnableAutoCommit = false — see Pairing A above.)
consumer.Commit(consumeResult);
And here's how to read back the group's current bookmark independent of any single message — useful for reasoning about where a restart will resume from:
// Ask the broker for this group's committed offset on a specific partition.
// This returns the checkpoint, not any information about individual messages.
var topicPartition = new TopicPartition("orders", partition: 0);
var committedOffsets = consumer.Committed(
new[] { topicPartition },
timeout: TimeSpan.FromSeconds(5));
var offset = committedOffsets[0].Offset;
// A group that has never committed on this partition returns Offset.Unset
// (-1001), not 0 — "no bookmark yet" is distinct from "bookmark at the start."
Console.WriteLine(offset == Offset.Unset
? "No committed offset — AutoOffsetReset decides where reading begins"
: $"Group will resume at offset: {offset}");
Both snippets are simplified to isolate the offset concepts — real consumer code needs exception handling around Consume() and a clean shutdown path, which the next section covers.
❌ Wrong thinking: "I called Commit(), so Kafka now knows message X succeeded."
✅ Correct thinking: "I called Commit(), so the group's resume point moved forward. Kafka has no opinion on whether message X — or anything before it — actually succeeded."
Why This Distinction Outlasts Any Specific API
It's worth dwelling on why the bookmark model exists rather than per-message acknowledgement, because it shapes everything downstream. Tracking a single integer per partition is cheap: the broker needs no growing table of acknowledgement flags for every message ever produced, and a consumer rejoining after a rebalance only asks "where do I resume?" rather than reconciling a status history. That efficiency is precisely the trade-off — you gain a lightweight, scalable checkpoint, and you give up selective acknowledgement within a batch. Kafka's ordering-by-partition guarantee reinforces this: because messages within a partition are strictly ordered and one consumer processes them in that order, a single "resume from here" number fully describes progress. That same property makes the crash-and-resume scenario deterministic rather than ambiguous — the replacement consumer knows exactly which messages come after the committed offset, even though it can't know which were already handled.
Building a Minimal Consumer with Confluent.Kafka
Everything discussed so far — partition ownership, offset semantics — is abstract until you write the code that joins a group and pulls messages off a partition. This section builds that code piece by piece using the official Confluent.Kafka client. The goal isn't a production-ready service; it's a correct, minimal skeleton you'll extend in later lessons.
Configuring the Consumer
Every consumer starts with a ConsumerConfig, a strongly typed wrapper around the settings the underlying client library accepts. Four of them matter before you write a line of consuming logic.
BootstrapServers is the initial contact point — one or more broker addresses the client uses to discover the rest of the cluster. It needn't list every broker; the client learns full cluster metadata after the first successful connection. A phone number to dial, not the whole address book.
GroupId determines which consumer group this instance joins. This single field turns an isolated client into a member of a coordinated group — every consumer sharing the same GroupId against the same topic has partitions divided among them per the one-partition-one-consumer rule. Get it wrong — a typo, a copy-paste error, an environment suffix left in by accident — and you silently create a second group that reads the entire topic independently; that failure mode gets its own treatment in "Common Pitfalls."
AutoOffsetReset controls what happens the very first time this group reads a given partition, when no committed offset exists to resume from. Earliest starts at the beginning of the retained log; Latest starts from whatever is produced from this moment on, ignoring history. It applies only when no valid committed offset exists; it does not override an existing checkpoint. A new group pointed at a long-running topic with AutoOffsetReset left at its default of Latest silently skips every message already in the log — a common surprise for anyone expecting to "replay from the start."
EnableAutoCommit and its companion EnableAutoOffsetStore decide when the bookmark moves. Both default to true, which as the previous section showed gives you at-most-once behavior. Pick one of the two pairings deliberately rather than inheriting the defaults:
using Confluent.Kafka;
var config = new ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "order-processing-group",
AutoOffsetReset = AutoOffsetReset.Earliest,
// Pairing A: we commit explicitly, after the handler succeeds.
// (Pairing B is EnableAutoCommit = true + EnableAutoOffsetStore = false.)
EnableAutoCommit = false
};
💡 Pro Tip: Treat GroupId as a deployment-time configuration value, not a hardcoded literal. Two services with slightly different GroupId strings pointed at one topic each receive their own full copy of the data — sometimes exactly what you want, sometimes a bug.
Constructing the Client and Subscribing
With configuration in hand, the client is built through ConsumerBuilder<TKey, TValue>, a generic builder that also lets you specify how keys and values are deserialized. For plain string payloads the built-in Deserializers.Utf8 is enough; typed payloads (Avro, JSON, Protobuf) plug into the same builder but are outside this section's scope.
using var consumer = new ConsumerBuilder<Ignore, string>(config)
.Build();
consumer.Subscribe("order-events");
The key type here is Ignore, meaning the code doesn't care about message keys — only the value matters. Calling Subscribe does not connect to a partition immediately; it registers interest in the topic and hands partition assignment to the group coordination protocol running broker-side. That protocol is the subject of the Consumer Group Mechanics lesson; for now it's enough that Subscribe is a declaration of intent, and the actual assignment arrives asynchronously once you start calling Consume.
⚠️ Common Mistake: Calling Subscribe repeatedly, expecting topics to accumulate. Subscribe replaces the current subscription set entirely rather than adding to it — for multiple topics, pass them in one call: consumer.Subscribe(new[] { "order-events", "order-events-retry" }).
The Poll Loop
Kafka consumers are pull-based: nothing is pushed to your process, so you must actively ask for the next message. This is the poll loop — a while loop repeatedly calling consumer.Consume(cancellationToken), blocking until a message is available or the token is cancelled, and returning a ConsumeResult<TKey, TValue> containing the message, its topic-partition, and its offset.
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) =>
{
e.Cancel = true; // prevent immediate process kill
cts.Cancel();
};
try
{
while (!cts.IsCancellationRequested)
{
try
{
var result = consumer.Consume(cts.Token);
Console.WriteLine(
$"Partition {result.Partition.Value}, " +
$"Offset {result.Offset.Value}: {result.Message.Value}");
// Business logic goes here. Keep it fast — a slow handler
// delays the next Consume() call and can trigger a rebalance,
// covered in "Common Pitfalls When Consuming with Confluent.Kafka."
consumer.Commit(result); // Pairing A: bookmark moves after the work
}
catch (ConsumeException ex)
{
Console.WriteLine($"Consume error: {ex.Error.Reason}");
// Log and continue — one malformed message shouldn't kill the loop.
}
}
}
catch (OperationCanceledException)
{
// Expected: cancelling the token unblocks Consume() by throwing.
}
finally
{
consumer.Close();
}
Walk through what this loop does. Each Consume returns one ConsumeResult — there is no batch API surfaced here, even though internally the client fetches from the broker in batches. The Offset.Value printed alongside each message is the log offset, distinct from the committed offset the group records. And note the two nested try blocks: the inner one catches per-message failures so the loop survives them, while the outer one catches the cancellation that ends the loop. OperationCanceledException is not a ConsumeException, so without that outer catch it escapes the method entirely after finally runs — an unhandled exception on every clean shutdown.
Poll loop lifecycle, one iteration:
consumer.Consume(token) blocks
↓
broker returns next message (or token cancelled → OperationCanceledException)
↓
ConsumeResult returned: { Partition, Offset, Message }
↓
handler code runs on the message
↓
offset committed (or stored, depending on pairing)
↓
loop repeats
Handling ConsumeException Without Crashing
ConsumeException is thrown when the client receives a message it cannot deserialize or hits a lower-level protocol error on that particular fetch. If it escapes the loop, the process terminates and every partition this consumer owned must be picked up by another group member — or sits unprocessed until this instance restarts — a disruption for what might be a single corrupt record. Catching it at the point of the Consume call keeps the loop alive so the next message is still attempted.
⚠️ Common Mistake: This inner catch is deliberately narrow. It's easy to widen it into a blanket catch (Exception) that also swallows business-logic failures inside your handler, silently discarding messages that needed a dead-letter queue, an alert, or a retry. Note also that a blanket catch would swallow OperationCanceledException too, turning your cancellation signal into a tight spin loop. That broader failure mode is covered as its own pitfall in the next section; the pattern here is scoped to just the exception the client raises.
Shutting Down Cleanly
The last piece is graceful shutdown, and it matters more than it looks. A CancellationTokenSource is wired to the process's cancel signal (Console.CancelKeyPress in a console app, or a hosted service's stopping token), and cancelling it causes the blocking Consume call to throw OperationCanceledException, which the outer catch absorbs. The critical step is in finally: calling consumer.Close().
Close() does two things that matter to the group. It commits any pending offsets one last time, and — more importantly here — it sends an explicit "leaving group" notification to the coordinator, which immediately reassigns this consumer's partitions to the remaining members. Skip Close() and the coordinator can't know the member is gone until its session timeout elapses, leaving those partitions unread for that entire window.
🎯 Key Principle: A poll loop is only "minimal" in business logic — its structural requirements (deliberate commit configuration, a cancellable loop, narrow exception handling around Consume, an outer catch for cancellation, and a guaranteed Close()) are not optional extras to layer on later. Every consumer you build from here sits on exactly this skeleton.
Putting the Pieces Together
using Confluent.Kafka;
var config = new ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "order-processing-group",
AutoOffsetReset = AutoOffsetReset.Earliest,
EnableAutoCommit = false // Pairing A: explicit commits
};
using var consumer = new ConsumerBuilder<Ignore, string>(config).Build();
consumer.Subscribe("order-events");
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };
try
{
while (!cts.IsCancellationRequested)
{
try
{
var result = consumer.Consume(cts.Token);
Console.WriteLine($"Offset {result.Offset.Value}: {result.Message.Value}");
consumer.Commit(result);
}
catch (ConsumeException ex)
{
Console.WriteLine($"Consume error: {ex.Error.Reason}");
}
}
}
catch (OperationCanceledException)
{
// graceful shutdown
}
finally
{
consumer.Close();
}
This is deliberately incomplete as a production artifact — no dead-letter routing, no batched commits, no protection against a slow handler starving the loop. What it gives you is a correct scaffold: every instance built this way registers with the group named by GroupId, receives a subset of the topic's partitions, records progress only after the work is done, and leaves the group cleanly on shutdown.
Common Pitfalls When Consuming with Confluent.Kafka
The minimal consumer works fine in a demo. Production traffic exposes a different set of failure modes. Most damage in real deployments comes from a handful of implementation habits that look harmless in isolation but corrupt consumer state, stall processing, or quietly drop data. Here are five, and how to recognize them before they reach production.
Pitfall 1: Sharing One IConsumer Across Threads
It's tempting to treat IConsumer<TKey,TValue> like any other injectable service — register it once, hand it to multiple worker threads, let them all call Consume(). This breaks immediately, because the Confluent.Kafka client is not thread-safe. The underlying librdkafka handle maintains internal state — assignment table, poll cursor, heartbeat scheduling — that assumes a single thread is driving it.
// ❌ Wrong thinking: "more threads calling Consume() means faster throughput"
var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe("orders");
// Spinning up several threads against the SAME consumer instance
for (int i = 0; i < 4; i++)
{
Task.Run(() =>
{
while (!cts.IsCancellationRequested)
{
var result = consumer.Consume(cts.Token); // corrupts internal state
Process(result);
}
});
}
This doesn't throw a clean exception every time — that's what makes it dangerous. You'll see intermittent KafkaException failures, partitions that appear to stop delivering, or assignment state that no longer matches what the broker thinks this consumer owns. Because the corruption is timing-dependent, it often passes local testing and only surfaces under load.
✅ Correct thinking: one IConsumer instance belongs to exactly one thread's poll loop. To parallelize across cores, either run multiple consumer instances (each with its own IConsumer, all sharing a GroupId so the group splits partitions between them), or keep a single-threaded poll loop that hands each ConsumeResult to a bounded worker pool while the loop keeps polling.
using System.Threading.Channels;
// One consumer, one poll-loop thread — parallelism happens downstream
var channel = Channel.CreateBounded<ConsumeResult<string, string>>(capacity: 100);
// Single thread owns the consumer and only calls Consume() here
_ = Task.Run(async () =>
{
using var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe("orders");
while (!cts.IsCancellationRequested)
{
var result = consumer.Consume(cts.Token);
await channel.Writer.WriteAsync(result, cts.Token);
}
});
// Separate worker tasks pull from the channel and do the actual processing
for (int i = 0; i < 4; i++)
{
_ = Task.Run(async () =>
{
await foreach (var result in channel.Reader.ReadAllAsync(cts.Token))
{
Process(result); // safe: consumer object itself is untouched
}
});
}
The rule to internalize: the IConsumer object is single-threaded property, but the messages it hands you are plain data, safe to fan out once they're off the poll loop.
⚠️ This hand-off pattern moves the offset problem rather than solving it. As written, it is deliberately incomplete on two fronts. First, once messages leave the poll loop, the loop keeps polling — so whatever advances the bookmark (auto-store, auto-commit, or a commit call in the loop) runs ahead of the workers, and a crash loses everything in flight. Second, several workers can process messages from the same partition concurrently, which discards the per-partition ordering the whole ownership model exists to protect. Making it correct means tracking completion per partition and storing only the offset of the highest contiguously completed message — real work, developed in the Rebalancing & Backpressure lesson. Use the simple version only where duplicate and out-of-order processing are both genuinely acceptable.
Pitfall 2: Long-Running Handlers Blocking the Poll Loop
Even with a correctly single-threaded consumer, it's easy to do too much work per iteration:
while (!cts.IsCancellationRequested)
{
var result = consumer.Consume(cts.Token);
await CallDownstreamApiAsync(result.Message.Value); // could take seconds
await SaveToDatabase(result.Message.Value);
}
It's worth knowing exactly which liveness check this trips, because the mechanism differs from the Java client. In librdkafka, group heartbeats are sent on a background thread, so a blocked handler does not immediately fail the session timeout the way it would on the JVM. What it trips is MaxPollIntervalMs (default 300000 ms — five minutes): the maximum gap allowed between your calls into Consume(). Exceed it and the client logs Application maximum poll interval exceeded, leaves the group, and your next commit fails because you no longer own the partition. The record gets reprocessed by whoever inherits it.
The practical takeaway: keep per-iteration work short, and if a message requires slow processing, hand it to a background worker (with the caveats above) rather than doing the slow work inline. Raising MaxPollIntervalMs is a legitimate second option when work genuinely takes minutes — the cost is that a truly stuck consumer goes undetected that much longer.
⚠️ Common Mistake: Teams discover this only after volume grows — a handler taking milliseconds against test data takes seconds against a real downstream service, and a smoothly running group starts rebalancing under load, exactly when it can least afford the disruption.
Pitfall 3: Skipping Close()/Dispose() on Shutdown
When a consumer process exits, the group finds out in one of two very different ways. If the consumer calls Close() first, it sends an explicit "I'm leaving" notification to the coordinator, which rebalances immediately and reassigns those partitions with minimal delay. If the process exits abruptly — crashes, gets killed, or simply never calls Close() — the broker can't know until it stops seeing heartbeats, then waits out the session timeout before declaring the member dead.
// ❌ No Close() — broker only notices via session timeout
var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe("orders");
try
{
while (!cts.IsCancellationRequested)
{
var result = consumer.Consume(cts.Token);
Process(result);
}
}
catch (OperationCanceledException)
{
// process just exits here — partitions sit unclaimed until timeout
}
// ✅ Explicit Close() in a finally block
var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe("orders");
try
{
while (!cts.IsCancellationRequested)
{
var result = consumer.Consume(cts.Token);
Process(result);
}
}
catch (OperationCanceledException)
{
// expected on graceful shutdown
}
finally
{
consumer.Close(); // leaves the group cleanly, triggers immediate rebalance
}
The practical cost of skipping Close() is a window during which nobody reads that consumer's partitions — messages queue up unprocessed until the session timeout expires. In a rolling deployment restarting instances one at a time, this repeats on every restart and adds up to real, avoidable latency. Dispose() releases the underlying client handle but does not commit pending offsets or send the leave-group notification, so a controlled shutdown wants Close() called explicitly before disposal.
Pitfall 4: Mistyped or Copy-Pasted GroupId Values
The GroupId string is the only thing telling the broker which consumers should share a topic's partitions. Because it's just a string in configuration, it's exactly the kind of value that gets copy-pasted between services, typo'd in an environment variable, or left as a tutorial placeholder.
// Service A
var configA = new ConsumerConfig
{
BootstrapServers = "broker:9092",
GroupId = "order-processor",
AutoOffsetReset = AutoOffsetReset.Earliest
};
// Service B — meant to join the same group, but has a trailing typo
var configB = new ConsumerConfig
{
BootstrapServers = "broker:9092",
GroupId = "order-processor ", // trailing space — a DIFFERENT group to the broker
AutoOffsetReset = AutoOffsetReset.Earliest
};
The broker treats "order-processor" and "order-processor " as two entirely distinct groups. Instead of splitting partitions between the two instances, each forms its own single-member group and reads every partition independently. No exception is thrown; both consumers appear to work and per-instance throughput looks fine — but downstream, every message is processed twice, and if the workload was split for capacity reasons, neither instance is relieved of any load.
💡 Pro Tip: Because this failure produces no errors, the reliable way to catch it is to check group membership directly rather than trust the config file. The Consumer Group Mechanics lesson covers the tooling for inspecting which consumers the broker actually considers part of a group — the fastest way to confirm your instances are sharing partitions as intended.
Pitfall 5: Swallowing ConsumeException Broadly
The minimal consumer wraps Consume() in try/catch (ConsumeException) so one malformed message doesn't crash the process. That's a reasonable baseline, but it's easy to take the defensive instinct too far — logging the exception and moving on with no record of which message caused it and no path for dealing with it later.
// ❌ Wrong thinking: "as long as we don't crash, we're fine"
while (!cts.IsCancellationRequested)
{
try
{
var result = consumer.Consume(cts.Token);
Process(result);
}
catch (ConsumeException e)
{
Console.WriteLine($"Error: {e.Error.Reason}"); // and then... nothing
}
}
The problem isn't that the process survives — it's that the poison message (malformed, corrupted, or otherwise undeserializable) simply vanishes from view. Depending on where in the flow the exception occurs and how offsets are configured, the group's bookmark may still advance past that message, meaning it will never be seen again; there is no automatic retry and no record of what went wrong. Anyone debugging a missing order later has no trail to follow.
// ✅ Correct thinking: capture context and route to a dead-letter path
while (!cts.IsCancellationRequested)
{
try
{
var result = consumer.Consume(cts.Token);
Process(result);
}
catch (ConsumeException e)
{
// Preserve enough detail to diagnose and reprocess later
await deadLetterSink.SendAsync(new
{
e.Error.Reason,
Partition = e.ConsumerRecord?.Partition.Value,
Offset = e.ConsumerRecord?.Offset.Value,
Timestamp = DateTimeOffset.UtcNow
}, cts.Token);
}
}
(Simplified — a production dead-letter path typically needs its own topic or durable store plus a way to correlate the failed record back to its original key and headers. The habit that matters is capture and route, not log and discard.)
⚠️ Common Mistake: Treating every ConsumeException the same way. A deserialization failure on one malformed record is a very different situation from a broker connectivity error that will resolve itself — collapsing both into one "log and continue" branch means transient infrastructure issues get silently absorbed alongside permanently bad data, and neither gets the response it needs.
Why These Pitfalls Compound
None of these five are exotic — each is a small, easy decision in otherwise ordinary consumer code. What makes them worth studying together is that they fail silently rather than loudly: a shared IConsumer doesn't throw "you violated thread safety"; a mistyped GroupId doesn't refuse to start; a swallowed ConsumeException doesn't page anyone. Each erodes a guarantee the group protocol is supposed to provide — exclusive ownership, timely rebalancing, a durable record of every message — without announcing it. Diagnose all of them the same way: verify actual group membership and partition assignment against what you intend, rather than trusting that the configuration you wrote matches the behavior you're getting.
Symptom you observe Likely pitfall to check first
─────────────────────────────────────────────────────────
Duplicate processing GroupId mismatch (Pitfall 4)
Unexplained rebalances Slow handler in poll loop (Pitfall 2)
Corrupted/erratic state Shared IConsumer across threads (Pitfall 1)
Delayed reassignment on Missing Close()/Dispose() (Pitfall 3)
deploy or restart
Messages disappearing Broad catch on ConsumeException (Pitfall 5)
with no trace
Messages lost on restart Default auto-store/auto-commit pairing
with no error (see "Offsets Are Checkpoints")
Keeping this mapping in mind turns a vague "the consumer is acting weird" investigation into a targeted check of one specific piece of code, which is usually where the fix ends up living.
Key Takeaways: Consumer Groups and Offsets
You've walked through why consumer groups exist, the ownership rule governing how they divide work, the difference between a message offset and a committed offset, and a working Confluent.Kafka consumer loop. Before moving into the mechanics that explain how Kafka enforces all this, it's worth consolidating the model into something you can check your own code against. This is a review pass, not new material — treat it as a checklist.
Recap: Ownership Caps Parallelism
The rule shaping everything else: within one consumer group, a partition is assigned to exactly one consumer instance at a time. Kafka never splits a partition's messages across two members of the same group, and never lets two members read the same partition concurrently. That's what separates consumer groups from a generic fan-out queue — the group divides partitions, not messages.
The consequence is a hard ceiling: a topic with 6 partitions supports at most 6 usefully active consumers in one group. A 7th joins but sits idle until one of the six leaves. Conversely, 2 consumers against 6 partitions means each owns 3 and interleaves work across them. Neither is an error — both are mechanical consequences of the ownership rule, and it's why partition count is the number you plan around before instance count.
🎯 Key Principle: Consumer group scaling is bounded by partition count, not by however many instances you're willing to deploy. Adding consumers past that ceiling adds no throughput — only idle processes waiting for a rebalance.
Recap: Offsets Checkpoint Progress, They Don't Acknowledge Messages
The second load-bearing idea is the distinction between a message's log offset — its fixed position in the partition — and the group's committed offset — the saved bookmark for where to resume. A commit doesn't mark individual messages done; it moves one number forward, and Kafka treats everything before it as handled, regardless of whether every message in between actually succeeded.
That's why commit timing determines your delivery semantics:
- Commit before processing → a crash mid-handler skips those messages on restart. Risks message loss.
- Commit after processing → a crash after the work but before the commit replays them. Risks duplicate processing, the foundation of at-least-once.
And the .NET-specific trap worth carrying out of this lesson: Confluent.Kafka's defaults give you the first shape, not the second. EnableAutoOffsetStore defaults to true, storing an offset the instant Consume() returns it — before your handler runs — and EnableAutoCommit commits whatever is stored on a 5-second timer. Getting at-least-once requires deliberately choosing one of two pairings:
<table> <thead><tr><th>Pairing</th><th>Config</th><th>Progress recorded by</th></tr></thead> <tbody> <tr><td>A — manual</td><td><code>EnableAutoCommit = false</code></td><td><code>Commit(result)</code> after the handler</td></tr> <tr><td>B — stored</td><td><code>EnableAutoCommit = true</code><br/><code>EnableAutoOffsetStore = false</code></td><td><code>StoreOffset(result)</code> after the handler</td></tr> <tr><td>❌ Neither</td><td><code>EnableAutoCommit = false</code> + <code>StoreOffset</code></td><td>Nothing — no committer runs</td></tr> </tbody> </table>
Neither pairing gives exactly-once for free — that requires idempotent handling or transactional writes, outside what committed offsets alone can guarantee. The full commit API surface and offset storage internals belong to the Offset Management lesson; what matters here is recognizing commit timing as a design decision rather than an implementation detail you can inherit.
Recap: The Minimal .NET Consumer Shape
Stripped to essentials, a working consumer needs five pieces: a GroupId, a deliberate commit configuration, a Subscribe() call, a Consume() loop that runs until told to stop, and a Close() on shutdown so the broker learns about departure immediately instead of waiting out a session timeout.
var config = new ConsumerConfig
{
BootstrapServers = "localhost:9092",
GroupId = "order-processing-group", // ties this instance to a specific group
AutoOffsetReset = AutoOffsetReset.Earliest,
EnableAutoCommit = false // deliberate: we commit after the work
};
using var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe("orders");
using var cts = new CancellationTokenSource();
Console.CancelKeyPress += (_, e) =>
{
e.Cancel = true; // let the finally block run Close() instead of killing the process
cts.Cancel();
};
try
{
while (!cts.IsCancellationRequested)
{
var result = consumer.Consume(cts.Token);
// handler logic goes here
consumer.Commit(result);
}
}
catch (OperationCanceledException)
{
// expected on shutdown — Consume() throws when the token is cancelled
}
finally
{
consumer.Close(); // leaves the group cleanly rather than waiting on a timeout
}
Each piece maps back to a concept: GroupId determines which ownership set this instance joins, the commit configuration determines whether you get loss or duplicates on a crash, Subscribe opts into partition assignment, the loop is where the work happens, and Close() lets the group react immediately rather than waiting on a failure-detection window.
💡 Mental Model: The pieces answer five questions — which group am I in (GroupId), when does my progress count (commit config), what am I reading (Subscribe), how do I get messages (the loop), and how do I leave politely (Close). A consumer missing any one is incomplete, not just suboptimal.
Review Checklist
Before treating a consumer as production-ready, walk five checks that map to failure modes covered above:
- Commit configuration — have you deliberately chosen Pairing A or B, or are you inheriting the at-most-once defaults? This is the check most likely to be wrong in code that otherwise looks correct.
- Thread-safety — is a single
IConsumer<TKey,TValue>instance ever touched from more than one thread? The client is not thread-safe, so sharing corrupts its state rather than throwing an obvious exception. - Handler duration — does the code inside the loop do slow, blocking work before looping back to
Consume()? ExceedingMaxPollIntervalMsbetween calls drops you out of the group. - Shutdown handling — does every exit path, including exceptions, reach
Close()? A process killed or throwing past thefinallyleaves the broker waiting out the session timeout. - GroupId correctness — is the value identical, byte-for-byte, across every instance meant to share partitions? A typo silently creates a second group instead of failing loudly.
<table><tr><th>🔧 Check</th><th>🎯 What it verifies</th><th>⚠️ Symptom if wrong</th></tr><tr><td>📍 Commit configuration</td><td>Progress recorded after the work</td><td>Silent message loss on crash or restart</td></tr><tr><td>🔒 Thread-safety</td><td>Single instance, single thread</td><td>Corrupted internal state, unpredictable errors</td></tr><tr><td>⏱️ Handler duration</td><td>Fast return to the poll loop</td><td>Max poll interval exceeded, unwanted rebalances</td></tr><tr><td>🚪 Shutdown handling</td><td>Close() on every exit path</td><td>Broker waits out session timeout to notice departure</td></tr><tr><td>🏷️ GroupId correctness</td><td>Identical value across instances</td><td>Each instance reads the full topic instead of sharing it</td></tr></table>
⚠️ Common Mistake: Treating this as a one-time setup review rather than something to re-check after refactors. A handler fast at launch can quietly grow slow when a new dependency call is added inside the loop, reintroducing exactly the problem the checklist was meant to catch.
What You Now Understand That You Didn't Before
At the start of this lesson, a single consumer reading a topic serially was the obvious approach and its throughput ceiling wasn't obviously a design problem. You now have the vocabulary to reason about why that ceiling exists, how a consumer group removes it up to the limit set by partition count, why an offset commit is a bookmark rather than a receipt, and why the .NET client's defaults quietly choose loss over duplication unless you tell them otherwise. Those ideas — ownership is partition-scoped, commits are checkpoints, and the checkpoint moves on a schedule you must configure — are what every later lesson assumes you already have.
Where This Goes Next
- Consumer Group Mechanics opens up the protocol behind partition assignment — how the coordinator decides who owns what, and what a group's internal state looks like.
- Offset Management goes deep on commit APIs, auto-commit intervals, and offset storage, giving you the tools to choose deliberately between at-least-once and at-most-once instead of getting whichever your configuration accidentally produces.
- Rebalancing & Backpressure explains what happens when a consumer is slow enough, or absent long enough, to trigger reassignment — and develops the per-partition completion tracking the worker hand-off pattern in Pitfall 1 needs to be correct.
💡 Pro Tip: As you move into those lessons, keep re-anchoring new detail back to the two recap ideas — partition-scoped ownership and offsets-as-checkpoints. Every mechanism you're about to learn is an elaboration of one of those, not a replacement.
0