Topics, Partitions & Keys

Introduction: The Topic-Partition-Key Mental Model

If you've built messaging into a .NET system before, you already have a mental model for what a "queue" does: a producer drops a message in, a consumer picks it up, and once it's processed, it's gone. MSMQ works this way. So does RabbitMQ. So does Azure Service Bus. Kafka calls its topics "topics," which sounds queue-adjacent, but the internal structure is genuinely different, and that difference has consequences from the very first line of producer code.

If you've completed "Core Kafka Mental Model," the next two subsections will be familiar — skim them and pick up at "Why the Partition, Not the Topic, Is the Real Unit." Everything after that is new: the physical anatomy of a topic, the exact routing formula the .NET client uses (which is not the one Java-oriented material describes), and the operational traps that partition and key decisions create.

Three questions frame the lesson. What actually gets stored when you publish a message? What decides which part of the system a given message ends up on? And why should a .NET developer care about the distinction? The answers hinge on three words — topic, partition, and key.

Kafka Topics Are Logs, Not Queues

A Kafka topic is not a container that holds messages until they're consumed. It's a named, append-only commit log — records are written to the end and stay there, in arrival order, identified by a numeric position called an offset. A topic isn't one log file, though: it's split into one or more partitions, each its own independent, ordered, append-only log, spread across the brokers in your cluster.

The consequence that matters most here is retention. In MSMQ, RabbitMQ, or Azure Service Bus, a message is transient — it exists to be delivered, and acknowledgment (or expiry) removes it. Kafka inverts this. A record stays at its offset for as long as the topic's retention policy allows, independent of whether zero consumers or a hundred have read it. A consumer doesn't take a message; it reads a copy and advances its own tracked offset. Another consumer group can read the same records from the beginning later.

💡 Mental Model: A queue is a checkout line — once you're served, you leave and the line forgets you. A Kafka partition is a numbered ledger — every entry stays on the page at its line number, and "reading" means glancing at line 47 without erasing it. When that ledger gets trimmed is a retention-policy question, covered in its own lesson, not a property of the log itself.

Why the Partition, Not the Topic, Is the Real Unit

Here's the detail that trips up newcomers: when people describe a topic's behavior — its ordering, parallelism, throughput — they're almost always describing partition behavior. The topic is a logical name you subscribe to; nearly everything operationally significant happens one level down:

🧠 Storage — each partition is a physically separate log on disk (replicated across brokers), not a shared structure. 📚 Ordering — offsets are assigned per partition, so "record 10 comes after record 9" holds within a single partition, never across the topic. (Precise guarantees and their limits are the dedicated Partition Ordering lesson; internalize for now that ordering is partition-scoped.) 🔧 Parallelism — a consumer within a group is assigned whole partitions, so partition count caps how many consumers in that group can work simultaneously. 🎯 Routing — every record lands in exactly one partition, and the mechanism deciding which is the partitioner, driven primarily by the record's key.

That last point is the hinge. If two records carry the same key, the partitioner routes them to the same partition — the same ordered log — so their relative order is preserved. Different keys (or no key) give no such guarantee. So "will my messages be processed in order?" is really "did my messages land on the same partition?", which is a direct function of key choice and the partitioning mechanism dissected in "How the Partitioner Assigns Keys to Partitions."

🎯 Key Principle: Kafka does not order a topic. It orders each partition independently. Any correctness argument that depends on message ordering has to be phrased in terms of "same partition," not "same topic."

A Running Example: Order Events

Every code sample in this lesson builds against one scenario: an e-commerce system publishing order events — OrderCreated, OrderPaid, OrderShipped — to a topic named order-events, keyed by order ID. That choice isn't arbitrary; it makes the partitioner observable, because events sharing a key land on the same partition and a consumer reading that partition sees them in production order. Whether OrderId is the best key (versus, say, CustomerId) is a strategic question deferred to the "Key Selection Strategy" lesson. Here it exists purely to make routing tangible.

// The event payload our producer will serialize as the message value.
// The OrderId will separately be used as the record's Kafka key.
public record OrderEvent(
    string OrderId,
    string EventType,   // e.g. "Created", "Paid", "Shipped"
    DateTimeOffset Timestamp,
    decimal Amount
);

And a preview of a produce call using Confluent.Kafka's IProducer<TKey, TValue> — full setup and delivery-result handling come later, but seeing the key/value split now makes the relationship concrete:

// Illustrative shape only — producer construction and config are covered later.
// Note that Key and Value are separate fields: the key drives partition routing,
// the value is the payload a consumer will deserialize.
var message = new Message<string, string>
{
    Key = orderEvent.OrderId,                       // routing input
    Value = JsonSerializer.Serialize(orderEvent)     // payload
};

var deliveryResult = await producer.ProduceAsync("order-events", message);
// deliveryResult.Partition tells you which partition the partitioner chose

Key and Value are distinct fields — a distinction that doesn't exist in the same form in queue-based .NET messaging APIs, where a body is just a body. In Kafka the key is not incidental metadata; it's the input to the routing decision that determines which ordered log the record joins.

What This Lesson Covers — and What It Deliberately Doesn't

This lesson walks through the physical anatomy of a topic in "Anatomy of a Topic: Partitions, Brokers, and Replicas"; the mechanics of turning a key into a partition number — and the places where the .NET client's behavior diverges from what Java-oriented documentation describes — in "How the Partitioner Assigns Keys to Partitions"; full producer/consumer code in "Producing and Consuming with Confluent.Kafka: A Worked Example"; and the operational mistakes these decisions cause in "Common Pitfalls: Partition Count, Skew, and Remapping."

Three closely related topics are deliberately out of scope: the precise guarantees and limits of per-partition ordering (Partition Ordering lesson), the strategic question of which field makes a good key (Key Selection Strategy), and how long records persist before deletion or compaction (Log Retention). Keeping those separate lets this lesson stay on one job: building the structural model — topic, partition, broker, key — that all three assume you already have.

💡 Real-World Example: Picture a PaymentProcessed event and a ShipmentDispatched event for two different orders, produced back to back. Because they carry different order IDs as keys, the partitioner may route them to different partitions entirely — and a consumer has no guarantee it will see them in production order, because they aren't competing for position in the same log. Reasoning about "did A happen before B" in Kafka always starts with "were A and B on the same partition?"

⚠️ Common Mistake: Treating a Kafka topic as if it behaves like an Azure Service Bus topic with subscriptions, where the middleware quietly manages delivery and cleanup for you. ❌ Wrong thinking: "I don't need to think about partitions — Kafka will just deliver each message to my consumer and clean up after itself." ✅ Correct thinking: "My topic is a set of independently ordered logs; my consumer is assigned specific partitions to read, my messages persist until a retention policy removes them, and which partition each message lands on is a routing decision I make (or delegate) through the key."

Anatomy of a Topic: Partitions, Brokers, and Replicas

A topic looks deceptively simple from the outside — you name it, you send to it, you read from it. But a topic is not a single file or a single queue on one machine. It is a logical name mapping onto one or more partitions, and those partitions are the physical units that get stored, replicated, and read. Understanding this split — topic as label, partition as reality — is the prerequisite for everything else, because every guarantee Kafka makes, and every guarantee it deliberately does not make, is a statement about partitions.

A Partition Is an Ordered, Immutable Log

Picture each partition as an append-only file. New records are written to the end, existing records are never modified, and every record carries a sequential integer called an offset marking its position within that specific partition.

Topic: order-events (3 partitions)

Partition 0:
  offset 0 → OrderCreated(id=1042)
  offset 1 → OrderShipped(id=1042)
  offset 2 → OrderCreated(id=1050)

Partition 1:
  offset 0 → OrderCreated(id=1043)
  offset 1 → OrderCancelled(id=1043)

Partition 2:
  offset 0 → OrderCreated(id=1044)
  offset 1 → OrderShipped(id=1044)
  offset 2 → OrderShipped(id=1046)
  offset 3 → OrderCreated(id=1051)

Notice that offset 1 exists in every partition and refers to three completely different records. Offsets are per-partition counters, not one sequence shared across the topic. There is no such thing as "offset 1 of order-events" — you must say "offset 1 of partition 1," because that's the only coordinate system Kafka maintains. A developer coming from a single-queue model (an MSMQ queue with one linear backlog) has to unlearn the idea that a topic has one running position counter; it has as many counters as it has partitions, all advancing independently.

🎯 Key Principle: A topic's ordering and storage guarantees exist at the partition level, not the topic level. "Topic" is an organizational label over a set of independently-ordered logs.

Leaders, Followers, and Replication

Partitions don't live on just one broker if you want fault tolerance — and in any real deployment, you do. For each partition, Kafka designates one broker as the leader, which handles every read and write for that partition. The remaining copies live on other brokers as follower replicas, continuously fetching from the leader but not serving client traffic under normal operation. The number of copies — leader plus followers — is the topic's replication factor.

Partition 0 (replication factor = 3)

  Broker 1: [Leader]    ← handles all reads/writes for Partition 0
  Broker 2: [Follower]  ← replicates Partition 0's log
  Broker 3: [Follower]  ← replicates Partition 0's log

If the broker hosting the leader fails, one of the in-sync followers is promoted, and clients transparently redirect. Replication factor 3 is the common production baseline because it lets you lose one broker and still have two copies of every record — which is what keeps a topic writable under the usual min.insync.replicas=2 setting. (Leader election picks from the in-sync replica set; it is not a majority vote, so a two-broker outage makes a partition unavailable rather than merely degraded. The mechanics of in-sync replica sets and acknowledgment settings belong to the durability lesson.)

The structural point for this lesson is simpler: replication factor is a per-partition property determining how many broker-local copies of that partition's log exist. A topic with 6 partitions and replication factor 3 doesn't store 3 copies of the topic — it stores 3 copies of each of the 6 partitions, for 18 physical partition-copies scattered across the cluster.

⚠️ Common Mistake: Assuming replication factor and partition count are the same knob. Partition count answers "how many parallel logs make up this topic"; replication factor answers "how many broker-local copies does each of those logs get." Setting replication factor to 1 outside local experimentation means a single broker failure makes that partition's data unavailable — there's no follower to promote.

Partition Count Sets the Ceiling on Parallelism

Because each partition can only be actively consumed by one consumer within a given consumer group at a time (assignment mechanics are covered in the worked example), partition count puts a hard ceiling on how many consumer instances in a single group can do useful work simultaneously. A topic with 3 partitions keeps at most 3 consumers in one group busy; a fourth sits idle, because there's no fourth partition to hand it. This is the structural reason partition count is a capacity-planning decision rather than a storage detail.

💡 Mental Model: Think of partitions as a fixed number of checkout lanes. Adding more cashiers than lanes doesn't speed anything up — the extras stand around. The number of lanes is decided when the store is built, and while Kafka lets you add lanes later, doing so has a real cost for keyed messages, as you'll see at the end of this lesson.

Creating a Topic from .NET Code

All of this structure has to be specified somewhere, and in a Confluent.Kafka application that's typically the IAdminClient interface rather than a broker's command-line tooling. CreateTopicsAsync declares partition count and replication factor programmatically, which is useful for integration tests, provisioning scripts, or any path where topic creation should be deployable code rather than a manual step.

using Confluent.Kafka;
using Confluent.Kafka.Admin;

var adminConfig = new AdminClientConfig
{
    BootstrapServers = "localhost:9092"
};

using var admin = new AdminClientBuilder(adminConfig).Build();

try
{
    await admin.CreateTopicsAsync(new[]
    {
        new TopicSpecification
        {
            Name = "order-events",
            NumPartitions = 6,          // parallelism ceiling for one consumer group
            ReplicationFactor = 3        // leader + 2 followers per partition
        }
    });

    Console.WriteLine("Topic 'order-events' created.");
}
catch (CreateTopicsException ex)
{
    // CreateTopicsAsync throws if the topic already exists or the
    // requested replication factor exceeds the number of available brokers
    Console.WriteLine($"Topic creation failed: {ex.Results[0].Error.Reason}");
}

Three things are worth calling out. NumPartitions and ReplicationFactor are the two structural properties this section has been building toward — everything else in TopicSpecification (retention and compaction overrides) is orthogonal to anatomy. CreateTopicsAsync is not idempotent: calling it for a topic that already exists throws a CreateTopicsException rather than silently succeeding, so provisioning code typically checks existing topics first or catches and inspects that exception. And requesting a replication factor higher than the number of brokers currently in the cluster fails outright — you cannot replicate onto more brokers than exist, which is a concrete illustration of replication factor as a physical constraint rather than a configuration number.

⚠️ Common Mistake: Hardcoding ReplicationFactor = 1 because that's what local single-broker Docker setups need, then deploying the same code unmodified against a multi-broker cluster. The topic still gets created, but it silently forfeits the fault tolerance the cluster exists to provide. Make replication factor a configuration value read from environment-specific settings, precisely so a local dev value can't leak into a production provisioning script.

Reading the Anatomy Back Out

Once a topic exists, the same admin client can describe its structure — confirming that the partition and replica layout matches what you requested, and revealing the current leader for a given partition. That workflow, using GetMetadata(), appears in the worked example later, when you'll have real data to inspect. For now the model to carry forward is architectural: a topic is a named collection of independently-ordered partitions, each with exactly one leader broker and zero or more follower replicas, and partition count is the hard limit on how much of that topic a single consumer group can process in parallel. Keys — which decide which partition a record lands in — are the mechanism connecting this physical layout to your application's data.

How the Partitioner Assigns Keys to Partitions

When a .NET producer calls ProduceAsync on a topic with several partitions, something decides which partition receives the record. That component is the partitioner, and understanding it is what lets you predict — rather than guess — where a message ends up. This section is deliberately narrow: it covers the mechanics of routing, not which field makes a good key (the Key Selection Strategy lesson).

It is also the section where Confluent.Kafka diverges most sharply from the Java client, in two ways that silently break systems. Read the divergence subsections even if you already know how Kafka partitioning works "in general."

The Default Partitioner: Hashing a Key

When you produce a message with a non-null key, the default partitioner applies a hash-modulo formula: it hashes the key's bytes and takes that hash modulo the current number of partitions, yielding a partition number between 0 and partitionCount - 1.

key bytes → hash function → hash value
hash value % partitionCount → partition number

The critical property is determinism: the same key, hashed against the same partition count, always produces the same partition number. If OrderId = "ORD-4471" hashes to partition 2 today, it will hash to partition 2 tomorrow and every time after — as long as the topic's partition count doesn't change. This is what makes keyed ordering possible: every producer instance, running independently, computes the identical mapping, so all records for that order converge on one partition without any coordination between producers.

using Confluent.Kafka;

var config = new ProducerConfig { BootstrapServers = "localhost:9092" };

using var producer = new ProducerBuilder<string, string>(config).Build();

// The key ("ORD-4471") is hashed by the default partitioner.
// Every message with this exact key lands on the same partition.
var result = await producer.ProduceAsync("order-events", new Message<string, string>
{
    Key = "ORD-4471",
    Value = "{ \"status\": \"Created\" }"
});

Console.WriteLine($"Delivered to partition {result.Partition.Value} at offset {result.Offset.Value}");

Run this three more times with the same key and different payloads ("Paid", "Shipped", "Delivered") and every one reports the same partition through result.Partition. The value being produced is irrelevant to routing; only the key bytes matter.

⚠️ Common Mistake: Assuming the partition number is stable across topic resizes. The modulo depends on the current partition count, so changing that count changes the formula's output for existing keys. The operational consequences are covered in "Common Pitfalls" later; the mechanism to remember is that the formula is hash(key) % partitionCount, not hash(key) % (some fixed number).

Divergence 1: Which Hash Function .NET Actually Uses

Here is the fact most Kafka material will not tell you, because most Kafka material is written for the JVM.

Confluent.Kafka wraps librdkafka, whose default partitioner setting is consistent_random — a CRC32-based hash. The Java producer uses murmur2. These are different functions, so the same key produces a different partition number depending on which client produced it.

That is fine in a homogeneous .NET shop and catastrophic in a mixed one. If a .NET service and a Java (or Kafka Streams, or Spark, or Flink) service both produce order-4471 events to the same topic, those events land on two different partitions, and the per-key ordering guarantee you thought you had never existed. Nothing errors. Nothing logs. A consumer just sees an order's history split across two logs with no relative order between them.

librdkafka exposes the alternatives through ProducerConfig.Partitioner:

<table> <thead><tr><th>⚙️ <code>Partitioner</code> value</th><th>🔑 Keyed routing</th><th>🕳️ Null key goes to</th></tr></thead> <tbody> <tr><td><code>ConsistentRandom</code> <em>(default)</em></td><td>CRC32</td><td>A random partition</td></tr> <tr><td><code>Consistent</code></td><td>CRC32</td><td>One fixed partition</td></tr> <tr><td><code>Murmur2Random</code></td><td>murmur2 — <strong>Java-compatible</strong></td><td>A random partition</td></tr> <tr><td><code>Murmur2</code></td><td>murmur2</td><td>One fixed partition</td></tr> <tr><td><code>Random</code></td><td>Random</td><td>A random partition</td></tr> </tbody> </table>

// Any topic that is ALSO produced to by a JVM client must use murmur2,
// or the two producers will disagree about where a key belongs.
var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    Partitioner = Partitioner.Murmur2Random
};

🎯 Key Principle: Cross-client key colocation is opt-in, not automatic. If a topic has producers on more than one client stack, pin the partitioner explicitly on every one of them and write down which you chose. Murmur2Random is the setting that matches the Java default's keyed routing.

⚠️ Common Mistake: Discovering this after the fact and "fixing" it by switching the .NET producer to Murmur2Random on a live topic. That's the same event as a partition-count change — every existing key's target moves, and records already written stay where they are. Treat it as a migration, not a config tweak.

Divergence 2: What Happens When the Key Is Null

Not every message needs a key. With Key = null there's no hash to compute, so the partitioner falls back to a different strategy — and this is the second place the .NET and Java clients differ.

Java-oriented material describes sticky partitioning (KIP-480): rather than rotating per message, the producer sends null-key messages to one partition until that batch fills or the linger interval elapses, then switches. The reasoning is throughput — producers batch per partition, and rotating on every message spreads each partition's queue thin.

librdkafka does not implement that. Under the default consistent_random, a null-key record is assigned a random partition per record. Batching still happens — records accumulate per partition and flush on linger.ms / batch.size — it just isn't deliberately concentrated on one partition the way the JVM's sticky partitioner concentrates it.

librdkafka default (consistent_random), null key:
  msg1 → P3, msg2 → P0, msg3 → P3, msg4 → P1, ...
  (random per record; batching still occurs, just not concentrated)

JVM client (KIP-480 sticky), null key:
  msg1..msg50  → P1   (one batch fills, then flushes)
  msg51..msg80 → P2   (next batch)
  (deliberately concentrates a batch on one partition)

Client defaults do shift between versions, so verify partitioner against the librdkafka version your package pins if this matters to your throughput budget. What does not shift is the guarantee: if you don't supply a key, you get reasonable load distribution and no ordering relationship whatsoever between any two unkeyed messages, since they may not even share a partition. That falls directly out of the routing mechanism; it isn't a separate rule.

💡 Mental Model: Think of the default partitioner as a two-mode switch. Key present → deterministic CRC32 routing. Key absent → random placement per record. Nothing more exotic happens by default.

Bypassing the Partitioner: Explicit Partition Targeting

Sometimes you already know which partition a record belongs on. Confluent.Kafka supports this through a ProduceAsync overload taking a TopicPartition instead of a topic name. The hashing partitioner is never invoked; the client sends the record straight to the partition you named.

using Confluent.Kafka;

var config = new ProducerConfig { BootstrapServers = "localhost:9092" };
using var producer = new ProducerBuilder<string, string>(config).Build();

// TopicPartition targets partition 3 directly.
// The key is still stored with the message, but it plays no role
// in routing here — the partitioner is bypassed entirely.
var topicPartition = new TopicPartition("order-events", new Partition(3));

var result = await producer.ProduceAsync(topicPartition, new Message<string, string>
{
    Key = "ORD-4471",
    Value = "{ \"status\": \"Refunded\" }"
});

Console.WriteLine($"Explicitly written to partition {result.Partition.Value}");

This is useful for migrating data between topics while preserving each record's original partition, or for diagnostic tooling that writes a test record to a specific partition. It's a narrow escape hatch rather than a routine pattern — manual assignment makes you responsible for whatever colocation guarantees your keys were supposed to provide.

⚠️ Common Mistake: Mixing explicit TopicPartition targeting with keyed produces for the same logical key. If one code path routes "ORD-4471" through the hash (landing on partition 2) while another forces it onto partition 3, you've silently broken the colocation guarantee keying was meant to provide — messages for one order now live on two partitions with two independent offset sequences.

Writing a Custom Partitioner

The default covers the overwhelming majority of cases, but you can override it entirely. Confluent.Kafka does this with a delegate, not an interface — ProducerBuilder.SetDefaultPartitioner (all topics) or SetPartitioner (one named topic), both taking a PartitionerDelegate:

Partition PartitionerDelegate(
    string topic, int partitionCount, ReadOnlySpan<byte> keyData, bool keyIsNull);

Here is one that reserves partition 0 for VIP traffic and hashes everything else across the rest:

using Confluent.Kafka;

// FNV-1a: deterministic across processes, machines, and restarts.
static uint Fnv1a(ReadOnlySpan<byte> data)
{
    const uint offsetBasis = 2166136261, prime = 16777619;
    uint hash = offsetBasis;
    foreach (var b in data)
    {
        hash ^= b;
        hash *= prime;
    }
    return hash;
}

static Partition VipAwarePartitioner(
    string topic, int partitionCount, ReadOnlySpan<byte> keyData, bool keyIsNull)
{
    // Single-partition topics have nothing to reserve against, and the
    // modulo below would divide by zero.
    if (partitionCount < 2) return new Partition(0);

    // Spread unkeyed records over the non-reserved partitions.
    if (keyIsNull) return new Partition(Random.Shared.Next(1, partitionCount));

    // UTF-8 literal — compares bytes without allocating a string.
    if (keyData.StartsWith("VIP-"u8)) return new Partition(0);

    return new Partition(1 + (int)(Fnv1a(keyData) % (uint)(partitionCount - 1)));
}

var config = new ProducerConfig { BootstrapServers = "localhost:9092" };

using var producer = new ProducerBuilder<string, string>(config)
    .SetDefaultPartitioner(VipAwarePartitioner)
    // ...or scope it to one topic:
    // .SetPartitioner("order-events", VipAwarePartitioner)
    .Build();

⚠️ Common Mistake: Reaching for key.GetHashCode() inside a custom partitioner. .NET randomizes string hash codes per process by default, so two instances of the same service — or the same service before and after a restart — will route an identical key to different partitions. The bug is invisible in a single-process test and destroys colocation the moment you scale out. A custom partitioner must use a hash that is stable across processes: FNV-1a as above, CRC32, or murmur2.

Custom partitioners are a specialized tool. Reach for one only when you have a routing requirement the hash formula can't express, and remember it inherits all the same colocation responsibilities — a hand-rolled partitioner can scatter one order's events across partitions just as easily as a misused built-in.

🎯 Key Principle: The partitioner answers "which partition?" It has no opinion on "which field should be the key?" That second question — OrderId versus CustomerId and the trade-offs between them — is scoped entirely to the Key Selection Strategy lesson. Here, treat the key as a given input and trace how it flows through hashing, random placement, explicit targeting, or custom logic to produce a partition number.

Producing and Consuming with Confluent.Kafka: A Worked Example

Everything so far is invisible until you run a producer and a consumer and watch the numbers come back. This section builds that end-to-end picture with a single scenario: order events keyed by OrderId. By the end you'll see exactly where partition and offset surface in the API, and confirm with real output that every event for an order lands on the same partition.

Building the Producer

The entry point is ProducerBuilder<TKey, TValue>. You configure broker addresses, specify key and value types as generic parameters, and Confluent.Kafka handles serialization with built-in serializers (Serializers.Utf8 for strings) or your own.

using Confluent.Kafka;

var producerConfig = new ProducerConfig
{
    BootstrapServers = "localhost:9092"
};

using var producer = new ProducerBuilder<string, string>(producerConfig)
    .SetKeySerializer(Serializers.Utf8)
    .SetValueSerializer(Serializers.Utf8)
    .Build();

// An order event: key = OrderId, value = a JSON payload (simplified as a string here)
var orderId = "ORD-48213";
var message = new Message<string, string>
{
    Key = orderId,
    Value = "{\"orderId\":\"ORD-48213\",\"status\":\"Created\"}"
};

DeliveryResult<string, string> result =
    await producer.ProduceAsync("order-events", message);

Console.WriteLine(
    $"Delivered to partition {result.Partition.Value} " +
    $"at offset {result.Offset.Value}");

ProduceAsync sends the keyed message and awaits the broker's acknowledgment. The returned DeliveryResult<string, string> is where routing becomes observable: result.Partition is the partition the default partitioner chose (recall hash(key) % partitionCount), and result.Offset is the exact position the record now occupies in that partition's log. Call it again with the same orderId and you get the same partition number back every time — determinism as a concrete integer in your console output.

⚠️ Common Mistake: Using the fire-and-forget Produce method (which takes a delivery-report callback instead of returning a Task) and then letting the process exit before the callback fires. Buffered messages are lost silently. Note the trade-off, though: await-ing every ProduceAsync individually serializes your pipeline into one round trip per record and collapses throughput. Use ProduceAsync where a request path genuinely needs the result inline; use Produce with a delivery handler plus an explicit Flush(timeout) on shutdown for bulk paths.

Building the Consumer

ConsumerBuilder<TKey, TValue> mirrors the producer's shape. The critical extra configuration is GroupId, placing this consumer into a consumer group — a named set of consumers that split a topic's partitions between them.

using Confluent.Kafka;

var consumerConfig = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "order-processing-service",
    AutoOffsetReset = AutoOffsetReset.Earliest
};

using var consumer = new ConsumerBuilder<string, string>(consumerConfig)
    .SetKeyDeserializer(Deserializers.Utf8)
    .SetValueDeserializer(Deserializers.Utf8)
    .Build();

consumer.Subscribe("order-events");

try
{
    while (!cancellationToken.IsCancellationRequested)
    {
        ConsumeResult<string, string> record = consumer.Consume(cancellationToken);

        Console.WriteLine(
            $"key={record.Message.Key} " +
            $"partition={record.Partition.Value} " +
            $"offset={record.Offset.Value} " +
            $"value={record.Message.Value}");
    }
}
catch (OperationCanceledException) { /* graceful shutdown */ }
finally
{
    consumer.Close();   // leaves the group immediately instead of waiting
                        // for the session timeout to expire
}

Each Consume call blocks until a record is available and returns a ConsumeResult<string, string>. The fields that matter for this model are record.Partition, record.Offset, and record.Message.Key. Note the symmetry: the partition and offset the consumer reports are exactly the values the producer's DeliveryResult reported when the record was written. Offsets are never reassigned or renumbered — they're a fixed coordinate in that partition's log.

Consumer Groups and Partition Assignment

A single consumer can read every partition of a topic, but production systems run several instances sharing a GroupId to parallelize. Kafka's rule is strict: within a consumer group, each partition is assigned to exactly one consumer at a time. With four partitions and two consumers, a typical split is two partitions each; scale to four consumers and each gets one; add a fifth and it sits idle, because a partition cannot be split between two group members.

order-events (4 partitions)
Consumer group: order-processing-service

Partition 0 ─┐
Partition 1 ─┼──► Consumer A
Partition 2 ─┐
Partition 3 ─┼──► Consumer B

Which consumer gets which partitions is decided by a pluggable partition assignor, configured via PartitionAssignmentStrategy. librdkafka defaults to range,roundrobin. The Range assignor hands out contiguous partition ranges per topic, which can produce mild imbalance when a group subscribes to several topics. The CooperativeSticky assignor minimizes partition movement during rebalances — when a member joins or leaves, it leaves as many existing assignments untouched as it can, rather than reshuffling everything, which shortens the processing pause a rebalance causes.

⚠️ Common Mistake: Switching an existing group to cooperative-sticky in a single deploy. Cooperative and eager assignors cannot coexist in one group; migrating requires a two-step rolling upgrade (first roll adds the cooperative assignor alongside the eager one, second roll removes the eager one). Flipping it in one go leaves the group unable to form.

💡 Mental Model: A consumer group is a team splitting a stack of numbered folders. Every folder is held by exactly one member at any moment — none shared, none skipped — but which member holds which can change as people join or leave.

Full Worked Example: Order Events Colocated by OrderId

Now put both sides together and confirm the claim that matters most: all events for the same order land on the same partition. Suppose order-events has 4 partitions, and you produce three events for ORD-48213 and one for ORD-90110:

string[] orderIds = { "ORD-48213", "ORD-48213", "ORD-48213", "ORD-90110" };
string[] statuses = { "Created", "PaymentReceived", "Shipped", "Created" };

for (int i = 0; i < orderIds.Length; i++)
{
    var msg = new Message<string, string>
    {
        Key = orderIds[i],
        Value = $"{{\"orderId\":\"{orderIds[i]}\",\"status\":\"{statuses[i]}\"}}"
    };

    var result = await producer.ProduceAsync("order-events", msg);
    Console.WriteLine($"{orderIds[i]} ({statuses[i]}) -> partition {result.Partition.Value}, offset {result.Offset.Value}");
}

Because the partitioner computes hash(key) % partitionCount and the key is identical for the first three sends, all three ORD-48213 events report the same partition, while ORD-90110 may land on the same or a different one depending on how its hash reduces modulo 4. Output might read: ORD-48213 (Created) -> partition 2, ORD-48213 (PaymentReceived) -> partition 2, ORD-48213 (Shipped) -> partition 2, ORD-90110 (Created) -> partition 0. The consumer confirms it from the other direction: whoever reads partition 2 sees the three ORD-48213 records in production order, because within one partition Kafka preserves write order. What this demonstrates is the mechanism that guarantee depends on — keying by OrderId is what physically colocates an order's history on one ordered log instead of scattering it.

⚠️ Common Mistake: Assuming that because all three events are in the topic, a consumer reading the whole topic sees them in produced order. ORD-48213's events (all on partition 2) are always read in order by whoever owns partition 2, but that says nothing about the relative timing of records on other partitions — the topic as a whole has no global order, only partition-local order.

Inspecting Partitions and Leaders with IAdminClient

Sometimes you need to inspect a topic's structure at runtime — confirming partition count before deciding how many consumer instances to run, or checking which broker leads a partition during troubleshooting. IAdminClient.GetMetadata does this.

using System.Linq;
using Confluent.Kafka;

var adminConfig = new AdminClientConfig { BootstrapServers = "localhost:9092" };
using var admin = new AdminClientBuilder(adminConfig).Build();

// Fetch metadata for a single topic (pass null to get metadata for all topics)
Metadata metadata = admin.GetMetadata("order-events", TimeSpan.FromSeconds(10));

var topicMetadata = metadata.Topics.Single(t => t.Topic == "order-events");

Console.WriteLine($"Topic: {topicMetadata.Topic}");
foreach (var partition in topicMetadata.Partitions)
{
    Console.WriteLine(
        $"  Partition {partition.PartitionId}: " +
        $"leader broker id = {partition.Leader}, " +
        $"replicas = [{string.Join(",", partition.Replicas)}]");
}

This prints one line per partition showing the current leader (the broker the producer and consumer clients actually talk to for that partition) and the replica set, tying back to the leader/replica structure from "Anatomy of a Topic." It's a genuinely useful debugging habit: if a consumer group is reading unevenly, checking GetMetadata first tells you the actual partition count and replica layout rather than relying on documentation that may have drifted.

⚠️ Common Mistake: Using GetMetadata with a topic name as an existence check. On a cluster with auto.create.topics.enable left on, requesting metadata for a topic that doesn't exist can create it — with the broker's default partition count and replication factor, which is almost never what you wanted. Use ListTopics or DescribeTopics when you're probing rather than confirming.

💡 Pro Tip: GetMetadata is synchronous and relatively cheap against the broker's cached cluster state, so it's fine in a startup health check or a diagnostic endpoint — but it's not a substitute for monitoring tooling and shouldn't be polled in a tight loop.

Taken together, these three pieces — a producer reporting where it wrote, a consumer reporting where it read, and an admin client reporting the topic's actual layout — give you a verifiable loop. You're no longer taking the partitioner on faith; you can watch a key resolve to a partition number in your own terminal.

Common Pitfalls: Partition Count, Skew, and Remapping

Every mistake in this section traces back to one habit: treating partition count and key choice as implementation details you set once and forget. Both decisions ripple through a topic's lifetime — they determine how much parallelism your consumers can ever have, what happens when traffic grows, and whether "same key, same partition" still holds after a change that felt harmless.

The Partition Count Trade-off

Partition count is the ceiling on how many consumers in a single group can process a topic in parallel, because each partition is assigned to exactly one consumer in a group at a time. That ceiling cuts both ways.

Too few partitions caps parallelism directly. If an orders topic has 3 partitions and you scale your consumer group to 6 instances hoping to double throughput, three instances sit idle — there's no partition left to assign them. You've paid for compute that does nothing.

Too many partitions looks free until you account for what each costs the brokers. Every partition is a set of log segment files the broker holds open file handles for, plus metadata the controller tracks. Each partition also needs leader election when a broker fails or restarts, and a cluster with tens of thousands of partitions makes those elections slower and more disruptive because there's more state to reconcile. There's no single correct number; it depends on target throughput per partition, consumer count, and how much broker overhead your cluster is provisioned to absorb.

Too few partitions:        Right-sized:              Too many partitions:

[P0]-> C1                  [P0]-> C1                 [P0..P29] spread across
[P1]-> C2                  [P1]-> C2                 6 consumers, but broker
[P2]-> C3                  [P2]-> C3                 now tracks 30x the file
        C4 idle            [P3]-> C4                 handles and leader
        C5 idle            [P4]-> C5                 metadata for one topic
        C6 idle            [P5]-> C6

🎯 Key Principle: Partition count sets the maximum parallelism a consumer group can ever reach, but every partition is standing broker overhead whether or not it carries meaningful traffic. Size for expected peak consumer parallelism, not for a round number.

⚠️ Common Mistake: Picking a partition count that exactly matches today's consumer instance count, leaving no room to scale later without a repartitioning operation. Since adding partitions has consequences (below) and removing them is impossible — Kafka supports increasing partition count and offers no operation to decrease it, so shrinking means creating a new topic and migrating — pad the initial count modestly above your near-term target.

What Breaks When You Add Partitions

This is the pitfall that catches teams who understand the partitioner but haven't connected it to a live topic. The default partitioner routes with hash(key) % partitionCount, which is deterministic only while partitionCount stays fixed. The moment you add partitions — say 6 to 12 — the modulus changes for every key in your system, immediately.

Concretely: with 6 partitions, hash("order-4471") % 6 might land on partition 2. With 12, hash("order-4471") % 12 will very likely land elsewhere; there's no guarantee of overlap. Kafka does not rehash and move existing data. Records already on partition 2 stay on partition 2. What changes is where new messages with that key are routed.

This matters enormously if your application depends on per-key colocation — often relied on for per-key event order or for co-locating related state in a stream-processing job. After a partition count change that guarantee breaks silently: old events for order-4471 sit on partition 2, new events appear elsewhere, and any consumer logic assuming "everything about this order is on one partition" is now wrong with no error raised anywhere.

// Adding partitions to an existing topic — this is a one-way operation.
// There is no CreatePartitions equivalent that decreases the count.
// After this call, hash(key) % partitionCount changes for every key,
// and Kafka does NOT move existing records to match the new mapping.
using var adminClient = new AdminClientBuilder(new AdminClientConfig
{
    BootstrapServers = "localhost:9092"
}).Build();

await adminClient.CreatePartitionsAsync(new[]
{
    new PartitionsSpecification
    {
        Topic = "orders",
        IncreaseTo = 12 // was 6 — every key's target partition can now change
    }
});

⚠️ Common Mistake: Assuming increasing partition count is purely additive and low-risk. It is additive for throughput headroom, but it's a breaking change for any consumer or downstream job relying on same-key-same-partition behavior across the transition. Kafka gives no warning and no migration path — the responsibility falls entirely on whoever runs the change to know which consumers depend on colocation first.

💡 Mental Model: Partition count is closer to a database's shard count than a queue's worker pool size. Resharding a database moves data; increasing Kafka partitions does not — it changes the routing table for records not yet written, leaving old records where they are.

Skewed Keys and Hot Partitions

Even a generously sized topic bottlenecks if its keys are unevenly distributed. Key skew is when a few key values account for a disproportionate share of traffic, so the partitions those keys hash to become hot partitions — carrying far more volume than their siblings while the rest sit comparatively idle.

Imagine an order-events topic with 12 partitions keyed by CustomerId. If one enterprise customer generates a large share of total volume — very plausible in B2B — every event for that customer routes to one partition, exactly as designed. The other 11 handle everyone else comfortably, but the hot one becomes the throughput ceiling: its consumer works as fast as it can while sibling consumers finish and wait. Aggregate partition count looks sufficient on paper; achievable throughput is bounded by the busiest single partition, not the average.

This is a distinct failure from having too few partitions — adding more doesn't help if the skewed key still hashes to one of them. Mitigations (salting hot keys, choosing a higher-cardinality key, deliberately splitting a hot key across partitions) belong to Key Selection Strategy. What matters here is recognizing the symptom: one partition's consumer lagging while others are caught up is key skew, not insufficient capacity.

💡 Real-World Example: A dashboard showing consumer lag per partition is the fastest way to catch this. If one or two partitions climb steadily while the rest hover near zero, that's a hot key — adding partitions or consumers won't move that line down. Topic-level average lag hides it completely.

Ordering Is Per Partition, Not Per Topic

A closely related misreading shows up constantly in teams migrating from Azure Service Bus, where a single queue implies a single global order. Kafka's ordering guarantee applies within a partition — records written to the same partition are read back in write order — with no guarantee about interleaving across partitions.

❌ Wrong thinking: "I published Order Created, then Order Shipped,
   to the 'orders' topic, so consumers will always see Created before
   Shipped."

✅ Correct thinking: "Created and Shipped will arrive in that order
   ONLY if both events share a key that hashes to the same partition.
   If they land on different partitions, no ordering is guaranteed
   between them."

This is why key choice and ordering are inseparable: keying both events by OrderId is what forces them onto the same partition and therefore into a guaranteed sequence. The full mechanics — including behavior across producer retries and consumer rebalances — are the Partition Ordering lesson. The instinct to fix here is narrower but non-negotiable: never reason about ordering at the topic level, only at the partition level.

Null and Accidentally-Constant Keys

The last pitfall is the quietest, because it throws no exception and survives code review — it shows up weeks later as a lopsided lag dashboard. A null key triggers random placement across partitions, which is generally healthy for distribution. The danger is a key that is not null but constant: every message carries a key, nothing about the produce call looks wrong, but the key never varies.

// Bug: keying every message by a hardcoded string instead of the
// per-order identifier. Every message hashes to the SAME partition,
// no matter how many partitions the topic has.
await producer.ProduceAsync("orders", new Message<string, string>
{
    Key = "order-event",           // constant literal, not OrderId
    Value = JsonSerializer.Serialize(orderEvent)
});

// Fix: key by the field that actually varies per message.
await producer.ProduceAsync("orders", new Message<string, string>
{
    Key = orderEvent.OrderId,       // varies per order, spreads across partitions
    Value = JsonSerializer.Serialize(orderEvent)
});

In the buggy version, hash("order-event") % partitionCount evaluates to the same index for every message the producer sends — a 12-partition topic degrades to the throughput of one partition, and 11 consumers sit permanently idle. This is easy to introduce when a key is built from a template string, a fixed event-type label, or a placeholder that never got replaced.

⚠️ Common Mistake: Copy-pasting a producer example that uses a literal string as the key for illustration, and shipping it without swapping in an actual per-entity field. Kafka doesn't reject a constant key — it's a perfectly valid key value — so nothing fails at compile time, deploy time, or under moderate test load. The collapse only becomes visible once real volume accumulates on the one partition it always hashes to.

Summary and Quick Reference

You've moved from "Kafka is another message queue" to seeing it as a set of partitioned, append-only logs distributed across brokers, with keys as the routing input deciding which log a record joins. Before the deeper guarantees built on this model, lock the vocabulary down — these five terms get used loosely, and imprecise use is where a lot of production confusion starts.

The Five Terms, Precisely

A topic is the logical name producers and consumers agree on — no inherent order, no single physical location. Order and location live one level down in the partition: an ordered, immutable append-only log, the actual unit Kafka stores, parallelizes, and routes to. Within a partition, a record's position is its offset, a per-partition integer meaningless outside it — offset 500 in partition 0 and offset 500 in partition 1 are unrelated records. Physically, each partition lives on a broker as a leader with zero or more replica followers copying it. And the key is the data attached to a record that the partitioner uses to pick a partition.

<table> <tr><th>Term</th><th>What it is</th><th>What it is NOT</th></tr> <tr><td>📚 Topic</td><td>A named, logical stream that groups related partitions</td><td>❌ Not itself ordered, not a single log</td></tr> <tr><td>🔒 Partition</td><td>An ordered, immutable append-only log; the real unit of storage and parallelism</td><td>❌ Not shared order across partitions</td></tr> <tr><td>🎯 Offset</td><td>A record's position within one specific partition</td><td>❌ Not a global sequence number across the topic</td></tr> <tr><td>🔧 Broker / Replica</td><td>A server hosting a partition's leader or a follower copy for fault tolerance</td><td>❌ Not a unit of routing or ordering</td></tr> <tr><td>🧠 Key</td><td>The routing input a producer supplies per record</td><td>❌ Not required, not a value the broker interprets</td></tr> </table>

🎯 Key Principle: Ordering, parallelism, and fault tolerance are each owned by a different one of these — order lives in the partition, parallelism is capped by partition count, fault tolerance comes from replicas, and none of the three belongs to the topic name itself.

The Partitioner Mechanism, Recapped

Routing has two paths. For a keyed record, the default partitioner hashes the key and reduces it modulo the partition count — hash(key) % numPartitions — deterministic for a fixed count, so the same key always lands on the same partition. For an unkeyed record, librdkafka's default consistent_random assigns a random partition per record; the sticky batching described in JVM-focused material is a Java client feature and does not apply to Confluent.Kafka.

And the divergence worth carrying out of this lesson: librdkafka hashes keys with CRC32, the Java client with murmur2. If a topic has producers on both stacks, set Partitioner = Partitioner.Murmur2Random on the .NET side or the two will disagree about where every key belongs.

using Confluent.Kafka;

var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    // Only needed if a JVM client also produces to this topic —
    // but harmless, explicit, and worth writing down either way.
    Partitioner = Partitioner.Murmur2Random
};

using var producer = new ProducerBuilder<string, string>(config).Build();

// Keyed: hash(key) % partitionCount picks the partition deterministically
var keyedResult = await producer.ProduceAsync("order-events",
    new Message<string, string> { Key = "order-482", Value = "OrderCreated" });
Console.WriteLine($"Keyed message -> partition {keyedResult.Partition.Value}");

// Null key: a partition is chosen at random for this record
var unkeyedResult = await producer.ProduceAsync("order-events",
    new Message<string, string> { Value = "HeartbeatPing" });
Console.WriteLine($"Unkeyed message -> partition {unkeyedResult.Partition.Value}");

Run this twice with the same key and the keyed message reports the same partition both times (given a stable partition count), while the unkeyed message's partition isn't predictable from the call site.

💡 Mental Model: The key is an address label and the partitioner a deterministic sorting machine — same label, same bin, every time, as long as the number of bins doesn't change and everyone feeding the machine uses the same hash function.

The Operational Facts, in One Line Each

Three things from "Common Pitfalls" are worth keeping in short-term memory while you design a topic. More partitions buy consumer parallelism but cost broker memory, file handles, and leader-election overhead, so size deliberately rather than maximize. Partition count can be increased but never decreased. And because the partitioner's modulo depends on the current count, adding partitions reshuffles hash(key) % numPartitions for every key and silently breaks any prior colocation guarantee.

⚠️ Common Mistake: Treating partition count as something you can freely tune later the way you'd scale a stateless web service. For Kafka, changing it retroactively changes routing for every existing key, not just future capacity.

Where the Deeper Guarantees Live

Everything here has been mechanism and vocabulary. Three follow-on lessons answer what this one deliberately left open:

• Partition Ordering turns "order lives in the partition" into a precise guarantee — what Kafka promises about the order consumers see within a partition, and exactly where that promise stops.
• Key Selection Strategy takes the routing mechanism above and asks the design question: for a domain like order events, what field should be the key — order ID, customer ID, something else — and how do you avoid the hot-partition skew this lesson only named.
• Log Retention vs Queue returns to the opening contrast with MSMQ and Azure Service Bus and covers how long records actually persist, which nothing here has addressed.

💡 Pro Tip: When designing a new topic, work the vocabulary table in order — name the topic, decide the partition count with the cost trade-off in mind, pick a key with the next lesson's strategy in mind, confirm the partitioner setting if anything else produces to the topic, and only then write the producer code.

Practical Next Steps

Four concrete actions. First, before writing your next producer, decide explicitly whether each message needs a key at all — a null key opts into random placement and gives up colocation, fine for independent events and wrong for anything needing per-entity ordering. Second, check whether any topic your .NET services produce to is also written by a JVM-based producer; if so, that's a Partitioner setting to reconcile today, not after an incident. Third, use DeliveryResult.Partition as a sanity check that your key is producing the colocation you expect rather than assuming it. Fourth, treat partition count as a decision made once, and if you suspect you'll need to change it, read "Partition Ordering" and "Key Selection Strategy" first, since both affect how costly that change will be.

0