KRaft & Modern Kafka

Why Modern Kafka Changed the Rules for .NET Developers

If you've ever run docker-compose up on a Kafka project and watched two separate containers fight for memory before your producer could send a single message, you've felt the tax that ZooKeeper used to charge every Kafka deployment. For most of Kafka's history, a Kafka cluster was never really "a cluster" — it was two distributed systems duct-taped together, and .NET developers writing Confluent.Kafka code had to know at least a little about both. Why did the Kafka project decide an entire second coordination system was expendable? What actually breaks — and what stays exactly the same — in your C# code when that system disappears? And why does a change that sounds purely operational end up touching your docker-compose.yml, your CI pipeline, and your IAdminClient calls? This lesson answers those questions by walking through KRaft (Kafka Raft), the ZooKeeper-free architecture that is now the only way to run a new Kafka cluster, and by tracing exactly where that shift shows up in .NET application and tooling code.

The Two-System Problem ZooKeeper Created

In the original Kafka architecture, brokers didn't just talk to each other — they depended on an external coordination service, Apache ZooKeeper, for three critical jobs: electing a controller broker (responsible for managing partition leadership and cluster metadata), tracking which brokers were alive and registered, and storing the metadata describing every topic and partition. ZooKeeper is a general-purpose distributed coordination service, not something purpose-built for Kafka, which meant every Kafka deployment was really an exercise in running and tuning two distributed systems side by side.

That had real costs. ZooKeeper needed its own quorum of nodes, its own JVM tuning, its own monitoring, and its own upgrade cadence, often out of sync with the broker upgrade cycle. Controller failover went through a ZooKeeper-mediated election involving watches and ephemeral znodes — a mechanism that worked, but added a layer of indirection between "a broker died" and "a new controller is active and metadata is consistent again." For a .NET developer this rarely showed up in producer or consumer code, but it showed up in the surrounding ecosystem: local dev tooling that shelled out to zkCli.sh, health checks that pinged ZooKeeper's client port, and runbooks with a section titled "if ZooKeeper is unhealthy, do not touch the brokers yet."

KRaft: One System Instead of Two

KRaft replaces ZooKeeper's job with a built-in Raft-based metadata quorum made up of Kafka nodes themselves. Instead of an external system tracking cluster metadata in znodes, the metadata lives in a replicated log inside Kafka, managed by the Raft consensus protocol. As of the Kafka 4.0 release, ZooKeeper support has been removed entirely — KRaft is no longer an opt-in alternative for the adventurous, it is the only supported mode for standing up a new cluster. If you're provisioning a fresh cluster today, whether that's a local container for integration tests or a production deployment, there is no ZooKeeper configuration to reach for.

🎯 Key Principle: KRaft doesn't change what Kafka's metadata is (which broker leads which partition, which topics exist, which configs apply) — it changes how that metadata is agreed upon and stored, replacing an external coordination service with a Raft log native to Kafka itself.

ZooKeeper-based cluster (removed in Kafka 4.0):
 Kafka Broker 1 ↔ ZooKeeper ensemble ↔ Kafka Broker 2
 Controller election, broker registration, and topic metadata
 all live in ZooKeeper znodes, external to the brokers.

KRaft-based cluster (the only supported mode going forward):
 Kafka Broker/Controller 1 ↔ Raft metadata log ↔ Kafka Broker/Controller 2
 Controller election and metadata live in a Kafka-native
 replicated log; no external coordination service exists.

This flattens a real distinction Kafka makes between broker nodes, controller nodes, and combined nodes that play both roles — the wiring of process.roles and the controller quorum settings that makes this concrete for a docker-compose.yml is covered in "Spinning Up and Targeting a KRaft Cluster from a .NET Project," and the mechanics of how the Raft quorum elects a leader and replicates its log are the subject of the dedicated KRaft Architecture lesson.

What Actually Changes for a .NET Developer

Here's the part that matters most for your day job: the shift is, in one sense, remarkably boring. Open a Confluent.Kafka producer or consumer loop and nothing about the message-passing code changes. Producing a message, consuming a batch, committing an offset — all of that talks to Kafka brokers over the Kafka wire protocol, and it never talked to ZooKeeper directly even in the old architecture. .NET client libraries were always insulated from ZooKeeper by design; only the brokers and a narrow set of admin-adjacent tooling touched it.

What does change is everything around that core — the seams where your .NET project meets cluster infrastructure:

🔧 Cluster bootstrap — how a cluster is initialized and assigned an identity now happens through a Kafka-native storage format step rather than auto-assignment via ZooKeeper. 🔧 AdminClient behavior — calls like DescribeClusterAsync() now surface controller and cluster-identity information that used to require ZooKeeper-adjacent tooling. 🔧 Local dev tooling — Docker Compose files and Testcontainers definitions need different environment variables and a different startup sequence than a ZooKeeper-plus-broker pair. 🔧 Operational runbooks — failure modes, health checks, and recovery steps need to reflect a single coordinated system instead of two independently-monitored ones.

A minimal way to see this insulation is to look at how little a basic .NET consumer cares. The following is identical whether the brokers behind it run KRaft or, on an older deployment, still coordinate through ZooKeeper — which is exactly the point:

using Confluent.Kafka;

var config = new ConsumerConfig
{
    // Only the Kafka protocol endpoint matters here.
    // There has never been a ZooKeeper setting on this config object.
    BootstrapServers = "localhost:9092",
    GroupId = "orders-processor",
    AutoOffsetReset = AutoOffsetReset.Earliest
};

using var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe("orders");

try
{
    while (!cts.IsCancellationRequested)
    {
        var result = consumer.Consume(cts.Token);
        Console.WriteLine($"Received: {result.Message.Value}");
    }
}
catch (OperationCanceledException) { /* graceful shutdown */ }
finally
{
    consumer.Close();
}

What is new is the kind of call that used to require ZooKeeper-side tooling to answer. Confirming which node is currently acting as controller is now something your .NET code asks the cluster directly:

using Confluent.Kafka;
using Confluent.Kafka.Admin;

var adminConfig = new AdminClientConfig { BootstrapServers = "localhost:9092" };
using var admin = new AdminClientBuilder(adminConfig).Build();

// DescribeClusterAsync reaches the cluster through the Kafka protocol itself —
// no external ZooKeeper client required. Note it is async-only; there is no
// synchronous DescribeCluster overload on IAdminClient.
var cluster = await admin.DescribeClusterAsync(
    new DescribeClusterOptions { RequestTimeout = TimeSpan.FromSeconds(5) });

Console.WriteLine($"Cluster ID: {cluster.ClusterId}");
Console.WriteLine($"Controller: {cluster.Controller?.Host}:{cluster.Controller?.Port}");

This is a preview, not the full treatment — a worked end-to-end producer/consumer/admin example against a live KRaft container is the focus of the next-but-one section. The point here is narrower: operations that used to require stepping outside the Kafka client entirely, into ZooKeeper CLI tools, now happen through the same IAdminClient your application already uses.

💡 Mental Model: Think of the ZooKeeper-to-KRaft transition less as "Kafka got a new feature" and more as "Kafka internalized a dependency it used to outsource." The job didn't disappear, it moved inside the system you're already talking to — which is exactly why your producer and consumer code doesn't need to change, but your infrastructure code does.

What This Lesson Covers, and What It Deliberately Doesn't

This lesson stays focused on the .NET-facing consequences: the configuration keys that replaced ZooKeeper settings, how to point local and CI tooling at a KRaft cluster, resilience patterns for AdminClient calls during topology changes, and the mistakes teams carry over from ZooKeeper-era habits.

Two related topics deserve their own treatment. The internals of the Raft-based controller quorum — how voters and observers are structured, how leader election works, how the metadata log is replicated — belong to the dedicated KRaft Architecture lesson; those internals matter more to operators than to application developers, but they explain why the client-visible behaviors here hold. Separately, tiered storage, which splits a topic's data between fast local disks and cheaper remote object storage, is a related-but-independent modernization; it changes how retention and disk sizing work but not how your producer or consumer code is written, and it gets its own Tiered Storage Concepts lesson.

⚠️ Common Mistake: Treating "ZooKeeper is gone" and "tiered storage exists" as the same upgrade, since both get bundled into the phrase "modern Kafka." They're independent: a cluster can run KRaft with no tiered storage configured at all, and the disk-filling consequences of forgetting that are covered later in "Common Mistakes .NET Teams Make with Modern Kafka Clusters."

From ZooKeeper Habits to KRaft Reality: What Client Code Needs to Know

If you've maintained a .NET service talking to Kafka for a few years, some of your instincts were formed by ZooKeeper's presence — even if your application code never called it. Config files, compose definitions, admin scripts, and mental models about "where cluster state lives" all quietly assumed a second distributed system behind the brokers. KRaft removes that assumption, and the differences show up in exactly the places a .NET developer touches.

Connection Strings: bootstrap.servers Is the Only Door

Under the ZooKeeper-era architecture, two connection strings coexisted in most Kafka projects. Producers and consumers used bootstrap.servers to reach brokers, but administrative tooling frequently used zookeeper.connect — pointing at the ZooKeeper ensemble instead. Tools like zkCli.sh, and broker-side scripts invoked with --zookeeper, could read and write cluster metadata by talking to ZooKeeper's znode tree directly, bypassing the Kafka protocol altogether.

That second door is gone. In a KRaft cluster, zookeeper.connect is not deprecated-but-tolerated — it has no meaning at all, because there is no ensemble to connect to. Every operation goes through the Kafka protocol against bootstrap.servers. It doesn't just mean "use a different connection string": an entire class of tooling that read cluster state by inspecting znodes has no equivalent path anymore and must be rewritten against the Kafka Admin API.

// ZooKeeper-era .NET tooling often needed BOTH of these,
// because some operations only existed via ZK-aware scripts:
// var zkConnect = "zk1:2181,zk2:2181,zk3:2181"; // no longer applicable

// KRaft-mode configuration: only the Kafka protocol endpoint matters
var adminConfig = new AdminClientConfig
{
    BootstrapServers = "broker1:9092,broker2:9092,broker3:9092"
};

using var admin = new AdminClientBuilder(adminConfig).Build();

This looks almost too simple to be the point — and that's the point. AdminClientConfig carries no ZooKeeper-related properties at all, because Confluent.Kafka's admin client has only ever spoken the Kafka wire protocol. What changed is that every administrative capability formerly reachable only through ZK-aware scripts is now guaranteed to be reachable this same way.

Where Cluster Identity Comes From Now

In the ZooKeeper architecture, cluster identity was assigned automatically: the first broker to start registered itself, a cluster ID was generated and stored in ZooKeeper, and other brokers discovered it there. There was no explicit provisioning step — the cluster "became" a cluster the moment brokers connected to a shared ensemble.

KRaft inverts this. Before a node can start in KRaft mode, its storage directory must be formatted with kafka-storage.sh format, supplying a cluster ID generated once (typically with kafka-storage.sh random-uuid) and reused across every node in that cluster. This is a deliberate, one-time provisioning act rather than an emergent side effect of nodes finding each other. Skip it and the node refuses to start rather than silently forming an ad hoc cluster.

💡 Pro Tip: You will not see this step in most Docker Compose files, and that's not an omission. The official apache/kafka image's entrypoint generates a cluster ID and formats storage on first start if the log directory is empty. That convenience is why local setups appear to skip provisioning — but it also means a container that loses its volume comes back with a different cluster ID, which is worth knowing before you debug why a persisted consumer group vanished.

Anything that used to confirm "which cluster am I talking to" by reading a znode under /cluster/id has no ZooKeeper tree to inspect. The Kafka protocol exposes this directly:

// Confirm cluster identity and membership via the Kafka protocol,
// replacing tooling that used to read ZooKeeper's /cluster/id znode.
// DescribeClusterAsync is async-only — there is no synchronous overload.
var description = await admin.DescribeClusterAsync(
    new DescribeClusterOptions { RequestTimeout = TimeSpan.FromSeconds(10) });

Console.WriteLine($"Cluster ID: {description.ClusterId}");
Console.WriteLine($"Controller: {description.Controller?.Host}:{description.Controller?.Port}");
foreach (var node in description.Nodes)
{
    Console.WriteLine($"Node {node.Id}: {node.Host}:{node.Port}");
}

DescribeClusterAsync() returns the cluster ID baked in at format time, the current controller, and the full node list — all over the same protocol your producers and consumers already use. Note that Controller can be null if the client hasn't yet learned who the controller is, which is why the null-conditional operators are there rather than decoration.

⚠️ Common Mistake: Reaching for GetMetadata() and looking for a cluster ID on the result. Metadata exposes Brokers, Topics, and OriginatingBrokerId/OriginatingBrokerName — the last of which identifies the broker that answered your request, not the cluster. DescribeClusterAsync() is the only call that returns ClusterId.

process.roles and the Controller Quorum: Why This Matters for Your docker-compose.yml

ZooKeeper-era Kafka had a clean process-level separation: ZooKeeper processes handled coordination, Kafka broker processes handled data. KRaft collapses coordination into Kafka itself, but does so by introducing an explicit role each Kafka process plays, set via process.roles. A node can start with process.roles=broker, process.roles=controller, or — common in small and local deployments — process.roles=broker,controller, meaning one JVM does both jobs.

Alongside that, a node needs to know how to find the controller quorum. Two settings can do this:

<table> <thead><tr><th>⚙️ Setting</th><th>🎯 Model</th><th>📌 When to use</th></tr></thead> <tbody> <tr><td><code>controller.quorum.voters</code></td><td>Static — every voter listed as <code>nodeId@host:port</code></td><td>Fixed quorum membership; simplest for local dev and still fully supported</td></tr> <tr><td><code>controller.quorum.bootstrap.servers</code></td><td>Dynamic — an entry point, membership managed at runtime</td><td>Kafka 3.9+ clusters using dynamic quorum membership, where controllers can be added or removed without a full-cluster config change</td></tr> </tbody> </table>

Static voters are what most examples and local setups still use, and they remain valid in Kafka 4.0 — the examples in this lesson use them for that reason. If you're provisioning a production cluster where you expect to resize the controller quorum, the dynamic form is worth reading up on; the mechanics belong to the KRaft Architecture lesson.

Either way, this is the setting that occupies the space zookeeper.connect used to fill:

## docker-compose.yml excerpt: a single combined broker+controller node,
## typical for local .NET integration test environments
services:
  kafka:
    image: apache/kafka:latest
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:9093"
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      # Required: CONTROLLER is a listener NAME, not a protocol — without
      # this mapping the node fails to start.
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
    ports:
      - "9092:9092"

The environment variables that didn't exist in a ZooKeeper-based compose file — KAFKA_PROCESS_ROLES and the quorum setting — are exactly the ones replacing what ZooKeeper used to provide implicitly.

Metadata Propagation: Fewer Stale-Metadata Surprises

One behavioral difference .NET developers notice without changing a line of code is how quickly the cluster's view of partition leadership becomes consistent after a change. Under ZooKeeper, metadata changes were written to ZooKeeper and propagated to brokers via watches, with the controller broker responsible for pushing updates out. That worked, but introduced multiple hops and windows where a client's cached metadata didn't yet reflect reality, producing NOT_LEADER_OR_FOLLOWER and similar errors on the next produce or fetch.

KRaft replaces the watch mechanism with a single Raft-replicated metadata log that all brokers consume directly. Instead of a controller reading from ZooKeeper and republishing, every broker tails the same ordered log of metadata records. One authoritative log rather than a coordination store plus a relay step means a shorter, more uniform propagation path, and disagreements between what a broker believes and what the controller just committed resolve faster.

The practical upshot: you may see fewer stale-metadata errors following a leadership change, and error windows around controller transitions tend to close more quickly. This does not mean such errors disappear — a client can still hold cached metadata momentarily behind the cluster's true state, and handling that is still the job of retry and timeout configuration, covered in "Writing Resilient .NET Clients for KRaft-Based Clusters." What changes is the frequency and duration of the condition, not its existence.

⚠️ Common Mistake: Assuming that because KRaft's metadata propagation is faster, a .NET service no longer needs sensible metadata-refresh or retry handling. Faster convergence reduces the odds of hitting a stale-metadata window during routine leadership changes; it does not eliminate transient errors during an active controller election, which is exactly the edge case explored later.

Spinning Up and Targeting a KRaft Cluster from a .NET Project

Once you understand that a KRaft cluster has no ZooKeeper ensemble behind it, the practical question is how to stand one up locally and how a .NET producer, consumer, or admin client finds it. The client-side wiring barely changes. The work that does change is entirely in configuring the broker container, because a KRaft node now has to know things ZooKeeper used to track on its behalf — its identity, its role, and who else votes on cluster metadata.

Configuring a Combined Broker+Controller Node for Local Development

For integration tests you don't need a multi-node quorum — a single node acting as both broker and controller via process.roles=broker,controller is the standard pattern. It gives you no controller fault tolerance at all, which is exactly why production clusters split these roles across multiple dedicated controller nodes.

A combined node needs three pieces of identity ZooKeeper used to hand out: a node ID, a cluster ID (stamped into the storage directory at format time, or auto-generated by the apache/kafka image entrypoint on first start), and the controller quorum setting listing which node IDs may vote, with their controller-listener addresses.

services:
  kafka:
    image: apache/kafka:latest
    container_name: kraft-broker
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      # Single-node cluster: no replication headroom, local dev/test only
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

The format 1@kafka:9093 means "node ID 1, reachable at host kafka on port 9093 for controller traffic." Note that the client-facing listener (PLAINTEXT on 9092) and the controller listener (CONTROLLER on 9093) are deliberately separate — application traffic and Raft consensus traffic never share a listener.

Testcontainers, and the Advertised-Listener Trap

⚠️ Common Mistake — the one that costs an afternoon. KAFKA_ADVERTISED_LISTENERS is the address the broker tells clients to use after bootstrap. If you bind Kafka's port to a random host port (WithPortBinding(9092, true), the Testcontainers default for avoiding collisions) but leave advertised listeners hardcoded to localhost:9092, your client connects to the random port, receives localhost:9092 back in the metadata response, and then fails to reach the broker. The symptom looks like a network or firewall problem and has nothing to do with either.

The cleanest fix is to use the dedicated Testcontainers.Kafka package, whose KafkaBuilder rewrites advertised listeners to match the mapped port for you:

using Testcontainers.Kafka;

// KafkaBuilder handles the advertised-listener rewrite and KRaft startup.
var kafkaContainer = new KafkaBuilder()
    .WithImage("confluentinc/cp-kafka:latest")
    .Build();

await kafkaContainer.StartAsync();
var bootstrapServers = kafkaContainer.GetBootstrapAddress();

If you need the raw ContainerBuilder — say, to pin the official apache/kafka image or set KRaft variables the module doesn't expose — bind a fixed host port so the advertised address stays truthful:

var kafkaContainer = new ContainerBuilder()
    .WithImage("apache/kafka:latest")
    .WithPortBinding(9092, 9092)          // fixed, NOT random — must match advertised
    .WithEnvironment("KAFKA_NODE_ID", "1")
    .WithEnvironment("KAFKA_PROCESS_ROLES", "broker,controller")
    .WithEnvironment("KAFKA_LISTENERS", "PLAINTEXT://:9092,CONTROLLER://:9093")
    .WithEnvironment("KAFKA_ADVERTISED_LISTENERS", "PLAINTEXT://localhost:9092")
    .WithEnvironment("KAFKA_CONTROLLER_LISTENER_NAMES", "CONTROLLER")
    .WithEnvironment("KAFKA_CONTROLLER_QUORUM_VOTERS", "1@localhost:9093")
    .WithEnvironment("KAFKA_LISTENER_SECURITY_PROTOCOL_MAP",
        "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT")
    .WithWaitStrategy(Wait.ForUnixContainer().UntilPortIsAvailable(9092))
    .Build();

await kafkaContainer.StartAsync();
var bootstrapServers = "localhost:9092";

A fixed port means parallel test runs on one machine will collide, which is the trade you're accepting for a simpler listener configuration.

The wait strategy above only checks that the port is open — not that the broker finished formatting storage and completed leader election for internal topics. For a suite that hits Kafka immediately after startup, pair it with a short retry loop on the first admin call rather than assuming readiness the instant the port responds.

Producing and Consuming Against a KRaft Cluster

This is the reassuring part: ProducerConfig and ConsumerConfig need only BootstrapServers. There is no zookeeper.connect equivalent and no separate discovery step — the client asks whatever broker it can reach for current cluster metadata, and that broker answers using information it learned from the Raft metadata log instead of from ZooKeeper watches. From the client's point of view this is the same bootstrap-and-discover flow Kafka has always used.

using Confluent.Kafka;

var bootstrapServers = "localhost:9092"; // from Testcontainers or docker-compose

// Produce a single message
var producerConfig = new ProducerConfig { BootstrapServers = bootstrapServers };
using (var producer = new ProducerBuilder<string, string>(producerConfig).Build())
{
    var result = await producer.ProduceAsync("orders",
        new Message<string, string> { Key = "order-42", Value = "placed" });
    Console.WriteLine($"Produced to {result.TopicPartitionOffset}");
}

// Consume it back
var consumerConfig = new ConsumerConfig
{
    BootstrapServers = bootstrapServers,
    GroupId = "orders-test-group",
    AutoOffsetReset = AutoOffsetReset.Earliest
};
using (var consumer = new ConsumerBuilder<string, string>(consumerConfig).Build())
{
    consumer.Subscribe("orders");
    var consumeResult = consumer.Consume(TimeSpan.FromSeconds(10));
    Console.WriteLine($"Consumed: {consumeResult?.Message.Value}");
    consumer.Close();
}

Nothing here is KRaft-specific — precisely the point. What KRaft changes is what happens before this code runs and what tooling you use to inspect cluster state.

Verifying Cluster State with DescribeClusterAsync()

Once the container is running, confirm — rather than assume — that your client sees the cluster as expected. DescribeClusterAsync() returns the broker list, the cluster ID, and which node is acting as controller. This matters more under KRaft than before, because there's no external zkCli session to peek at; the AdminClient call is your window into cluster metadata now.

using Confluent.Kafka;
using Confluent.Kafka.Admin;

var adminConfig = new AdminClientConfig { BootstrapServers = bootstrapServers };
using var admin = new AdminClientBuilder(adminConfig).Build();

var clusterInfo = await admin.DescribeClusterAsync(
    new DescribeClusterOptions { RequestTimeout = TimeSpan.FromSeconds(5) });

Console.WriteLine($"Cluster ID: {clusterInfo.ClusterId}");
Console.WriteLine($"Controller: {clusterInfo.Controller?.Id} " +
                  $"({clusterInfo.Controller?.Host}:{clusterInfo.Controller?.Port})");
foreach (var node in clusterInfo.Nodes)
{
    Console.WriteLine($"Node {node.Id}: {node.Host}:{node.Port}");
}

Against the single-node combined setup above, Nodes should contain exactly one entry and Controller.Id should match it, since one process plays both roles. Scale up to a multi-voter quorum later and this same call is how you confirm — from application code, a CI health check, or a diagnostic script — which node currently holds the controller role.

Checking Client/Broker API Version Compatibility

Confluent.Kafka wraps librdkafka, which negotiates a supported API version range with the broker on connect — this is how the two agree on request/response formats. ⚠️ Common Mistake: pinning an old Confluent.Kafka NuGet package (and the librdkafka build bundled with it) that predates broker-side changes introduced for KRaft-era clusters. An outdated client can still connect but may fall back to older protocol assumptions and mishandle newer error codes, producing errors that look unrelated to versioning.

Inspect what actually got negotiated rather than guessing:

var debugConfig = new AdminClientConfig
{
    BootstrapServers = bootstrapServers,
    Debug = "broker,protocol"
};
using var debugAdmin = new AdminClientBuilder(debugConfig)
    .SetLogHandler((_, message) => Console.WriteLine($"[{message.Level}] {message.Message}"))
    .Build();

await debugAdmin.DescribeClusterAsync(
    new DescribeClusterOptions { RequestTimeout = TimeSpan.FromSeconds(5) });

The protocol debug context logs the ApiVersions request/response exchange, including the version ranges each side advertises. Treat mismatches as a signal to check your NuGet version — a stale package pin is a far more common root cause than an actual protocol incompatibility with a current broker.

One Connection Configuration, Two Environments

Because BootstrapServers is the only mandatory setting, keep it and any security settings entirely in configuration, never hardcoded, so one compiled binary can point at a local KRaft container in dev and a managed cluster in production. KRaft's simplified bootstrap model makes this easier to get right, since there's one moving part instead of two.

{
  "Kafka": {
    "BootstrapServers": "localhost:9092",
    "SecurityProtocol": "Plaintext"
  }
}

public class KafkaOptions
{
    public string BootstrapServers { get; set; } = string.Empty;
    public string SecurityProtocol { get; set; } = "Plaintext";
}

// Program.cs / composition root
builder.Services.Configure<KafkaOptions>(
    builder.Configuration.GetSection("Kafka"));

builder.Services.AddSingleton<IProducer<string, string>>(sp =>
{
    var options = sp.GetRequiredService<IOptions<KafkaOptions>>().Value;
    var config = new ProducerConfig
    {
        BootstrapServers = options.BootstrapServers,
        SecurityProtocol = Enum.Parse<SecurityProtocol>(options.SecurityProtocol)
    };
    return new ProducerBuilder<string, string>(config).Build();
});

In production, an environment variable or secrets store overrides Kafka:BootstrapServers and typically flips SecurityProtocol to SaslSsl with credentials — the code building the producer never changes.

Writing Resilient .NET Clients for KRaft-Based Clusters

A cluster that fails over its controller in well under a second sounds like it should make client-side resilience less necessary. In practice the opposite happens: because the failure window is short, it's tempting to skip defensive tuning entirely, and then a deployment pipeline or production producer trips over the one request that lands during that narrow window. The goal here is to make that window survivable by construction.

Faster Failover Doesn't Mean No Failover

Under ZooKeeper coordination, a controller failover could take several seconds because a new controller had to win a session-timeout-driven election and then re-read broker and partition state from ZooKeeper before acting. KRaft's Raft-based quorum typically completes leader election in a fraction of that time, because the new active controller already has the metadata log locally. The client-facing consequence: .NET producers, consumers, and admin clients see shorter error windows around controller changes, not zero-length ones.

A shorter window still needs a plan. librdkafka's timeouts are generous by default — tuned so that a transient problem doesn't fail a request, at the cost of taking a long time to surface a real one. Against a cluster that recovers in under a second, several of them are worth tightening so your client reacts as fast as the cluster does:

<table> <thead><tr><th>🔧 Setting</th><th>📦 librdkafka default</th><th>✅ Suggested</th><th>💬 Why</th></tr></thead> <tbody> <tr><td><code>message.timeout.ms</code><br/>(<code>MessageTimeoutMs</code>)</td><td>300000 (5 min)</td><td>30000</td><td>Total delivery budget per message; 5 minutes is far longer than any KRaft transition and delays real failures</td></tr> <tr><td><code>socket.connection.setup.timeout.ms</code></td><td>30000</td><td>10000</td><td>Move on to another broker sooner when one isn't coming back</td></tr> <tr><td><code>topic.metadata.refresh.interval.ms</code></td><td>300000 (5 min)</td><td>10000–30000</td><td>Notice leadership changes in seconds rather than minutes</td></tr> <tr><td><code>retry.backoff.ms</code></td><td>100</td><td>250–500</td><td>Avoid hammering a broker mid-election</td></tr> <tr><td><code>message.send.max.retries</code></td><td>2147483647</td><td>leave as-is</td><td>Already effectively unbounded — let the timeout be the real budget</td></tr> </tbody> </table>

🎯 Key Principle: Treat message.timeout.ms as the single source of truth for "how long is this operation allowed to take," and leave the retry count effectively unbounded underneath it. A transient controller change is then absorbed automatically instead of needing every retry knob tuned in lockstep.

(librdkafka accepts delivery.timeout.ms as an alias for message.timeout.ms, which is why Java-oriented material and .NET material sometimes appear to name different settings for the same budget.)

Idempotent Producers and a Resilient AdminClient Wrapper

Producer-side resilience starts with the idempotent producer: the broker deduplicates retried writes using a producer ID and sequence number, so a retry after a timeout doesn't create duplicates. Pairing idempotence with acks=all and a capped in-flight request count keeps ordering intact even when retries fire.

using Confluent.Kafka;

var producerConfig = new ProducerConfig
{
    BootstrapServers = "broker1:9092,broker2:9092,broker3:9092",
    EnableIdempotence = true,       // dedupes retried writes at the broker
    Acks = Acks.All,                // wait for all in-sync replicas
    MaxInFlight = 5,                // capped so ordering survives retries
    MessageTimeoutMs = 30000,       // overall delivery budget per message
    RetryBackoffMs = 300,           // spacing between retry attempts
    TopicMetadataRefreshIntervalMs = 30000
};

using var producer = new ProducerBuilder<string, string>(producerConfig).Build();

MaxInFlight caps unacknowledged requests outstanding on one connection; 5 or fewer preserves ordering when idempotence is on, since a higher number risks reordering across retried batches. (Enabling idempotence in librdkafka enforces this bound and acks=all for you, and will fail at construction if you set a conflicting value — setting them explicitly documents intent rather than adding behavior.)

AdminClient operations need the same posture but fail differently. Instead of a message timing out, a call like CreateTopicsAsync can throw when it reaches a broker that briefly can't answer authoritatively. A CI provisioning step that lets that bubble up is treating a sub-second condition as a hard error.

using System.Linq;
using Confluent.Kafka;
using Confluent.Kafka.Admin;

static async Task CreateTopicWithRetryAsync(
    IAdminClient adminClient,
    TopicSpecification spec,
    int maxAttempts = 5)
{
    // Transient around a controller transition; anything else is a real failure.
    static bool IsTransient(ErrorCode code) =>
        code == ErrorCode.NotController ||
        code == ErrorCode.RequestTimedOut ||
        code == ErrorCode.CoordinatorNotAvailable;

    var delay = TimeSpan.FromMilliseconds(500);

    for (var attempt = 1; attempt <= maxAttempts; attempt++)
    {
        try
        {
            await adminClient.CreateTopicsAsync(new[] { spec });
            return; // success
        }
        catch (CreateTopicsException ex) when (
            ex.Results.Any(r => r.Error.Code == ErrorCode.TopicAlreadyExists))
        {
            return; // idempotent for a provisioning step: already exists is fine
        }
        // Admin failures can surface either as a top-level error or as a
        // per-topic result, so check both before deciding to retry.
        catch (CreateTopicsException ex) when (
            attempt < maxAttempts &&
            (IsTransient(ex.Error.Code) || ex.Results.Any(r => IsTransient(r.Error.Code))))
        {
            await Task.Delay(delay);
            delay *= 2;
        }
        catch (KafkaException ex) when (attempt < maxAttempts && IsTransient(ex.Error.Code))
        {
            await Task.Delay(delay);
            delay *= 2;
        }
    }

    throw new InvalidOperationException(
        $"Failed to create topic '{spec.Name}' after {maxAttempts} attempts.");
}

The clauses do different jobs: "already exists" counts as success for a CI step that may run twice; the transient checks retry only the error codes that show up around a controller transition, and check both the top-level error and the per-topic results because admin failures can arrive either way. ⚠️ Common Mistake: catching KafkaException broadly and retrying on any code — that also retries genuine configuration errors (an invalid replication factor, say) that will never succeed, turning a fast clear failure into a slow confusing one.

Tuning Metadata Refresh and Connection Setup for Fast Recovery

Retry logic only helps if the client's view of the cluster is current enough for the retry to succeed. Two settings control how quickly a client notices topology changes: the metadata refresh interval, bounding how long a cached snapshot is trusted, and socket.connection.setup.timeout.ms, bounding how long the client waits on a TCP connection before trying another broker.

Left at defaults, a .NET client can be slower than the cluster itself to recover — the broker has elected a new controller in under a second while the client still holds five-minute-old metadata or waits 30 seconds on a socket to a broker that isn't coming back.

var adminConfig = new AdminClientConfig
{
    BootstrapServers = "broker1:9092,broker2:9092,broker3:9092",
    SocketConnectionSetupTimeoutMs = 10000,      // fail fast on an unreachable broker
    TopicMetadataRefreshIntervalMs = 15000,      // notice leadership changes sooner
    MetadataMaxAgeMs = 60000                     // hard ceiling on cached metadata age
};

using var admin = new AdminClientBuilder(adminConfig).Build();

All three are strongly typed on ClientConfig, so they're available on AdminClientConfig, ProducerConfig, and ConsumerConfig alike — no string-keyed Set() needed. In librdkafka, topic.metadata.refresh.interval.ms is the knob that actually drives periodic refreshes; metadata.max.age.ms is the outer bound and should stay comfortably larger than it.

💡 Pro Tip: Apply these to producer and consumer configs, not just AdminClient — a consumer holding stale leadership metadata keeps sending fetch requests to a broker that's no longer the leader until its own refresh catches up.

Edge Case: Provisioning Topics During an Active Controller Election

The retry wrapper exists for exactly this: a CI step calls CreateTopicsAsync at the moment a controller election is in progress, and the broker it reaches either isn't the controller anymore or can't answer authoritatively yet. The client sees a not-controller or request-timeout error — not a malformed request, not a permissions problem, just "ask me again in a moment."

Without the wrapper, one unlucky request fails an entire deployment pipeline over a condition that resolves in under a second. With it, the request is retried after a short backoff and almost always succeeds on the second or third attempt. The fix isn't clever engineering — it's refusing to treat a known, transient, documented error code as fatal.

Harder Variant: Provisioning Topics Mid Rolling Upgrade

A tougher version appears when a provisioning step runs against a cluster mid rolling upgrade — not a single election, but a sustained period where different brokers sit at different points in the metadata log. In that state CreateTopicsAsync can return success from the broker that answered, while a different broker a later pipeline step talks to hasn't caught up and briefly reports the topic as not found.

The retry-on-error-code pattern doesn't cover this, because the create call itself succeeds — the danger is a false negative in a later verification step. The fix is to confirm convergence rather than trust a single success response:

using System.Linq;

static async Task VerifyTopicVisibleAsync(
    IAdminClient adminClient,
    string topicName,
    int maxAttempts = 6)
{
    var delay = TimeSpan.FromMilliseconds(500);

    for (var attempt = 1; attempt <= maxAttempts; attempt++)
    {
        // Note: on a cluster with auto.create.topics.enable left on, a metadata
        // request for a missing topic can create it. That's benign here because
        // we only call this for a topic we just created deliberately.
        var metadata = adminClient.GetMetadata(topicName, TimeSpan.FromSeconds(5));
        var topic = metadata.Topics.FirstOrDefault(t => t.Topic == topicName);

        if (topic is not null && topic.Error.Code == ErrorCode.NoError)
        {
            return; // topic is visible with no per-topic error
        }

        await Task.Delay(delay);
        delay *= 2;
    }

    throw new InvalidOperationException(
        $"Topic '{topicName}' was created but is not yet visible cluster-wide.");
}

Calling CreateTopicWithRetryAsync then VerifyTopicVisibleAsync gives a pipeline two defenses: the first absorbs errors thrown during creation, the second absorbs creation succeeding before the cluster has converged on that fact. ⚠️ Skipping verification and treating a successful create as the end of the story is a common way a CI pipeline passes while a smoke test against a lagging broker fails moments later — two failures that look unrelated unless you know they share a metadata-log catch-up window.

Taken together these patterns share one posture: assume the cluster will occasionally answer "not yet" rather than "yes" or "no," and treat "not yet" as a reason to wait and ask again rather than to fail. That costs a handful of retry lines and turns KRaft's fast-but-nonzero failover into something a .NET service never has to notice.

Common Mistakes .NET Teams Make with Modern Kafka Clusters

Migrating a cluster to KRaft is usually the easy part. The mistakes that bite .NET teams show up afterward, buried in manifests, package references, and CI pipelines nobody revisits once the cluster "just works."

Mistake 1: Zombie ZooKeeper Configuration ⚠️

The most common mistake isn't a KRaft misconfiguration — it's leftover ZooKeeper configuration nobody deleted. Teams migrate bootstrap.servers, confirm producers and consumers work, and call the migration done. But the compose file, Helm chart, or Kubernetes manifest that stood up the old cluster often still contains a zookeeper.connect variable, a separate ZooKeeper container, and — critically — a readiness or liveness probe pinging ZooKeeper's port before marking the broker pod healthy.

❌ Wrong thinking: "The broker starts fine, so the leftover ZK block is harmless dead weight." ✅ Correct thinking: A stale ZK health check is an active failure mode, not inert clutter — it can block pod readiness, fail CI health checks, or cause an orchestrator to restart a perfectly healthy broker because a service it no longer depends on isn't responding.

## ⚠️ Leftover from a pre-KRaft manifest — this probe targets a ZooKeeper
## port that no longer exists once process.roles is broker,controller
readinessProbe:
  tcpSocket:
    port: 2181   # ZooKeeper's client port
  initialDelaySeconds: 10
  periodSeconds: 5

If the ZooKeeper container was removed but this probe wasn't, the pod never reports ready, and any integration test or CI step waiting on readiness times out with an error that looks like a networking problem. The fix is a deliberate audit: grep every compose file, Helm values file, and manifest for zookeeper, zkClient, and port 2181, and remove them alongside --zookeeper flags in CI shell scripts.

Mistake 2: Treating the Controller Quorum Size as a Formality ⚠️

It's tempting to fill in the controller quorum with whatever node count feels convenient — one for a quick dev setup, or an even number because it "seemed reasonable" — without registering that this setting is the fault-tolerance boundary of the cluster's metadata plane.

A single controller voter gives zero redundancy: if that process dies, no new controller can be elected and metadata operations — topic creation, partition reassignment, ISR updates — stall until it's replaced. An even number doesn't buy anything over the next odd number down, because Raft-style quorums need a strict majority to make progress. Three voters tolerate one failure; four voters also tolerate only one, while requiring an extra machine and an extra vote in every round. Five tolerate two.

## ✅ Three-voter quorum: tolerates one controller failure
controller.quorum.voters=1@ctrl-1:9093,2@ctrl-2:9093,3@ctrl-3:9093

## ❌ Single voter: any controller loss halts metadata operations
controller.quorum.voters=1@ctrl-1:9093

💡 Pro Tip: Treat quorum size like a database replica count — pick an odd number (three is the common baseline, five when you need to survive two simultaneous failures) and revisit it whenever you change the number of controller-eligible nodes.

Mistake 3: Pinning a Pre-KRaft Client Library Version ⚠️

Teams pin Confluent.Kafka (and transitively librdkafka) in a .csproj and leave it there, especially in services that "work fine." When that pin predates KRaft-aware protocol handling, pointing it at a KRaft-mode broker produces confusing symptoms: connection failures that look like network issues, UNSUPPORTED_VERSION from the AdminClient, or metadata requests returning stale or incomplete broker lists.

The underlying issue is API version negotiation. Kafka's protocol evolves per-API; client and broker each advertise supported version ranges and settle on the highest mutual one. Older librdkafka builds shipped before admin and controller API changes tied to KRaft existed, and may assume broker behavior that no longer holds.

<!-- ❌ Old, unmaintained pin left in a .csproj for months -->
<PackageReference Include="Confluent.Kafka" Version="1.4.2" />

<!-- ✅ A current version, pinned explicitly and verified against your broker.
     Prefer an exact version over a floating range so builds stay reproducible. -->
<PackageReference Include="Confluent.Kafka" Version="2.11.0" />

⚠️ Common Mistake: Assuming that because produce and consume still work, the library is fully compatible. Simple produce/consume paths are the most backward-compatible part of the protocol; AdminClient operations and metadata-heavy paths are where mismatches surface first. The negotiated-API-versions check from the previous section is the concrete way to confirm rather than guess.

💡 Real-World Example: A nightly CI job calling CreateTopicsAsync against a freshly upgraded KRaft cluster starts throwing KafkaException with obscure codes, while the same team's produce/consume smoke tests pass. Root cause: a Confluent.Kafka version older than the cluster's Kafka release, whose admin protocol assumptions don't match what the newer broker advertises.

Mistake 4: Sizing Local Retention as If Tiered Storage Is Already On ⚠️

A subtler mistake: teams read about tiered storage — the split between fast local disk for recent segments and cheaper remote object storage for older ones — and assume it changes their retention math automatically. The .NET-facing error is specific: configuring retention as if remote offload is happening when the feature was never enabled.

Three settings are involved, and conflating them is the trap:

<table> <thead><tr><th>⚙️ Setting</th><th>📍 Scope</th><th>🎯 Meaning</th></tr></thead> <tbody> <tr><td><code>remote.log.storage.system.enable</code></td><td>Broker</td><td>Turns the tiered storage subsystem on at all</td></tr> <tr><td><code>remote.storage.enable</code></td><td>Topic</td><td>Opts this topic into remote offload</td></tr> <tr><td><code>local.retention.ms</code> / <code>local.retention.bytes</code></td><td>Topic</td><td>How much stays on local disk once offload is active</td></tr> </tbody> </table>

Set a generous 30-day retention.ms on a high-throughput topic reasoning that "old segments offload to cheap storage anyway," without the broker-level flag actually on, and every byte accumulates on local disk. The volume fills, and the broker takes the log directory offline — which surfaces to clients not as a tidy "disk full" but as produce failures and leadership errors on the affected partitions, since a broker with an offline log dir stops serving them.

using Confluent.Kafka;
using Confluent.Kafka.Admin;

// Verification, not configuration: check what the broker actually reports
// before trusting retention math that depends on offload.
using var admin = new AdminClientBuilder(new AdminClientConfig
{
    BootstrapServers = "localhost:9092"
}).Build();

var results = await admin.DescribeConfigsAsync(new[]
{
    new ConfigResource { Type = ResourceType.Topic, Name = "orders" }
});

foreach (var result in results)
{
    foreach (var entry in result.Entries.Values)
    {
        if (entry.Name.Contains("remote", StringComparison.OrdinalIgnoreCase) ||
            entry.Name.StartsWith("local.retention", StringComparison.OrdinalIgnoreCase))
        {
            Console.WriteLine($"{entry.Name} = {entry.Value}");
        }
    }
}

Note DescribeConfigsAsync — like DescribeClusterAsync, there is no synchronous overload. The fix for the underlying mistake is procedural: size local retention on confirmed broker-side state, not on the theoretical existence of the feature, and treat dev and CI environments — which almost always skip object storage setup — as local-disk-only unless proven otherwise.

Mistake 5: Treating KRaft as Invisible Plumbing and Skipping Integration Tests ⚠️

The last mistake is the most conceptual and the one that causes production incidents rather than dev annoyances. Because the client-facing API surface barely changes, it's easy to conclude KRaft is purely internal and never needs testing against directly.

❌ Wrong thinking: "We already have integration tests against a Kafka container from before the migration; the client code didn't change, so those tests still validate production behavior." ✅ Correct thinking: Tests last validated against a ZooKeeper-mode cluster can pass while missing real behavioral differences that only appear under KRaft, particularly around AdminClient timing and error codes.

The concrete gap is admin API error codes and timing. A team that never updated its Testcontainers image from a ZooKeeper-based Kafka image to a KRaft-mode one may never observe the faster-but-different controller failover, or the specific exception shapes thrown when an admin call races a controller change, until that race happens in production during a rolling upgrade.

🎯 Key Principle: If your service calls the AdminClient for anything beyond trivial reads — topic creation, partition reassignment, config updates — your integration suite should run against the same KRaft-mode topology (a single combined node is fine) that production uses, not a leftover ZK-era image kept around out of inertia.

Each of these shares a shape: something true under ZooKeeper — a health check target, a client library assumption, a retention calculation, a belief that internals don't matter — quietly stops being true under KRaft, and nothing forces a team to notice until it fails. The fix in every case is the same discipline: audit infrastructure files explicitly rather than assuming they were updated, verify broker-side state rather than trusting configuration intent, and test against the topology you actually run.

Modern Kafka Mental Model: Recap and Where to Go Next

You've walked through the config-level differences between ZooKeeper-era and KRaft-era clusters, spun up a KRaft node from a .NET solution, and hardened a producer/consumer/AdminClient trio against the new failure modes. The mental model underneath is simple to state and easy to forget under deadline pressure: your .NET code has exactly one door into the cluster, and ZooKeeper was never that door for application traffic — it's just gone as an option entirely now.

The One-Door Model

In the ZooKeeper era, a surprising amount of tooling and mental overhead existed because two systems needed reasoning about. Even though producers and consumers never spoke the ZooKeeper protocol, plenty of adjacent work did — health checks, migration scripts, diagnostics that shelled out to zkCli.sh or passed --zookeeper. That second system is what's been removed, not a second connection string your client code maintained.

Your .NET service
    ↓
bootstrap.servers  (initial connection, protocol-level only)
    ↓
Kafka protocol requests: produce, fetch, metadata, admin RPCs
    ↓
Brokers (some of which may also serve as controllers)

Every operation — producing, consuming, or calling DescribeClusterAsync() — travels this same path. No second protocol, no second port, no second client library. If you find yourself reaching for a ZooKeeper client package or a zookeeper.connect setting in a modern Kafka project, that's a signal you're solving a problem that no longer exists.

Quick-Reference Checklist for a .NET Project

<table> <tr><th>✅ Check</th><th>🔍 What to verify</th><th>🛠️ How to verify it from .NET</th></tr> <tr><td>🆔 cluster.id present</td><td>Cluster was formatted, not left default</td><td><code>DescribeClusterAsync().ClusterId</code></td></tr> <tr><td>🎭 process.roles correct</td><td>Broker/controller roles match topology intent</td><td>Docker/K8s manifest review, not a client call</td></tr> <tr><td>📡 Advertised listeners match</td><td>The address brokers hand back is reachable from the client</td><td>Connect and produce one message; failure here is almost always this</td></tr> <tr><td>📦 Client version verified</td><td>librdkafka negotiates expected API versions</td><td><code>Debug = "broker,protocol"</code> trace</td></tr> <tr><td>⏱️ Resilience timeouts reviewed</td><td>Message timeout and metadata refresh set deliberately</td><td>Config review against librdkafka defaults</td></tr> </table>

If DescribeClusterAsync() ever returns an unexpected cluster ID in staging, that usually means someone pointed a client at a different or freshly re-formatted node rather than a genuine client bug — worth ruling out before debugging retry logic.

using Confluent.Kafka;
using Confluent.Kafka.Admin;

var adminConfig = new AdminClientConfig
{
    BootstrapServers = "localhost:9092" // the one door: no zookeeper.connect exists
};

using var admin = new AdminClientBuilder(adminConfig).Build();

// Cluster identity, controller, and reachable node count in one call.
// A client built against an incompatible protocol version fails here, not later.
var description = await admin.DescribeClusterAsync(new DescribeClusterOptions
{
    RequestTimeout = TimeSpan.FromSeconds(5)
});

Console.WriteLine($"Cluster ID: {description.ClusterId}");
Console.WriteLine($"Controller: {description.Controller?.Host}:{description.Controller?.Port}");
Console.WriteLine($"Reachable nodes: {description.Nodes.Count}");

Drop this into a startup routine or CI job so a misconfigured cluster fails fast with a clear message instead of surfacing as a mysterious timeout three services downstream.

0

Resilience Configuration Is the Default Posture, Not a Special Case

It's tempting to treat retry/backoff and idempotent-producer patterns as extra hardening you add once a service is important enough to justify the effort. That gets the risk backwards. KRaft's controller failover is fast, but "fast" still means a real window during which metadata can be stale or an admin call can hit a not-controller error. A service that adds retry logic only after its first outage has, by definition, already paid the cost the default posture was designed to avoid.

// The same defensive shape from "Writing Resilient .NET Clients" — repeated to
// underline that it's a baseline, not an advanced option for high-traffic services.
var producerConfig = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    EnableIdempotence = true,
    Acks = Acks.All,
    MaxInFlight = 5,
    MessageTimeoutMs = 30000,                 // tightened from the 5-minute default
    TopicMetadataRefreshIntervalMs = 30000,   // tightened from the 5-minute default
    RetryBackoffMs = 300
};

⚠️ Common Mistake: Copying producer and consumer configuration from an older reference project without revisiting the timeouts. Those values were often tuned around ZooKeeper session timeouts that no longer apply, and leaving them unexamined means a KRaft cluster's faster recovery never actually benefits your service — the client still waits as long as it always did.

Where the Deeper Internals Live

This lesson stayed at the boundary between your .NET code and the cluster. Two pieces belong to their own lessons.

The controller quorum — how voters reach consensus, what a Raft log entry looks like for a partition reassignment, how leader election unfolds among voters and observers, and how dynamic membership (controller.quorum.bootstrap.servers) differs from static voters — is covered in the KRaft Architecture lesson. You don't need it to configure a .NET client correctly, but it becomes valuable when debugging quorum sizing or reasoning about failure tolerance during a rolling upgrade.

The split between local (hot) storage and remote (cold) storage — tiered storage — changes how you think about retention, disk sizing, and read latency for older data, but not how producer or consumer code is written. That's the Tiered Storage Concepts lesson. The connection back here is narrow but important: a dev environment with retention sized for a tiered-storage production cluster will fill its disk, because the feature was never enabled locally.

🎯 Key Principle: This checklist is a starting heuristic for catching the most common KRaft-related misconfigurations quickly — not an exhaustive audit. Cluster-specific topologies (multi-datacenter controller placement, custom authentication, managed-service abstractions) introduce failure modes these items won't catch.

1

Practical Next Steps

Three actions turn this recap into something you apply. First, audit one existing service's Kafka configuration against the checklist — most teams find at least one leftover ZooKeeper-era assumption, whether a stale health check, an unreviewed timeout, or an unpinned client version. Second, if your integration tests still spin up a ZooKeeper container alongside a broker, replace that with a single combined process.roles=broker,controller node — a smaller compose file and a faster test startup. Third, before your next deployment touching Kafka, apply the idempotent-producer-plus-retry-wrapper pattern even if the service has never had a Kafka incident; treat it as the default, not the fix.

💡 Remember: ZooKeeper's removal simplified the cluster's internals without changing the shape of your application code — the door your .NET service walks through was always bootstrap.servers and the Kafka protocol, and KRaft just closed off every other door that used to exist alongside it.