KIP index

The Kafka Improvement Proposals this library actually implements, degrades on, or deliberately does not implement. Ordered by number.

Legend: ✅ implemented · ⚠️ partial or degraded · ❌ not implemented · 🚫 blocked upstream

Each KIP is searchable by number on the Apache Kafka KIP index.

The ones worth expanding on

KIP-848 — the new consumer group protocol ✅

Kafka 4.x's default group protocol, and the reason group kinds is its own chapter. Assignment moves server-side: the broker computes it and the client acknowledges, replacing the JoinGroup/SyncGroup dance entirely.

Both halves are implemented. ConsumerGroupDescribe renders these groups completely — epochs, assignor, per-member assignment — and ConsumerGroupHeartbeat joins one, with a client-generated member id and a broker-computed assignment. This was adopted before the classic protocol on purpose: server-side assignment removes the byte-compatibility problem that makes classic hard, so it was the cheaper of the two to get right first.

KIP-932 — share groups ⚠️

Queue semantics on top of Kafka: multiple consumers on the same partition, per-record acknowledgement, no partition-exclusive ownership.

ShareGroupDescribe is implemented. Note that librdkafka has no KIP-932 support at all, so rdkafka cannot even generate a share-group fixture — the tests drive kafka-console-share-consumer.sh in the container instead.

KIP-405 and KIP-1005 — tiered storage ⚠️

Both sentinels this library can reach are surfaced distinctly rather than collapsed, and that matters more on a tiered cluster than anywhere else: EARLIEST and EARLIEST_LOCAL_TIMESTAMP differ by exactly the data that has been offloaded to remote storage, which on a tiered cluster is most of it. A UI that treats them as interchangeable reports wrong retention.

The third tiered sentinel, KIP-1023's -6, is blocked upstream.

KIP-227 — incremental fetch sessions ✅ in one crate, deliberately not the other

kafka-consume establishes and maintains sessions, which is what a steady-state consumer needs: after the first full fetch, subsequent requests carry only what changed.

kafka-read deliberately does not. crates/kafka-read/src/fetch.rs pins session_id = 0, session_epoch = -1 — Java's FetchMetadata.LEGACY sentinel — because a browse-shaped scan is one-shot, and an incremental session would make each scan depend on the last for no benefit.

The split is the point: the same KIP is right for one crate and wrong for the other, which is why they are separate crates.

KIP-98 — exactly-once ✅ · KIP-447 — exactly-once v2 ✅

Both halves of KIP-98 are here. On the read side, Visibility::CommittedOnly sets read_committed and the client filters aborted records using the AbortedTransactions list the broker returns — the broker does not filter for you. On the write side, kafka-produce claims a producer id, tracks per-partition sequences, and drives InitProducerId/AddPartitionsToTxn/EndTxn behind init_transactions, begin_transaction, commit_transaction and abort_transaction — including the epoch bump KIP-890 hides inside EndTxn. DescribeTransactions, ListTransactions and DescribeProducers inspect the resulting state.

KIP-447 closes the loop. Producer::send_offsets_to_transaction commits a consumer's offsets inside the producer's transaction, which is what makes a consume-process-produce cycle exactly-once end to end. Two hops in one transaction, and the order is not optional: AddOffsetsToTxn to the transaction coordinator enrols the __consumer_offsets partition backing the group, then TxnOffsetCommit to the group coordinator stores the offsets pending the marker.

The consumer supplies its own identity — Consumer::group_metadata, GroupConsumer::group_metadata, ClassicConsumer::group_metadata — as ConsumerGroupMetadata, which lives in kafka-meta because it is the one value that travels from a consumer to a producer and kafka-produce does not depend on kafka-consume. It carries one generation field for both group kinds, because the classic generation_id and KIP-848's member_epoch are the same wire field.

What proves it is the aborted case, not the committed one: an offset commit that merely runs alongside a transaction passes a commit-only test and fails the moment an abort is supposed to take the offsets with it. Both halves run in crates/kafka-produce/tests/transactions.rs and against a real cluster in crates/livetest/src/produce.rs, where the two coordinators are genuinely different machines.

Closed by #10.

KIP-255 — SASL/OAUTHBEARER ✅ · KIP-768 — OIDC token retrieval ✅

Split across an issue boundary on purpose, because the second half is the one that changes the dependency tree.

KIP-255 is the mechanism: RFC 7628's %x01-separated initial client response, the failure challenge, and a TokenProvider the exchange asks again on every KIP-368 re-authentication rather than a token captured at construction. No new dependencies, and useful on its own to anyone with a token source.

KIP-768 is the client_credentials flow: fetch from a token endpoint, cache, refresh at 80% of the lifetime. It needs an HTTP client, so it is behind the oidc cargo feature — a caller who brings its own tokens does not pay for hyper in the crate everything else here sits on. It decodes no JWTs: the token response carries expires_in, and an access token is opaque to a client by design.

Neither one implements authorization-code or device flows. A Kafka client is a machine; when a human is in the loop the token arrives some other way, and SaslConfig::oauth_bearer takes it.

The framing traps, the second round trip, and why OAuth-over-PLAIN is not a substitute are in TLS, SASL and re-authentication.

Closed by #12 and #13.

KIP-699 — batched FindCoordinator ⚠️

The batched form resolves many coordinators in one round trip. kafka-meta uses the single-key form, which is correct on every broker version and costs an extra round trip per group on a cold cache. Worth revisiting for a UI rendering hundreds of groups at once.