KIP-482 — Flexible versions and tagged fields

Status: implemented — back to the KIP index.

What the KIP changes in Apache Kafka

From Kafka 2.4, each API declares a cutover version at which its encoding becomes "flexible": strings, bytes, and arrays switch to compact (unsigned-varint) length prefixes, request headers gain a v2 shape and response headers a v1 shape, and every structure ends in a tagged-field block — an extensible (tag, size, value) section that lets either side attach optional data without bumping the API version. Unknown tags must be skipped, which is what makes the scheme forward-compatible.

How kaas implements it

The whole mechanism lives in crates/kaas-codec:

• crates/kaas-codec/src/tagged.rs — the envelope: uvarint(num_fields) || (uvarint(tag), uvarint(len), bytes)*. read consumes and discards every tag (Apache's documented forward-compat contract — no handler inspects a specific tag today); write_empty emits the single-byte uvarint(0) that is by far the most common case on the wire, and the only tagged block kaas writes.
• crates/kaas-codec/src/primitives.rs — compact strings, bytes, and arrays with the KIP's count + 1 length encoding (read_compact_string, read_compact_array_len, and friends).
• crates/kaas-codec/src/headers.rs — the HeaderVersion V0/V1/V2 split: flexible requests use v2 headers (client id + tagged block), flexible responses use v1 (correlation id + tagged block).
• crates/kaas-codec/src/api/registry.rs — each ApiSpec carries min_flexible: Option<i16>; is_flexible(version), request_header and response_header derive everything else from it, as Apache's generated ApiMessageType does. Deriving it is the fix for gh #272: per-module copies had given every legacy-version response a tagged-fields byte. The "Flexible" column of the API matrix is generated from this table, so the docs cannot disagree with what ApiVersions advertises.

One honest asymmetry: kaas reads tagged fields everywhere but never writes a non-empty block — every response ends in write_empty. That is also what Apache 3.7 does on most paths; kaas simply has no optional server-side tags to attach yet. See Wire protocol & framing for how this sits alongside the byte-opacity contract.

How it's verified

• crates/kaas-codec/src/tagged.rs unit tests: empty_block_roundtrips, unknown_tags_discarded_by_read, truncated_field_value_is_eof.
• crates/kaas-codec/src/headers.rs: request_v2_flexible_tagged_block, response_v1_flexible_with_tag_block (asserts the exact 5-byte shape).
• crates/kaas-codec/src/api/registry.rs: flex_predicate pins the flexible-from version for ApiVersions itself; header_versions_match_apache pins both header versions against Apache 3.7's values, including the ApiVersions exception.
• crates/kaas-protocol/src/dispatch.rs: legacy_versions_get_a_v0_response_header; bins/kaas/tests/auth_smoke.rs: legacy_sasl_authenticate_response_has_no_tagged_header_byte parses a SaslAuthenticate v1 reply off a real socket.
• Every per-API codec module round-trips its flexible versions; the leave_group.rs v5 Java-client fixture is a real regression pin — see KIP-800 for the bug it caught.
• scripts/kafka-broker-api-versions.sh exercises the ApiVersions response (itself a flexible API) against a live broker.