KIP-354 — tombstone retention

Status: partial — config surface only; tombstone expiry is not yet enforced. See the KIP index.

What the KIP changes in Apache Kafka

Upstream, KIP-354 (Kafka 2.3) is titled "Add a Maximum Log Compaction Lag" and adds max.compaction.lag.ms: an upper bound on how long a record — including a tombstone — can sit uncompacted, so deletion is guaranteed to happen within a bounded time (the GDPR-shaped use case). The tombstone-lifetime knob itself, delete.retention.ms, predates the KIP by years: it bounds how long a delete marker survives after its segment is cleaned, so lagging consumers replaying the compacted log still observe the deletion.

What kaas actually has

kaas's source (crates/kaas-storage/src/topicconfig.rs) attributes its delete.retention.ms field to KIP-354 — the shared goal is bounded tombstone lifetime — but to be precise: kaas implements the delete.retention.ms config surface, and max.compaction.lag.ms does not exist anywhere in kaas (no config field, no CRD entry, no defaults row). The declared kaas semantics also differ from Apache in two ways:

• Expiry granularity is per batch, keyed on the batch baseTimestamp, where Apache decides per record — a consequence of the storage engine never opening batches (Storage hot path).
• 0 is documented as "tombstones live forever" (crates/kaas-broker/src/topic_config_defaults.rs), where Apache's 0 means tombstones are removable as soon as the segment is cleaned.

Like KIP-58, what ships today is the plumbing, not the enforcement. The field flows from KafkaTopic.spec.config through the operator into /data/<topic>/.config.json (delete_retention_ms, topicconfig.rs), IncrementalAlterConfigs accepts the key (crates/kaas-broker/src/topic_cr_writer.rs), and DescribeConfigs advertises a default of 86400000 (24 h, matching Apache) from topic_config_defaults.rs. But the compactor that would drop expired tombstones does not exist yet: crates/kaas-storage/src/cleaner.rs implements size- and time-based delete-policy retention and marks the compactor honouring both knobs as follow-up work (tracked as gh #158). No compaction runs, so no tombstone is ever compacted away — but the outcome is no longer safely conservative. The retention cleaner does not consult cleanup.policy, and the enforced retention.ms default of 7 days means a compacted topic's closed segments — tombstones and live keys alike — age out unless retention.ms is set to -1. With respect to the KIP's guaranteed-removal goal: nothing is guaranteed to disappear within delete.retention.ms, and until the compactor lands everything old enough disappears wholesale under delete-style retention.

How it's verified

Config-plumbing level only: roundtrip_preserves_unset_vs_zero in crates/kaas-storage/src/topicconfig.rs round-trips a set delete_retention_ms next to unset siblings, and scripts/kafka-configs.sh covers the --describe / --alter surface against a live broker. Tombstone-expiry behaviour has no tests because the behaviour is not implemented.