Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Observability

OTLP metrics and tracing pushed to Prometheus, and the /healthz endpoint's rich runtime state.

Apache Kafka exposes its metrics as JMX MBeans, and most deployments bolt a Prometheus JMX exporter onto every broker to scrape them. kaas is OpenTelemetry-native instead: brokers push OTLP metrics straight to Prometheus's native OTLP receiver, emit OTel spans, and correlate every log line with the active trace — no sidecar, no exporter agent.

Bootstrap: push-mode OTLP

The OTel SDK is wired from environment variables (all emitted by the Helm chart):

  • OTEL_EXPORTER_OTLP_METRICS_ENDPOINT — metrics are pushed via OTLP/HTTP (http/protobuf) to Prometheus's native OTLP receiver (/api/v1/otlp/v1/metrics). Prometheus speaks only http/protobuf on that path — exporting gRPC just gets an h2 GoAway.
  • OTEL_EXPORTER_OTLP_ENDPOINT — traces via OTLP/gRPC.
  • KAAS_METRIC_EXPORT_INTERVAL — a duration string (30s, 1m); default 30s. The SDK default of 60 s made rate([1m]) panels see a single sample and gap.
  • OTEL_SERVICE_VERSION, MY_POD_NAME, KAAS_NAMESPACE — folded into resource attributes so dashboards can filter by broker and namespace.
  • OTEL_TRACES_SAMPLER_ARG — trace sampling ratio in [0, 1], default 0.1.

With neither endpoint set the SDK still runs and exports go nowhere — safe for tests and local dev. Before bootstrap runs, the global metrics handle is a no-op registry, so pre-boot code and tests can record without nil checks.

One metrics-pipeline invariant is load-bearing enough to repeat here: gauge callbacks (high watermark, log start offset) read the storage engine through a lock-free snapshot, never the partition mutex. Before that split, a stuck NFS fsync holding the mutex stalled the OTel gauge callback and all broker metrics vanished from Prometheus until the stall cleared — exactly when you needed them most.

/healthz and /readyz

Both are served on port 8080:

  • /readyz is the kubelet readiness probe, and it is honest: listeners bound, the main runtime alive, and — in cluster mode — every partition the assignment gives this broker open in the storage engine, i.e. takeover complete. It is served from a dedicated thread and runtime, never the main runtime it reports on, so a wedged broker answers unready instead of hanging the probe. The full story — the two signals, and how readiness paces a rolling update — is Honest readiness & rollout pacing.
  • /healthz returns a JSON runtime view: controller identity (is_controller, controller_id, controller_epoch), heartbeat health (heartbeat_rtt_ms, heartbeat_age_ms), assignment state (assignment_version, assignment_age_ms), partition counts (partitions_led, partitions_assigned, partitions_recovering), and storage_stalled — true when any partition's most recent committer fsync tripped the stall watchdog, surfacing "storage backend wedged" before the broker accumulates enough queued appenders to look outwardly idle.

Fields with no measurement yet return -1 internally and render as JSON null, so dashboards show "not measured" rather than a misleading zero. In local-dev mode (no cluster runtime) the whole runtime view is absent and the handler serves zero-valued fields — the right answer when no controller, coordinator, or heartbeat client is running.

partitions_led sources from the same assignment.json-backed view of leadership as the Produce/Fetch ownership check — /healthz never invents its own answer to "who leads what".

Byte-opacity tripwires

The broker's load-bearing invariant is "kaas is a byte mover, not a byte interpreter": no code path decodes individual records or re-encodes a RecordBatch (see Storage engine hot path). Two counters give that invariant teeth — codec_record_decode and codec_batch_reencode — which must stay at zero in steady state. Every increment names the offending call site in a site attribute, and the KaasByteOpacityViolated alert fires on non-zero.

As of today no kaas code path calls these functions; the counters exist so the first violation is an alert, not an archaeology project.

Tracing

One global tracing stack is installed at boot: a filter honouring KAAS_LOG_LEVEL, a log formatter honouring KAAS_LOG_FORMAT (json or text), and an OTel layer that emits every span through the tracer from bootstrap. The correlation contract: every log line carries trace_id + span_id whenever a span is active, so a log grep pivots straight into the trace view. Span sampling defaults to 10% — set OTEL_TRACES_SAMPLER_ARG=1.0 for a debugging session.

Implementation notes (for contributors)

  • Everything lives in crates/kaas-observability: bootstrap.rs (OTel SDK bring-up), health.rs (the axum /healthz//readyz surface; the JSON view is the RuntimeState trait), byteopacity.rs (tripwire counters), tracing.rs (install_tracing, the tracing-subscriber stack).
  • Codec-side tripwire counterparts: crates/kaas-codec/src/tripwires.rs; CI assertion: bins/kaas/tests/byte_opacity.rs.
  • Two standing rules: pick KAAS_METRIC_EXPORT_INTERVAL with the Prometheus rate() window in mind, and gauge callbacks must read the engine's ArcSwap snapshot, never the partition mutex.
  • gh #95 — the fsync stall watchdog behind storage_stalled; gh #208/#211 — the honest, serving-gated /readyz (see readiness & rollout).