cursus

Bounded diagnostic observations

Design and evidence

The October 20 incident artifacts identify shared I/O stalls and synchronous diagnostic paths, not a proven deadlock or a regression in checkpoint suppression. The starting revision is fa218d884c02f43e927705748ab8b64a6a554d45. HealthState.report invokes callbacks synchronously; its context cannot interrupt callbacks that ignore it. Disk readiness performs stat and statfs; disk runtime collection also waits on handler locks held by fsync. Topic and transaction snapshots can indirectly wait on I/O too.

The selected approach runs fixed, sequential diagnostic workers and publishes immutable memory snapshots. A per-request timeout goroutine was rejected because uncancellable calls would accumulate. Replacing every storage lock with atomics was rejected because it expands the durability change surface unnecessarily.

Kafka comparison (before implementation)

These references describe implemented Kafka behavior. The expiry policy and workers below are the Cursus design, not a claim of Kafka equivalence.

Contract and regression criteria

Availability and lifecycle limits

There is no hard wall-clock guarantee if the process/OS cannot schedule threads or the client cannot read the response. The guarantee is isolation from storage I/O and application locks in diagnostic handlers. Context cancellation cannot interrupt an OS stat/statfs or mutex wait. Stop must cancel future work without joining such a call; at most the fixed worker count remains blocked. Normal broker durable shutdown can still wait for fsync; this change does not bypass it.

Sampling introduces a bounded detection/recovery delay and freshness expiry can reduce availability under load. This is intentional: an old healthy result cannot certify current storage. There are three sequential workers per exporting broker: readiness, runtime metrics and default process/counter gathering. Only the readiness worker runs when the exporter is disabled. Each worker has at most one outstanding attempt. Prometheus may fan out its fixed set of default collectors within that one attempt. All dynamic runtime metrics may be omitted when their collection is blocked; diagnostic status and independently sampled process/counter metrics remain available. The default registry is also sampled to keep Linux procfs access out of the request path. Observation failures count attempts, not HTTP scrapes.

Snapshot consumption and compatibility

NewHealthState and NewCollector do not perform work. Their owners call Start and defer Close; the ordinary broker and both recovery diagnostics paths do so. External callers using the exported constructors must also adopt this lifecycle. metrics.Handler callers outside the server helper must close its returned handler; the server helper closes it on shutdown, listener failure and Serve exit. Close never signals that durable storage shutdown has completed.

cursus_observation_*{component="readiness|runtime|process"} exports validity, in-progress, deadline overrun, stale status, last completion, age, last failure, and completed failure count. Timestamp/age/last-failure series are absent until the first completion. A stuck first attempt exports valid=0, in_progress=1 and (after two seconds) timed_out=1 without inventing a completed result. A storage stat failure invalidates the runtime observation instead of exposing partial usage/capacity as zero. Missing segment/index paths trigger one membership refresh and resample: completed retention can remove the segment, and truncate recreates its index before releasing the mutation lock. Persistent absence after that bounded retry still invalidates the observation; it is never counted as measured zero. Successfully measured capacity below the configured headroom threshold is different: it remains a valid runtime observation with actual free/total bytes and headroom readiness 0, while broker readiness and write admission still fail. Both Helm charts retain the disk-headroom alert and add CursusObservationUnavailable when any component’s observation stays invalid for one minute, including initial absence, failure, expiry and deadline overrun. A successful HTTP scrape alone does not establish healthy storage. Existing metric names and labels remain unchanged when the observation is valid. Removed dynamic labels disappear on the next valid sample, or when the previous observation expires.

Readiness adds an observation JSON object and keeps existing status and checks fields. New check registration/replacement invalidates the earlier result until a pass over that check set completes. The two-second budget now covers the entire sequential health pass. Cluster topology, broker registration, metadata recovery and other existing dependency checks retain their meaning.

Local validation