The October 20 incident artifacts identify shared I/O stalls and synchronous
diagnostic paths, not a proven deadlock or a regression in checkpoint suppression.
The starting revision is fa218d884c02f43e927705748ab8b64a6a554d45.
HealthState.report invokes callbacks synchronously; its context cannot interrupt
callbacks that ignore it. Disk readiness performs stat and statfs; disk runtime
collection also waits on handler locks held by fsync. Topic and transaction
snapshots can indirectly wait on I/O too.
The selected approach runs fixed, sequential diagnostic workers and publishes immutable memory snapshots. A per-request timeout goroutine was rejected because uncancellable calls would accumulate. Replacing every storage lock with atomics was rejected because it expands the durability change surface unnecessarily.
These references describe implemented Kafka behavior. The expiry policy and workers below are the Cursus design, not a claim of Kafka equivalence.
/live stays process liveness (200); storage failures do not request restart.
/ready retains startup/dependency gating, GET/HEAD, JSON status/checks, and
200/503. Initial, expired, over-budget and stopped observations fail closed.There is no hard wall-clock guarantee if the process/OS cannot schedule threads or the client cannot read the response. The guarantee is isolation from storage I/O and application locks in diagnostic handlers. Context cancellation cannot interrupt an OS stat/statfs or mutex wait. Stop must cancel future work without joining such a call; at most the fixed worker count remains blocked. Normal broker durable shutdown can still wait for fsync; this change does not bypass it.
Sampling introduces a bounded detection/recovery delay and freshness expiry can reduce availability under load. This is intentional: an old healthy result cannot certify current storage. There are three sequential workers per exporting broker: readiness, runtime metrics and default process/counter gathering. Only the readiness worker runs when the exporter is disabled. Each worker has at most one outstanding attempt. Prometheus may fan out its fixed set of default collectors within that one attempt. All dynamic runtime metrics may be omitted when their collection is blocked; diagnostic status and independently sampled process/counter metrics remain available. The default registry is also sampled to keep Linux procfs access out of the request path. Observation failures count attempts, not HTTP scrapes.
NewHealthState and NewCollector do not perform work. Their owners call Start
and defer Close; the ordinary broker and both recovery diagnostics paths do so.
External callers using the exported constructors must also adopt this lifecycle.
metrics.Handler callers outside the server helper must close its returned
handler; the server helper closes it on shutdown, listener failure and Serve exit.
Close never signals that durable storage shutdown has completed.
cursus_observation_*{component="readiness|runtime|process"} exports validity,
in-progress, deadline overrun, stale status, last completion, age, last failure,
and completed failure count. Timestamp/age/last-failure series are absent until
the first completion. A stuck first attempt exports valid=0, in_progress=1 and
(after two seconds) timed_out=1 without inventing a completed result. A storage
stat failure invalidates the runtime observation instead of exposing partial
usage/capacity as zero. Missing segment/index paths trigger one membership refresh
and resample: completed retention can remove the segment, and truncate recreates
its index before releasing the mutation lock. Persistent absence after that bounded
retry still invalidates the observation; it is never counted as measured zero. Successfully
measured capacity below the configured headroom threshold is different: it remains
a valid runtime observation with actual free/total bytes and headroom readiness
0, while broker readiness and write admission still fail. Both Helm charts retain
the disk-headroom alert and add CursusObservationUnavailable when any component’s
observation stays invalid for one minute, including initial absence, failure,
expiry and deadline overrun. A successful HTTP scrape alone does not establish
healthy storage. Existing metric names and labels remain unchanged when
the observation is valid. Removed dynamic labels disappear on the next valid
sample, or when the previous observation expires.
Readiness adds an observation JSON object and keeps existing status and
checks fields. New check registration/replacement invalidates the earlier
result until a pass over that check set completes. The two-second budget now
covers the entire sequential health pass. Cluster topology, broker registration,
metadata recovery and other existing dependency checks retain their meaning.
go test -race including real HTTP probes while both storage
observation paths remain blocked. Windows targeted snapshot/endpoint tests
also pass with normal local permissions; the sandbox denies loopback sockets.go test -race ./pkg/... ./cmd/... run passed, including topic
checkpoint crash/restart, replication/FSM recovery and ACK regression suites.
Affected packages were rerun after follow-up changes.go build ./cmd/... passes on Linux; changed-package golangci-lint reports
zero issues. No workflow trigger exceptions or production workloads were used.bash test/monitoring/check-rules.sh STANDALONE_RENDER.yaml CLUSTER_RENDER.yaml
with monitoring enabled and namespace brokers; the deployment CI does so.