Cursus exposes health and Prometheus endpoints on HTTP listeners separate from the client protocol listener.
| Default port | Endpoint | Response | Purpose |
|---|---|---|---|
9080 |
GET /live |
JSON | Process liveness |
9080 |
GET /ready |
JSON | Broker and dependency readiness |
9100 |
GET /metrics |
Prometheus/OpenMetrics | Broker metrics when the exporter is enabled |
Only GET and HEAD are accepted by health endpoints. Other methods return
405 Method Not Allowed.
The HTTP listeners bind before broker readiness is published. A health or metrics bind failure fails broker startup instead of being logged as a background-only failure.
/live returns 200 while the health HTTP server can execute requests. It does
not check topic storage, consumer groups, or cluster quorum.
{"status":"live"}
/ready returns 200 only after the client listener, command handler, worker
pool, and enabled HTTP services have initialized and all dynamic checks pass.
Consumer group member counts are not readiness checks. A registered group with
zero members leaves the broker ready; only broker-owned consumer metadata
recovery participates in readiness.
Standalone response:
{"status":"ready","checks":{"consumer_metadata":"ok","storage":"ok","topic_metadata":"ok"}}
In distributed mode, readiness also requires a resolvable cluster leader,
active local broker registration, successful node-local topic, replica, and
transaction-state materialization, and a topology that can safely admit
writes. An offline leader, incomplete assignment, unsatisfied effective
minimum ISR, or active replica-recovery marker makes every broker unready.
Ordinary under-replication alone does not fail readiness when the leader is
active, the assignment is complete, and the remaining ISR still satisfies the
effective minimum. CLUSTER_STATUS remains stricter: healthy=true requires a
full active ISR and no pending recovery or materialization.
During replica-gap recovery, CLUSTER_STATUS reports
recovery_pending_partitions > 0; the affected partition lists
recovery_replicas and the replica_recovery_pending reason. New writes to
that partition fail closed until bounded catch-up and prefix verification
re-admit the quarantined replica. Use these fields with /ready rather than
treating process liveness or an otherwise sufficient ISR as recovery success.
{
"status": "not_ready",
"checks": {
"cluster_leader": "no cluster leader available",
"storage": "ok"
}
}
Only /live and /ready are registered. /health and / return HTTP 404.
Recommended Kubernetes probes:
livenessProbe:
httpGet:
path: /live
port: 9080
readinessProbe:
httpGet:
path: /ready
port: 9080
Do not use /live for traffic admission. It intentionally remains successful
when the broker cannot safely serve client work.
Runtime gauges come from expiring background snapshots. Each completed sample replaces the preceding series, including labels for deleted topics, departed groups and old partition leaders.
| Metric | Type | Meaning |
|---|---|---|
cursus_broker_client_connections_total |
Counter | Accepted client TCP connections |
cursus_broker_client_connections_active |
Gauge | Connections currently handled |
cursus_broker_commands_total{command,result} |
Counter | Completed text command dispatches |
cursus_broker_command_duration_seconds{command} |
Histogram | Command dispatch latency |
cursus_broker_command_errors_total{command,code} |
Counter | Wire errors by bounded command and error code |
cursus_broker_client_response_write_failures_total{reason} |
Counter | Client responses that could not be delivered, with a bounded failure reason |
cursus_broker_publish_acknowledgements_total{ack_mode,result} |
Counter | Publish requests by normalized acknowledgement mode and bounded result |
cursus_broker_async_replication_failures_total{topic,error_class} |
Counter | Follower failures after an acks=1 leader acknowledgement |
cursus_broker_replication_retries_total{topic,ack_mode,error_class} |
Counter | Retried partition replication attempts, including pending duplicate commit barriers |
cursus_broker_log_compaction_runs_total{result,reason} |
Counter | Completed, skipped, or failed compaction passes with bounded reasons |
broker_messages_processed_total |
Counter | Messages accepted by the topic manager |
broker_message_latency_seconds |
Histogram | Topic manager publish latency |
broker_seqnum_gap_total{topic,partition,producer_id} |
Counter | Detected producer sequence gaps |
broker_seqnum_duplicate_total{topic,partition} |
Counter | Detected duplicate producer sequences |
Command duration ends when a streaming command is accepted. It does not include the lifetime or payload transfer time of a stream.
| Metric | Meaning |
|---|---|
cursus_broker_ready |
Current readiness result (1 or 0) |
cursus_broker_topics |
Topics loaded on this broker |
cursus_topic_metadata_restored_topics |
Topics restored from the durable standalone manifest during startup |
cursus_broker_partitions |
Partitions loaded on this broker |
cursus_partition_log_start_offset{topic,partition} |
Earliest retained offset |
cursus_partition_log_end_offset{topic,partition} |
Next allocated offset |
cursus_partition_high_watermark{topic,partition} |
Next offset visible to committed readers |
cursus_streams_active |
Registered streaming consumers |
cursus_storage_handlers |
Open partition storage handlers |
cursus_storage_segments |
Segment files represented by open handlers |
cursus_storage_bytes |
Segment and offset-index bytes represented by open handlers |
cursus_storage_pending_writes |
Messages queued for disk writes |
cursus_storage_active_readers |
Current segment readers |
cursus_storage_stat_failures |
0 in a valid runtime observation; failed measurements invalidate the snapshot and omit this series |
cursus_storage_segment_cache_entries |
Memory-mapped segment readers retained by open handlers |
cursus_storage_segment_cache_hits |
Cache hits accumulated by open handlers |
cursus_storage_segment_cache_misses |
Cache misses accumulated by open handlers |
cursus_storage_segment_cache_evictions |
Cache evictions accumulated by open handlers |
cursus_wire_protocol_failures_total{reason} |
Rejected Wire v2 frames by bounded protocol reason |
cursus_wire_decompression_rejections_total{reason} |
Rejected compressed payloads by bounded reason |
cursus_storage_bytes covers open topic handlers. It is not filesystem capacity
or free-space telemetry; collect those values with the node or container
runtime exporter.
Runtime metrics use background snapshots. Initial, failed, expired or over-budget
observations omit their values; missing series must not be interpreted as zero.
Use cursus_observation_valid{component=~"readiness|runtime|process"},
cursus_observation_stale, cursus_observation_timed_out,
cursus_observation_in_progress and cursus_observation_last_failed to distinguish
observation health from broker health. Completion timestamp, age and last-failure
series are absent before the first completed sample. Both Helm charts alert on
invalid observations lasting one minute with CursusObservationUnavailable.
A successful capacity measurement below the configured free-space reserve still
exports cursus_storage_filesystem_headroom_ready=0 and the measured free/total
bytes. This preserves CursusStorageDiskHeadroomLow without treating a failed
filesystem query as a measured zero. See snapshot lifecycle and defaults.
| Metric | Meaning |
|---|---|
cursus_consumer_metadata_recovery_ready{phase} |
1 only after durable group/offset replay completes without error |
cursus_consumer_metadata_restored_groups |
Groups materialized during startup |
cursus_consumer_metadata_restored_offsets |
Committed (group,topic,partition) next-offset keys restored |
cursus_consumer_metadata_replayed_records |
Active internal log records scanned |
cursus_consumer_metadata_orphan_records |
Records fenced or superseded by authoritative lifecycle/revision state |
cursus_consumer_metadata_corrupt_records |
Malformed or inconsistent records that stopped recovery |
In diagnostics-only mode, /ready includes the retained consumer_metadata failure and cursus_consumer_metadata_recovery_ready remains 0; the client command listener is not opened. Use the restored counts together with the offline standalone storage recovery inventory.
| Metric | Type / unit | Meaning |
|---|---|---|
cursus_consumer_group_members{topic,group} |
Gauge / members | Current member count, emitted only by the authoritative coordinator |
cursus_consumer_group_state{topic,group,state} |
Gauge / boolean | One-hot authoritative state: stable or empty |
cursus_consumer_group_coordinator_up{topic,group} |
Gauge / boolean | 1 only on the broker that successfully resolves itself as current coordinator; known groups emit 0 or 1, while unavailable group observation can omit the series |
cursus_consumer_group_last_activity_timestamp_seconds{topic,group} |
Gauge / Unix seconds | Latest accepted heartbeat or group lifecycle activity known by the authoritative coordinator; 0 means unknown, including after standalone durable recovery |
cursus_consumer_group_last_rebalance_timestamp_seconds{topic,group} |
Gauge / Unix seconds | Latest completed membership rebalance known by the authoritative coordinator; 0 means unavailable or no rebalance has completed since recovery |
cursus_consumer_group_observation_failures_total{topic,group,reason} |
Counter / failures | Per-broker observation failures; reason is one of coordinator_lookup, group_lookup, or topic_lookup, and a series appears only after its first failure |
cursus_consumer_group_generation{group,topic} |
Gauge / generation | Authoritative coordinator’s current group generation |
cursus_consumer_group_assigned_partitions{group,topic} |
Gauge / partitions | Authoritative coordinator’s assignments held by active members |
cursus_consumer_group_committed_offset{group,topic,partition} |
Gauge / offsets | Known durable next offset on the authoritative coordinator, with an observed local partition |
cursus_consumer_group_lag{group,topic,partition} |
Gauge / messages | Authoritative max(HWM - committedNextOffset, 0); absent when an input is unknown |
cursus_consumer_group_offset_out_of_range{group,topic,partition} |
Gauge / boolean | Authoritative commit is below local log start or above local HWM; absent when an input is unknown |
cursus_storage_filesystem_free_bytes |
Gauge / bytes | Space currently available on the filesystem containing log_dir |
cursus_storage_filesystem_total_bytes |
Gauge / bytes | Total capacity of the filesystem containing log_dir |
cursus_storage_filesystem_headroom_ready |
Gauge / boolean | 1 while the configured byte and percentage free-space reserve permits new writes |
cursus_broker_requests_inflight |
Gauge / requests | Decoded requests queued or processed across the client and internal admission pools |
cursus_broker_request_bytes_inflight |
Gauge / bytes | Reserved encoded and decoded request payload memory |
cursus_broker_request_admission_waiters |
Gauge / connections | Connections blocked before payload allocation by their listener’s request budget |
cursus_broker_request_admission_rejections_total{reason} |
Counter / requests | Admission failures caused by bytes or connection context cancellation |
In standalone mode the local coordinator is authoritative. In distributed
mode, replicated membership can remain present on a broker that no longer owns
the group, while heartbeat activity is coordinator-local. Such a broker emits
coordinator_up = 0 and omits the authoritative lifecycle and offset gauges. It does not
present its retained snapshot as healthy.
For a converged three-broker scrape, exactly one target emits authoritative
lifecycle gauges and the sum of coordinator_up is one. Gate each max by
that exact authority count; an unguarded max can retain an old positive value
during an overlapping coordinator view. Use sum for the per-broker failure
counters:
max by (topic, group) (cursus_consumer_group_members)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
max by (topic, group, state) (cursus_consumer_group_state)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
max by (topic, group) (
cursus_consumer_group_last_activity_timestamp_seconds
)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
max by (topic, group) (
cursus_consumer_group_last_rebalance_timestamp_seconds
)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
sum by (topic, group) (cursus_consumer_group_coordinator_up)
sum by (topic, group, reason) (
rate(cursus_consumer_group_observation_failures_total[5m])
)
sum of an authoritative lifecycle gauge is also exact after coordinator ring
convergence because only one broker emits it. A coordinator_up sum of zero
means no scraped target currently claims authority; a value above one reveals
an overlapping ring view. Both cases remove the gated lifecycle value instead
of allowing a stale value to mask a member-count decrease.
Heartbeat and rebalance timestamps are runtime lifecycle observations, not a
durable heartbeat journal. Cluster snapshots can carry them across coordinator
materialization, but standalone metadata recovery intentionally restores no
members and reports 0 until a new lifecycle event occurs. Persisting every
heartbeat would add storage write amplification to the consumer hot path.
An unknown group has no Cursus series. Cursus does not fabricate a zero-valued
group because it does not know which application groups are required. Use an
external cluster-config catalog to fill that missing actual state with zero.
See the consumer lifecycle alert runbook
for the catalog contract, alert rules, and a 1 -> 0 -> 1 canary procedure.
Lag uses the high watermark rather than the local log end so uncommitted replica tail data is not reported as consumable work. Both exact-topic and wildcard groups expose only known stored offsets with available local partition data. No stored commit means no offset/lag/out-of-range sample; a stored offset of zero remains a valid sample. Fetch/reset defaults do not imply a durable commit.
Names and labels are unchanged, but series from non-coordinators and fabricated
default-zero commits are removed. During rolling upgrades, filter the old
brokers’ metrics by authority on the same scrape target before aggregation.
For example, record the following expression as cursus:consumer_group_lag
(use the same expression with the out-of-range or committed-offset metric):
max by (cluster, group, topic, partition) (
cursus_consumer_group_lag
and on (cluster, job, instance, group)
(cursus_consumer_group_coordinator_up == 1)
and on (cluster, job, instance)
(up{job="cursus"} == 1)
and on (cluster, job, instance)
(cursus_observation_valid{component="runtime"} == 1)
)
and on (cluster, group)
(sum by (cluster, group) (cursus_consumer_group_coordinator_up) == 1)
Adapt cluster, job and instance to labels that uniquely identify your
cluster and broker scrape target; preserve tenant/namespace labels if required.
The group is broker-unique. Do not join on topic for wildcard or multi-topic
subscriptions: coordinator labels describe the subscription, whereas offset
labels describe the concrete topic. An exact authority count rejects overlapping
owner views. Samples across brokers still have sampling/scraping delay; this is
not an atomic cluster snapshot.
Alert separately on sum by (cluster,group) (...coordinator_up) != 1 and on
absent expected groups. A concrete missing-owner check for a known group is:
absent(cursus_consumer_group_coordinator_up{cluster="prod", group="workers"})
or
(sum(cursus_consumer_group_coordinator_up{cluster="prod", group="workers"}) != 1)
Also detect missing expected offset partitions with absent() or an external
expected-group/partition inventory. coordinator_up=1 alone does not prove that
every partition/commit is observable. Do not use or vector(0) for missing lag
or out-of-range data. Monitor up, runtime observation validity and consumer
metadata recovery separately. A missing group is not evidence that it is healthy.
Apply range functions to the authority-filtered recording rule; a direct
max_over_time over legacy per-broker series retains old false positives for
the whole window. These are query migration examples, not deployed alert changes.
See the design and regression comparison
for handoff/recovery behavior and the limitations of local partition observations.
Wire v2 exposes only cursus_consumer_group_lag; the former
broker_consumer_lag alias is not registered.
The in-process Go SDK exposes separate counters for physical anomalies and
expected compacted holes. cursus_consumer_offset_gap_total{topic,group}
increments only when a non-compacted topic jumps forward unexpectedly.
cursus_consumer_compacted_offsets_skipped_total{topic,group} counts logical
offsets removed by a cleanup policy that includes compact.
| Metric | Meaning |
|---|---|
cursus_distribution_enabled |
Distributed mode is enabled |
cursus_cluster_brokers |
Brokers in replicated metadata |
cursus_cluster_has_leader |
This broker resolves a cluster leader |
cursus_cluster_is_leader |
This broker is the current cluster leader |
cursus_cluster_offline_partitions |
Partitions without an active assigned leader |
cursus_cluster_under_replicated_partitions |
Partitions where active ISR size is below the topic replication factor |
cursus_cluster_assignment_deficient_partitions |
Partitions whose distinct assignment does not match the topic replication factor |
cursus_cluster_inactive_replica_partitions |
Partitions assigned to an inactive or unknown broker |
cursus_cluster_inactive_replicas |
Replica assignments that reference inactive or unknown brokers |
cursus_cluster_min_insync_unsatisfied_partitions |
Partitions whose active ISR size is below effective min.insync.replicas |
cursus_cluster_partition_replicas{topic,partition} |
Actual assigned replica count |
cursus_cluster_partition_expected_replicas{topic,partition} |
Replica count declared by the topic definition |
cursus_cluster_partition_active_replicas{topic,partition} |
Assigned replicas backed by active brokers |
cursus_cluster_partition_in_sync_replicas{topic,partition} |
Current active ISR size |
cursus_cluster_partition_min_insync_replicas{topic,partition} |
Effective min.insync.replicas |
cursus_cluster_partition_topology_healthy{topic,partition} |
Assignment, leader, and ISR satisfy the durable topic contract |
cursus_cluster_partition_leader_epoch{topic,partition} |
Current leader epoch |
cursus_cluster_partition_leader{topic,partition,broker_id} |
Current leader identity (1) |
cursus_cluster_isr_catchup_proofs_total{outcome,reason} |
ISR catch-up proofs accepted or rejected by bounded reason |
cluster_replication_lag_seconds{topic,partition,broker} |
Successful follower acknowledgement latency |
cluster_replication_lag_seconds is the only operation-time cluster metric.
Use the cursus_cluster_* scrape-time metrics for current topology and health.
| Metric | Meaning |
|---|---|
cursus_transaction_recovery_ready |
Transaction state recovery completed before serving |
cursus_transactions{state} |
Transactions retained by coordinator state |
cursus_transactions_expired |
Expired transaction identities awaiting replacement or compaction |
cursus_transaction_oldest_active_seconds |
Age of the oldest open or committing transaction |
Use these gauges with cursus_broker_command_errors_total{command,code} and the
transaction alert runbook.
health_check_port: 9080
enable_exporter: true
exporter_port: 9100
Equivalent environment variables are HEALTH_CHECK_PORT, ENABLE_EXPORTER,
and EXPORTER_PORT.
Prometheus example:
scrape_configs:
- job_name: cursus
scrape_interval: 15s
static_configs:
- targets: ["broker-1:9100", "broker-2:9100", "broker-3:9100"]
The standalone and three-member Helm charts can create a metrics Service,
ServiceMonitor, and PrometheusRule baseline. The Prometheus Operator CRDs
must already be installed. Add any labels required by the operator’s selectors;
for kube-prometheus-stack this is commonly:
monitoring:
enabled: true
labels:
release: kube-prometheus-stack
monitoring.serviceMonitor.enabled and monitoring.prometheusRule.enabled
control the two CRD-backed resources independently. Tune
consumerLag, transactionOldestSeconds, and the alert durations to the
application’s throughput and transaction timeout before enabling paging.
monitoring.grafanaDashboard.enabled installs the Cursus production
overview dashboard as a ConfigMap. The default grafana_dashboard: "1"
label is compatible with the common Grafana sidecar selector and can be
replaced through monitoring.grafanaDashboard.labels. The dashboard uses a
Prometheus datasource variable and a namespace variable; select the datasource
that scrapes the Cursus ServiceMonitor after import. See the
production incident runbook for the
response attached to each panel and alert.
# Scrape target unavailable
up{job="cursus"} == 0
# Process is reachable but cannot serve client work
cursus_broker_ready == 0
# Writes are blocked before the filesystem reserve is consumed
cursus_storage_filesystem_headroom_ready == 0
# No cluster leader
cursus_distribution_enabled == 1 and cursus_cluster_has_leader == 0
# Replication safety degraded
cursus_cluster_under_replicated_partitions > 0
# Durable definition and actual assignment disagree
cursus_cluster_assignment_deficient_partitions > 0
# Effective write quorum cannot be satisfied
cursus_cluster_min_insync_unsatisfied_partitions > 0
# Group commit no longer points into retained data
cursus:consumer_group_offset_out_of_range == 1
# The converged broker set has no single authoritative group coordinator
sum by (topic, group) (cursus_consumer_group_coordinator_up) != 1
# Sustained consumer backlog
max_over_time(cursus:consumer_group_lag[10m]) > 10000
# Storage writer backlog
min_over_time(cursus_storage_pending_writes[10m]) > 0
# Client responses could not be delivered
increase(cursus_broker_client_response_write_failures_total[5m]) > 0
# Compaction errors (inspect the bounded reason label)
increase(cursus_broker_log_compaction_runs_total{result="error"}[10m]) > 0
Tune lag and pending-write thresholds to topic throughput and retention. A single non-zero sample can be normal during bursts.
Health and metrics endpoints do not perform client authentication. Bind and publish these ports only on a trusted operations network, or place them behind an authenticated reverse proxy. Metrics include topic, consumer group, and broker identifiers.
Do not expose the metrics listener as a public application endpoint.
curl -fsS http://localhost:9080/live
curl -fsS http://localhost:9080/ready
curl -fsS http://localhost:9100/metrics | grep '^cursus_'