cursus

Broker Observability

Contract

Cursus exposes health and Prometheus endpoints on HTTP listeners separate from the client protocol listener.

Default port Endpoint Response Purpose
9080 GET /live JSON Process liveness
9080 GET /ready JSON Broker and dependency readiness
9100 GET /metrics Prometheus/OpenMetrics Broker metrics when the exporter is enabled

Only GET and HEAD are accepted by health endpoints. Other methods return 405 Method Not Allowed.

The HTTP listeners bind before broker readiness is published. A health or metrics bind failure fails broker startup instead of being logged as a background-only failure.

Liveness And Readiness

/live returns 200 while the health HTTP server can execute requests. It does not check topic storage, consumer groups, or cluster quorum.

{"status":"live"}

/ready returns 200 only after the client listener, command handler, worker pool, and enabled HTTP services have initialized and all dynamic checks pass. Consumer group member counts are not readiness checks. A registered group with zero members leaves the broker ready; only broker-owned consumer metadata recovery participates in readiness.

Standalone response:

{"status":"ready","checks":{"consumer_metadata":"ok","storage":"ok","topic_metadata":"ok"}}

In distributed mode, readiness also requires a resolvable cluster leader, active local broker registration, successful node-local topic, replica, and transaction-state materialization, and a topology that can safely admit writes. An offline leader, incomplete assignment, unsatisfied effective minimum ISR, or active replica-recovery marker makes every broker unready. Ordinary under-replication alone does not fail readiness when the leader is active, the assignment is complete, and the remaining ISR still satisfies the effective minimum. CLUSTER_STATUS remains stricter: healthy=true requires a full active ISR and no pending recovery or materialization.

During replica-gap recovery, CLUSTER_STATUS reports recovery_pending_partitions > 0; the affected partition lists recovery_replicas and the replica_recovery_pending reason. New writes to that partition fail closed until bounded catch-up and prefix verification re-admit the quarantined replica. Use these fields with /ready rather than treating process liveness or an otherwise sufficient ISR as recovery success.

{
  "status": "not_ready",
  "checks": {
    "cluster_leader": "no cluster leader available",
    "storage": "ok"
  }
}

Only /live and /ready are registered. /health and / return HTTP 404.

Recommended Kubernetes probes:

livenessProbe:
  httpGet:
    path: /live
    port: 9080
readinessProbe:
  httpGet:
    path: /ready
    port: 9080

Do not use /live for traffic admission. It intentionally remains successful when the broker cannot safely serve client work.

Metrics Semantics

Runtime gauges come from expiring background snapshots. Each completed sample replaces the preceding series, including labels for deleted topics, departed groups and old partition leaders.

Broker Traffic

Metric Type Meaning
cursus_broker_client_connections_total Counter Accepted client TCP connections
cursus_broker_client_connections_active Gauge Connections currently handled
cursus_broker_commands_total{command,result} Counter Completed text command dispatches
cursus_broker_command_duration_seconds{command} Histogram Command dispatch latency
cursus_broker_command_errors_total{command,code} Counter Wire errors by bounded command and error code
cursus_broker_client_response_write_failures_total{reason} Counter Client responses that could not be delivered, with a bounded failure reason
cursus_broker_publish_acknowledgements_total{ack_mode,result} Counter Publish requests by normalized acknowledgement mode and bounded result
cursus_broker_async_replication_failures_total{topic,error_class} Counter Follower failures after an acks=1 leader acknowledgement
cursus_broker_replication_retries_total{topic,ack_mode,error_class} Counter Retried partition replication attempts, including pending duplicate commit barriers
cursus_broker_log_compaction_runs_total{result,reason} Counter Completed, skipped, or failed compaction passes with bounded reasons
broker_messages_processed_total Counter Messages accepted by the topic manager
broker_message_latency_seconds Histogram Topic manager publish latency
broker_seqnum_gap_total{topic,partition,producer_id} Counter Detected producer sequence gaps
broker_seqnum_duplicate_total{topic,partition} Counter Detected duplicate producer sequences

Command duration ends when a streaming command is accepted. It does not include the lifetime or payload transfer time of a stream.

Topic And Storage State

Metric Meaning
cursus_broker_ready Current readiness result (1 or 0)
cursus_broker_topics Topics loaded on this broker
cursus_topic_metadata_restored_topics Topics restored from the durable standalone manifest during startup
cursus_broker_partitions Partitions loaded on this broker
cursus_partition_log_start_offset{topic,partition} Earliest retained offset
cursus_partition_log_end_offset{topic,partition} Next allocated offset
cursus_partition_high_watermark{topic,partition} Next offset visible to committed readers
cursus_streams_active Registered streaming consumers
cursus_storage_handlers Open partition storage handlers
cursus_storage_segments Segment files represented by open handlers
cursus_storage_bytes Segment and offset-index bytes represented by open handlers
cursus_storage_pending_writes Messages queued for disk writes
cursus_storage_active_readers Current segment readers
cursus_storage_stat_failures 0 in a valid runtime observation; failed measurements invalidate the snapshot and omit this series
cursus_storage_segment_cache_entries Memory-mapped segment readers retained by open handlers
cursus_storage_segment_cache_hits Cache hits accumulated by open handlers
cursus_storage_segment_cache_misses Cache misses accumulated by open handlers
cursus_storage_segment_cache_evictions Cache evictions accumulated by open handlers
cursus_wire_protocol_failures_total{reason} Rejected Wire v2 frames by bounded protocol reason
cursus_wire_decompression_rejections_total{reason} Rejected compressed payloads by bounded reason

cursus_storage_bytes covers open topic handlers. It is not filesystem capacity or free-space telemetry; collect those values with the node or container runtime exporter.

Runtime metrics use background snapshots. Initial, failed, expired or over-budget observations omit their values; missing series must not be interpreted as zero. Use cursus_observation_valid{component=~"readiness|runtime|process"}, cursus_observation_stale, cursus_observation_timed_out, cursus_observation_in_progress and cursus_observation_last_failed to distinguish observation health from broker health. Completion timestamp, age and last-failure series are absent before the first completed sample. Both Helm charts alert on invalid observations lasting one minute with CursusObservationUnavailable.

A successful capacity measurement below the configured free-space reserve still exports cursus_storage_filesystem_headroom_ready=0 and the measured free/total bytes. This preserves CursusStorageDiskHeadroomLow without treating a failed filesystem query as a measured zero. See snapshot lifecycle and defaults.

Consumer Metadata Recovery

Metric Meaning
cursus_consumer_metadata_recovery_ready{phase} 1 only after durable group/offset replay completes without error
cursus_consumer_metadata_restored_groups Groups materialized during startup
cursus_consumer_metadata_restored_offsets Committed (group,topic,partition) next-offset keys restored
cursus_consumer_metadata_replayed_records Active internal log records scanned
cursus_consumer_metadata_orphan_records Records fenced or superseded by authoritative lifecycle/revision state
cursus_consumer_metadata_corrupt_records Malformed or inconsistent records that stopped recovery

In diagnostics-only mode, /ready includes the retained consumer_metadata failure and cursus_consumer_metadata_recovery_ready remains 0; the client command listener is not opened. Use the restored counts together with the offline standalone storage recovery inventory.

Consumer Groups

Metric Type / unit Meaning
cursus_consumer_group_members{topic,group} Gauge / members Current member count, emitted only by the authoritative coordinator
cursus_consumer_group_state{topic,group,state} Gauge / boolean One-hot authoritative state: stable or empty
cursus_consumer_group_coordinator_up{topic,group} Gauge / boolean 1 only on the broker that successfully resolves itself as current coordinator; known groups emit 0 or 1, while unavailable group observation can omit the series
cursus_consumer_group_last_activity_timestamp_seconds{topic,group} Gauge / Unix seconds Latest accepted heartbeat or group lifecycle activity known by the authoritative coordinator; 0 means unknown, including after standalone durable recovery
cursus_consumer_group_last_rebalance_timestamp_seconds{topic,group} Gauge / Unix seconds Latest completed membership rebalance known by the authoritative coordinator; 0 means unavailable or no rebalance has completed since recovery
cursus_consumer_group_observation_failures_total{topic,group,reason} Counter / failures Per-broker observation failures; reason is one of coordinator_lookup, group_lookup, or topic_lookup, and a series appears only after its first failure
cursus_consumer_group_generation{group,topic} Gauge / generation Authoritative coordinator’s current group generation
cursus_consumer_group_assigned_partitions{group,topic} Gauge / partitions Authoritative coordinator’s assignments held by active members
cursus_consumer_group_committed_offset{group,topic,partition} Gauge / offsets Known durable next offset on the authoritative coordinator, with an observed local partition
cursus_consumer_group_lag{group,topic,partition} Gauge / messages Authoritative max(HWM - committedNextOffset, 0); absent when an input is unknown
cursus_consumer_group_offset_out_of_range{group,topic,partition} Gauge / boolean Authoritative commit is below local log start or above local HWM; absent when an input is unknown
cursus_storage_filesystem_free_bytes Gauge / bytes Space currently available on the filesystem containing log_dir
cursus_storage_filesystem_total_bytes Gauge / bytes Total capacity of the filesystem containing log_dir
cursus_storage_filesystem_headroom_ready Gauge / boolean 1 while the configured byte and percentage free-space reserve permits new writes
cursus_broker_requests_inflight Gauge / requests Decoded requests queued or processed across the client and internal admission pools
cursus_broker_request_bytes_inflight Gauge / bytes Reserved encoded and decoded request payload memory
cursus_broker_request_admission_waiters Gauge / connections Connections blocked before payload allocation by their listener’s request budget
cursus_broker_request_admission_rejections_total{reason} Counter / requests Admission failures caused by bytes or connection context cancellation

In standalone mode the local coordinator is authoritative. In distributed mode, replicated membership can remain present on a broker that no longer owns the group, while heartbeat activity is coordinator-local. Such a broker emits coordinator_up = 0 and omits the authoritative lifecycle and offset gauges. It does not present its retained snapshot as healthy.

For a converged three-broker scrape, exactly one target emits authoritative lifecycle gauges and the sum of coordinator_up is one. Gate each max by that exact authority count; an unguarded max can retain an old positive value during an overlapping coordinator view. Use sum for the per-broker failure counters:

max by (topic, group) (cursus_consumer_group_members)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1

max by (topic, group, state) (cursus_consumer_group_state)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1

max by (topic, group) (
  cursus_consumer_group_last_activity_timestamp_seconds
)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1

max by (topic, group) (
  cursus_consumer_group_last_rebalance_timestamp_seconds
)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1

sum by (topic, group) (cursus_consumer_group_coordinator_up)

sum by (topic, group, reason) (
  rate(cursus_consumer_group_observation_failures_total[5m])
)

sum of an authoritative lifecycle gauge is also exact after coordinator ring convergence because only one broker emits it. A coordinator_up sum of zero means no scraped target currently claims authority; a value above one reveals an overlapping ring view. Both cases remove the gated lifecycle value instead of allowing a stale value to mask a member-count decrease.

Heartbeat and rebalance timestamps are runtime lifecycle observations, not a durable heartbeat journal. Cluster snapshots can carry them across coordinator materialization, but standalone metadata recovery intentionally restores no members and reports 0 until a new lifecycle event occurs. Persisting every heartbeat would add storage write amplification to the consumer hot path.

An unknown group has no Cursus series. Cursus does not fabricate a zero-valued group because it does not know which application groups are required. Use an external cluster-config catalog to fill that missing actual state with zero. See the consumer lifecycle alert runbook for the catalog contract, alert rules, and a 1 -> 0 -> 1 canary procedure.

Lag uses the high watermark rather than the local log end so uncommitted replica tail data is not reported as consumable work. Both exact-topic and wildcard groups expose only known stored offsets with available local partition data. No stored commit means no offset/lag/out-of-range sample; a stored offset of zero remains a valid sample. Fetch/reset defaults do not imply a durable commit.

Consumer offset authority and query migration

Names and labels are unchanged, but series from non-coordinators and fabricated default-zero commits are removed. During rolling upgrades, filter the old brokers’ metrics by authority on the same scrape target before aggregation. For example, record the following expression as cursus:consumer_group_lag (use the same expression with the out-of-range or committed-offset metric):

max by (cluster, group, topic, partition) (
  cursus_consumer_group_lag
  and on (cluster, job, instance, group)
    (cursus_consumer_group_coordinator_up == 1)
  and on (cluster, job, instance)
    (up{job="cursus"} == 1)
  and on (cluster, job, instance)
    (cursus_observation_valid{component="runtime"} == 1)
)
and on (cluster, group)
  (sum by (cluster, group) (cursus_consumer_group_coordinator_up) == 1)

Adapt cluster, job and instance to labels that uniquely identify your cluster and broker scrape target; preserve tenant/namespace labels if required. The group is broker-unique. Do not join on topic for wildcard or multi-topic subscriptions: coordinator labels describe the subscription, whereas offset labels describe the concrete topic. An exact authority count rejects overlapping owner views. Samples across brokers still have sampling/scraping delay; this is not an atomic cluster snapshot.

Alert separately on sum by (cluster,group) (...coordinator_up) != 1 and on absent expected groups. A concrete missing-owner check for a known group is:

absent(cursus_consumer_group_coordinator_up{cluster="prod", group="workers"})
or
(sum(cursus_consumer_group_coordinator_up{cluster="prod", group="workers"}) != 1)

Also detect missing expected offset partitions with absent() or an external expected-group/partition inventory. coordinator_up=1 alone does not prove that every partition/commit is observable. Do not use or vector(0) for missing lag or out-of-range data. Monitor up, runtime observation validity and consumer metadata recovery separately. A missing group is not evidence that it is healthy.

Apply range functions to the authority-filtered recording rule; a direct max_over_time over legacy per-broker series retains old false positives for the whole window. These are query migration examples, not deployed alert changes. See the design and regression comparison for handoff/recovery behavior and the limitations of local partition observations.

Wire v2 exposes only cursus_consumer_group_lag; the former broker_consumer_lag alias is not registered.

The in-process Go SDK exposes separate counters for physical anomalies and expected compacted holes. cursus_consumer_offset_gap_total{topic,group} increments only when a non-compacted topic jumps forward unexpectedly. cursus_consumer_compacted_offsets_skipped_total{topic,group} counts logical offsets removed by a cleanup policy that includes compact.

Cluster State

Metric Meaning
cursus_distribution_enabled Distributed mode is enabled
cursus_cluster_brokers Brokers in replicated metadata
cursus_cluster_has_leader This broker resolves a cluster leader
cursus_cluster_is_leader This broker is the current cluster leader
cursus_cluster_offline_partitions Partitions without an active assigned leader
cursus_cluster_under_replicated_partitions Partitions where active ISR size is below the topic replication factor
cursus_cluster_assignment_deficient_partitions Partitions whose distinct assignment does not match the topic replication factor
cursus_cluster_inactive_replica_partitions Partitions assigned to an inactive or unknown broker
cursus_cluster_inactive_replicas Replica assignments that reference inactive or unknown brokers
cursus_cluster_min_insync_unsatisfied_partitions Partitions whose active ISR size is below effective min.insync.replicas
cursus_cluster_partition_replicas{topic,partition} Actual assigned replica count
cursus_cluster_partition_expected_replicas{topic,partition} Replica count declared by the topic definition
cursus_cluster_partition_active_replicas{topic,partition} Assigned replicas backed by active brokers
cursus_cluster_partition_in_sync_replicas{topic,partition} Current active ISR size
cursus_cluster_partition_min_insync_replicas{topic,partition} Effective min.insync.replicas
cursus_cluster_partition_topology_healthy{topic,partition} Assignment, leader, and ISR satisfy the durable topic contract
cursus_cluster_partition_leader_epoch{topic,partition} Current leader epoch
cursus_cluster_partition_leader{topic,partition,broker_id} Current leader identity (1)
cursus_cluster_isr_catchup_proofs_total{outcome,reason} ISR catch-up proofs accepted or rejected by bounded reason
cluster_replication_lag_seconds{topic,partition,broker} Successful follower acknowledgement latency

cluster_replication_lag_seconds is the only operation-time cluster metric. Use the cursus_cluster_* scrape-time metrics for current topology and health.

Transaction State

Metric Meaning
cursus_transaction_recovery_ready Transaction state recovery completed before serving
cursus_transactions{state} Transactions retained by coordinator state
cursus_transactions_expired Expired transaction identities awaiting replacement or compaction
cursus_transaction_oldest_active_seconds Age of the oldest open or committing transaction

Use these gauges with cursus_broker_command_errors_total{command,code} and the transaction alert runbook.

Configuration

health_check_port: 9080
enable_exporter: true
exporter_port: 9100

Equivalent environment variables are HEALTH_CHECK_PORT, ENABLE_EXPORTER, and EXPORTER_PORT.

Prometheus example:

scrape_configs:
  - job_name: cursus
    scrape_interval: 15s
    static_configs:
      - targets: ["broker-1:9100", "broker-2:9100", "broker-3:9100"]

The standalone and three-member Helm charts can create a metrics Service, ServiceMonitor, and PrometheusRule baseline. The Prometheus Operator CRDs must already be installed. Add any labels required by the operator’s selectors; for kube-prometheus-stack this is commonly:

monitoring:
  enabled: true
  labels:
    release: kube-prometheus-stack

monitoring.serviceMonitor.enabled and monitoring.prometheusRule.enabled control the two CRD-backed resources independently. Tune consumerLag, transactionOldestSeconds, and the alert durations to the application’s throughput and transaction timeout before enabling paging.

monitoring.grafanaDashboard.enabled installs the Cursus production overview dashboard as a ConfigMap. The default grafana_dashboard: "1" label is compatible with the common Grafana sidecar selector and can be replaced through monitoring.grafanaDashboard.labels. The dashboard uses a Prometheus datasource variable and a namespace variable; select the datasource that scrapes the Cursus ServiceMonitor after import. See the production incident runbook for the response attached to each panel and alert.

Alert Baseline

# Scrape target unavailable
up{job="cursus"} == 0

# Process is reachable but cannot serve client work
cursus_broker_ready == 0

# Writes are blocked before the filesystem reserve is consumed
cursus_storage_filesystem_headroom_ready == 0

# No cluster leader
cursus_distribution_enabled == 1 and cursus_cluster_has_leader == 0

# Replication safety degraded
cursus_cluster_under_replicated_partitions > 0

# Durable definition and actual assignment disagree
cursus_cluster_assignment_deficient_partitions > 0

# Effective write quorum cannot be satisfied
cursus_cluster_min_insync_unsatisfied_partitions > 0

# Group commit no longer points into retained data
cursus:consumer_group_offset_out_of_range == 1

# The converged broker set has no single authoritative group coordinator
sum by (topic, group) (cursus_consumer_group_coordinator_up) != 1

# Sustained consumer backlog
max_over_time(cursus:consumer_group_lag[10m]) > 10000

# Storage writer backlog
min_over_time(cursus_storage_pending_writes[10m]) > 0

# Client responses could not be delivered
increase(cursus_broker_client_response_write_failures_total[5m]) > 0

# Compaction errors (inspect the bounded reason label)
increase(cursus_broker_log_compaction_runs_total{result="error"}[10m]) > 0

Tune lag and pending-write thresholds to topic throughput and retention. A single non-zero sample can be normal during bursts.

Security Boundary

Health and metrics endpoints do not perform client authentication. Bind and publish these ports only on a trusted operations network, or place them behind an authenticated reverse proxy. Metrics include topic, consumer group, and broker identifiers.

Do not expose the metrics listener as a public application endpoint.

Validation

curl -fsS http://localhost:9080/live
curl -fsS http://localhost:9080/ready
curl -fsS http://localhost:9100/metrics | grep '^cursus_'