Cursus reports the actual lifecycle state held by the current consumer group coordinator. It does not decide whether a group is required.
A zero-member group is normal when, for example, a lazy consumer has not joined
an empty topic. The same value is an outage when an application contract says
the group must remain active. The cluster-config catalog owns that contract,
and Prometheus owns the comparison and alert duration. Broker /ready remains
independent of both values.
Publish application expectations as catalog metrics from cluster-config or its existing metrics bridge. This repository does not add those group entries.
cursus_required_consumer_group_active{topic="<topic>",group="<group>"} 1
cursus_required_consumer_group_min_members{topic="<topic>",group="<group>"} 1
Use active = 0 to retain a catalog entry without enforcing it. Set
min_members to the minimum number of concurrently joined members required by
that application’s availability contract.
Collapse the three broker targets only while exactly one broker claims
authority. This prevents an overlapping old coordinator from masking a 1 ->
0 transition through max:
max by (topic, group) (cursus_consumer_group_members)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
A required group might never have been registered, so a direct comparison would have no left-hand series and would not alert. Fill missing actual values from the catalog before comparing:
(
(
(
max by (topic, group) (cursus_consumer_group_members)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
)
or on (topic, group)
(0 * cursus_required_consumer_group_min_members)
)
< on (topic, group)
cursus_required_consumer_group_min_members
)
and on (topic, group)
(cursus_required_consumer_group_active == 1)
Apply an alert for duration longer than the configured consumer session
timeout, heartbeat check interval, normal coordinator movement, and scrape
interval. This prevents a planned rebalance from paging while still bounding
detection time.
Alert separately when a required group’s authoritative coordinator is absent or overlapping:
(
(
sum by (topic, group) (cursus_consumer_group_coordinator_up)
or on (topic, group)
(0 * cursus_required_consumer_group_active)
) != 1
)
and on (topic, group)
(cursus_required_consumer_group_active == 1)
Observation failures are per scraped broker, so aggregate counters with sum:
sum by (topic, group, reason) (
increase(cursus_consumer_group_observation_failures_total[10m])
) > 0
The reason label is bounded to coordinator_lookup, group_lookup, and
topic_lookup. Member IDs, client addresses, broker endpoints, and raw errors
are never metric labels. A failure-counter series is created on its first
failure rather than emitting three zero-valued reason series for every group.
Use an isolated canary <topic> and <group> whose catalog contract is
active = 1 and min_members = 1. Do not reuse a production group with other
members, because its aggregate would not reach zero.
Confirm all three brokers are ready and the authority sum is one:
min(cursus_broker_ready) == 1
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
Start one canary consumer. Wait until the authoritative member count is one and the state is stable:
(
max by (topic, group) (cursus_consumer_group_members) == 1
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
)
(
max by (topic, group) (cursus_consumer_group_state{state="stable"}) == 1
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
)
Stop the canary. Use graceful leave for the fast path, then repeat once with
an abrupt stop to exercise heartbeat expiry. Wait for member count zero and
state empty:
(
max by (topic, group) (cursus_consumer_group_members) == 0
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
)
(
max by (topic, group) (cursus_consumer_group_state{state="empty"}) == 1
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
)
Verify the required-member alert reaches firing after its configured
for duration. During this interval, verify every available broker remains
ready; consumer absence must not change /ready or cursus_broker_ready.
Restart the canary with the same catalog contract. Wait for member count one and stable state, then verify the alert resolves.
Review both timestamps. Activity should advance after the new join or heartbeat, and rebalance time should advance on the transitions to zero and back to one:
max by (topic, group) (
cursus_consumer_group_last_activity_timestamp_seconds
)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
max by (topic, group) (
cursus_consumer_group_last_rebalance_timestamp_seconds
)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
This procedure validates actual Cursus state and the external expectation join without adding application-specific group names to the broker configuration.
consumer_snapshot_busy_total counts nonblocking FSM snapshot refusals.
consumer_snapshot_busy_consecutive counts those refusals since a successful
FSM snapshot capture, and resets on process restart. Successful capture does
not imply successful snapshot persistence; also monitor Raft persistence errors
and log disk usage. For example, add this rule to the broker Prometheus rules:
- alert: CursusConsumerSnapshotBusy
expr: consumer_snapshot_busy_consecutive >= 3
for: 5m
labels:
severity: warning
annotations:
summary: Consumer state repeatedly prevents Raft snapshot capture
description: Check offset-write latency, snapshot errors and Raft log disk growth on this broker.
Tune the threshold and duration to the configured Raft snapshot interval. The
counter can show shorter bursts using increase(consumer_snapshot_busy_total[15m]).
Sustained contention delays compaction and increases retained recovery work.
Investigate slow durable consumer writes and reduce excessive offset commit
frequency if needed. Retrying with a wait inside FSM Snapshot would block the
same goroutine needed to apply the pending write, so retries use Raft’s ordinary
snapshot scheduling.