Cursus reports the actual lifecycle state held by the current consumer group coordinator. It does not decide whether a group is required.
A zero-member group is normal when, for example, a lazy consumer has not joined
an empty topic. The same value is an outage when an application contract says
the group must remain active. The cluster-config catalog owns that contract,
and Prometheus owns the comparison and alert duration. Broker /ready remains
independent of both values.
Publish application expectations as catalog metrics from cluster-config or its existing metrics bridge. This repository does not add those group entries.
cursus_required_consumer_group_active{topic="<topic>",group="<group>"} 1
cursus_required_consumer_group_min_members{topic="<topic>",group="<group>"} 1
Use active = 0 to retain a catalog entry without enforcing it. Set
min_members to the minimum number of concurrently joined members required by
that application’s availability contract.
Collapse the three broker targets only while exactly one broker claims
authority. This prevents an overlapping old coordinator from masking a 1 ->
0 transition through max:
max by (topic, group) (cursus_consumer_group_members)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
A required group might never have been registered, so a direct comparison would have no left-hand series and would not alert. Fill missing actual values from the catalog before comparing:
(
(
(
max by (topic, group) (cursus_consumer_group_members)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
)
or on (topic, group)
(0 * cursus_required_consumer_group_min_members)
)
< on (topic, group)
cursus_required_consumer_group_min_members
)
and on (topic, group)
(cursus_required_consumer_group_active == 1)
Apply an alert for duration longer than the configured consumer session
timeout, heartbeat check interval, normal coordinator movement, and scrape
interval. This prevents a planned rebalance from paging while still bounding
detection time.
Alert separately when a required group’s authoritative coordinator is absent or overlapping:
(
(
sum by (topic, group) (cursus_consumer_group_coordinator_up)
or on (topic, group)
(0 * cursus_required_consumer_group_active)
) != 1
)
and on (topic, group)
(cursus_required_consumer_group_active == 1)
Observation failures are per scraped broker, so aggregate counters with sum:
sum by (topic, group, reason) (
increase(cursus_consumer_group_observation_failures_total[10m])
) > 0
The reason label is bounded to coordinator_lookup, group_lookup, and
topic_lookup. Member IDs, client addresses, broker endpoints, and raw errors
are never metric labels. A failure-counter series is created on its first
failure rather than emitting three zero-valued reason series for every group.
Use an isolated canary <topic> and <group> whose catalog contract is
active = 1 and min_members = 1. Do not reuse a production group with other
members, because its aggregate would not reach zero.
Confirm all three brokers are ready and the authority sum is one:
min(cursus_broker_ready) == 1
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
Start one canary consumer. Wait until the authoritative member count is one and the state is stable:
(
max by (topic, group) (cursus_consumer_group_members) == 1
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
)
(
max by (topic, group) (cursus_consumer_group_state{state="stable"}) == 1
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
)
Stop the canary. Use graceful leave for the fast path, then repeat once with
an abrupt stop to exercise heartbeat expiry. Wait for member count zero and
state empty:
(
max by (topic, group) (cursus_consumer_group_members) == 0
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
)
(
max by (topic, group) (cursus_consumer_group_state{state="empty"}) == 1
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
)
Verify the required-member alert reaches firing after its configured
for duration. During this interval, verify every available broker remains
ready; consumer absence must not change /ready or cursus_broker_ready.
Restart the canary with the same catalog contract. Wait for member count one and stable state, then verify the alert resolves.
Review both timestamps. Activity should advance after the new join or heartbeat, and rebalance time should advance on the transitions to zero and back to one:
max by (topic, group) (
cursus_consumer_group_last_activity_timestamp_seconds
)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
max by (topic, group) (
cursus_consumer_group_last_rebalance_timestamp_seconds
)
and on (topic, group)
sum by (topic, group) (cursus_consumer_group_coordinator_up) == 1
This procedure validates actual Cursus state and the external expectation join without adding application-specific group names to the broker configuration.