cursus

Go SDK consumer coordinator observability

Scope and current routing behavior

The Go consumer resolves a group coordinator with FIND_COORDINATOR through ConsumerClient.ConnectWithFailoverContext. That bootstrap connection may be made to any configured broker. A successful response supplies the broker address used by group commands; the bootstrap or Kubernetes ClusterIP is not the coordinator route after successful discovery.

The legacy consumer uses the coordinator route for JOIN_GROUP, SYNC_GROUP, HEARTBEAT, FETCH_OFFSET, COMMIT_OFFSET, BATCH_COMMIT, and LEAVE_GROUP. Heartbeat and batched commit each reuse their own connection; the remaining legacy paths normally open a bounded request connection. The cooperative consumer also keeps separate membership and operation connections. GROUP_STATUS is not emitted by the production Go consumer. It appears in broker-facing integration fixtures, so a broker-side GROUP_STATUS NOT_COORDINATOR series cannot by itself be attributed to this SDK.

Before this change, NOT_COORDINATOR with host and port overwrote Consumer.coordinatorAddr, including when the returned address was identical to the stale address. An address-less response left the stale cache untouched. The surrounding command paths had different bounds: offset fetch used five attempts, group join used 30 attempts with backoff, batch commit had its existing bounded transport retry and commit queue, and heartbeat retried on its normal periodic lifecycle. This made a same-address or address-less redirect capable of recurring without evidence that the cache ever converged.

The cache contract is now:

CoordinatorAddr remains an optional initial/pinned deployment override, but a broker redirect can replace it. Production Kubernetes brokers advertise their stable StatefulSet DNS names, so a ClusterIP is suitable for bootstrap only. If a deployment advertises a load-balanced ClusterIP as every broker’s identity, direct coordinator routing is impossible and repeated redirects are expected until the advertised-address configuration is corrected.

Apache Kafka comparison

The comparison baseline is Apache Kafka 4.1.0, using the versioned AbstractCoordinator, ConsumerCoordinator, and the 4.1 protocol error table. Kafka treats NOT_COORDINATOR as retriable, marks the coordinator unknown for heartbeat, sync, offset fetch, and offset commit responses, and performs rediscovery inside the caller’s timer and retry backoff.

Cursus adopts the safety property that an invalid dynamically discovered coordinator is not kept as a valid cache entry and that retries remain deadline/budget bounded. It differs by retaining Cursus Wire v2’s redirect-address optimization: a different broker address in the structured error can refresh the cache without a second lookup. It also keeps an explicit CoordinatorAddr deployment pin when an error has no replacement address, because Kafka’s broker identity is directly reachable while a Cursus host-port mapping can intentionally differ from the advertised address. A usable structured redirect can still replace the pin. No wire protocol or public method signature changed.

Command recovery work is bounded by the existing command budget. A usable redirect costs one failed command response, one replacement connection, and one command retry. A missing redirect on a dynamic route, or a same-address redirect, additionally costs one framed FIND_COORDINATOR request/response on the next existing attempt. A missing redirect on an explicit pin skips that lookup and reuses the pin on the caller’s next existing attempt or heartbeat tick. The exact byte count depends on group-name length, Wire v2 framing, authentication, and compression; the change does not add background polling, payload reads, or a new retry loop. Recovery bytes therefore grow with at most that single bounded lookup per invalidated dynamic attempt, not with topic data volume.

Metrics

All metrics use the SDK-owned Prometheus registry returned by MetricsHandler. Prometheus target labels such as job, service, namespace, and pod identify the application deployment and are intentionally not duplicated as SDK labels.

Metric Type Labels Meaning
cursus_consumer_coordinator_commands_total Counter command, client_role, route_source, result Completed coordinator command attempts, including success and failure. This is the denominator for redirect ratios.
cursus_consumer_coordinator_redirects_total Counter command, client_role, route_source Structured NOT_COORDINATOR responses.
cursus_consumer_coordinator_resolution_total Counter result, reason, client_role Coordinator lookup or classified reconnect outcomes.
cursus_consumer_coordinator_cache_refresh_total Counter result, reason, client_role Cache refresh outcomes. A missing redirect on a dynamic route, or a same-address redirect, is counted when its subsequent lookup succeeds or fails. An address-less configured-pin retry is not a refresh.
cursus_consumer_coordinator_retry_total Counter command, result, client_role Outcome of the recovery command sent on a route learned from a redirect. After the first success, later commands on that persistent connection use the normal cache route and do not increment this counter.
cursus_consumer_coordinator_resolution_duration_seconds Histogram result, reason, client_role Lookup or reconnect latency.

Bounded command values are join_group, sync_group, leave_group, heartbeat, fetch_offset, commit_offset, batch_commit, group_status, and other. route_source is one of bootstrap, cache, retry, or reconnect. result is success or failure; reason is initial, redirect, reconnect, or cache_miss.

ConsumerConfig.ClientRole is optional and defaults to consumer. Its public bounded values are consumer, cooperative_consumer, processor, saga, and observer; configuration rejects other values. Use the scrape target’s job/service label for the application and client_role only when one process runs multiple bounded roles.

Remote IP/port, broker address, client ID, request ID, payload, token, group, member, topic, and retry number are not Prometheus labels. There is no per-client cache-valid gauge because multiple clients sharing one process and role would overwrite one another; the refresh, retry, and command counters retain correct aggregate semantics.

PromQL

Application and role redirect rate:

sum by (job, service, client_role) (
  rate(cursus_consumer_coordinator_redirects_total[5m])
)

Command redirect rate:

sum by (job, service, client_role, command) (
  rate(cursus_consumer_coordinator_redirects_total[5m])
)

Redirect ratio over all completed coordinator commands:

sum by (job, service, client_role) (
  rate(cursus_consumer_coordinator_redirects_total[5m])
)
/
clamp_min(
  sum by (job, service, client_role) (
    rate(cursus_consumer_coordinator_commands_total[5m])
  ),
  1e-9
)

Share of cache-routed commands that redirect:

sum by (job, service, client_role) (
  rate(cursus_consumer_coordinator_redirects_total{route_source="cache"}[5m])
)
/
clamp_min(
  sum by (job, service, client_role) (
    rate(cursus_consumer_coordinator_commands_total{route_source="cache"}[5m])
  ),
  1e-9
)

Redirect-triggered resolution success ratio:

sum by (job, service, client_role) (
  rate(cursus_consumer_coordinator_resolution_total{reason="redirect",result="success"}[5m])
)
/
clamp_min(
  sum by (job, service, client_role) (
    rate(cursus_consumer_coordinator_resolution_total{reason="redirect"}[5m])
  ),
  1e-9
)

Outcome of commands retried after a redirect:

sum by (job, service, client_role, command, result) (
  rate(cursus_consumer_coordinator_retry_total[5m])
)

Reconnect activity and redirects after reconnect:

sum by (job, service, client_role, result) (
  rate(cursus_consumer_coordinator_resolution_total{reason="reconnect"}[5m])
)

sum by (job, service, client_role, command) (
  rate(cursus_consumer_coordinator_redirects_total{route_source="reconnect"}[5m])
)

An alert should combine a sustained ratio, a minimum coordinator command rate, and a for duration. Do not ship an arbitrary absolute redirects-per-second default. Measure the deployment baseline first, then substitute its reviewed ratio_threshold, minimum_request_rate, and duration into both conditions:

(
  sum by (job, service, client_role) (rate(cursus_consumer_coordinator_redirects_total[5m]))
  /
  clamp_min(sum by (job, service, client_role) (rate(cursus_consumer_coordinator_commands_total[5m])), 1e-9)
) > ratio_threshold
and
sum by (job, service, client_role) (rate(cursus_consumer_coordinator_commands_total[5m])) > minimum_request_rate

Interpretation and structured logs

A redirect is a routing correction, not proof that publish, consume, or commit ultimately failed. A small burst during coordinator failover can be healthy if the cache-refresh or resolution success counter increases once, retry success follows, and the redirect ratio returns to baseline. Redirects that continue on route_source="cache", repeated retry failures, or repeated same-address rediscovery after traffic has stabilized indicate stale or unusable advertised routing.

The previously observed aggregate of about 4 redirects/s cannot be classified as normal initial discovery from the broker counter alone. Successful Go SDK startup resolves before its first group command, and the production Go SDK does not send GROUP_STATUS. A sustained warm-state rate across FETCH_OFFSET, GROUP_STATUS, and HEARTBEAT therefore points to multiple applications, an older/custom client, a pinned stale address, or an advertised address that load-balances back to non-coordinators. The new SDK metrics can prove or exclude this Go process by job/service, client_role, command, and route source.

Rate-limited key/value logs supplement metrics with event, command, client_role, route_source, reason, result, group, topic, member, old_coordinator, new_coordinator, same_coordinator, retry_attempt, and retry_limit. These fields are for targeted diagnosis only. Logs never include credentials, tokens, payloads, request IDs, or TLS material.