cursus

Truncate cleanup and group re-registration

Cause

Baseline: a49138e7922aa66b14cb4482a8b084e5b9c1b8cd.

The cleanup worker captures current local groups, not groups at the truncate commit. It accepts only a tombstone newer than the captured registration and all observed live records. A re-registration therefore replaces the proof needed by an old truncate. An owner that already materialized that truncate has no pending operation to generate another tombstone. Generating one would destroy new state.

Topic lifecycle epochs, group registration epochs, and membership generations are independent domains. Registration epochs order one group’s registrations and deletions; membership generations do not order topic operations. Ordinary metadata replay also copies historical membership into a new registration when no matching lifecycle snapshot exists; the baseline applies its epoch check only to strict replay. This can explain a local generation 14 versus an owner’s fresh generation 0, but production history must be inspected to attribute those exact values conclusively.

Kafka comparison

The Cursus boundary captures the authoritative committed end of every consumer-metadata partition in the serialized Raft truncate application. Bind the boundary to the topic incarnation/lifecycle and internal topic identity. Persist it in snapshots, never infer it from wall-clock timestamps or current HWM. During the bounded scan, classify registration records on either side of this boundary. Preserve a later registration only with a complete validated prefix and evidence that any previous live group was deleted before the new registration. The final local cleanup check must use the same proof. Missing boundary/history must not authorize a new deletion or exempt a live registration.

The snapshot addition is optional recovery evidence. Old snapshots that already discarded the truncate command cannot reconstruct its historical boundary from the current HWM. They remain fail closed; retained Raft replay can reconstruct it. No production state mutation or destructive workaround is part of this work.

Implemented behavior

The scanner retains the highest live group epoch before the cut, the latest validated deletion epoch, and the latest registration’s position relative to the cut. A later same-name registration survives only when a committed scoped tombstone proves the previous live epoch was deleted. A first registration with no preceding deletion remains fail closed: the FSM has no deterministic proof that historical Raft-only groups were absent. Local compatibility state or lock contention must never become persisted Raft evidence. Timestamp values never order the operation. Shared and pattern subscriptions retain their existing fail-closed scope protection; registration initial offsets do not authorize preservation.

An already superseded local group can be discarded using its own tombstone; normal ownership recovery installs the later group from unchanged durable records. No tombstone is written for a protected registration. Only a recovered, fenced owner may write a deletion for a group proven live before the cut.

The final Raft-apply check consumes the same validated operation-bound proof and nonblockingly checks the current partition identity, generation, committed HWM and snapshot status without reading disk. Rollback or replacement invalidates it. The check also holds consumer recovery admission nonblockingly: an empty overlay during snapshot installation cannot establish that all old groups were removed. Snapshot restore and definition updates leave truncation pending until that check succeeds. Membership generations are no longer copied across registration epochs during ordinary replay. The normal same-epoch compatibility path remains. If a failed handler open leaves no local topic during Raft replay, retry restores the authoritative definition before performing the same cleanup check. It also restores the authoritative HWM fence, so missing new-generation data must be recovered by replication before readiness can succeed. Startup replay retains its existing finalization phase; a background retry finalizes its staged watermark immediately instead of waiting for that already completed callback. Replica eligibility still checks local data against the authoritative committed watermark.

Checkpoint version 2 binds the cut and accumulated evidence to the current targets and durable log generation. Version 1, damaged or mismatched checkpoints are discarded and reread. Record/byte/time limits, cancellation and the existing scheduler/backoff remain. Completed batches retain per-registration proofs so more than 64 later groups cannot starve subsequent targets. A restart may repeat completed batches; cursors for the active matching batch remain reusable. These proofs are disposable and never replace the log or coordinator startup replay.

Compatibility and deployment conditions

Reproduction evidence

The deployed commit reproduces a fully scanned three-record history ending at Next=End=3, with epoch 3 registration evidence, while repeatedly returning waiting for a newer committed group tombstone. A separate baseline regression fails because a fresh registration inherits membership generation 14; the fixed replay returns generation 0 with no inherited members.

The isolated three-broker snapshot scenario passes with a real new-generation message, offset 1 and the same active generation/member after Ready/full ISR recovery. The fixture holds the target’s handler open across restart until consumer metadata recovery is ready, then releases local materialization. This forces cleanup to encounter the recovered registration instead of completing against a temporarily empty startup overlay. The fixed run took 119.48 seconds; follower cleanup scanned 8 records / 3,931 frame bytes in 2 reads after restart. Other recovery work included 5,918 catch-up transfer bytes and 4,928 retained-scan read bytes / 10 frames. These are small-fixture counters, not estimates for the reported 286,710,274-byte production scan.

The same snapshot sequence on the baseline fails: restore completes topic materialization without revisiting consumer cleanup. At the failed assertion, the replica is still at LEO/HWM 0 with committed HWM 1. The paired retained-Raft baseline also fails to converge (topic_pending=1, replica_pending=1). The suite exercises both replay paths in the existing cleanup recovery CI job. Final validation results are recorded in the PR.

Regression and fault checks defined before implementation