| Persistent artifact | Current writer | Older reader support | Upgrade rule |
|---|---|---|---|
| Application record | CDM6 (CRC32C, leader epoch and broker append time), CDM5 until activation | CDM2 through CDM5 remain readable | See the durable retention upgrade contract: capability activation, local Raft marker 12 and standalone manifest fencing prevent unsafe downgrade. |
| Topic manifest | 4 | version 3 remains readable without invented instance/receipt evidence | Validate an offline copy before coordinated upgrade; no downgrade after version 4 is written. |
| Transaction journal | 3 plus checksummed cut manifest | version-1 and version-2 records remain readable | Preserve the journal and its manifest with all partition and offset data. Version 3 writes bounded deltas between full checkpoints. |
| Event snapshot catalog | v2 (CRC32C) | legacy snapshot files remain readable beside v2 | Preserve both snapshot files until a clean bootstrap removes the legacy boundary. |
| Consumer metadata records | 5 | versions 1-4 remain replayable | Upgrade readers before writing offset-reservation records. |
| Consumer history compaction | lifecycle protocol 10, local Raft marker 13 | marker 12 can be replayed and upgraded before capability advertisement; old binaries reject marker 13 | Coordinated upgrade only; durable activation requires all registered brokers and voters. See consumer history operations. Disabling maintenance does not permit downgrade. |
| Cooperative broker/Go runtime | lifecycle protocol 11, local Raft marker 14, activated snapshot 13 | markers 9–13 can be replayed and upgraded; history-only protocol 10 brokers cannot activate cooperative membership | All registered brokers and voters must support the runtime. Both history and cooperative activation remain durable. See runtime and validation. Disabling the trigger flag does not permit downgrade. |
| Transaction history writer and replay evidence | lifecycle protocol 12, local Raft marker 15, activated snapshot 14 | markers 9–14 replay before atomic upgrade; protocol 11 cannot activate transaction history | Independent from the cooperative gate. Snapshot 14 preserves both activations in either order; older binaries reject marker 15. In-place downgrade is unsupported. See transaction history contract. |
| Raft FSM snapshot | 11 by default, 12 with cooperative ownership, 13 after cooperative runtime activation, or 14 after transaction-history activation | versions 9–14 are accepted through the supported one-way recovery path; transaction-history activation requires version 14 | Snapshot version is distinct from local Raft marker 15. Never downgrade written volumes; restore complete compatible backups into isolated storage. |
| Instance-bound truncate | broker lifecycle protocol 7, Wire command 61 | legacy TRUNCATE keeps its conflict contract | Upgrade all brokers before binding identities or using durable retries. |
| Truncate consumer cleanup boundary | broker lifecycle protocol 8, optional snapshot-11 evidence | omitted while any registered broker advertises an older capability | Upgrade every registered broker, including offline members, before relying on snapshot-persistent cleanup ordering. See truncate registration recovery. |
| Replica-gap recovery control state | broker lifecycle protocol 6 | An older active broker blocks a new quarantine transition and cannot register while a durable recovery marker exists | Complete the rolling upgrade before relying on automatic gap quarantine and proof-based ISR re-admission. |
Mixed-version rolling upgrades across a format boundary are unsupported. A normal restart of the same compatible release is supported; a format-changing upgrade is a coordinated whole-cluster operation.
For older consumer snapshots with unknown recovery authority, use the legacy compatibility recovery contract. Recovery requires complete committed source coverage and preserves unknown authority. Routing capability alone does not establish recovery readiness.
For an instance-bound truncate, save the CREATE response’s instance_id and
revision with the requested operation before sending it. Retry only that pair
after a timeout or lost connection. replayed=true acknowledges the retained
historical result; it does not mean today’s topic is still empty. Never replace
the revision with a newly fetched value to resolve an unknown outcome.
The broker bounds ordinary Raft command admission at 5 seconds. It gives the
synchronous TOPIC_TRUNCATE/TOPIC_DELETE storage lifecycle path 60 seconds to
enqueue and a separate 60 seconds to complete after acceptance, for a bounded
120-second worst case. Set the operator client’s command timeout above that
worst case. A client timeout or lost response is still an unknown outcome:
inspect the saved instance/revision receipt, lifecycle epoch and partition
offsets before replaying the exact request.
cleanup_pending=true means intent committed but the responding broker has not
finished local materialization. Inspect every broker’s readiness and topic/replica
materialization before resuming traffic; a missing flag is not a cluster-wide
barrier. Keep a pending standalone manifest, journal and epoch marker together;
restart resumes the committed cleanup. Do not remove the manifest or manually
advance the marker. See the verification ledger
for tested crash boundaries, costs and remaining acceptance checks.
If a post-truncate snapshot predates persisted consumer cleanup boundaries, the current HWM cannot reconstruct the missing historical cut. Retain the authoritative Raft history containing the truncate and preceding state, or a snapshot that actually contains the cut. Without either, cleanup must remain pending. A binary replacement alone is not proof that such an incident will recover. See the current acceptance audit.
CDM5 checksums cover every serialized authoritative field, including topic, partition, offset, producer and transaction metadata, event identity, and payload. CDM5 also stores the partition leader epoch used for divergent-tail recovery. Existing CDM2/CDM3/CDM4 records are an explicit legacy read boundary and are not rewritten in place. New appends use CDM5, so a segment may contain legacy and checksummed records after an upgrade. Backup validation, startup recovery, ordinary reads, compaction, and replica transfer all use the same decoder and reject a CDM5 checksum mismatch. Treat a reported segment and offset as unavailable; restore an immutable validated backup or repair it from a checksum-valid committed replica instead of editing bytes in place.
Each broker exclusively locks its log directory before opening metadata or
recovering partitions. A second broker using the same directory fails startup;
each partition also rejects a second storage handler. The filesystem must
support the operating system’s exclusive file locks. Locks are released on
normal shutdown or process exit, including a crash. The .cursus.lock and
partition_<id>.lock files remain on disk and are safe to retain in backups.
Do not delete or replace them while a broker is running: the file’s existence
does not indicate a stale lock, and removing it can defeat mutual exclusion.
For a standalone broker, validate a copied backup before using it for a rollback or recovery. The command is read-only: it verifies the explicit topic manifest against the persisted partition layout, scans topic and consumer metadata records, validates the transaction journal against its checksummed cut manifest, and checks every event snapshot record without starting a broker.
cursus-storage backup validate --log-dir /backup/cursus-logs
It exits non-zero if the manifest is missing or if validation finds a storage problem. Distributed brokers keep topic and group metadata in Raft and do not write standalone topic manifests. Do not apply this validator to a distributed PVC: its missing-manifest error does not establish corruption or restorable state. For a distributed backup, stop all members, copy every member’s complete log directory into one immutable generation, compare source/copy file digests, and start the previous image on writable clones of all member copies in an isolated cluster. Verify topics, group registrations, committed offsets, and a known record before upgrading the original volumes.
For the fixed three-member Kubernetes topology, use the separate Kubernetes cluster runbook. Its StatefulSet keeps one PVC per member and requires a one-member-at-a-time restart; never combine PVCs from different backup generations.
Before changing any binary or configuration, stop writes and record the target
release, git/image digest, configuration checksum, member list, leader, ISR,
groups and committed offsets, and available disk space. Take one immutable
backup generation containing every storage artifact present in the member
directories. Standalone backups include a topic manifest, transaction journal
and its .manifest, event snapshot files, consumer-offset logs, every .log,
its matching .index, and each .log.compacted-<size> sidecar.
For standalone backups, run these read-only checks against the copy, not the production volume:
cursus-storage manifest inspect --log-dir /var/lib/cursus/logs > inventory.json
cursus-storage consumer-metadata inspect --log-dir /var/lib/cursus/logs > consumer-records.json
Do not proceed if either output reports problems, orphaned topic directories, or an invalid compaction sidecar. For distributed backups, use the full-cluster clone-and-restore check above. Keep the inventory and backup generation identifier with the change record.
LOG_DIR=/data/logs, which overrides YAML
log_dir; explicitly set LOG_DIR to the mounted persisted data path and
verify that effective value in each broker’s startup log.__consumer_offsets is empty.
Validate topics, transaction recovery, and an acknowledged test
publish/consume before restoring client traffic.For a format-compatible Kubernetes rolling restart, delete one Pod at a time and wait for the
replacement to become Ready, regain its expected replica state, and return to
ISR before continuing. The StatefulSet uses OnDelete; changing the image does
not itself restart Pods. Use the production dashboard and baseline alerts as
the release gate, and abort the rollout on request admission, disk headroom,
recovery, or replication safety alerts.
Replica-gap quarantine requires every active broker to advertise lifecycle protocol 6. During a mixed-protocol rolling window, do not rely on that recovery path: drain affected writes, finish or abort the rollout, and preserve the first gap evidence instead of retrying truncation or editing offsets. If a durable recovery marker already exists, an older broker is fenced from registration until catch-up and prefix-proof admission clear the marker.
A follower can retain an uncommitted record after a replication attempt fails
and the leader reconciles its local tail to the authoritative committed HWM.
A different append at that offset, in the same leader epoch, then returns
replica_offset_conflict. Previously this was reported as the terminal
replica_append_failed, leaving the divergent follower in ISR without starting
catch-up. Storage read errors remain distinct from confirmed record conflicts.
The partition leader now quarantines the conflicting follower through the
existing fenced Raft transition. An acks=all attempt fails without advancing
the committed HWM, even when the remaining ISR satisfies minISR. A write
already acknowledged with acks=1 keeps its tail and write ownership while
waiting for catch-up. The existing recovery marker fences new writes and keeps
readiness closed until catch-up and prefix-proof admission complete. The
follower removes only its uncommitted tail at the authoritative boundary;
consumer offsets are not reset or skipped. This adds no storage format change.
After a conflict, check CLUSTER_STATUS for the affected partition’s
recovery_pending, recovery_replicas, ISR, and committed_hwm. Look for
partition replica quarantined after offset conflict and the
replica_offset_conflict retry label. Recovery is complete only after the
marker clears, the expected ISR returns, readiness succeeds, and a subsequent
write/consume advances normally. A persistent committed-prefix mismatch must
remain unavailable; do not delete records or force an offset reset to clear it.
The dedicated fault image can reproduce a follower failing after a durable append, followed by a conflicting append in the same epoch:
RUN_E2E_CHAOS=1 go test -v -count=1 -timeout=20m ./test/e2e-cluster \
-run '^TestReplicaConflictRecoveryAfterPartialAppend$'
This runs both acks=all and acks=1 with an isolated three-broker Docker
fixture and verifies quarantine,
closed readiness while catch-up is paused, follower restart, committed-prefix
preservation, resumed writes, and consumer membership/offset stability for more
than twice the fixture’s session timeout. It models the durable-tail failure
state; it does not establish the cause of an earlier network or transaction
timeout in a particular deployment.
If a follower’s prefix digest differs but the source has no retained epoch boundary after the follower’s epoch, recovery falls back to the same fenced, one-record rollback used for legacy records. This also covers non-zero epochs and legacy records encountered while scanning the source. Each subsequent request must prove the preceding prefix before records can be transferred; the authoritative source log is never truncated. Storage or checksum failures, unavailable prefix records, stale fences, and invalid committed HWMs still stop recovery.
Current brokers advertise recovery protocol 2 only after the source, consumer metadata cache, replica receiver and authenticated internal listener are ready. Direct and leader-forwarded registration use the same payload. Lifecycle protocol remains 6; protocol-1 peers retain indexed recovery and protocol-0 peers use the legacy digest path. There is no operator activation switch.
Consumer routing activates durably only when every registered broker, including inactive registrations, and every Raft voter has a compatible current process registration. Check these registrations if activation remains pending after an upgrade. Once active, the cluster rejects legacy or incomplete registrations; do not downgrade binaries on the same volumes without a supported migration.
Capable replicas use retained-prefix proofs for compacted histories and fenced, restartable bootstrap when the common retained history is unavailable. Source, receiver, leader and lifecycle checks remain mandatory through installation and ISR admission. Missing authoritative data or invalid proofs leave the partition unavailable. Do not choose a larger anchor, edit offsets, or truncate committed records to bypass a bootstrap requirement.
The log remains authoritative; partition_N.lineage is a derived sidecar.
Startup and invalidation require cold reconstruction on an explicit proof
request. Sidecar publication failures stop proof publication without reversing
acknowledged appends. Preserve the log and inspect the source error rather than
editing sidecars to manufacture agreement.
Cold proof reconstruction and bootstrap can still scan retained records. Resumable chunks and bounded caches reduce repeated work but cannot interrupt an OS read or fsync already in progress. After routing activation, consumer recovery uses fixed remote history and the latest owned commits under a global replay gate. Record actual transfer bytes, scan/hash work and elapsed time alongside request counts when diagnosing slow recovery.
See the recovery design for proof bounds, checkpoint invalidation, recovery budgets and regression coverage, and bootstrap storage for durable installation guarantees.
The same-epoch restart regression commits a replacement on the two surviving brokers before restarting a divergent follower, then checks exact raw-record equality on all three replicas and persistence through a second restart:
RUN_E2E_CHAOS=1 go test -v -count=1 -timeout=20m ./test/e2e-cluster \
-run '^TestReplicaRecoveryWithoutLaterEpochBoundary$'
If the candidate cannot read or safely operate on the persisted format, stop the complete cluster. Do not live-downgrade binaries. Restore the validated single backup generation and start the previously compatible release as a quorum. Re-run the readiness and read-only checks above before admitting clients. A partial file restore, a missing compaction sidecar, or mixing files from generations is not a supported recovery procedure.
The storage recovery rehearsal verifies
acknowledged data and consumer offsets after guest power loss, standalone backup
restore, authoritative-record corruption and derived-index damage. Its distributed
case stops all three writers, compares source/copy digests, and restores one
complete backup generation into fresh writable member volumes. A separate
coordinated-upgrade scenario writes with fixed revision c7384a5, stops every
writer, and restores fresh clones with the recorded candidate image. Both paths
verify records, consumer offsets, new writes and the immutable backup. This
compatible revision pair does not extend the support matrix above or establish
mixed-version operation or in-place downgrade support.