Kafka 4.1’s flush guidance distinguishes replication durability from per-write filesystem synchronization. Its versioned LogLoader rebuilds missing or corrupt derived indexes from the log during recovery. Cursus adopts the distinction between authoritative records and derived indexes, but retains its own standalone synchronous acknowledgement contract: complete checksummed record corruption must fail closed, not be treated as a partial tail that can be discarded. This trades availability for preserving acknowledged data and forensic evidence. No storage format or acknowledgement policy changes here.
The Kafka upgrade procedure separates binary rollout from feature activation. Cursus still requires a coordinated upgrade across incompatible persistent formats. A same-binary restore rehearsal cannot establish mixed-version compatibility or authorize a downgrade.
The coordinated-upgrade check uses the fixed source revision
c7384a51299a2519900966d3ed786b9cbe0e3f4c and the candidate checkout. Unlike
Kafka’s supported rolling transitions, this check stops all old writers before
starting any candidate broker. It adopts explicit source/target provenance and
post-upgrade validation while retaining Cursus’s whole-cluster upgrade rule.
The candidate must use a different recorded image from every source member.
The regression requires complete file inventories before cloning, all 32 old
records and consumer offset 13 after candidate startup, one new acknowledged
record, and an unchanged original backup. Copied/hash bytes and recovery time
are recorded as in same-binary restoration. Availability is deliberately lost
during the stop/clone/start window; no mixed-version or in-place downgrade
guarantee follows from this compatible revision pair.
The regression boundaries are defined before implementation:
Record copied/validated file bytes, payload bytes, record count and recovery elapsed time. Copy and hash bytes are logical test work, not physical device I/O. Guest power loss covers QEMU/ext4 page-cache loss with flush-respecting writeback; it does not certify a physical controller, host power loss, XFS or multi-rack loss. Standalone validation does not validate a distributed Raft/PVC backup.
The first hosted guest run retained all 32 acknowledged records but restarted with HWM 26 and LEO 32. The asynchronous HWM checkpoint had lagged the durable log; a duplicate retry did not advance that stale visibility boundary.
Kafka 4.1 ReplicaManager checkpoints HW periodically, while Partition recomputes it from leader/ISR state on leadership changes, including an ISR of one. Cursus should likewise distinguish a checkpoint from the authority that determines current visibility. Its standalone log has no replication commit decision: every recovered complete append can advance HWM. Distributed mode must retain its separate committed-watermark authority and never promote an uncommitted local tail merely because it survived.
The correction restores standalone HWM from the recovered durable end, after the existing pending-bootstrap guard. Transaction visibility still requires its recovered markers and coordinator decision. This avoids an additional checkpoint fsync per ACK or a second log scan: recovery adds only a scalar boundary assignment, with zero additional log bytes read. A complete append whose response was lost may also survive, as already allowed by retry semantics. Before implementation, regression checks require stale versioned and legacy checkpoints to recover standalone visibility, the same distributed checkpoints to preserve their boundary/fencing, open transactions to remain invisible, and the unchanged guest power-loss oracle to read all acknowledged records.
The deterministic stale-checkpoint test failed before the correction for both
versioned and legacy standalone checkpoints. After the correction, the focused
checkpoint/bootstrap/committed-read suite passed in 103.405 seconds and the
native process-crash/backup suite passed in 37.385 seconds. The checkpoint crash
test still checks the exact old-or-new file image separately from reconstructed
standalone visibility. The hosted rerun at cfded66 passed all seven guest
boundaries. The acknowledged-prefix reboot verified all 32 records (8,768 payload
bytes), committed offset 13, duplicate suppression and new end 33 in 1.514 seconds.
Its full-cluster restore also passed: 157,699,848 copied bytes, matching source,
backup and clone inventories, and 69.522 seconds for recovery, stable ISR and
complete readback. These are separate measurements from the local runs below.
On 2026-11-07 UTC, the native standalone process-crash and backup suite passed
in 10.144 seconds. It preserved all 32 acknowledged records (8,768 payload bytes),
the committed offset 13 and the empty group’s offset 0. Retrying producer sequence
32 kept end 32; sequence 33 advanced end to 33. The backup contained 24 files and
52,444,653 bytes, including preallocated index files. Complete record corruption
kept /ready at 503 with the checksum error, left the client port closed and
preserved the damaged log. Index reconstruction and validated-clone restore both
passed complete public readback.
The guest campaign passed all seven boundaries: acknowledged prefix, journal
sync/rename/directory sync, and manifest write/sync/directory sync. It used QEMU
7.2.22, Debian Linux 6.1.190, e2fsprogs 1.47.0 and ext4 data=ordered with QEMU
cache=writeback. The acknowledged-prefix reboot/readback phase completed in
1.60 seconds inside the guest; host boot time is separate. The transcript contains
the final ACK cut marker and all verification assertions. These timings describe
this small fixture and are not recovery-time objectives for production volumes.
The full-cluster rehearsal copied 157,699,845 bytes across three stopped members. Source, backup and fresh-volume file inventories matched. Each member restarted with its own recorded image ID; per-service builds need not have identical image IDs. Restore, the stable-ISR window and full readback completed in 72.987 seconds. All 32 original acknowledged records, consumer offset 13 and the new 33rd record were preserved, and the backup inventories remained unchanged. The first local rehearsal stopped at an overly strict cross-member image equality assertion; it was corrected to pin and compare each member’s own image before the successful run. No mixed-version upgrade was exercised.
The subsequent coordinated-upgrade run passed in 122.49 total test seconds.
It wrote with source c7384a5 and restored with candidate checkout 80ee305
(the hosted test merge of 22e83e1 into c7384a5), image
sha256:ebd42a4bde6d608687a762a46dc9d23f7bef6d5d1c1edaef41deb085a3e40ec9.
Its complete backup contained 157,699,842 bytes. Source/copy/clone inventories
matched; candidate recovery, stable ISR and readback took 69.453 seconds and
verified 33 acknowledged records and consumer offset 13. The original backup
digests remained unchanged. The same-image case in that run also passed in
132.09 total seconds, with 157,699,848 copied bytes and a 70.054-second recovery
and verification window. Both Linux test drivers used the race detector; broker
images were ordinary production builds. This validates
the recorded compatible revision change, not mixed-version operation, historical
format migrations, or an in-place downgrade.
go test -v -count=1 ./cmd/broker \
-run '^TestStandaloneBroker(AcknowledgedPrefixPowerRecovery|BackupCorruptionRestore)$'
docker build -f test/power-loss/Dockerfile -t cursus-power-rehearsal .
mkdir -p power-results
docker run --rm -v "$PWD/power-results:/results" cursus-power-rehearsal
RUN_E2E_ACK_FAULTS=1 CURSUS_ACK_FAULT_RESULTS=/absolute/path/to/results \
go test -v -count=1 -timeout 20m ./test/e2e-cluster \
-run '^TestAcknowledgedClusterBackupRestore$'
# Coordinated upgrade from an archived, fixed source revision.
baseline=c7384a51299a2519900966d3ed786b9cbe0e3f4c
git fetch --no-tags --depth=1 origin "$baseline"
source_dir=$(mktemp -d)
git archive "$baseline" | tar -xf - -C "$source_dir"
printf '%s\n' "$baseline" > "$source_dir/.upgrade-source-revision"
revision=$(git rev-parse HEAD)
candidate="cursus-backup-upgrade-candidate:$revision"
docker build --label "org.opencontainers.image.revision=$revision" \
-f test/e2e-cluster/Dockerfile.ack-faults -t "$candidate" .
RUN_E2E_ACK_FAULTS=1 RUN_E2E_ACK_UPGRADE=1 \
CURSUS_BACKUP_UPGRADE_SOURCE="$source_dir" \
CURSUS_BACKUP_UPGRADE_IMAGE="$candidate" \
CURSUS_BACKUP_UPGRADE_REVISION="$revision" \
CURSUS_ACK_FAULT_RESULTS=/absolute/path/to/upgrade-results \
go test -v -count=1 -timeout 20m ./test/e2e-cluster \
-run '^TestAcknowledgedClusterCoordinatedUpgrade$'
Run only one acknowledged-write fixture at a time. It owns a dedicated Compose
project and ports; on Linux reserve ports 42001–42003 from ephemeral allocation,
as the workflow does. Restore helper containers use --rm and no network.
The test destroys only its own containers and original/cloned data volumes after
capturing results. It never copies live writers or reuses a production volume.