The two implementation layers are independently gated and reviewed: the transaction-aware replay index and format contract, followed by certified history reclamation. An index hit is not evidence of physical reclamation. The authoritative retained log must remain sufficient when every derived index is removed. See the format contract and operational contract.
78b0bf566b4cd23d2d9317c417fc377654374205.7c85be902f8e07c532e918be7d2afc5309ac35f4, based on official main
d5b8d773ad215362c80ee46a2a8764c37ead9728.88b3e7d57a292efc22328574e3b5312cb3e897d7.533afcb82011cc232b88724892426218e0693a5b, tested merge
bdbe7fcfc6b75f385a1a1a368ee106df9dbe963f. All 34 reported checks passed,
including the complete scaling job and another replay-crash run.fd66ae9352e55ca4e6a4aec6f33ea54607bada90 for 88b3e7d.
The tests listed below passed in that run unless a different source is
explicitly stated. All 33 reported checks passed, including bootstrap,
snapshot restore, deployment, Helm and the dedicated crash experiment.The local ordinary-broker experiment uses a separate frozen, clean checkout;
later instrumentation/report changes do not silently change its source. Local
Docker timing samples and hosted Linux CI samples are different environments.
The tested merge and PR head 88b3e7d have the same Git tree
c017799dadb5e197a58125d7b40cb3ae655d235a.
An earlier local full-package run exceeded the 20-minute controller/disk budgets under overlapping I/O and exposed a stale protocol-11 fixture. The fixture now advertises the transaction protocol constant; focused controller publication/restore and lifecycle-gate regressions passed locally in 57.112 s. The current CI package race run passed. The timed-out local run is not counted as a successful full suite.
| Requirement | Passing regression evidence | Boundary checked |
|---|---|---|
| Exact committed position and current-state suffix | TestConsumerReplayIndexRealDiskStartupAndSuffix, TestConsumerReplayIndexSuffixAndBounds, TestConsumerReplayIndexRejectsChangingGeneration |
Partition/topic identity, committed cut and durable generation; no certification from a live mutable projection. |
| Missing, corrupt, incompatible or replaced index | TestConsumerReplayIndexInvalidation, TestConsumerReplayIndexOldSuffixCannotHideCorruption, TestConsumerReplayIndexCrossPartitionSuffixFallsBack |
Full retained-log fallback; stale state cannot hide suffix corruption or cross-partition evidence. |
| Index publication crash and failure | TestConsumerReplayIndexCrashPublication, TestConsumerReplayIndexPublicationFailurePreservesPrevious |
Reopen after durable publication stages; retain the prior valid file or fall back safely. |
| Deleted groups and re-registration | TestConsumerReplayIndexTombstoneAndReRegistration, TestConsumerHistoryLifecycleLadder |
Four reused live names and one deleted-name fence; 1,202 / 12,002 / 120,002 input records all reduce to nine retained and replayed records. This is an in-memory semantic fixture, not physical I/O timing. |
| Pending transactions, decisions and provenance | TestConsumerHistoryPlanPreservesPendingIntentAndDecisionControl, TestConsumerHistoryPlanPreservesEpochReservationsAndSources, TestConsumerReplayIndexTransactionEvidenceAndDecisionChange |
Pending reservation, epoch, terminal decision and compatibility provenance remain exact. Visibility uses the current committed view. |
| Transaction retirement and intermediate logs | TestConsumerHistoryRetiredTransactionHistoryBound, TestConsumerHistoryTransactionMarkerWaitsForOtherSegments, TestConsumerHistorySweepEveryIntermediateReplay, TestConsumerHistoryCurrentOpenEpochRemainsProtected |
Superseded v7 evidence requires a committed higher-epoch witness. Controls outlive their intents; open/current epochs remain protected. |
| Ordinary and transactional rewrite crashes | TestConsumerHistoryDiskCrashStages, TestConsumerHistoryTransactionCrashStages, TestConsumerHistoryPruneCrashStages |
Nine rewrite durability stages, old-or-new complete publication, repeated reopen without a cache, preserved uncommitted tail and original offsets. |
| Concurrent commits and invalidated certification | TestConsumerHistoryDiskConcurrentAppend, TestConsumerHistorySweepCannotSkipIntentOrAcceptExternalMutation, TestConsumerHistoryWorkDetachedAdmissionAndSuffix |
Appended suffixes survive; external replacement invalidates the proof. |
| Bootstrap and snapshot replacement | TestConsumerHistoryBootstrapPreservesCompactedCommittedState, TestConsumerHistoryControllerPublishesAndRestores, TestConsumerHistoryRemoteProofRefreshAndReplacement |
Transfer only committed retained state, invalidate old generations, restore exact offsets. The disk bootstrap fixture measures logical local transfer, not network throughput. |
| Unrelated membership and ownership clocks | TestConsumerHistoryPlanPreservesProducerAndLiveClocks, TestConsumerReplayIndexCooperativeOwnershipAndClocks, TestConsumerReplayPreservesOnlyUnchangedOwnerSession |
Heartbeat, activity, ownership and cooperative grant/revocation state remain intact for unchanged sessions; recreated registrations cannot inherit old sessions. |
| Repeated replay interruption | TestConsumerReplayRepeatedProcessCrashDuringScan, TestConsumerHistoryReplayCrashResources |
Actual source reads before each interruption, no incomplete index publication, exact final recovery and distinct per-process accounting. |
| Activation and downgrade | TestConsumerTransactionHistoryActivationAndRestoreGate, TestConsumerTransactionHistoryAndCooperativeGatesRemainDistinct |
All-broker protocol gate, snapshot persistence, independent cooperative activation and fail-closed incompatible formats. |
The current history recovery CI passed in 181.78 s. Four fixed groups acknowledged 400 commits in 34.263 s. Three sequential cache-absent starts reached ready/full ISR in 7.694 / 7.647 / 7.644 s. OS caches were not dropped, and shutdown/cache removal were outside those restart timers. The fixture verifies physical removal, exact offsets, ownership movement, DELETE/TRUNCATE, re-registration and the unrelated live member. This is correctness evidence, not a baseline/fix cold startup comparison.
The isolated transactional run
at 7c85be9 passed in 599.06 s. One hundred transaction epochs, including owner
stop/movement/return, took 448.575 s with 207 admission retries. Every replica
met the retained-intent bound, and three cache-absent restarts recovered exact
offset 100. An earlier run overlapping heavy local package I/O failed the
two-minute reclamation deadline with ten intents where eight were allowed;
it is not counted as passing evidence and the assertion was not weakened.
The RF2 nonreplica run at
d86c2c1 passed in 261.85 s before the final base refresh. Four hundred commits
took 83.182 s with 1,549 explicit availability retries. Node 3, an actual
nonreplica, reported 5,597 logical metadata transfer bytes after cache-absent
restart. DELETE/TRUNCATE and re-registration preserved the unrelated group.
This historical source demonstrates the remote path; it is not presented as a
current-head network performance comparison. Current package tests separately
exercise remote proof refresh/replacement and controller installation.
The replay-crash resource report retains every interrupted process’s CPU, I/O, records, allocation and sync costs, survivor interval deltas, final readiness and shutdown-inclusive exit accounting. Its final restart is warm after partial replay.
The whole-broker report completes the 100 / 1,000 / 10,000 baseline/fix matrix on hosted Linux. Every replica met the fixed retained-log bound; absent, valid, corrupt and unsupported-version indexes, interrupted startup, ownership movement and exact offsets all passed. Setup, seeding, maintenance, request latency and resource costs are retained in its raw JSONL and CSV. At 10,000, per-replica metadata decreased from 17,593,264 to 139,824 bytes and cache-absent replay work across three brokers decreased from 240,072 to 1,902 decoded records. These are repeated work counters, not unique-record counts.
That row’s whole startup changed from 15.217 to 12.941 s and failover from
6.396 to 4.997 s. These are single observations with advisory file-cache
eviction and unfrozen Raft/background progress, not an isolated causal speedup
or hardware-cold guarantee. Maintenance including recreation took 33.677 s,
used 42.890 broker CPU seconds and returned one NOT_COORDINATOR error in
1,341 foreground fetches. Sampled startup heap increased from 290,204,368 to
307,984,000 bytes despite lower replay work. No availability or memory cost
is hidden by the retained-history reduction.
The supported bound is for fixed logical identities and supported formats, not unlimited creation of distinct deleted group names or producer/transaction identities. Their authoritative fences remain retained. Legacy protected prefixes and non-compositional histories have explicitly higher recovery and certification costs. The one-second tiny-segment stress failure remains part of the operational evidence; no universal reclamation-latency guarantee or hardware-cold speedup is established by the current tests.