This document describes the Cursus partition-log files. The runtime accepts only the documented current metadata versions; an incompatible format requires an explicit offline converter or clean bootstrap.
Each topic partition has independent segment and sparse offset-index files:
{log_dir}/{topic}/
partition_{partition}_segment_{baseOffset:020d}.log
partition_{partition}_segment_{baseOffset:020d}.index
The segment number is the first logical record offset in that segment, not an ordinal file number. The default log_segment_bytes is 1 GiB and is configurable. A segment also rolls when its index would exceed log_index_size_bytes or when the configured time-based roll interval expires.
A log segment is a sequence of length-prefixed records:
uint32_be serializedLength
byte[serializedLength] serializedDiskMessage
The maximum accepted serialized record size is 16 MiB. Legacy-time records use
CDM5; durable broker-time activation writes CDM6. CDM6 adds an eight-byte,
positive Unix-millisecond broker append time immediately before the checksum,
and includes that field and its magic in CRC32C. Zero time continues to use CDM5;
CDM6 with zero, negative, overflowing or corrupt time is rejected. See the
retention contract for activation,
legacy holds, replication preservation and downgrade fences.
The CDM5 serialized message uses big-endian fixed-width integers and length-prefixed strings/bytes in this order:
| Field | Encoding |
|---|---|
| topic | uint16 length + UTF-8 bytes |
| partition | uint32 |
| logical offset | uint64 |
| producer id | uint16 length + bytes |
| producer sequence | uint64 |
| producer epoch | uint64 representation of non-negative int64 |
| partition leader epoch | uint64 representation of int64 |
| payload | uint32 length + bytes |
| event type | uint16 length + bytes |
| schema version | uint32 |
| aggregate version | uint64 |
| metadata | uint16 length + bytes |
| key | uint16 length + bytes |
| transactional id/state/marker | three uint16 length-prefixed values |
| control batch type | uint16 length + bytes |
| control batch version | int16 |
| control coordinator epoch | int64 |
| control key/value | two uint16 length-prefixed byte arrays |
| event id/payload digest | two uint16 length-prefixed values |
| checksum | CRC32C over every preceding byte, including CDM5 |
The transaction/control fields are empty for ordinary records. Control records remain in the raw log and are filtered by read_committed. CDM2, CDM3, and CDM4 records remain readable, but they do not contain the CDM5 leader epoch required to find a divergent replica boundary. A mismatch against such a legacy tail requires a clean bootstrap.
Each .index file contains fixed 16-byte entries:
uint64_be logicalOffset
uint64_be bytePosition
An entry is written after the configured byte interval (default 4096 bytes). Reads locate the segment and nearest indexed byte position, validate that entry against the log, then scan records. Active segments require contiguous logical offsets. Compacted closed segments retain original offsets in strictly increasing order and may contain holes. Segment data is opened through mmap for reads.
Asynchronous writes enter a buffered writeCh (default capacity 1024). flushLoop serializes and writes a batch when any of these occurs:
disk_flush_batch_size records are ready (default 50),linger_ms expires (default 50 ms),WriteBatch flushes the Go buffered data and index writers to the operating-system file descriptors. syncLoop calls Sync for data and index files at disk_flush_interval_ms (default 500 ms) and advances the partition durability callback after a successful sync. Explicit Flush, segment rotation, and shutdown also sync data. Initial and rolled segment/index filenames are published by syncing their parent directories before the handler can acknowledge a write. A failure at that boundary makes the handler unavailable for writes until restart.
A write being visible in the process page cache is different from surviving power loss. Client acknowledgements and replicated HWM advancement must be interpreted according to the selected publish/replication path and the synced committed-tail contract, not merely successful channel enqueue.
Startup recovery scans the active segment, validates every length and decoded record, checks contiguous offsets, and truncates an invalid or partial tail. It rebuilds or opens the sparse index and clamps recovered committed high watermarks to the durable tail.
Partition-owned side checkpoints include:
Transaction visibility indexes are rebuilt from durable transaction records and markers. The durable transaction coordinator decision remains a separate metadata authority.
A standalone broker stores the authoritative topic registry in {log_dir}/__topic_metadata.json. The only supported shape is version 3:
{
"version": 3,
"topics": [
{
"name": "orders",
"revision": 3,
"partitions": 4,
"replication_factor": 3,
"lifecycle_epoch": 1,
"idempotent": true,
"event_sourcing": false,
"policy": {
"cleanup_policy": "delete",
"partitioner": "hash_key",
"auth_policy": "acl",
"read_acl": ["game-server"],
"write_acl": ["game-server"],
"retention_hours": 168,
"retention_bytes": 0
}
}
]
}
The encoded manifest is limited to 16 MiB. Updates write a mode-0600 same-directory temporary file, sync it, atomically replace the committed manifest, and sync the parent directory where supported. Startup reads only the committed path; abandoned .tmp files are not authoritative. Runtime parsing accepts only manifest version 3, disallows unknown fields, and rejects duplicate names, missing revision/replication-factor/lifecycle-epoch fields, invalid definitions, and trailing content. A missing manifest with persisted topic logs, or a valid manifest that omits a persisted topic directory, is an integrity error. The broker does not infer ACL or event-sourcing mode from segment filenames when a manifest is absent or invalid.
Backups of a standalone broker must keep this manifest with topic partition directories, __transaction_state.journal, and the consumer offset log. Restoring only segment files cannot reconstruct the full topic policy.
__consumer_offsets stores version-1 group registration, complete committed-next-offset snapshot, and group tombstone JSON payloads inside ordinary segment frames. Stable semantic keys support compaction; lifecycle epochs fence delete/re-create, and snapshot revisions make replay deterministic across physical internal partitions. Runtime replay rejects unversioned single/bulk offset JSON records.
The internal topic is forced to compact cleanup and unlimited time/size retention regardless of broker defaults, manifest input, or application CREATE. Registration/commit acknowledgement synchronously flushes and fsyncs its authoritative log. A partial or undecodable final record is treated as a torn write and truncated; corruption before the tail, conflicting or regressing offsets, unversioned records, and key mismatches fail readiness instead of being skipped. See Standalone Clean-Bootstrap Recovery for the supported reset boundary.
A standalone broker stores coordinator snapshots in {log_dir}/__transaction_state.journal. This file is separate from partition segments. Each record uses the following versioned frame:
uint32_be payloadLength
byte[payloadLength] JSON {"version":1,"transaction":{...}}
uint32_be crc32(payload)
The encoded payload is limited to 32 MiB. Every accepted transition is appended and fsynced. Initial creation syncs the journal file and its directory entry before the journal becomes available. Existing journals receive the same directory sync during startup, and append/recovery never recreate a missing authoritative path. Before appending, the broker truncates bytes beyond the last validated record so a failed partial write cannot hide later acknowledged state. Startup repairs only a torn or checksum-corrupt final frame and rejects corruption before the tail. Every runtime record must use the version-1 envelope; bare transaction snapshots are rejected.
The journal appends every acknowledged transition and automatically compacts only superseded transaction snapshots when removable record or byte debt crosses its threshold. A large useful latest-state set is retained without repeated rewrites. Backups must keep the journal consistent with partition logs and the standalone consumer offset store.
Event-sourcing snapshots are append-only files under {log_dir}/{topic}/partition_{partition}/. Creation syncs the new file, its containing partition directory, and the topic directory before the store accepts SAVE_SNAPSHOT. Each successful save fsyncs the appended record before updating the in-memory snapshot index or returning success. A creation-sync failure leaves the store unavailable, and a record-sync failure does not publish the failed snapshot through reads.
log_cleanup_policy accepts delete, compact, or delete,compact. Time/size deletion removes complete eligible closed segments. Standalone compaction rewrites closed segments with same-directory .compacting files, fsyncs them, and atomically replaces the authoritative log followed by its rebuilt sparse index. Parent-directory metadata is synced where the platform supports it.
The active segment is preserved by both operations. Active readers defer maintenance. A compacted closed log has a versioned .log.compacted-<size> sidecar; offset holes are accepted only when the sidecar is valid and its size matches the log. Backups must preserve the log, its matching sidecar, and its index as one generation. A stale sparse index after an interrupted replacement is correctness-safe because every candidate entry is validated and invalid entries fall back to a scan. Startup removes abandoned compaction temporary files and stale sidecars.
Compaction preserves logical offsets and therefore permits holes only in closed segments. It is rejected for distributed and event-sourcing topics. The full selection and recovery contract is documented in Log Compaction.
DiskHandler separates metadata/file ownership (mu), serialized writer operations (ioMu), and index reader/writer state (indexMu). Write paths acquire metadata before I/O locks. Readers copy the segment list under a short lock and release it before mmap scanning.
Linux builds can apply sequential-access advice and zero-copy transfer where the selected path supports it. Windows uses portable file operations. The broker does not rely on O_DIRECT or per-batch O_SYNC as a universal format guarantee.
| Setting | Default |
|---|---|
| Segment size | 1 GiB |
| Index size | 10 MiB |
| Sparse index interval | 4096 bytes |
| Async channel capacity | 1024 records |
| Flush batch | 50 records |
| Linger | 50 ms |
| File sync interval | 500 ms |
| Cleanup policy | delete |
| Compaction check interval | 300000 ms |
| Minimum cleanable dirty ratio | 0.5 |