This document describes the Cursus partition-log files. The runtime accepts only the documented current metadata versions; an incompatible format requires an explicit offline converter or clean bootstrap.
Each topic partition has independent segment and sparse offset-index files:
{log_dir}/{topic}/
partition_{partition}_segment_{baseOffset:020d}.log
partition_{partition}_segment_{baseOffset:020d}.index
The segment number is the first logical record offset in that segment, not an ordinal file number. The default log_segment_bytes is 1 GiB and is configurable. A segment also rolls when its index would exceed log_index_size_bytes or when the configured time-based roll interval expires.
A log segment is a sequence of length-prefixed records:
uint32_be serializedLength
byte[serializedLength] serializedDiskMessage
The maximum accepted serialized record size is 16 MiB. The serialized message uses big-endian fixed-width integers and length-prefixed strings/bytes in this order:
| Field | Encoding |
|---|---|
| topic | uint16 length + UTF-8 bytes |
| partition | uint32 |
| logical offset | uint64 |
| producer id | uint16 length + bytes |
| producer sequence | uint64 |
| producer epoch | uint64 representation of non-negative int64 |
| payload | uint32 length + bytes |
| event type | uint16 length + bytes |
| schema version | uint32 |
| aggregate version | uint64 |
| metadata | uint16 length + bytes |
| key | uint16 length + bytes |
| transactional id/state/marker | three uint16 length-prefixed values |
| control batch type | uint16 length + bytes |
| control batch version | int16 |
| control coordinator epoch | int64 |
| control key/value | two uint16 length-prefixed byte arrays |
The transaction/control fields are empty for ordinary records. Control records remain in the raw log and are filtered by read_committed.
Each .index file contains fixed 16-byte entries:
uint64_be logicalOffset
uint64_be bytePosition
An entry is written after the configured byte interval (default 4096 bytes). Reads locate the segment and nearest indexed byte position, validate that entry against the log, then scan records. Active segments require contiguous logical offsets. Compacted closed segments retain original offsets in strictly increasing order and may contain holes. Segment data is opened through mmap for reads.
Asynchronous writes enter a buffered writeCh (default capacity 1024). flushLoop serializes and writes a batch when any of these occurs:
disk_flush_batch_size records are ready (default 50),linger_ms expires (default 50 ms),WriteBatch flushes the Go buffered data and index writers to the operating-system file descriptors. syncLoop calls Sync for data and index files at disk_flush_interval_ms (default 500 ms) and advances the partition durability callback after a successful sync. Explicit Flush, segment rotation, and shutdown also sync data.
A write being visible in the process page cache is different from surviving power loss. Client acknowledgements and replicated HWM advancement must be interpreted according to the selected publish/replication path and the synced committed-tail contract, not merely successful channel enqueue.
Startup recovery scans the active segment, validates every length and decoded record, checks contiguous offsets, and truncates an invalid or partial tail. It rebuilds or opens the sparse index and clamps recovered committed high watermarks to the durable tail.
Partition-owned side checkpoints include:
Transaction visibility indexes are rebuilt from durable transaction records and markers. The durable transaction coordinator decision remains a separate metadata authority.
A standalone broker stores the authoritative topic registry in {log_dir}/__topic_metadata.json. The only supported shape is version 3:
{
"version": 3,
"topics": [
{
"name": "orders",
"revision": 3,
"partitions": 4,
"replication_factor": 3,
"lifecycle_epoch": 1,
"idempotent": true,
"event_sourcing": false,
"policy": {
"cleanup_policy": "delete",
"partitioner": "hash_key",
"auth_policy": "acl",
"read_acl": ["game-server"],
"write_acl": ["game-server"],
"retention_hours": 168,
"retention_bytes": 0
}
}
]
}
The encoded manifest is limited to 16 MiB. Updates write a mode-0600 same-directory temporary file, sync it, atomically replace the committed manifest, and sync the parent directory where supported. Startup reads only the committed path; abandoned .tmp files are not authoritative. Runtime parsing accepts only manifest version 3, disallows unknown fields, and rejects duplicate names, missing revision/replication-factor/lifecycle-epoch fields, invalid definitions, and trailing content. A missing manifest with persisted topic logs, or a valid manifest that omits a persisted topic directory, is an integrity error. The broker does not infer ACL or event-sourcing mode from segment filenames when a manifest is absent or invalid.
Backups of a standalone broker must keep this manifest with topic partition directories, __transaction_state.journal, and the consumer offset log. Restoring only segment files cannot reconstruct the full topic policy.
__consumer_offsets stores version-1 group registration, complete committed-next-offset snapshot, and group tombstone JSON payloads inside ordinary segment frames. Stable semantic keys support compaction; lifecycle epochs fence delete/re-create, and snapshot revisions make replay deterministic across physical internal partitions. Runtime replay rejects unversioned single/bulk offset JSON records.
The internal topic is forced to compact cleanup and unlimited time/size retention regardless of broker defaults, manifest input, or application CREATE. Registration/commit acknowledgement synchronously flushes and fsyncs its authoritative log. Corrupt, truncated, conflicting, regressing, unversioned, or key-mismatched records fail readiness instead of being skipped. See Standalone Clean-Bootstrap Recovery for the supported reset boundary.
A standalone broker stores coordinator snapshots in {log_dir}/__transaction_state.journal. This file is separate from partition segments. Each record uses the following versioned frame:
uint32_be payloadLength
byte[payloadLength] JSON {"version":1,"transaction":{...}}
uint32_be crc32(payload)
The encoded payload is limited to 32 MiB. Every accepted transition is appended and fsynced. Before appending, the broker truncates bytes beyond the last validated record so a failed partial write cannot hide later acknowledged state. Startup repairs only a torn or checksum-corrupt final frame and rejects corruption before the tail. Every runtime record must use the version-1 envelope; bare transaction snapshots are rejected.
The journal is append-only and currently has no automatic compaction. Backups must keep it consistent with partition logs and the standalone consumer offset store.
log_cleanup_policy accepts delete, compact, or delete,compact. Time/size deletion removes complete eligible closed segments. Standalone compaction rewrites closed segments with same-directory .compacting files, fsyncs them, and atomically replaces the authoritative log followed by its rebuilt sparse index. Parent-directory metadata is synced where the platform supports it.
The active segment is preserved by both operations. Active readers defer maintenance. A compacted closed log has a versioned .log.compacted-<size> sidecar; offset holes are accepted only when the sidecar is valid and its size matches the log. Backups must preserve the log, its matching sidecar, and its index as one generation. A stale sparse index after an interrupted replacement is correctness-safe because every candidate entry is validated and invalid entries fall back to a scan. Startup removes abandoned compaction temporary files and stale sidecars.
Compaction preserves logical offsets and therefore permits holes only in closed segments. It is rejected for distributed and event-sourcing topics. The full selection and recovery contract is documented in Log Compaction.
DiskHandler separates metadata/file ownership (mu), serialized writer operations (ioMu), and index reader/writer state (indexMu). Write paths acquire metadata before I/O locks. Readers copy the segment list under a short lock and release it before mmap scanning.
Linux builds can apply sequential-access advice and zero-copy transfer where the selected path supports it. Windows uses portable file operations. The broker does not rely on O_DIRECT or per-batch O_SYNC as a universal format guarantee.
| Setting | Default |
|---|---|
| Segment size | 1 GiB |
| Index size | 10 MiB |
| Sparse index interval | 4096 bytes |
| Async channel capacity | 1024 records |
| Flush batch | 50 records |
| Linger | 50 ms |
| File sync interval | 500 ms |
| Cleanup policy | delete |
| Compaction check interval | 300000 ms |
| Minimum cleanable dirty ratio | 0.5 |