Cursus throughput comes from partition parallelism, buffered batch writes, sparse indexes, mmap reads, and optional platform-specific I/O paths. Tune only after measuring the real durability, retention, transaction, and replication settings used in production.
| Setting | Default | Effect |
|---|---|---|
channel_buffer_size |
1024 | Pending records per partition storage writer. |
partition_channel_buffer_size |
10000 | In-process partition dispatch capacity. |
consumer_channel_buffer_size |
1000 | In-process consumer receive capacity. |
broadcast_channel_buffer_size |
10000 | Broadcast/fan-out capacity. |
disk_flush_batch_size |
50 | Records accumulated before a batch write. |
linger_ms |
50 | Maximum wait for a partial async batch. |
disk_flush_interval_ms |
500 | Data/index Sync interval. |
disk_write_timeout_ms |
10 | Buffered enqueue timeout. |
log_segment_bytes |
1 GiB | Data segment roll limit. |
log_index_size_bytes |
10 MiB | Sparse index roll limit. |
log_index_interval_bytes |
4096 | Approximate bytes between sparse entries. |
writeCh is buffered, not unbuffered. A larger buffer absorbs short bursts but uses more memory and can widen the amount of work queued inside the process.
A larger batch or linger can improve sequential write efficiency and throughput. It also increases queueing latency and the number of records waiting for the next flush. A smaller value reduces latency at the cost of more writer/syscall overhead.
The batch writer flushes Go buffers to the file descriptor; the sync loop controls periodic file Sync. Shortening the interval reduces the unsynced power-loss window and increases storage pressure. Do not describe a successful async enqueue as durable.
Publish acknowledgements also depend on the selected acks mode, replication quorum, and HWM path. Test broker kill, host restart, and leader failover when changing durability-related settings.
Larger segments reduce file turnover but lengthen worst-case active-tail scans and increase retention granularity. A smaller sparse-index interval speeds seeks while increasing index writes and file size. Segment rolls also occur on index capacity or time.
Raw storage reads use the sparse index followed by mmap scanning. Closed segment mappings are retained in a bounded, reference-counted LRU cache, while the growing active segment is mapped per read so its visible length cannot become stale. Retention, compaction, truncation, and shutdown invalidate cached mappings before replacing or removing files.
read_committed adds HWM/LSO checks and transaction visibility filtering. Its append-maintained in-memory transaction index is read in place, so each poll does not clone the full index or rescan an accumulated result set. Resolved entries below the retention floor are pruned; unresolved transactions remain indexed and can hold the stable boundary at or above that floor.
Large poll batches reduce command overhead but increase handler latency and retry work. Stream batches reduce reconnect overhead but still require heartbeats, generation fencing, and offset commit discipline.
Partition signals are monotonic broadcast generations, so one append wakes every stream waiting on that partition instead of being consumed by only one waiter. One manager-owned scheduler drives periodic catch-up polls, keepalives, and timeout checks for all active streams; streams do not allocate independent polling or monitor tickers.
Partition count is the main unit of producer, consumer, and disk parallelism. Too few partitions cap concurrency; too many create files, goroutines, coordinator assignments, metrics, and replication work. Keyed records preserve ordering only inside the selected partition. Increasing partition count can remap future records for the same key.
A broker transaction adds:
read_committed visibility checks.Batch related records and offsets into a meaningful transaction, but avoid unbounded transactions: an open transaction holds the stable visibility boundary on touched partitions. All staged source offsets must belong to one consumer scope.
Coordinator synchronization currently persists the complete staged transaction snapshot after each mutation. Very large transactions therefore amplify standalone journal and distributed metadata traffic; keep transaction batches bounded and measure coordinator latency as well as partition throughput. A standalone encoded snapshot record is limited to 32 MiB. The standalone journal is append-only and does not yet compact superseded snapshots automatically.
Standalone keyed compaction scans a snapshot of closed segments twice before rewriting only segments that contain removable records. Cleaner memory is proportional to the number of distinct keys and producer IDs in that closed snapshot, not just the active segment. Temporary disk demand can approach the retained size of the largest segment being rewritten.
Producer appends continue during scanning and temporary-file construction. The partition metadata lock is held only while a prepared log/index pair replaces the closed segment and the directory entry is synced. Larger segments reduce roll frequency but increase cleaner scan time, peak temporary space, and the delay before an active key update becomes eligible.
channel_buffer_size: 10000
disk_flush_batch_size: 500
linger_ms: 10
disk_flush_interval_ms: 250
log_segment_bytes: 1073741824
log_index_interval_bytes: 8192
channel_buffer_size: 1024
disk_flush_batch_size: 10
linger_ms: 2
disk_flush_interval_ms: 100
log_index_interval_bytes: 4096
These are starting points, not safe production defaults. Validate storage latency, recovery time, quorum behavior, and power-loss tolerance.
TXN_STATUS (state-count and committing-age metrics are not exported yet),Run benchmarks with a cold and warm storage cache, retained history, the intended isolation level, and both standalone and cluster modes. A run is valid only when publish count, unique consumed count, missing count, duplicate count, commit errors, and process exit status agree. Container build/compose noise should be separated from the benchmark summary, but broker/consumer error lines must remain available for diagnosis.
See Benchmarks and Benchmark Verification.