Rust edition: this page follows the live DeepWiki structure but treats the current Rust crates as implementation authority. Non-Rust surfaces are identified at their boundary and are not presented as Rust APIs.

Event Sourcing and Event Store

Relevant Rust source files

• crates/event_store/src/capture/adapter.rs
• crates/event_store/src/writer/mod.rs
• crates/event_store/src/reader/mod.rs
• crates/event_store/src/verifier/mod.rs
• crates/event_store/src/bin/verify.rs
• crates/event_store/src/kernel.rs
• crates/system/src/event_store.rs
• crates/event_store/src/snapshot.rs
• crates/event_store/src/replay.rs
• crates/event_store/src/retention.rs
• crates/event_store/src/markers/mod.rs

This page documents the nautilus-event-store crate, which serves as the durable authority for all state-affecting history within NautilusTrader. It covers the core components like BusCaptureAdapter, EventStoreWriter, EventStoreReader, and various backends, explaining how they contribute to capturing, storing, and replaying the system's operational history. The design contract emphasizes the event store as the source of truth, with the cache acting as a write-through projection, enabling state reconstruction through replay of captured events.

Core Philosophy and Purpose

Event sourcing provides NautilusTrader with a durable, ordered record of messages that change the engine's state. The event store captures these messages at the system boundary, allowing readers, replay tools, and verifiers to reconstruct past states and understand system behavior.

The core philosophy is:

• The event store is the durable authority for state-affecting history.
• The cache is a write-through projection, not the source of truth.
• Cache replay rebuilds state by applying captured history to cache-owned state.
• Market data remains in the data catalog; the event store records messages that affect state.
• External I/O becomes replayable only when Nautilus captures it as commands, raw reports, or other state-affecting inputs.

This approach provides a durable basis to:

• Prove the integrity of a sealed run before replay or archiving.
• Inspect the exact command, report, and event sequence behind an order or component intent.
• Rebuild cache state from captured history, including a snapshot anchor and the run tail.
• Trace an intent through subsequent engine-side messages.
• Seal stale run files after a process exit or writer halt.

Key Terms

• Run: One kernel session for one instance, binary, and configuration.
• Entry: One captured message plus replay metadata.
• seq: The per-run sequence assigned by the writer, used as the replay order.
• High-watermark: The largest seq durably acknowledged by the backend.
• Snapshot anchor: The high-watermark recorded with a cache snapshot.
• Headers: Correlation and causation metadata propagated with captured messages.

What the Store Records

The event store records state-affecting message bus traffic for a single trading instance and run. A run begins when the kernel starts and ends upon clean process termination or crash.

Captured entries include:

• Execution commands (e.g., submit, modify, cancel).
• Data subscription commands.
• Fired time events and generated order, position, and account events.
• Raw venue execution reports before reconciliation.
• Reconciliation outputs.
• Request and response messages, or their audit-relevant metadata, that cross the bus and affect state.
• Run lifecycle entries such as RunStarted and RunEnded.

The store does not replace the data catalog, which continues to store market-data observations.

Capture Flow

Event capture occurs at the message bus dispatch boundary, before downstream handlers process the message. This ensures that any handler mutating state only sees messages already accepted by the event store.

The operational steps are:

1. A producer (engine, adapter, strategy, or component) publishes or sends a state-affecting message.
2. The bus capture tap builds an event-store entry.
3. The EventStoreWriter assigns the next seq, writes a batch, and advances the high-watermark after the backend acknowledges durability.
4. Downstream handlers run after the captured entry has reached the writer boundary.
5. EventStoreReader instances scan sealed or running backends without exposing append operations.

The writer uses a bounded channel. If it stalls past its configured threshold, Nautilus halts to prevent dropped entries or unaudited state changes.

Event Capture Flow

The event-store lane captures state-affecting traffic at the bus boundary and assigns durable sequence order before replay. It remains distinct from market-history projection and cache snapshots, which answer different recovery questions.

Event Store Components

The nautilus-event-store crate provides several key components:

BusCaptureAdapter

The BusCaptureAdapter (crates/event_store/src/capture/adapter.rs) acts as the seam between message bus dispatch and the event store writer. It intercepts messages published to the MessageBus and converts them into EventStoreEntry drafts for persistence. It uses an EncoderRegistry to serialize messages into EncodedPayloads.

EventStoreWriter

The EventStoreWriter (crates/event_store/src/writer/mod.rs) is responsible for the append path to the event store. It batches entries, advances the high-watermark, and signals fail-stop conditions. It operates asynchronously, receiving EntryDrafts via a channel and committing them to the EventStore backend. The writer ensures durability by only advancing the high-watermark after the backend acknowledges the write.

EventStoreReader

The EventStoreReader (crates/event_store/src/reader/mod.rs) provides read-only access to the event store. It supports range scans, point lookups, and is the primary interface for replay. It can scan entries within a seq range in either forward or backward direction.

Backends: RedbBackend and MemoryBackend

The nautilus-event-store crate supports different storage backends:

• RedbBackend: This is the default on-disk backend, using redb (a pure-Rust ACID key-value store) with one .redb file per run. Every commit uses Durability::Immediate to ensure data integrity, and the high-watermark only advances after a durable acknowledgment.
• MemoryBackend: An in-process backend used primarily for focused tests and simulation-style capture. It provides the same bus tap and writer semantics as RedbBackend but stores data in memory.

Verifier

The Verifier (crates/event_store/src/verifier/mod.rs) is a library surface for integrity checks over a single run. It performs checks such as recomputing entry hashes, detecting gaps in seq ordering, validating table-key and embedded-seq agreement, cross-checking secondary indices, and validating manifest status and high-watermark fields.

verify Binary

A standalone binary (crates/event_store/src/bin/verify.rs) for process-isolated verification of sealed run files. It delegates the scan to a worker subprocess to prevent a corrupted redb file from aborting the caller.

Run Lifecycle

The EventStoreLifecycle (crates/event_store/src/kernel.rs:17) manages the event store's operational lifecycle, including opening new runs, recovering from crashes, and sealing runs.

EventStoreLifecycleOptions

EventStoreLifecycleOptions (crates/event_store/src/kernel.rs:111) provides non-serialized, process-local configuration for advanced callers. It allows specifying a custom EncoderRegistry or a custom backend opener (e.g., MemoryBackend for testing).

open_run and recover_predecessors

Before a new run starts, the system scans for crashed predecessors. recover_predecessors (crates/event_store/src/kernel.rs:79) identifies and seals any runs that were left in a Running state, marking them as CrashedRecovered or Quarantined. The open_run function then initializes a fresh run, writing a RunStarted entry and setting the run status to Running.

seal

When a run concludes gracefully, the seal method (crates/system/src/event_store.rs:103) writes a RunEnded entry and updates the RunManifest to reflect the terminal status.

HaltSignal

The writer's halt callback is wrapped in a typed HaltSignal (crates/event_store/src/kernel.rs:22). This allows the kernel to poll for fail-stop conditions and initiate a graceful shutdown rather than a panic if the event store writer encounters an unrecoverable error.

Entry Format and seq Ordering

Each event-store entry (EventStoreEntry) is a captured message along with metadata.

The key fields are:

• seq: The per-run replay-order authority.
• ts_init: The domain timestamp on the captured message.
• ts_publish: The bus-accepted time.
• topic: The bus topic or logical endpoint.
• payload_type: The encoded message type.
• payload: The encoded message bytes.
• headers: Correlation and causation metadata.
• entry_hash: The canonical hash over the entry content.

The seq field dictates the replay order, and timestamps provide contextual information but do not override seq.

Note on Format Changes: Starting in v1.230, the event_store format has changed. Direct use of bincode for on-disk envelopes has been removed to improve robustness, and existing beta stores from v1.227-v1.229 must be regenerated.

Correlation Model

Nautilus records three levels of identity for correlation:

• correlation_id: The logical workflow or chain, often a component's intent_id.
• causation_id: The direct parent message that caused the current message.
• command_id, event_id, or report_id: The identity of the specific message.

Correlation Flow

Snapshot Anchors and Cache Replay

A SnapshotAnchor (crates/event_store/src/snapshot.rs:18) records the high-watermark of the event store at the time a cache snapshot is taken. This allows for efficient state recovery.

When restoring state, restore_cache_snapshot_and_replay_tail (crates/event_store/src/replay.rs:109) first restores the cache from a snapshot blob, then replays the event store entries from the snapshot anchor's seq up to the current high-watermark. This ensures that the cache is brought up to date with all state-affecting events that occurred after the snapshot.

The CacheReplayReport (crates/event_store/src/replay.rs:95) summarizes the outcome of a cache snapshot-tail replay, including the number of applied and ignored entries.

Retention Planning

The RetentionMode (crates/system/src/event_store.rs:133) defines how sealed run files are pruned. The kernel records this choice in the manifest, but actual retention enforcement is handled by a separate supervisor process.

Available retention modes:

• Full: Keeps every sealed run.
• Bounded: Keeps a specified number of the most recent sealed runs (keep_last).
• SnapshotAnchored: Keeps the manifest plus a snapshot anchor and the tail since the anchor; older entries are reclaimed once a newer anchor is durable.

The plan_redb_retention function (crates/event_store/src/retention.rs:100) provides a non-destructive planner for identifying sealed run files that are candidates for reclamation.

Data Markers

The markers module (crates/event_store/src/markers/mod.rs) provides a sidecar capture mechanism for data markers. These markers record the observed order of streaming data at the message-bus boundary as compact cursor snapshots, which can be joined back to catalog rows. No market-data payload is persisted within the event store itself.

Key components include:

• DataMarkerCapture: Captures data markers.
• DataMarkerExtractor: Extracts markers from messages.
• MarkerWriter: Writes markers to a MarkerBackend.
• MarkerReader: Reads markers.
• RedbMarkerBackend and MemoryMarkerBackend: Backends for storing markers.