Rust edition: this page follows the live DeepWiki structure but treats the current Rust crates as implementation authority. Non-Rust surfaces are identified at their boundary and are not presented as Rust APIs.
Data System
Relevant Rust source files
crates/data/src/engine/mod.rscrates/data/src/client.rscrates/data/src/aggregation.rscrates/common/src/messages/data/mod.rs
The Data System is responsible for all market data ingestion, processing, aggregation, storage, and retrieval within NautilusTrader. This system orchestrates data flow from external sources (exchanges, data providers, historical files) through processing pipelines to consumers (strategies, actors, indicators). The architecture supports both live streaming and historical backtesting with the same interfaces.
For execution and order management, see Execution System. For strategy implementation that consumes this data, see Trading Components. For backtest-specific data handling, see Backtesting System.
Architecture Overview
The Data System follows a hub-and-spoke pattern with DataEngine as the central orchestrator. Data flows from DataClient instances through the engine to subscribers via the MessageBus. The system maintains state in Cache and can persist data to a catalog for historical analysis.
Core Components
Sources:
DataEngine - Central Orchestrator
The DataEngine is the core component responsible for managing multiple DataClient instances, routing commands, processing incoming market data, and managing bar aggregators and order books. It employs a fan-in fan-out messaging pattern to execute DataCommand messages and process DataResponse or market data objects
The native Rust data system is implemented by nautilus-data and consumes the shared messages and model types from nautilus-common and nautilus-model.
For details, see DataEngine Architecture.
Key Responsibilities
| Responsibility | Implementation |
|---|---|
| Client Management | add_client(), register_venue_routing() |
| Command Routing | execute(), _execute_command() |
| Data Processing | process(), _handle_data() |
| Request Handling | request(), _handle_request() |
| Subscription Management | _handle_subscribe(), _handle_unsubscribe() |
Sources:
Data Client Framework
The DataClient abstraction provides a uniform interface for connecting to various data sources. MarketDataClient extends this for venue-specific market data subscriptions. These clients bridge the gap between external venue protocols (WebSocket/REST) and the internal DataEngine processing. In backtesting, BacktestMarketDataClient provides simulated connectivity
For details, see Data Client and Adapter Framework.
Subscription Command Flow
When a strategy or actor subscribes to data, the command flows through the system:
Sources:
Actors and Component Lifecycle
Actor is the base class for custom components that consume and produce data. It provides a high-level API for data subscriptions, indicator registration, and state management. In Rust, the DataActor trait and DataActorCore provide similar functionality for native components
For details, see Actors and Component Lifecycle.
Data Event Handlers
Actors override event handler methods to receive data:
| Event Type | Rust handler |
| --- | --- | --- |
| Quote Ticks | on_quote_tick(tick) | on_quote_tick(&mut self, tick) |
| Trade Ticks | on_trade_tick(tick) | on_trade_tick(&mut self, tick) |
| Bars | on_bar(bar) | on_bar(&mut self, bar) |
| Order Book | on_order_book(book) | on_book(&mut self, book) |
| Custom Data | on_data(data) | on_data(&mut self, data) |
Sources:
Bar Aggregation System
Bar aggregation transforms raw ticks into OHLCV bars. Rust aggregators cover the aggregation modes implemented in crates/data/src/aggregation.rs; callers must verify a requested specialized mode against that module rather than assuming parity with another surface.
For details, see Bar Aggregation System.
Bar Builder Mechanics
The BarBuilder accumulates price/size updates and builds bars:
Sources:
Order Book Management
The Data System maintains real-time order books using OrderBook instances. It supports L1 (BBO), L2 (Market-by-Price), and L3 (Market-by-Order) data types. The system uses granular OrderBookDelta objects to update book state efficiently It also tracks OwnOrderBook for strategy-specific order tracking.
For details, see Order Book Management.
Delta Flags and Boundaries
Each OrderBookDelta carries flags using RecordFlag bitmask values to signal event boundaries:
F_LAST: Marks the final delta in a logical event group.F_SNAPSHOT: Marks deltas that belong to a snapshot.
Sources:
Data Catalog and Parquet Storage
NautilusTrader uses a Rust Parquet/Arrow catalog for historical storage and retrieval and supports additional Rust codecs such as Cap'n Proto and SBE. Custom persisted data uses the Rust persistence macro and a stable Arrow schema.
For details, see Data Catalog and Parquet Storage.
Catalog File Organization
Data is organized by type and identifier in a directory structure. The ParquetDataCatalog manages the physical layout and indexing of these files while the DataEngine provides methods to register and access these catalogs
Sources: