Rust edition: this page follows the live DeepWiki structure but treats the current Rust crates as implementation authority. Non-Rust surfaces are identified at their boundary and are not presented as Rust APIs.

DataEngine Architecture

Relevant Rust source files

• crates/data/src/engine/mod.rs

Purpose and Scope

The DataEngine is the central data orchestrator in NautilusTrader, implementing a fan-in fan-out messaging pattern to manage interactions between multiple DataClient instances and data consumers (strategies, actors, other engines).

Core Responsibilities:

• Fan-In: Receives DataCommand and RequestData messages from multiple sources via the MessageBus, routing them to appropriate DataClient instances based on venue or client_id
• Fan-Out: Receives market data from clients via process(), publishes to MessageBus topics where multiple subscribers (strategies, actors) consume the data
• Subscription Management: Tracks active subscriptions, manages bar aggregators for internally aggregated bars, and handles order book lifecycle
• Request Coordination: Manages request/response cycles including long requests (pagination) and join requests (scatter-gather across multiple clients)
• Synthetic Instrument Support: Updates synthetic instruments from component feeds, generating synthetic quotes and trades

This page documents the engine's routing logic, subscription mechanisms, aggregation integration, and message flow patterns.

---

DataEngine Position in System

The DataEngine sits between data sources (via DataClient adapters) and data consumers. It is instantiated and managed by the NautilusKernel and communicates exclusively through the MessageBus.

Diagram: DataEngine Fan-In Fan-Out Architecture

The selected provider view replaces generic exchange examples with the Rust adapter actually in scope. Client factories terminate at the standard DataClient boundary; DataEngine owns routing, aggregation, cache updates, and fan-out.

---

Core Component Structure

The native DataEngine is the Rust struct implemented under crates/data/src/engine/.

Responsibility	Implementation
Client Management	_clients dict maps ClientId to DataClient, _routing_map maps Venue to DataClient
Subscription Routing	Routes Subscribe* commands to appropriate clients based on venue or explicit client_id
Data Distribution	Receives data via process(), publishes to MessageBus topics for fan-out
Request Handling	Manages Request* command lifecycle including long requests and join requests
Bar Aggregation	Manages BarAggregator instances, feeds ticks to aggregators, publishes completed bars
Order Book Management	Buffers OrderBookDelta, manages snapshots at intervals

Configuration from DataEngineConfig :

• time_bars_interval_type: "left" or "right" for time bar bounds.
• time_bars_build_with_no_updates: Build bars even with no ticks.
• buffer_deltas: Buffer OrderBookDeltas before processing.
• emit_quotes_from_book: Generate QuoteTicks from book state.

---

Client Registration and Routing

Client Registration

Clients are registered via register_client() :

def register_client(self, DataClient client):
    self._clients[client.id] = client
    if client.venue is None:
        self._default_client = client
    else:
        self._routing_map[client.venue] = client

Command Routing Logic

The engine uses a three-tier fallback mechanism in _execute_command() :

Diagram: Client Routing Implementation Flow

Routing uses an explicit client ID when supplied, then a venue association, then a configured default client. A missing route is an actionable failure; it is not silently redirected to an unrelated provider.

---

Subscription Management

Subscription Flow

Diagram: Subscription Request Processing with Code Entities

Subscription commands travel from an actor through the message bus to DataEngine, which resolves the selected client and calls its typed subscription method. Returning QuoteTick, TradeTick, bar, book, instrument, and status values then follow the live path shown above.

---

Request/Response Handling

Simple Request Flow

Diagram: Request-Response Cycle

Bounded historical requests: Large Rust historical queries must use the request/client facilities actually exposed by the selected provider and apply an explicit time or cursor bound. Generator pagination from another API surface is not claimed here.

---

Bar Aggregation System

The DataEngine creates and manages BarAggregator instances when strategies subscribe to aggregated bar types

Aggregator Types

• TIME: TimeBarAggregator triggered by timer interval
• TICK: TickBarAggregator triggered by tick count threshold
• VOLUME: VolumeBarAggregator triggered by cumulative volume

Aggregation Logic: If bar_type.is_internally_aggregated() is true, the engine creates an aggregator and subscribes to underlying ticks/quotes to feed it The BarBuilder class provides the low-level logic for OHLCV updates

---

Order Book Management

Delta Buffering and Snapshots

The DataEngine can buffer OrderBookDelta updates and apply them in batch if buffer_deltas=True It also manages snapshots at regular intervals via subscribe_book_at_interval()

The Rust implementation uses BookUpdater and BookSnapshotter to handle these high-performance operations

---

Performance Considerations

• Topic Caching: Uses TopicCache to avoid repeated string formatting on hot publish paths
• Counters: Tracks command_count, data_count, and request_count for operational metrics
• Rust Core: High-performance implementations in crates/data/src/engine/mod.rs handle the same logic with zero-cost abstractions
• Deferred Commands: The Rust engine uses a DeferredCommandQueue to handle subscription requests that occur within data ticks, avoiding re-entrancy issues