Data Catalog and Parquet Storage
Relevant Rust source files
crates/persistence/src/backend/catalog.rscrates/persistence/src/backend/catalog_operations.rscrates/persistence/src/backend/session.rscrates/persistence/src/backend/custom.rscrates/persistence/src/parquet.rscrates/persistence/src/config.rscrates/persistence/macros/src/custom.rs
ParquetDataCatalog is the Rust catalog for columnar historical data. It organizes Nautilus model values by data type and identifier, writes Arrow record batches to Parquet, reads time-bounded data through DataFusion/object-store facilities, and exposes consolidation and deletion operations.
Storage model
Catalog paths encode the logical data family and optional identifier. Filenames encode timestamp ranges so the catalog can reason about coverage without decoding every file. Helpers such as timestamps_to_filename, parse_filename_timestamps, are_intervals_disjoint, and are_intervals_contiguous make those invariants explicit.
Paths are normalized for local and object-store use. Identifier sanitization is part of the storage boundary; raw provider symbols must not become unchecked paths or SQL identifiers.
Writes
The Rust catalog converts supported Nautilus values into Arrow batches and writes Parquet through the selected object store. Writes should preserve instrument identity, event and initialization timestamps, numeric precision, and schema metadata. Overlapping intervals require an explicit consolidation policy rather than last-writer-wins ambiguity.
Custom data uses the persistence macro and helpers under backend/custom.rs. prepare_custom_data_batch adds the data-type discriminator and identifier-aware path information. Decode functions reconstruct typed Nautilus Data values from record batches. A custom type therefore needs a stable schema and a deliberate evolution story.
Queries
DataBackendSession and build_query form the lower-level query path. The session orders results using initialization time and merges batches from multiple files. DataQueryResult exposes the result stream without requiring callers to load an entire catalog into memory.
Query predicates should include data type, identifier, and time bounds whenever possible. Infinite-history clients should page by a stable cursor derived from stored time and identity, not by mutable row offsets.
Consolidation and deletion
ConsolidationQuery describes compaction of multiple compatible fragments into larger files. DeleteOperation scopes deletion. Both are catalog operations with a larger blast radius than ordinary reads or appends; run them against immutable inputs and verify the resulting interval map before removing source fragments.
Object stores and configuration
create_object_store_from_path selects local or supported remote object storage from a normalized URI. DataCatalogConfig and StreamingConfig carry catalog/runtime configuration. Credentials remain deployment concerns and must not be embedded in catalog paths or metadata.
The Parquet catalog is an analytical and historical store. It does not replace the in-memory cache, event-store run log, or transactional database projection. Choose the authority for each datum explicitly: durable committed history, replay ledger, hot operating state, and disposable analytical files have different recovery contracts.