Workloads this design serves¶

SharedData sits in the same workload category as tick stores such as kdb+: capture records and time series once, keep the working set mapped for many local consumers, and serve remote clients over a network API. What it adds on top of the store is the control plane described in orchestration — the same system schedules and supervises the programs that produce the data.

The patterns below illustrate how applications can combine long-running collectors, scheduled batch jobs and dependency ordering across worker hosts. They describe reusable capabilities rather than the size or topology of a particular installation.

Workload stages: capture, normalize, derive, decide and operate; batch dependencies wait for completion, while realtime dependencies wait for running routines.

Real-time capture¶

Collector routines attach to exchange, broker and vendor feeds — sockets, websockets, streaming APIs and HTTP polls — and write ticks, quotes, trades or snapshots into mapped tables or append-only streams. Bar-building routines consume the same data and publish aligned time series. Because the store is shared page cache, a collector and ten readers on the host see the same records without copies, queues or a serialization step between them.

Realtime path: a collector writes a stream or the mapped table; an application consumer writes stream data into the store; local readers share pages and remote clients use the API.

The stream-to-mapped-container consumer in this diagram is application code. The built-in SharedData.IO.StreamsPersist writes partitioned collections; SharedData.IO.StreamsCache updates the cache.

Scheduled ETL chains¶

File drops and vendor APIs rarely arrive ready to use. Batch routines form download → parse → normalize → publish chains: one routine fetches the files, the next extracts records, the next resolves them against the instrument master, the next builds the adjusted series. Dependencies, not wall-clock timing, order the chain — a slow download delays its parsers instead of letting them read stale input.

Consolidation into one canonical dataset¶

When several vendors cover the same instrument, consolidation routines vote or pick a preferred source per field and write one canonical record set: symbols, calendars, corporate actions, quotes. Downstream work reads the canonical set and never needs to know which vendors were running that day.

Derived analytics¶

Factor, mark-to-market, curve and surface jobs read the canonical data and write derived containers — value-at-risk inputs, implied-volatility surfaces, carry measures. They are ordinary batch routines with dependencies, so they wait for their batch prerequisites to complete, including late arrivals. Recomputation after a completed run needs another scheduled run or an explicit restart; changes to input containers do not trigger it.

Signals, allocations and backtests¶

Strategy routines consume factors and write signals; allocation routines turn signals into target portfolios; backtest jobs replay the same logic over history. The dependency graph keeps the order strategy → allocation → portfolio → backtest. Every step can read SharedData containers, but mapped records remain mutable: reproducible backtests need application-managed input snapshots or versioned datasets.

Account and order synchronization¶

Routines that talk to brokers or order-management endpoints keep local position, order and account containers in step with the external system — both as scheduled reconciliations and as realtime streams. Because positions are just containers, risk and reporting routines read them like any other dataset.

Backup, archive and retention¶

The data is the asset, so the schedule protects it with the same machinery: snapshot routines copy containers to object storage, a replication routine mirrors them offsite, and retention jobs prune or roll older partitions. Each step depends on the previous one completing, so an offsite copy never runs against a half-finished snapshot.

Serving humans and agents¶

The API server, websocket endpoints and web pages are realtime routines too — the Scheduler watches its own infrastructure. Humans get dashboards for schedules, logs, processes and data browsing; agent clients use the authenticated API or MCP to answer data questions against the same permissions model.

Where this is not the right tool¶

Distributed shared memory. Mapped pages are per host. Cross-machine sharing goes through the API and streams, with serialization.
Ad-hoc SQL analytics at warehouse scale. The store is record- and matrix-oriented; it is not a query planner over columnar files.
Transactional snapshots across records or containers. Mapped writes do not provide a transactional snapshot of a table or an atomic write across containers. Coordinate consistent reads and recovery in the application; see reader guarantees.
A managed service. You run the fleet; the project provides the engine, the control plane and the rules that keep them coherent.

If those limits fit, the combination — one mapped working set plus one dependency-aware scheduler watching a fleet of workers — replaces the usual patchwork of cron, message queues and bespoke supervisors with a single system whose state is itself queryable data.