Keep your data close to your compute.

SharedData is an open-source store for records and time series — in the workload category of tick stores such as kdb+ — plus the scheduler and worker fleet that produce the data. Processes on the same host share memory-mapped files through the operating system’s page cache; an HTTP API connects remote clients; a dependency-aware scheduler keeps hundreds of routines in step across the machines you control.

from SharedData.SharedData import SharedData
import pandas as pd

store = SharedData(__file__, user="example")
index = pd.MultiIndex.from_tuples(
    [(pd.Timestamp("2026-01-02"), "EXAMPLE")],
    names=["date", "symbol"],
)
rows = pd.DataFrame({"close": [100.0]}, index=index)
records = store.table(
    "MarketData", "D1", "DEMO", "PRICES",
    value=rows,
)
print(records.loc["2026-01-02", "EXAMPLE"])

Records. Structured NumPy arrays with primary-key indexes.
Time series. Memory-mapped matrices with timestamps and named columns.
Remote access. Authenticated HTTP and MCP interfaces connect authorized clients.

Illustrative data and namespace. The quick start sets up isolated storage before running this example.

Why it exists

Data producers, analysis and applications often need the same records. Repeated serialization and separate in-memory copies add work between them.

SharedData maps the file into each local process. Hot record pages are shared by the operating system. Indexed writes coordinate through a per-table process lock; remote access uses the HTTP API.

Your data stays on the storage you choose.

A store and the fleet that fills it

A routine is any program the Scheduler owns: a feed collector, a parser, a consolidation job, an API server. A schedule lists the routines that should stay alive or run at their times. Commands travel through a per-host worker pool that validates the requested source revision before every launch; routines report heartbeats and lifecycle messages through the logging plane; the Scheduler writes queryable state into the store.

Sources feed routines on worker hosts; the server hosts the API, scheduler and mapped store; consumers read locally or remotely.

One day of work is a dependency graph

A schedule can combine long-running collectors and scheduled batch jobs across worker hosts and source repositories. Batch routines wait for their batch dependencies to complete; realtime routines wait for their realtime dependencies to be running.

routines scheduled per day
realtime daemons kept alive
batch jobs on dependency edges
dependency edges in the schedule

A day of scheduled work as a dependency graph across five stages: capture, normalize, derive, decide and operate. — Capture, normalize, derive, decide, operate — ordered by dependencies, not wall-clock guesses.

What it gets used for

Realtime capture: Feed collectors write ticks and quotes to mapped tables; readers share the pages.
ETL chains: Downloads, parsers and consolidations ordered by dependencies, not wall clock.
Canonical data: One record set chosen across vendors; downstream never sees the merge.
Derived analytics: Factors, marks and surfaces run after their batch prerequisites complete.
Signals to backtests: Strategy, allocation and backtest pipelines over shared containers.
Ops and agents: Backups, dashboards and MCP clients on the same control plane.

Generalized from a live deployment — read workloads this design serves and how the scheduler and worker pool run them.

Install

Python 3.11 or 3.12. See the manual for native requirements and platform limits.

python -m pip install shareddata

Start with the quick start and configuration guide. Local records do not require a remote database or cloud account. Configure optional services when your application needs them.

Tables: Row-oriented records, indexed upserts and append-only realtime data.
Time series: Float matrices with a timestamp index and named columns.
HTTP API: Authenticated container access and semantic data queries.
Scheduler: Dependency-aware schedules; realtime daemons restarted on silence.
Workers: Tracked routines and batch jobs, one worker per host.
Integrations: MongoDB collections, Redis caches, Kafka streams and S3 storage.
MCP: A data interface for compatible agent clients.

Two shapes, one local working set

A table serves keyed, mixed-type records. A time series serves aligned numeric matrices. Both reuse mapped files instead of loading a separate copy of the dataset for each local process.

See table capacity and performance for local memmap results, observed HTTP latency and payload rates, and the strengths and limits of WebSocket and TCP table synchronization. The manual separately reports synthetic local-container p50/p95 measurements with a reproducible benchmark.

Platform and storage guarantees

Memory mapping shares data within one host; it does not create distributed shared memory. Visibility in another process and durability on disk are separate guarantees.

Linux and Windows use different rewrite paths. Read the storage guarantees and Windows limitations before choosing a deployment.

Project resources

Documentation	SharedData manual
Package	SharedData on PyPI
Contact and source	Public channel availability