Container API latency benchmark (synthetic DEMO data)¶

Observed read/write latencies for SharedData's two disk container APIs — indexed tables (Table records, memmap-backed) and time series tags (TimeseriesContainer / TimeSeriesDisk) — produced by scripts/benchmark_public_containers.py from this repository on fully synthetic, domain-neutral DEMO data.

What this is not. Not an SLA and not a production benchmark. It measures one single-process, warm-cache path on a disposable temporary storage folder. Cold cache, concurrent writers/contention, network or remote storage, and fsync-level durability are out of scope. Latencies are always reported as a pair (p50 and p95); a single "latency = X" number would say very little here.

Downloadable sanitized reports of the two runs below: run 1 · run 2.

What is measured (actual API surface)¶

Operation key	Actual call being timed
`table:loc_single_key`	`records.loc[key]` — primary-key read on a `(symbol)`-keyed `Symbols/D1` table
`table:loc_date_range_for_symbol`	`records.loc[(slice(start, stop), symbol)]` — date-range read on a `(date, symbol)`-keyed `MarketData/D1` table
`table:upsert_single_record`	`records.upsert({...})` — single dict upsert (update/insert)
`table:upsert_batch`	`records.upsert([…100 dicts…])` — batch upsert, mmap-visible only (no persist)
`table:upsert_batch_plus_write`	`records.upsert(batch)` plus the explicit `records.write()` persist
`timeseries:loc_single_date`	`frame.loc[date]` on the `pd.DataFrame` returned by `SharedData.timeseries(...)`
`timeseries:loc_date_list`	`frame.loc[[…60 dates…]]` (list selection)
`timeseries:loc_date_slice`	`frame.loc[start:stop]` (label slice, 125 rows)
`timeseries:write_values_batch_flush_false`	`TimeSeriesDisk.write_values(index, columns, values, add_columns=False, flush=False)`
`timeseries:write_values_batch_flush_true`	same with `flush=True` (adds an `msync` of the mapping)

Nothing else runs inside a measured interval: the value blocks and the write_values date-index slice are materialised beforehand, warm-up runs before the first sample, and every check (assertions, checksums, read-backs) happens after the stop timestamp of its own sample. The timings therefore describe the public path a caller experiences, not a raw mmap micro-benchmark.

Method¶

Isolation. The command you run is a pure-stdlib parent that spawns a child interpreter with a sanitized environment (fixed allowlist) and a disposable temporary storage folder (DATABASE_FOLDER/DATABASE_FOLDERS pinned there, S3 disabled, logging off). SharedData is never imported in the parent.
Child guards (self-checked, best effort). Before importing SharedData, the child installs a no-op dotenv stub (so no .env / ~/.shareddata.env file is consulted) and an in-process socket guard that blocks connect/connect_ex, create_connection, getaddrinfo, gethostbyname, gethostbyname_ex, gethostbyaddr and connectionless sends (sendto/sendmsg) on every non-local socket family; only local AF_UNIX remains. The self-check aborts the run if any patched entry point is still reachable, and performs a real AF_UNIX round-trip so "local traffic still works" is tested rather than assumed. The JSON report carries a guards: {dotenv_blocked, outbound_sockets_blocked, sharedata_import_after_guards} record of what was self-checked. The guard lives inside the child process — it is best effort, not a kernel-level sandbox.
Storage. By default the run creates its own temporary folder. --storage is accepted only if it resolves — after every symlink in every ancestor — to a fresh or empty directory inside the system temp directory; populated folders, files and symlinks are refused. The rule is positive ("what counts as disposable scratch"), so it never needs to name particular mount points or disk devices.
Data. Domain-neutral synthetic values (seeded NumPy RNG) under a DEMO namespace: 5 000-row (symbol) records table (capacity 10 000), 10 000-record (date, symbol) history table over 500 business days × 20 symbols (capacity 20 000), and one D1 time series tag with 8 channel_NN columns. The tag uses a recent-year startDate (1 January of the previous year) so the pre-created D1 index spans 523 rows — modest and bounded on purpose (intraday periods such as M15/M1 pre-create huge grids and are out of scope).
Timer and statistics. time.perf_counter_ns, reported in microseconds. Quantiles are nearest-rank, non-interpolated: the p-th percentile of n ascending-sorted samples is the value at index ceil(p/100·n) − 1 (0-based).
Setup/JIT excluded. Container creation, seeding, index building, JIT compilation and a warm-up phase (max(3, samples//5) calls per operation) happen before any sample is taken; the whole setup phase is reported separately as setup_seconds.
Validation. Every sampled read is compared against the seeded expected values and every sampled write is read back and compared — after the stop timestamp of its own sample; a mismatch aborts the run. The report's verification counters therefore equal the sample counts (per key for batch writes).
Flush vs durability. write_values(flush=False) provides shared-memory/page-cache visibility, not a durability promise; the flush=True variant additionally msyncs the mapping and is timed separately. Table write() is the persist step and is timed both with and without it. No fsync-level durability is claimed for any variant.

Reproduce¶

# from a checkout of the SharedData repository, in an environment with the
# SharedData runtime dependencies installed:
python scripts/benchmark_public_containers.py            # full bounded preset
python scripts/benchmark_public_containers.py --quick    # small smoke preset
python scripts/benchmark_public_containers.py --json-out build/report.json

The parent validates the child report — its sanitizer rejects (rather than silently cleanses) any report containing home/mount/disk paths, hostname, account, model or volume labels, credential-looking fields or host-specific tokens, which the test suite verifies — and removes the temporary storage on exit. Raw reports from the runs below live in the gitignored build/ folder; the published copies under docs/assets/ are the same reports after passing that sanitizer.

Observed results — 2026-10-31 (UTC)¶

Full bounded preset, seed 20261001, two independent runs back-to-back. Generic runtime facts only (no host identity): Linux x86_64, 32 CPUs, Python 3.12.3, NumPy 1.26.4, pandas 2.3.3, SharedData 6.86.0. Setup (seed + JIT + warm-up, excluded from samples) took 0.50 s / 0.51 s. All verification counters passed in both runs: 1160/1160 checked reads and 8950/8950 write read-backs (counted per key). Run 1 at 2026-10-31T23:48:54Z, run 2 at 2026-10-31T23:48:57Z.

Operation	batch	samples	p50 µs (run 1)	p95 µs (run 1)	p50 µs (run 2)	p95 µs (run 2)
`table:loc_single_key`	1	400	46.89	77.28	47.72	72.61
`table:loc_date_range_for_symbol`	125	120	195.56	314.58	203.08	261.61
`table:upsert_single_record`	1	300	79.87	122.17	82.22	121.41
`table:upsert_batch`	100	60	225.38	296.95	191.18	250.55
`table:upsert_batch_plus_write`	100	25	6132.67	6328.46	6060.99	7277.40
`timeseries:loc_single_date`	1	400	7.48	8.64	9.08	14.13
`timeseries:loc_date_list`	60	120	147.27	163.90	170.08	268.64
`timeseries:loc_date_slice`	125	120	12.74	19.80	12.90	14.51
`timeseries:write_values_batch_flush_false`	100 rows × 8 cols	100	78.22	107.45	82.28	120.08
`timeseries:write_values_batch_flush_true`	100 rows × 8 cols	50	2340.77	2516.82	2293.26	2931.98

The machine was not dedicated to benchmarking. These numbers describe the measured environment, not fixed properties of the API; this experiment does not identify the cause of timing variation. Storage layout and device identity are intentionally omitted, so the results cannot support hardware comparisons.

Reading these numbers¶

Point reads are the cheapest path. Single-key table reads and single-date series reads sit in the tens-of-microseconds range or below.
Contiguous is cheaper than scattered. A label slice (125 rows in one region) costs far less per row than a 60-date list selection or a secondary-key range read, which each do more index lookups and per-row gathering.
Visibility and persist are different operations. The batch upsert alone (mmap-visible) and the batch upsert plus records.write() are timed and reported separately, as are write_values(flush=False) and flush=True; in these runs the persist/flush variants are an order of magnitude above their visibility-only counterparts.
Storage, scheduling and cache state feed into the persist/flush rows; these runs do not attribute the cost to any one component, and agreement between two adjacent runs does not establish repeatability across machines. Even those rows remain non-durable claims (no fsync).

What the report contains¶

The JSON report (schema: shareddata-public-benchmark/1) carries: a UTC timestamp, generic software versions (Python/NumPy/pandas/SharedData, OS family and CPU count), dataset sizes and seed, the method description (quantile definition, timed-scope, flush semantics), per-operation batch/sample counts, the verification counters, and the guards record. The parent-side sanitizer is a rejection gate — tested against the identity of the machine that runs it, so a report carrying private paths, hostname, account, model or volume labels, credential-looking fields or host-specific tokens fails instead of being published — which is why the published reports carry measurements only.

Not covered¶

No SLA or SLO is implied. Not covered: cold page cache (no cache flush is performed, on purpose), multiple concurrent writers/readers or lock contention, network-attached storage, remote (S3) synchronization, the HTTP API layer, intraday periods (M15/M1), and fsync-level durability. Use these numbers to characterize the shape — read vs write, point vs range, visible vs persisted — of the container API on synthetic DEMO data, on one machine, in one process.