Data by symbol: the semantic data API

The low-level API addresses a container: database/period/source/tablename, plus its type and, for intraday data, its partition. The semantic data API addresses the data: bars of EXAMPLE at 15 minutes since Monday. The server finds the source, the container and the partitions, and says in the answer where the rows came from.

Three entry points run the same code (SharedData.DataAPI.service.dispatch):

Entry point	Example
HTTP	GET /api/data/marketdata/EXAMPLE?period=M15&start=2026-10-05
Client	ClientAPI.get_marketdata('EXAMPLE', period='M15', start='2026-10-05')
Local	shdata.dataframe('marketdata', 'EXAMPLE', period='M15', start='2026-10-05')

This is a convenience, not the fast path. Each call resolves the dataset again and copies the rows it returns. A producer, a backtest or a realtime consumer keeps using shdata.table(), shdata.timeseries() and shdata.collection() directly.

It never writes: no row, no container, no index, no file.

Operations

Route	Client method	Answers
/api/data/sources	get_data_sources	sources, their kind, what each publishes
/api/data/symbols	get_data_symbols	instruments one product actually holds
/api/data/symbol/{symbol}	get_data_symbol	identity of an instrument, its spelling per product
/api/data/marketdata/{symbol}	get_marketdata	bars and other market-data products; period=RT is the latest state
/api/data/signals/{symbol}	get_signals	signal matrices and allocation tables
/api/data/portfolios/{portfolio}	get_portfolios	performance, strategies, holdings
/api/data/riskfactors/{factor}	get_riskfactors	factor metrics, optionally its positions
/api/data/vol/{symbol}	get_vol	tenor, listed and strike surfaces, derived series

Parameters are listed in the API docs (/apidocs, tag Data). A parameter that takes several values (columns, metrics) accepts them comma-delimited (metrics=ret,open) or as a repeated key (metrics=ret&metrics=open); the two forms can be mixed.

/api/data/symbols lists the instruments of the newest date of the dataset that would answer. On a weekend that date holds only the small set of rows written then; date= asks for the instruments of another day.

What every answer carries

{"provenance": {"dataset": "master/MarketData/M15/SRC/table/QUOTES", "partitions": ["2026"],
                "source": "SRC", "source_kind": "consolidator", "store": "table", "semantics": "history",
                "symbol": {"query": "EXAMPLE", "canonical": "EXAMPLE@XDEMO", "alias": "EXAMPLE@XDEMO", "alias_form": "canonical"},
                "key": ["date", "symbol"], "fields": [{"name": "close", "dtype": "<f8", "unit": "price"}],
                "row_provenance": "source", "range": {"start": "...", "end": "...", "defaulted": false},
                "alternatives": [{"dataset": "...", "reason": "not needed"}], "rows": 2, "truncated": false},
 "data": [{"date": "2026-10-05T13:30:00", "symbol": "EXAMPLE@XDEMO", "source": "DEMO", "close": 1.0}]}

row_provenance names the column that holds the vendor of each row in a consolidated source. semantics is history for tables, timeseries and collections and latest for a cache: a cache keeps the last state of each instrument and no history.

How a request is resolved

1. Catalog. Every container the server can see is listed (names only, cached 60 s) and classified by naming convention into a category and a product. A container no convention covers stays visible as uncategorized.
2. Symbol. The registry gives the instrument's identity; each dataset says how it spells an instrument in its key, so the caller never builds EXAMPLE_1M or EXAMPLE_20260116.
3. Source. The configured preference decides; source= overrides it for one request. A store the configuration deprecates, or a table not written for stale_after_days, is tried last. When the preferred source has no rows for the request, the next one answers and the answer says so.
4. Partitions. The date range becomes partition names from the dataset's own format (none for daily data, a year for 15-minute data, a month for 1-minute data).
5. Permissions. A token reads through these routes only what it can read through the low-level route of each container.

Limits

Limit	Default	Configuration key
Rows of one answer	10,000 (at most 100,000)	default_rows, max_rows
Fields a listing shows per dataset (field_count has the total)	50	listed_fields
Range of a read with no symbol or entity	31 days	scan_max_days
Range of a strike-surface read	7 days	strike_max_days
Time of one collection query	20 s	query_seconds

A collection read always runs on the collection's key index. A collection without it is refused with status 409; python -m SharedData.DataAPI.indexes reports which ones lack it and --apply creates them (an operator's step: building an index on a large collection takes hours).

Configuration

SHAREDDATA_DATA_API_CONFIG names a JSON file; docs/data-api-config.example.json is an example. Without it no source is preferred and none is a consolidator. The file is read again when it is replaced; while it is missing or cannot be read, the configuration read last stays in effect and a warning is logged.

Key	Meaning
source_preference	{"marketdata": ["SRC"], "vol/tenor": ["SRC"]}: sources to try first, per category or per category/product
source_kinds	{"SRC": "consolidator"}: kind of a source, over what its datasets suggest
deprecated	dataset paths to use only when nothing else answers; they do not decide the period of a request that names none
stale_after_days	a table not written for longer ranks last (30)
registry_metadata, registry_collection	the instrument master file and the provider-symbol collection
canonical_fields, ticker_fields, alias_fields	registry columns that hold the registry symbol, the ticker and vendor aliases
riskfactor_sources	source names of the factor namespace
units	{"field": "unit"} over the built-in units