SharedData mmap + index lock

SharedData uses a page-cache record store with an in-place hash index. This page describes the storage model and coordination required by indexed operations.

1. What SharedData is

A disk table is one file of row-oriented structured records, mapped shared into the process:

data.bin
  [Header: hdrdtype]   count, recordssize, isidxcreated, hasindex, …
  [Records: recdtype × recordssize]

See Table and TableDisk.malloc() (src/SharedData/TableDisk.py): np.memmap header then np.memmap records at the header offset.

MAP_SHARED means every process’s PTE points at the same page-cache pages. A store is visible to other local processes without msync. msync / flush is durability, not IPC. When the page is resident, access is RAM; when it is not, the kernel faults from local NVMe. That is the speed contract. A second copy (immutable part, WAL replay, sealed Plasma object) is extra IO, not sophistication.

Indexes are more mmap files next to the table (TableIndex.py):

• pkey.bin — int64 hash table
• dateidx.bin, symbolidx.bin, portidx.bin — last-slot hash + prev-link lists

Lookups and upserts are Numba JIT over those arrays (get_loc_*_jit, upsert_*_jit).

This is not columnar. Timeseries tags are a separate Fortran-order matrix (TimeSeriesDisk). Do not collapse the two.

2. Two planes

Plane	Surface	Synchronization
Data	records[:count], hdr['count']	Lockless. Publish count after the rows are stored.
Index	pkey + chain arrays + isidxcreated + rememmap	Exclusive per-table process lock.

get_loc / upsert / create_index / sort_index are index operations. They take the mutex. That is required: create_index() and index.initialize() relocate the mmap arrays (see comments on SharedNumpy.get_loc and SharedNumpy.upsert). A walker holding a stale pkey view is use-after-remap. Upsert writes hash slots and prev-links in place; a concurrent probe can miss or follow a torn chain.

hasindex=False tables (typical append-only live feeds) never enter the hash. extend appends rows and publishes count. That lock is writer-serialize of the header, not an index lock.

The data plane is already lockless; “adding lockless readers” would solve nothing.

3. The mutex

Implementation (SharedData.acquire / release, src/SharedData/SharedData.py):

• One shm object per table: {pid, type, isloaded}
• CAS via cpp.long_compare_and_swap (Python and C++ use the same word)
• Waiters spin with time.sleep(0.000001)
• Steal if psutil.pid_exists(holder) is false (after 1 s / 15 s)
• SCHEMA only: abort after 30 s (TimeoutError) so a stuck holder does not hang every table() open

Table.acquire() refreshes self.pid = os.getpid() on every call (Table.py, TimeSeriesDisk.py, TimeseriesContainer.py), then passes that pid into SharedData.acquire. After fork(), the first acquire claims with the child pid. The mutex is still non-reentrant.

This is a process lock. It matches the model: one OS process owns the index critical section. CPython 3.12 GIL + Numba @njit without nogil=True already serializes Python/Numba upsert inside one process. Gunicorn gthread --nthreads N overlaps socket I/O; it is not N parallel hash writers.

SharedData.mutex() acquires as a side effect. Table.__init__ holds that lock until finally: release(). Everything reachable from init (malloc, create_index, read_header, …) runs with the lock already held. A second acquire() on the same table self-deadlocks (the CAS waits for pid==0; the field holds our own pid). That is documented on SharedData.acquire as non-reentrant.

Brief-hold contract (same docstring): the mutex protects one upsert, one get_loc, one init. Do not hold it across slow I/O. SCHEMA is special because every table open resolves its folder via schema.get_loc().

4. Why textbook alternatives do not fit

Each of these recommendations fights the paradigm of this store:

Recommendation	Why it is wrong here
Replace tables with ClickHouse MergeTree / kdb+ HDB parts	Scan engines. Extra files, extra IO. You already have the bytes in RAM.
LMDB-style COW pages as the data model	Different unit of mutation (page vs record). Extra copy on write.
Arrow Plasma create/seal	Extra copy. Removed from Arrow 12.
Make get_loc lockless / RCU hash	Requires atomic pointer publish of chain links and epoch before freeing the old mmap. That is a new index, not a faster use of pkey.bin.
Rewrite the mutex as pthread + thread-id re-entry	Wrong grain. Process lock is the contract with C++. Init holds the lock on purpose.
--nthreads 1 as the architecture	Only removes an in-process GIL-drop race. Does not fix grow-under-lock or SCHEMA fork.

A sound concurrent-hash design for an in-place chained table that relocates on rebuild is a coarse exclusive lock around every index op. Fine-grained / RCU / relativistic hashes are a different structure (Linux rhashtable, Triplett/McKenney, Erlang ETS DB_FINE_LOCKED).

5. What is actually wrong

These sit next to the design. They do not replace it.

5.1 Index lock held across grow

SharedNumpy.upsert acquires, then may call auto_grow → Table.malloc() / posix_fallocate (about 100 MB, page-aligned), then isidxcreated = 0 and index.initialize() which remaps every index array — still under the same acquire (src/SharedData/SharedNumpy.py).

SharedNumpy.extend (hasindex=0) acquires, then posix_fallocate + np.memmap resize, then insert(..., acquire=False).

Every waiter that needs the index (or the extend lock) spins at 1 µs for the duration of a syscall. That is the one place the design leaves RAM speed. Grow is capacity. The lock is for index bytes (or a short count publish).

Textbook for this lock: drop or never take it across fallocate; pre-size so grow is rare; rememmap index after grow under a short re-acquire. Boost.Interprocess managed_mapped_file::grow is explicit: grow is not a concurrent operation.

5.2 Gunicorn freezes SCHEMA’s locker pid

src/SharedData/API/ServerGunicorn.py:

flask_app = create_app()   # module import
…
"preload_app": False,
GunicornEmbedded(flask_app, gunicorn_opts).run()

create_app() constructs SharedData('SharedData.API', …). If DATABASE_FOLDERS has more than one entry, SharedData.__init__ calls init_schema() and opens the SCHEMA table. That sets Table.pid to the parent pid.

preload_app=False does not reopen the app in each worker: the parent already imported the module and built flask_app. Workers fork and inherit the Table objects, including self.pid.

Without a pid refresh, acquire(self.mutex, self.pid, …) in a worker still presents the master’s pid. All workers look like one SCHEMA owner.

The implementation sets self.pid = os.getpid() in acquire() before the CAS; the process lock stays the contract with C++ and create_app() stays at module import.

5.3 Steal by pid_exists, no generation

Dead holder → CAS holder → me. No generation. A recycled PID can look like a live holder or be stolen from. A live stuck holder on SCHEMA wedges the machine until the 30 s abort — that abort is damage control.

Same process-lock grain, better identity: robust futex (EOWNERDEAD) and/or a generation on the shm word. Userspace psutil is not that.

5.4 Date-loc helpers skip the lock

SharedNumpy.get_date_loc acquires (and may create_index). get_date_loc_gte / get_date_loc_lte / get_date_loc_gte_lte walk datelastidx / dateprevidx with no acquire.

If another process can rememmap those arrays (upsert auto-grow does), that is the stale-pointer bug already documented for pkey. Either those helpers take the index lock, or they are restricted to tables that never rebuild after first create.

5.5 Ops around the lock

• Many concurrent processes plus a C++ writer on one indexed table is one exclusive queue. Correct exclusivity; a writer-topology cost, not a broken mutex. hasindex=False + one writer (or short count CAS) is the append path.
• Supervisor orphans that keep writing the same file are two mutators that ignore the design.
• trim() / os.replace does not retarget live maps. Anyone still mapped is on a ghost. Closed dates should be treated as immutable maps; compaction is a new file + epoch, then old maps die.

6. Code map

Concern	Where
Process lock, brief-hold, SCHEMA 30 s	src/SharedData/SharedData.py — acquire, release, mutex, _is_schema_relpath
Pid refresh on acquire	Table.py / TimeSeriesDisk.py / TimeseriesContainer.py — acquire()
SCHEMA open at API start	src/SharedData/SharedData.py — init_schema; API/__init__.py — create_app; API/ServerGunicorn.py — import-time create_app()
Index lock on probe / upsert	src/SharedData/SharedNumpy.py — get_loc, upsert, extend
Remap on rebuild / grow	src/SharedData/TableIndex.py — create_index, rememmap, initialize; SharedNumpy.auto_grow
Date/symbol loc lock	SharedNumpy.py — get_date_loc*, get_symbol_loc, get_portfolio_loc, get_tag_loc (acquire=)
Layout	Table; src/SharedData/TableDisk.py

7. If you change this

• Keep the two planes. Do not lock the data plane. Do not unlock the index plane.
• Keep the process lock shared with C++.
• Keep grow on the index lock (fallocate stays inside the hold).
• Do not open SCHEMA (or any table) before Gunicorn fork, or refresh Table.pid on acquire.