Skip to content

Table and TimeSeries on Windows

Table and TimeSeries operations are available on Windows. Linux behaves exactly as before: whatever Windows needs differently sits behind TableRewrite.rewrite_supported(), which is true only where the staged, crash-safe rewrite can run (Linux POSIX file APIs).

Why Windows is different

Linux publishes a rewrite as a complete stage file installed with one os.replace over the live file. Windows refuses to rename, delete or truncate a file that any process has mapped, and every worker and routine keeps the table files mapped. So on Windows nothing is replaced: the shared Table mutex is taken in STRUCT mode, the live files are rewritten in place through mapped views, files are only ever extended, and the indexes are rebuilt.

Operation Linux Windows
sort_index, explicit key cleanup stage file + os.replace rows permuted in the mapped image, index rebuilt
overwrite=True on an existing table stage + os.replace header and committed rows zeroed in place, index and BSON files kept
index files (pkey/dateidx/symbolidx/portidx) growing tmp file + os.replace file extended in place; maps use the table's capacity, an oversized file is harmless
reload() (tail or whole image) stage built beside the live file, then os.replace remote data checked in a private file first, then copied into the live files through mapped views
BSON compaction (defragment_bson) pair published as BSON + records compacted stream copied over the live BSON, bson_ptr rewritten in place
TimeSeries extend rows / delete columns / V1 migration temp file + os.replace columns moved inside the same file (last to first when growing, first to last when shrinking)
trim() shrinks the file skipped (warning): only returns unused space
Table / timeseries deletion files removed removed with a retry for this process's own mappings

While an in-place operation runs, readers wait (STRUCT) instead of reading, as before 6.85.

Known limitations on Windows

  • Not crash-safe. A process or machine that dies inside an in-place rewrite leaves the table in STRUCT/DIRTY. The next owner rebuilds the indexes but cannot undo a half-moved range of rows; a half-permutation can leave duplicate keys, which the index rebuild then refuses ("explicit cleanup required"). The staged publication that recovers from this exists only on Linux. Windows also forgets STRUCT/DIRTY when every process has exited (the shared mutex dies with its last handle), so a restart cannot detect an interrupted rewrite.
  • Files do not shrink while another process maps them. BSON compaction and trim give the space back only if no other process maps the file; otherwise the file keeps its size. Compaction still zeroes the stale bytes behind the new end so the upload stays small, and readers use the sizes in the header.
  • Raw views are not stable across a structural operation. A NumPy view or a DataFrame taken before a sort, cleanup, reload, overwrite, compaction or timeseries relayout may show moved data; take a new view afterwards. Lookups (loc, get_loc) are re-read when the generation changes.
  • Physical deletion needs every holder to let go. delete_table / delete_timeseries fail with the operating system's error while another process still maps one of the files, and also while the calling process still holds an array of the table: free() cannot unmap a file under a live array, so drop the reference (del tbl) before deleting.
  • os.fsync of a mapped file needs a writable handle; the library opens one. FlushViewOfFile does not promise the data reached the disk, only that other mappings see it.
  • The native lock (cpp CAS) is built for Python 3.11 on Windows. Other Windows interpreters need their own build.
  • Staged crash-safe tests (fork-injected process death, stage-file identity, /proc) run on Linux only; each operation has a Windows test in tests/test_windows_rewrite_fallback.py, which also runs on Linux with rewrite_supported() forced to false.
  • Outside this platform's scope: reload_remote() of a disk table (refused on every platform) and header migration of legacy disk tables (copy_file, refused on every platform) behave the same on both platforms.