Skip to content

Schedules: where they live, and the SCHED page

The Scheduler keeps its schedules in two SQLite containers on the disk of the server it runs on (the API server's host). It reads and writes nothing else: a slow or absent object store cannot delay it.

What Container Key
Definition of schedule <NAME>: one row per routine Hashs/RT/SCHEDULES/<NAME> hash = routine (user@COMPUTER:Repo/folder/code, user@COMPUTER:SharedData.Module)
State of every day: one row per routine and day Text/D1/SCHEDULES/<NAME> date = the Scheduler's calendar day, hash = routine

Definition columns: name (group), isexternal, isrealtime, runtimes (HH:MM, several separated by commas; empty for a realtime routine), args, dependencies (routines, comma-separated), computer, seq (order). State columns: those, plus script, status, lastmsg, lastmsgts and the other message times (UTC), seq (run order).

They are ordinary containers: listed on the DATA page, served by /api/sqlite/..., and subject to the usual permissions. No address or name is treated specially.

Reading

from SharedData.IO.ClientAPI import ClientAPI
definition = ClientAPI.get_sqlite('Hashs', 'RT', 'SCHEDULES', 'MASTER', sort=[['seq', 1]])
today = ClientAPI.get_sqlite('Text', 'D1', 'SCHEDULES', 'MASTER',
                             query={'date': '2026-10-06'}, sort=[['seq', 1]])

On the server itself: SharedData.Routines.ScheduleStore.definition(name), .state(name, 'YYYYMMDD'), .names(), .days(name).

Editing a definition

ClientAPI.post_sqlite('Hashs', 'RT', 'SCHEDULES', 'MASTER',
                      [{'hash': 'user@HOST:Repo/code', 'name': 'GROUP', 'isrealtime': True, 'args': '--fast'}])
ClientAPI.delete_sqlite('Hashs', 'RT', 'SCHEDULES', 'MASTER', query={'hash': 'user@HOST:Repo/old'})

Or with a spreadsheet, on the server:

python -m SharedData.Routines.ScheduleStore export MASTER --xlsx master.xlsx   # sheet "static"
python -m SharedData.Routines.ScheduleStore import MASTER --xlsx master.xlsx   # replaces the definition
python -m SharedData.Routines.ScheduleStore list

As before, the Scheduler reads a definition when it starts and when the day changes; an edit takes effect then.

Coming from the metadata store

Schedules used to be the metadata SCHEDULES/<name> (definition) and SCHEDULES/<name>/<YYYYMMDD> (state), kept in the object store.

  • First start. A schedule that is not in the store yet is copied once from the old place. python -m SharedData.Routines.ScheduleStore import <name> does the same by hand.
  • The old place becomes a copy. A background thread of the Scheduler writes the definition (when it changed) and the state (at most every 10 s) to the old place, for what still reads it there: a Dashboard on an older SharedData, scripts using ClientAPI.get_metadata('SCHEDULES/...'). The Scheduler never waits for it. An edit made in the old place is not applied and is overwritten; import applies it on purpose.
  • Switching the copy off. SHAREDDATA_SCHEDULER_METADATA_COPY=off in the Scheduler's environment, once nothing reads the old place. The Scheduler then makes no call to the object store at all.

The Scheduler's two tasks, and who sends the commands

Every few seconds, for each schedule, the Scheduler does two things:

  1. status: reads what the routines said in the log and works out how each one is doing;
  2. commands: starts and restarts the routines that are due.

By default it does both: routines are started and restarted automatically. The second task can be switched off and on again while the Scheduler runs.

  • The switch. One per schedule, by the API, ClientAPI, or the toggle button on the SCHED page:
ClientAPI.set_schedule_mode('MASTER', 'observe')    # status only: no command is sent
ClientAPI.set_schedule_mode('MASTER', 'control')    # back to the default
ClientAPI.get_schedule_mode('MASTER')               # what was asked, by whom, what the Scheduler reports
ClientAPI.list_schedule_modes()

GET  /api/schedules
GET  /api/schedules/<name>/mode
POST /api/schedules/<name>/mode      {"mode": "observe"}     (master token)

The Scheduler adopts a switch during its polling loop, also in the middle of a round: it looks at the switch right before it sends. Switched off, it goes on keeping the status (and the copy for old readers); what was due is started as soon as it is switched on again. The switch stays as it was left across a restart of the Scheduler. Restart and Stop of one routine on the SCHED page still work while it is off: those are a person's commands. What was asked, by whom, and what the Scheduler reports it is doing are kept in Hashs/RT/SCHEDULER/CONTROL (one row per schedule). - Starting with the commands off. python -m SharedData.Routines.Scheduler NAMES --mode observe (or SHAREDDATA_SCHEDULER_MODE=observe) starts the Scheduler with the switch of its schedules off. It is the same switch: it can be switched on at any time, and then that Scheduler is in charge. This is for running it next to the Scheduler that controls the routines, for instance the new code beside an older Scheduler: until it is switched on it also leaves the old metadata place and the Scheduler's own heartbeat to that one. Through a worker it is a routine of its own:

ClientAPI.post_workerpool({'sender': 'MASTER', 'target': 'user@COMPUTER', 'job': 'routine',
                           'repo': 'SharedData', 'routine': 'Routines.Scheduler',
                           'args': 'MASTER --mode observe'})      # 'job': 'kill' ends it

Run one Scheduler of this version per schedule: two of them would both write the same state, and switched on, both would send commands.

With SHAREDDATA_SCHEDULER_MODE=observe in the environment of a whole server (a copy that watches another server's routines by reading its log folder, mounted read-only), every Scheduler it starts begins with the commands off. Its SCHED page is the same page, Restart, Stop and Cancel included, so it can be tried there; what is sent from it is recorded on that server only (its worker pool is not the one those routines listen to), and the page says so in the question and in the notice.

A Scheduler never sends a command for itself, even when its schedule lists it. Its Worker keeps it running; a start command for it made the Worker run a second Scheduler beside it.

The SCHED page

SCHED (/admin/sched, masters) shows today's routines of every schedule kept on the server, read again every 5 seconds.

  • Tabs 1) Realtime and 2) Batch (keys 1, 2), each with its count and how many fail.
  • Colours: failing red (ERROR, DELAYED, or a realtime routine that was silent for 30 s, shown as SILENT), starting amber, waiting orange, completed green, running neutral.
  • The filters are behind the Filters button (also on LOGS and on the Jobs tab), shut until asked for; the button says how many are set (Filters · 2), and open or shut is remembered in the browser. Filter by text (key /, which opens them) and by status; Problems only (key P); group by schedule group, computer or status, or in run order.
  • A routine opens to its details: full name, arguments (the value of a flag that looks like a secret is hidden), what it waits for, times, the last message.
  • Restart ends the routine if it runs and starts it again. Stop ends it; the Scheduler starts a realtime routine again once it has been silent for 45 s to 5 minutes, so to keep one stopped, remove it from the definition. Both ask first and are logged with who sent them (SCHED: restart of <routine> sent by <user>).
  • If the Scheduler has not written its state for a minute, a banner says so and every routine of that schedule is shown as not known (WAS RUNNING): nobody is watching them.
  • Under the tabs, each schedule's toggle button with its name: Commands on (the Scheduler starts and restarts its routines) or Commands off (observing only). Who switched it off and when is shown on hover. The button asks before it switches.
  • Grouping includes By schedule.
  • A third tab, 3) Jobs (key 3), lists the worker pool's batch jobs: see docs/job-store.md.
  • ?tab=1|2|3 (or realtime, batch, jobs) in the address opens that tab; in the terminal, typing the tab's number in the command line does the same. A tab chosen in the page is written to the address.
  • Realtime groups start shut. A group's header carries the status of the group as a whole: RUNNING with a green dot when every routine of it runs, otherwise the most pressing status among them (failing, starting, waiting, not known, running, scheduled, completed) with the count of each. A group opened by hand stays open in that browser; a filter opens the groups it found something in.
  • − and + at the top right of every page of this kind (SCHED, LOGS, DATA, WORKERS, USERS, ROUTES) make its text smaller or larger, from 80% to 200%, remembered in the browser for all of them.
  • A Scheduler listed in a schedule has no Restart or Stop: see "Restarting only the Scheduler" below.

Next to a Dashboard, its own page of that name stays reachable as DASHSCHED; a user who is not a master keeps SCHED for the Dashboard's page.

Operating

  • Restarting only the Scheduler. End its process (kill -TERM <pid>); the Worker that owns it (--schedules) starts it again within seconds. Routines keep running.
  • Going back. Install the previous version and restart the Scheduler the same way. While the copy is on, the old place is current, so the old Scheduler continues from it.

Limits

  • The store is local disk, on the host that runs the Scheduler (which is the API server's host). A Scheduler on another host is not supported.
  • Only the Scheduler was freed from the object store. A server whose object store is configured but does not answer still does not finish starting, as before.
  • Worker election (RUN_MASTER_ELECTION) still keeps its heartbeats in the metadata store.