Skip to content

Batch jobs: the monthly job store, and the Jobs tab

Batch jobs are what ClientAPI.post_workerpool({'job': 'batch', 'command': {...}}) posts and workers started with --batchjobs N take, run and report on.

Where they live

Before With the job store switched on
Place one schemaless table Hashs/D1/WORKERPOOL/JOBS SQLite containers Text/RT/WORKERPOOL/JOBS/YYYYMM
Keys hash date (when the job is due, UTC), hash (of its command)
Active jobs one record (ACTIVE_JOBS) holding all of them, written again at every change rows with an active status, found through an index
Growth every job ever posted in one file with a fixed number of rows one file a month; an old month is archived or deleted as a file

Columns of a job: status, user, computer (who may take it, ANY by default), target (who took it), job, source-revision metadata, routine, command, dependencies, stderr, changed (last change, UTC) and more (whatever else was posted or reported, such as exit_code and run_time).

Nothing changes for who posts, fetches or reports: same calls, same statuses (NEW -> WAITING -> PENDING -> FETCHED -> RUNNING -> COMPLETED / ERROR / CANCEL), same rules. One active job per command: posting it again replaces the one still waiting. A finished run stays as history under its own date. A job due late in a month that finishes in the next one stays in the month it was due.

The store is off until it is switched on, server by server. Until then everything works on the old table as before.

Switching it on

On the server that runs the API, with the API and the routine SharedData.Routines.UpdateActiveJobs stopped, so that no job changes while it is carried over:

python -m SharedData.Routines.JobStore migrate
python -m SharedData.Routines.JobStore status

migrate carries over the jobs that are active and the finished ones they depend on, empties the old record of active jobs, and leaves a marker file (<DATABASE_FOLDER>/master/Text/RT/WORKERPOOL/sqlite/.jobs-in-sqlite). From then on every process of this version on the server uses the store. Workers on other hosts need nothing: they talk to the API. A job a worker is running during the switch is reported into the store when it ends.

What is not carried over: the finished jobs of the old table. The old table and its file are left as they are; nothing reads or writes them afterwards, and they can be archived or deleted when convenient.

Going back: stop the API, delete the marker file, start it. Jobs posted since the switch stay in the store and are not seen by the old table.

Anything that reads the old table directly (WorkerPool.get_job_table), or runs on the server with an older SharedData version, sees no active jobs after the switch. Use WorkerPool.get_active_jobs, WorkerPool.get_batch_job, JobStore.find, or the Jobs tab.

Reading

from SharedData.Routines import JobStore
JobStore.find()                                    # active jobs: rows, total, counts by status, months
JobStore.find(month='202610', statuses=['ERROR'])  # every failed job of a month
JobStore.find(month='202610', text='example_job', limit=50, offset=50)
JobStore.get(job_hash)                             # the latest job with that hash

or through the API, as any SQLite container (ClientAPI.get_sqlite('Text', 'RT', 'WORKERPOOL', 'JOBS/202610', query={'status': 'ERROR'})).

Keeping it small

A month is one file: <DATABASE_FOLDER>/master/Text/RT/WORKERPOOL/sqlite/JOBS/YYYYMM.db. To drop an old month, delete the container (ClientAPI.delete_sqlite) or move the file away. Active jobs are looked for in last month, this month and any later one; a job left active in an older month is no longer handed out.

The Jobs tab

SCHED has a third tab, 3) Jobs (key 3), with the number of active jobs.

  • The tab reads the groups only: for each, its status as a whole and how many jobs of each status it has. The jobs of a group are read when the group is opened, 200 at a time (Show more), and again every 5 seconds while it stays open. Grouped by time and routine (the minute the jobs are due) by default; also by routine, status or computer, or as one plain list.
  • Active now covers the jobs waiting or running. With the job store on, the chooser also offers each month, with every job of it.
  • Filter by text (routine, source revision, computer, job id, arguments) and by status.
  • A job opens to its details: id, who may take it and who took it, what it waits for and how that ended, arguments (a value whose name looks like a secret is hidden), and what it wrote when it failed. The line of a failed job shows the error itself.
  • Cancel: Cancel N on a group's header cancels the active jobs of that group, all of them, read or not; Cancel all (N) those of the whole list as filtered; ticked jobs go with Cancel selected. Each asks first. A job still waiting is no longer handed to a worker. A job a worker already runs is not interrupted: it shows as cancelled until that worker reports how it ended (as before, the report replaces the mark).
  • Addresses behind it: GET /api/web/sched/jobs/groups?by=time|routine|status|computer, GET /api/web/sched/jobs?group=<the key of a group>, POST /api/web/sched/jobs/cancel with {jobs: [...]} or {where: {month, status, q, group}}.
  • Before the job store is on, the tab shows the old table's active jobs, without months and without the count on the tab.

Limits

  • A job whose dependency ended in ERROR or CANCEL waits forever (as before): cancel it.
  • A job that a worker took and that says nothing for an hour ends in ERROR (as before).
  • The store is on the API server's disk; it is not copied to the object store.