Logging templates¶

The Scheduler is the process that is allowed to speak on a timer. A routine speaks at the boundary of a job: started, one line per batch committed, completed, or failed. Neither of them logs inside a loop over rows, symbols, or bars.

Logger.log writes a file and posts to the API when SHAREDDATA_ENDPOINT and a logging token (SHAREDDATA_TOKEN or SHAREDDATA_LOG_TOKEN) are configured. That call is I/O. One call per row stalls the loop and fills the log store the schedule page reads, so a healthy job looks like a flood of INFO lines.

SharedData(source, user=...) connects the logger once. Do not call Logger.connect again on every iteration.

When to log¶

Place	Log
Process start	one `info`: what started, for which schedule
Scheduler, once, when it will send commands	`ROUTINE STARTED!`
Scheduler heartbeat	one `debug` line `#heartbeat#schedule:<names>` at most every 15 seconds
A schedule that failed to load, then loaded	one `error` per failed attempt, later one `info` that it recovered
Mode switch (`observe` / `control`)	one `info`
An exception that aborts the tick	one `error`, then sleep. The next tick does not repeat it unless it fails again
A routine, when the job begins	`ROUTINE STARTED!`
A routine, after a batch is upserted or written	one `info` with the count
A routine, when the job finishes or raises	`ROUTINE COMPLETED!` or one `error`

When not to log¶

The Scheduler while True body. tick() and wait() run every few seconds and write nothing on the success path. The heartbeat is the only periodic line, and it is gated by the 15 second clock (src/SharedData/Routines/Scheduler.py).
A for symbol in ... or for row in ... loop. Count the rows. Log the count after the loop.
upsert / write_values / extend on each row. Build the batch, write it once, log once.
Debug prints left in a tight loop "just for now". They become the production log.

TimeSeriesDisk follows the same rule for the constructor path: rows that miss the time index are skipped, and one warning reports N of M. It does not warn per row.

Template: a long-running scheduler loop¶

Copy this shape when you own the outer loop. Log the state change. Do not log the tick.

import time
from SharedData.Logger import Logger
from SharedData.SharedData import SharedData

HEARTBEAT_SECONDS = 15


def main():
    SharedData(__file__, user='example')
    Logger.log.info('scheduler starting')
    Logger.log.info('ROUTINE STARTED!')
    last_heartbeat = time.time()
    while True:
        try:
            if time.time() - last_heartbeat > HEARTBEAT_SECONDS:
                last_heartbeat = time.time()
                Logger.log.debug('#heartbeat#schedule:MYJOB')
            tick()
            time.sleep(5)
        except Exception as exc:
            Logger.log.error('loop exception: %s' % exc)
            time.sleep(60)


def tick():
    # Decide, dispatch, return. No Logger.log on the quiet path.
    return None

The file scheduler_loop.py next to this page is the same template.

Template: a routine the Scheduler starts¶

The Scheduler already records that it launched you. Inside the routine, log the job, not the items.

from SharedData.Logger import Logger
from SharedData.SharedData import SharedData


def main():
    shdata = SharedData(__file__, user='example')
    Logger.log.info('ROUTINE STARTED!')
    written = 0
    try:
        records = shdata.table('MarketData', 'D1', 'DEMO', 'PRICES/2026')
        for batch in batches():
            rows = build_rows(batch)
            records = records.upsert(rows)
            written += len(rows)
            Logger.log.info('committed %d rows (%d total)' % (len(rows), written))
        Logger.log.info('ROUTINE COMPLETED!')
    except Exception as exc:
        Logger.log.error('ROUTINE ERROR: %s' % exc)
        raise


def build_rows(batch):
    rows = []
    for symbol in batch:
        rows.append(compute(symbol))
    return rows


def batches():
    return []


def compute(symbol):
    # Supply a valid MarketData key (date and symbol) and the table's values.
    raise NotImplementedError('Implement the row computation for this symbol')

build_rows has no log call. The info sits after upsert, once per batch. If the batches are small and frequent, log every N batches or only the total at the end. The file routine_batch.py next to this page is the same template.

What the running Scheduler actually does¶

Routines/Scheduler.py main:

The Scheduler's SharedData instance connects the logger for its configured namespace.
One info names the schedules and whether commands are off.
ROUTINE STARTED! only when this process is the one sending commands.
The loop heartbeats at debug, at most every 15 seconds, and only while it is in control or a schedule is not on standby.
tick and wait are silent when nothing changed.
A failed load is retried each tick and logs an error for each failed attempt, then one info on recovery. These retries are not deduplicated.
An exception in the loop is one error, then a 60 second sleep.

Match that. A second heartbeat from the routine, on a tighter timer, fights the one the Scheduler already emits.