Skip to content

Credential redaction

SharedData.Redaction is the central, standard-library-only consumer module. It imports no storage or HTTP client and changes no global logging factory/class. It is compatible with Python 3.11 and 3.12.

from SharedData.Redaction import install_redaction, redact_credentials
install_redaction(logger)

install_redaction(logger) protects existing and later stream/file handlers. Custom handlers opt in explicitly with install_redaction(handler); SharedData's API and buffer handlers do this in their constructors. Other telemetry handlers and logger filters remain untouched. Install on each owned ancestor handler used by propagation. A handler assigned directly to logger.handlers must be installed explicitly. Reinstall after replacing a formatter. An unprotected child handler emits before an ancestor can protect its output.

Record preservation and sink ownership

Redaction occurs on private sink projections or final rendered output. Original msg, args, exc_info, exc_text, stack_info, metadata and extras remain intact. The exception objects and their cause/context relationships stay live for telemetry, regardless of whether it runs before or after an owned sink. Existing handler levels, locks, filter order and inner formatters are retained. The handler wrappers substitute copies before filtering and emission, including on Python 3.11. A final projection also sanitizes fields added by owned filters.

redact_record(record) caches the sanitized message/exception projection by weak original-record identity and current environment values. Lazy arguments are rendered once into the projection; they are never replaced on the caller record. Fresh projections copy mutable extras so one sink cannot modify another sink's view. This is a snapshot for an emission, not a mutable-record replay facility. The projection contains sanitized exception text, including useful cause/context and exception type diagnostics, and no live exc_info that a JSON formatter can use to regenerate the raw traceback. Malformed formatting retains the sanitized format and error type and omits unassociated arguments. API shipping includes sanitized exception/stack diagnostics without rewriting ordinary apostrophes.

Public interfaces:

  • redact_credentials(value) -> str: safe text for recognized credential shapes.
  • redact_fields(value): bounded, recursive copies of structured log fields.
  • redact_record(record) -> LogRecord: a safe projection; the input is preserved.
  • install_redaction(logger_or_handler): idempotent sink installation.
  • RedactingFormatter(formatter=None): wraps text/JSON formatters; direct calls also preserve the input record and sanitize final formatter constants.
  • RedactingFilter(name=''): returns a projection, without mutating its input. Attaching this filter alone is insufficient on Python 3.11; use installation or the formatter. Installation does not depend on 3.12 filter replacement.
  • StreamRedactor.redact(text): Worker continuation-aware string redaction.
  • RedactingStream.write(text), .flush(), .finish(): protected WSGI writer.
  • install_flask_redaction(app): Flask logger and request-local error adapter.

SharedData protects its owned console/file/API/buffer sinks, Flask API/Web error paths, Gunicorn error handlers, Auth/DataAPI loggers, Worker stdout/stderr and records/documents immediately before daily-file and LOGS/state-table writes. Clean messages, routinenames, commands, recognized argument fields, status/file metadata, WorkerPool heartbeat/command diagnostics and seven daily-file fields remain readable. Caller containers, HTTP request credentials/parameters/payloads, and original response objects are not changed by logging redaction.

Scanner and WSGI buffer

The forward scanner classifies labels before parsing values. Ordinary fields never trigger a search of their remaining values; credential values are consumed once. The scanner yields replacement spans, which the string helper and spilled stream renderer share. URL regions are bounded by whitespace and adjacent complete schemes. Raw and escaped JSON slash representations, including nested JSON/repr forms, retain valid enclosing containers. Scheme colon/slash Unicode escapes are also recognized. Compound auth/cookie values use header semantics with either colon or equals, except for URL query parameters, which retain query boundaries. Quoted scheme values, quoted header parameters and trailing cookie subvalues are masked. Percent-encoded names are decoded at most twice.

Known values are checked at emission time in literal and URL-encoded forms: SHAREDDATA_TOKEN, SHAREDDATA_LOG_TOKEN, SHAREDDATA_SECRET_KEY and FLASK_SECRET_KEY. Values shorter than eight characters are not globally replaced; labeled credentials still mask short values. URL userinfo is removed in full. Unquoted values conservatively include internal commas/semicolons; trailing punctuation before whitespace and enclosing container closers are preserved.

WSGI writes scan only the new input for complete lines. Incomplete fragments append to a private spooled buffer; after 256 KiB it uses an anonymous temporary file. A large line is rendered from replacement spans over a read-only mapping, with bounded output chunks, without reading the entire line into a Python string. Deep quote delimiters are represented by counts rather than materialized runs. UTF-8 diagnostic text and whitespace boundaries survive spill rendering. Pending Python heap is bounded for these paths; the caller's input, downstream sink, spill file size, mapped address space and OS page cache remain proportional to the logical line. Temporary-storage exhaustion can raise an I/O error; it is not handled by dropping the message or emitting its unredacted content.

flush() flushes already emitted output and retains the unfinished logical line: a credential may span writes or flush calls. finish() drains that final line and releases the private buffer; close() calls it without closing the borrowed error stream. The Flask adapter finishes on response completion, error or close, including a response closed before iteration. Standalone users must call finish() at their stream boundary. This replaces the former partial-line marker, which discarded clean diagnostics, with lossless delayed emission.

Dashboard adapter contract

Dashboard should depend on the reviewed/released central module and import redact_credentials, redact_record, RedactingFormatter or install_redaction. It should remove its copied scanner rather than maintaining another grammar. Install on explicitly owned sinks; custom emitters read a projection, and unrelated telemetry retains original records. Do not rely on an upstream logger filter or on attaching RedactingFilter alone under Python 3.11. Preserve existing handler configuration and use the WSGI finish lifecycle if adapting an error stream.

Intentional limits and acceptance

The guarantee covers configured owned sinks and recognized credential shapes and known values. It is not a universal secret detector. Unknown unlabeled values, unregistered sinks, direct writes outside owned streams and arbitrary encoded syntax remain outside it. Existing persisted logs are not rewritten. Unterminated structured credential quotes intentionally mask continuation data until closed. Ambiguous unescaped password punctuation can conservatively mask additional URL context. Generic diagnostics such as max_tokens, cache_key, hbkey, pagination and asset-token labels are preserved.

The existing 16-case gap matrix is retained with strict expected failures. Whitespace-only CLI/prose associations, bare Basic values, YAML blocks/lists, HTML/URL-encoded assignment separators, unusual Unicode label/separator syntax, unlabeled PEM/JWT and XML elements remain unsupported. Explicit CLI password options, YAML password blocks, XML password elements and private-key envelopes are high-confidence credentials, not harmless or ambiguous prose. Their exclusion limits any broad claim that commands or arbitrary logs can never contain secrets; consumers producing them need separate scoped grammar work before such a claim. The independently confirmed record, escaped-JSON-URL, cookie-equals and WSGI blockers are required acceptance cases and are not folded into this gap matrix.

Focused tests cover original-record identity, telemetry order, exception chains, direct emit/format, helper/Worker/file/API shapes, chunk/flush/request-close boundaries, spill parity, speed/heap scaling and WorkerPool/file headers. Browser checks require Playwright; skipped browser checks are not verified behavior.