# SP18 — Structured JSON Logging

**Date:** 2026-06-21
**Branch:** `sp18-structured-logging`
**Status:** Shipped
**Scope:** Backend only. No frontend changes.

---

## 1. Why this exists

Cyclone has ~150 `logging.getLogger(__name__).*` call sites and
zero of them are structured. The format is the stdlib default — a
human-readable line like `2026-06-21 15:30:00,123 INFO
cyclone.scheduler: Processed inbound foo.x12: parser=parse_999 claims=3`
— which is fine for `tail -f` in dev but unparseable for anything an
operator actually wants to do with logs:

* **Find all errors in the last 24h.** `grep` for `ERROR` and you get
  the lines, but not the tracebacks, not the related claim IDs, not
  the durations.
* **Correlate scheduler ticks with API requests.** Impossible without
  a request/correlation id in every line.
* **Detect PII leaks.** No scrubbing means a single accidental
  `log.info("claim=%s", claim)` can dump PHI to stderr and the
  README won't catch it.

The completeness review calls this out as gap #5 (`no structured
logging`).

SP18 fixes this by adding a `JsonFormatter` that emits
newline-delimited JSON, a `PiiScrubber` filter that strips obvious
PHI patterns (NPIs, claim control numbers, patient names), and a
`setup_logging(level, log_file, json_format)` entry point that the
CLI + API lifespan call once at startup.

## 2. Output format

Default (json_format=True):

```json
{"ts": "2026-06-21T15:30:00.123Z", "level": "INFO", "logger": "cyclone.scheduler", "msg": "Processed inbound foo.x12", "extra": {"parser": "parse_999", "claims": 3}}
{"ts": "2026-06-21T15:30:00.456Z", "level": "ERROR", "logger": "cyclone.api", "msg": "backup failed", "extra": {"reason": "BackupError: no passphrase"}}
```

Optional (json_format=False, the dev-friendly format):

```
2026-06-21T15:30:00.123Z INFO  cyclone.scheduler Processed inbound foo.x12 parser=parse_999 claims=3
```

The dev format is the `CycloneDevFormatter` — same fields, tabular.
Useful when `tail -f`-ing the API in dev.

## 3. PII scrubbing

The `PiiScrubber` is a logging `Filter` that walks the log record's
message + extra fields and replaces known PHI patterns with
`<redacted:npi>` etc.:

| Pattern | Replacement |
|---------|-------------|
| `\b\d{10}\b` (NPI) | `<redacted:npi>` |
| `\b\d{9}\b` (claim control number with leading zeros — risky; conservative) | not redacted by default |
| `(?i)patient[_ ]?name[:=]\s*\S+` | `<redacted:patient_name>` |
| `(?i)ssn[:=]\s*\d{3}-?\d{2}-?\d{4}` | `<redacted:ssn>` |
| `(?i)dob[:=]\s*\d{4}-\d{2}-\d{2}` | `<redacted:dob>` |

The default scrubber is conservative — we redact the unambiguous
patterns only. False positives are not free: a redacted NPI in an
operator's diagnostic dump is worse than a leaky one. The scrubber
can be disabled (`setup_logging(scrub_pii=False)`) for tests.

## 4. Files

* `cyclone.logging_config` — new module (~200 LOC).
  * `JsonFormatter` — `logging.Formatter` subclass that JSON-encodes the record.
  * `CycloneDevFormatter` — `logging.Formatter` subclass, tabular.
  * `PiiScrubber` — `logging.Filter` subclass, regex rewriter.
  * `setup_logging(level, log_file, json_format, scrub_pii)` — entry point.
* CLI: every `click.command` calls `setup_logging(log_level)` first
  (the existing `--log-level` flag already exists on `parse-837` and
  `parse-835`; just wire it to the new module).
* API: the FastAPI lifespan calls `setup_logging(level=os.environ.get("CYCLONE_LOG_LEVEL", "INFO"))` before any other setup.
* Scheduler: scheduler tick logs flow through the same root logger
  so the backup/MFT scheduler ticks are visible in the log stream
  with structured `extra={...}`.

## 5. Operator surface

| Env var | Default | Meaning |
|---------|---------|---------|
| `CYCLONE_LOG_LEVEL` | `INFO` | Root logger level. DEBUG for troubleshooting, WARNING to quiet. |
| `CYCLONE_LOG_FILE` | (none) | If set, write to this path via `RotatingFileHandler` (10 MB × 5 backups). |
| `CYCLONE_LOG_JSON` | `true` | If `false`, use the dev formatter. |
| `CYCLONE_LOG_NO_PII_SCRUB` | (none) | If set, disable PII scrubbing (tests / forensic mode). |

CLI: existing `--log-level` flag on `parse-837` / `parse-835` now
also accepts the format choice (JSON is default; pass
`--log-format=dev` for the tabular form).

## 6. Migration strategy

SP18 does **not** rewrite every log call. It:

1. Adds the formatter + filter to the root logger.
2. Migrates the ~25 highest-value log sites to use `extra={...}`
   for structured fields (e.g. `log.info("processed inbound",
   extra={"filename": f.name, "parser": "parse_999", "claims": 3})`
   instead of f-string concatenation).
3. Keeps backward compatibility — `log.info("foo %s", x)` still works.

This is a deliberate scope cut. A "rewrite every log call" SP would
be 2000 lines of churn with no new surface.

## 7. Tests

* `test_logging_formatter.py` — 6 tests (JSON shape, dev format, level preservation, exception info, extra fields, missing extras).
* `test_logging_scrubber.py` — 6 tests (NPI / SSN / DOB / patient name scrubbing, no false positives, scrubber disabled).
* `test_logging_setup.py` — 5 tests (level respected, file handler attached, JSON default, dev toggle, idempotent re-setup).

Total: 17 new tests.

## 8. Out of scope

* A real log aggregator (Loki / ELK / Vector). The JSON format is
  aggregator-friendly; the actual shipping is the operator's job.
* Per-logger log levels via config file. `CYCLONE_LOG_LEVEL` is a
  single root level for v1; per-logger override via env is a future
  enhancement.
* OpenTelemetry / Prometheus instrumentation. That's a different
  SP (observability) for later.