feat(sp18): structured JSON logging + PII scrubber

Cyclone previously emitted stdlib default-formatted log lines that
operators couldn't parse with anything beyond grep. SP18 replaces
that with:

- JsonFormatter: newline-delimited JSON, ISO-8601 ms timestamps,
  structured "extra" dict, exception tracebacks serialized as a
  single string.
- CycloneDevFormatter: tabular format for tail -f in dev.
- PiiScrubber: logging.Filter that redacts NPIs, SSNs, DOBs,
  patient names — both inline in the message ("npi 1881068062")
  and via PHI-keyed extras ({"dob": "1980-04-12"}).
- setup_logging(): idempotent entry point used by the API lifespan
  and CLI main; respects CYCLONE_LOG_LEVEL/FILE/JSON/NO_PII_SCRUB.
- CLI --log-format=json|dev + --log-file=… flags.
- Migrate 9 highest-value log sites in scheduler/backup_scheduler/
  backup_service to use extra={...} (input_filename, claims, parser,
  backup_id, db_fingerprint, etc.).

34 new tests (test_logging_formatter + test_logging_scrubber +
test_logging_setup). All 867 backend tests pass.
This commit is contained in:
Tyler
2026-06-21 09:59:34 -06:00
parent daf3206d55
commit 47e0f80786
12 changed files with 1586 additions and 19 deletions
+75 -1
View File
@@ -370,6 +370,68 @@ fingerprints and the post-rotation table count. The old key is
retained in the `cyclone.db.key.previous` Keychain account for a
grace period so a botched rotation can be rolled back by hand.
## Structured logging (SP18)
Cyclone emits newline-delimited JSON to stderr by default — readable
by `jq`, Loki, Vector, ELK, or any log shipper. Every record carries
`ts` (ISO 8601 ms UTC), `level`, `logger`, `msg`, and an optional
`extra` dict for structured fields. Exceptions render as a
`traceback` string.
```
{"ts":"2026-06-21T15:30:00.123Z","level":"INFO","logger":"cyclone.scheduler","msg":"Processed inbound file","extra":{"input_filename":"ACK_999.x12","parser":"parse_999","claims":3}}
{"ts":"2026-06-21T15:30:01.456Z","level":"ERROR","logger":"cyclone.api","msg":"Backup create failed","extra":{"reason":"BackupError: passphrase mismatch"},"traceback":"Traceback ..."}
```
### PII scrubbing
A `PiiScrubber` filter is attached to the root logger and rewrites
obvious PHI patterns to `<redacted:npi>` / `<redacted:ssn>` /
`<redacted:dob>` / `<redacted:patient_name>` before any handler sees
the record:
| Pattern | Replacement |
|---------|-------------|
| `\b\d{10}\b` | `<redacted:npi>` |
| `\b\d{3}-\d{2}-\d{4}\b` or `\b\d{9}\b` at phrase boundary | `<redacted:ssn>` |
| `(dob\|date_of_birth)[:=]\s*\d{4}-\d{2}-\d{2}` | preserves the key, redacts the date |
| `patient_name=...` | full chunk redacted |
| Extras with key `dob`/`ssn`/`npi`/`patient_name`/… | value redacted regardless of shape |
The scrubber is conservative — bare ISO dates without a `dob=` prefix
are **not** scrubbed (they're too often timestamps or batch IDs), and
11+ digit numbers are left alone (they can't be NPIs). Disable for
forensic mode with `CYCLONE_LOG_NO_PII_SCRUB=1`.
### Knobs
| Env var | Default | Meaning |
|---------|---------|---------|
| `CYCLONE_LOG_LEVEL` | `INFO` | Root logger level. `DEBUG` for troubleshooting. |
| `CYCLONE_LOG_FILE` | (none) | Write to this path via `RotatingFileHandler` (10 MB × 5 backups). |
| `CYCLONE_LOG_JSON` | `true` | `false` uses the dev tabular formatter. |
| `CYCLONE_LOG_NO_PII_SCRUB` | (none) | `1` disables scrubbing. |
CLI:
```
cyclone --log-format=dev parse-837 sample.x12 --output-dir out/ # tabular for tail -f
cyclone --log-file=/var/log/cyclone.log backup create # JSON to rotating file
```
The `parse-837` / `parse-835` subcommands also accept `--log-level`
which re-runs `setup_logging()` so the per-invocation level overrides
the group default.
### Files
* `cyclone.logging_config``JsonFormatter`, `CycloneDevFormatter`,
`PiiScrubber`, `setup_logging()`.
* `tests/test_logging_formatter.py` (11), `test_logging_scrubber.py`
(13), `test_logging_setup.py` (10) — 34 new tests.
* `cyclone.api` lifespan calls `setup_logging()` first; the CLI
`main` group does the same.
## Encrypted Backups (SP17)
The BackupService takes an online consistent snapshot of the live
@@ -537,7 +599,7 @@ backup API).
## Roadmap
Sub-projects 2 through 17 are **shipped**. See the [completeness
Sub-projects 2 through 18 are **shipped**. See the [completeness
review](docs/reviews/2026-06-20-cyclone-completeness-review.md) for
the honest gap analysis against the industry definition of a HIPAA
clearinghouse — the short version is that the local-only,
@@ -548,6 +610,18 @@ scope.
Shipped sub-projects (most recent first):
- **Sub-project 18 (shipped) — Structured JSON logging.** All logs
emitted by the API, CLI, scheduler tick loop, and backup service
flow through a `JsonFormatter` (newline-delimited JSON, ISO-8601 ms
timestamps) by default. A `PiiScrubber` filter redacts obvious PHI
(NPIs, SSNs, DOBs, patient names) from message + extras — both via
inline patterns (`npi 1881068062`) and via PHI-keyed extras
(`extra={"dob": "1980-04-12"}`). Configurable via env vars
(`CYCLONE_LOG_LEVEL`, `CYCLONE_LOG_FILE`, `CYCLONE_LOG_JSON`,
`CYCLONE_LOG_NO_PII_SCRUB`) and CLI flags
(`--log-format=json|dev`, `--log-file=…`); a tabular `CycloneDevFormatter`
is the opt-out for `tail -f` in dev. See
[Structured logging](#structured-logging-sp18) below.
- **Sub-project 17 (shipped) — Encrypted DB backups.** Automated
encrypted backups via AES-256-GCM (PBKDF2-HMAC-SHA256, 200k iters).
The operator sets a separate passphrase in the macOS Keychain