feat(sp18): structured JSON logging + PII scrubber
Cyclone previously emitted stdlib default-formatted log lines that
operators couldn't parse with anything beyond grep. SP18 replaces
that with:
- JsonFormatter: newline-delimited JSON, ISO-8601 ms timestamps,
structured "extra" dict, exception tracebacks serialized as a
single string.
- CycloneDevFormatter: tabular format for tail -f in dev.
- PiiScrubber: logging.Filter that redacts NPIs, SSNs, DOBs,
patient names — both inline in the message ("npi 1881068062")
and via PHI-keyed extras ({"dob": "1980-04-12"}).
- setup_logging(): idempotent entry point used by the API lifespan
and CLI main; respects CYCLONE_LOG_LEVEL/FILE/JSON/NO_PII_SCRUB.
- CLI --log-format=json|dev + --log-file=… flags.
- Migrate 9 highest-value log sites in scheduler/backup_scheduler/
backup_service to use extra={...} (input_filename, claims, parser,
backup_id, db_fingerprint, etc.).
34 new tests (test_logging_formatter + test_logging_scrubber +
test_logging_setup). All 867 backend tests pass.
This commit is contained in:
@@ -370,6 +370,68 @@ fingerprints and the post-rotation table count. The old key is
|
||||
retained in the `cyclone.db.key.previous` Keychain account for a
|
||||
grace period so a botched rotation can be rolled back by hand.
|
||||
|
||||
## Structured logging (SP18)
|
||||
|
||||
Cyclone emits newline-delimited JSON to stderr by default — readable
|
||||
by `jq`, Loki, Vector, ELK, or any log shipper. Every record carries
|
||||
`ts` (ISO 8601 ms UTC), `level`, `logger`, `msg`, and an optional
|
||||
`extra` dict for structured fields. Exceptions render as a
|
||||
`traceback` string.
|
||||
|
||||
```
|
||||
{"ts":"2026-06-21T15:30:00.123Z","level":"INFO","logger":"cyclone.scheduler","msg":"Processed inbound file","extra":{"input_filename":"ACK_999.x12","parser":"parse_999","claims":3}}
|
||||
{"ts":"2026-06-21T15:30:01.456Z","level":"ERROR","logger":"cyclone.api","msg":"Backup create failed","extra":{"reason":"BackupError: passphrase mismatch"},"traceback":"Traceback ..."}
|
||||
```
|
||||
|
||||
### PII scrubbing
|
||||
|
||||
A `PiiScrubber` filter is attached to the root logger and rewrites
|
||||
obvious PHI patterns to `<redacted:npi>` / `<redacted:ssn>` /
|
||||
`<redacted:dob>` / `<redacted:patient_name>` before any handler sees
|
||||
the record:
|
||||
|
||||
| Pattern | Replacement |
|
||||
|---------|-------------|
|
||||
| `\b\d{10}\b` | `<redacted:npi>` |
|
||||
| `\b\d{3}-\d{2}-\d{4}\b` or `\b\d{9}\b` at phrase boundary | `<redacted:ssn>` |
|
||||
| `(dob\|date_of_birth)[:=]\s*\d{4}-\d{2}-\d{2}` | preserves the key, redacts the date |
|
||||
| `patient_name=...` | full chunk redacted |
|
||||
| Extras with key `dob`/`ssn`/`npi`/`patient_name`/… | value redacted regardless of shape |
|
||||
|
||||
The scrubber is conservative — bare ISO dates without a `dob=` prefix
|
||||
are **not** scrubbed (they're too often timestamps or batch IDs), and
|
||||
11+ digit numbers are left alone (they can't be NPIs). Disable for
|
||||
forensic mode with `CYCLONE_LOG_NO_PII_SCRUB=1`.
|
||||
|
||||
### Knobs
|
||||
|
||||
| Env var | Default | Meaning |
|
||||
|---------|---------|---------|
|
||||
| `CYCLONE_LOG_LEVEL` | `INFO` | Root logger level. `DEBUG` for troubleshooting. |
|
||||
| `CYCLONE_LOG_FILE` | (none) | Write to this path via `RotatingFileHandler` (10 MB × 5 backups). |
|
||||
| `CYCLONE_LOG_JSON` | `true` | `false` uses the dev tabular formatter. |
|
||||
| `CYCLONE_LOG_NO_PII_SCRUB` | (none) | `1` disables scrubbing. |
|
||||
|
||||
CLI:
|
||||
|
||||
```
|
||||
cyclone --log-format=dev parse-837 sample.x12 --output-dir out/ # tabular for tail -f
|
||||
cyclone --log-file=/var/log/cyclone.log backup create # JSON to rotating file
|
||||
```
|
||||
|
||||
The `parse-837` / `parse-835` subcommands also accept `--log-level`
|
||||
which re-runs `setup_logging()` so the per-invocation level overrides
|
||||
the group default.
|
||||
|
||||
### Files
|
||||
|
||||
* `cyclone.logging_config` — `JsonFormatter`, `CycloneDevFormatter`,
|
||||
`PiiScrubber`, `setup_logging()`.
|
||||
* `tests/test_logging_formatter.py` (11), `test_logging_scrubber.py`
|
||||
(13), `test_logging_setup.py` (10) — 34 new tests.
|
||||
* `cyclone.api` lifespan calls `setup_logging()` first; the CLI
|
||||
`main` group does the same.
|
||||
|
||||
## Encrypted Backups (SP17)
|
||||
|
||||
The BackupService takes an online consistent snapshot of the live
|
||||
@@ -537,7 +599,7 @@ backup API).
|
||||
|
||||
## Roadmap
|
||||
|
||||
Sub-projects 2 through 17 are **shipped**. See the [completeness
|
||||
Sub-projects 2 through 18 are **shipped**. See the [completeness
|
||||
review](docs/reviews/2026-06-20-cyclone-completeness-review.md) for
|
||||
the honest gap analysis against the industry definition of a HIPAA
|
||||
clearinghouse — the short version is that the local-only,
|
||||
@@ -548,6 +610,18 @@ scope.
|
||||
|
||||
Shipped sub-projects (most recent first):
|
||||
|
||||
- **Sub-project 18 (shipped) — Structured JSON logging.** All logs
|
||||
emitted by the API, CLI, scheduler tick loop, and backup service
|
||||
flow through a `JsonFormatter` (newline-delimited JSON, ISO-8601 ms
|
||||
timestamps) by default. A `PiiScrubber` filter redacts obvious PHI
|
||||
(NPIs, SSNs, DOBs, patient names) from message + extras — both via
|
||||
inline patterns (`npi 1881068062`) and via PHI-keyed extras
|
||||
(`extra={"dob": "1980-04-12"}`). Configurable via env vars
|
||||
(`CYCLONE_LOG_LEVEL`, `CYCLONE_LOG_FILE`, `CYCLONE_LOG_JSON`,
|
||||
`CYCLONE_LOG_NO_PII_SCRUB`) and CLI flags
|
||||
(`--log-format=json|dev`, `--log-file=…`); a tabular `CycloneDevFormatter`
|
||||
is the opt-out for `tail -f` in dev. See
|
||||
[Structured logging](#structured-logging-sp18) below.
|
||||
- **Sub-project 17 (shipped) — Encrypted DB backups.** Automated
|
||||
encrypted backups via AES-256-GCM (PBKDF2-HMAC-SHA256, 200k iters).
|
||||
The operator sets a separate passphrase in the macOS Keychain
|
||||
|
||||
Reference in New Issue
Block a user