- conftest.py example now uses SAMPLE_837P constant so sample_837p_bytes
reads the file (not the directory).
- Note that Python 3.13 satisfies requires-python >= 3.11 if 3.11 isn't
installed.
Adds pure local validators for the 10-digit NPI Luhn checksum (CMS-
published algorithm with the '80840' NPPES prefix) and 9-digit EIN
format (rejects reserved prefixes 00/07/80-89). No NPPES round-trip,
no IRS e-file lookup — catches the 99% typo case at parse time.
Surface:
- cyclone.npi.is_valid_npi / is_valid_tax_id / normalize_tax_id
- CLI: 'cyclone validate-npi <npi>' and 'cyclone validate-tax-id <ein>'
- API: GET /api/admin/validate-provider?npi=&tax_id=
- Parser validator: new R021_npi_checksum rule (warning, not error,
to keep test fixtures with placeholder NPIs ingestible)
- minimal_837p.txt fixture NPI updated from '1234567890' to the
Luhn-valid '1993999998' so strict-mode CLI parses still pass
Tests:
- test_npi.py — 27 cases (Luhn math, valid/invalid NPIs, EIN cases,
normalize_tax_id edge cases)
- test_api_validate_provider.py — 4 cases (both valid, both invalid,
omitted NPI, omitted tax_id)
- test_cli_validate.py — 8 cases (valid/invalid for both subcommands,
exit codes, malformed inputs)
- test_validator.py — 4 new R021 cases (valid Luhn silent, bad Luhn
warning, skipped when format bad, skipped when NPI missing)
Total: 923 tests pass.
13 tasks implementing the spec: pre-flight baseline, atomic-move of
store.py into store/__init__.py, 10 module extractions (exceptions,
records, orm_builders, ui, write, batches, claim_detail, acks, backups,
inbox, providers), and final verification + single atomic commit.
Working tree stays dirty through Tasks 1-11; final commit happens at
end of Task 12 per the spec's single-atomic-commit requirement.
Each task ends with a focused test-suite verification (not full baseline
— that's Task 12 Step 1's job) so regressions are caught early.
Step 4 of the file-split refactor series. Targets backend/src/cyclone/store.py
(2,412 lines) which sits on the hot path of parse-999, parse-277ca,
reconciliation, and inbox match/unmatch.
Decisions locked:
- Public surface: cyclone/store/ subpackage + thin facade (__init__.py
re-exports every currently-importable name).
- Class shape: module functions for bodies, CycloneStore keeps its current
method signatures as 1-line delegations.
- Helper distribution: dedicated utility modules (records, orm_builders,
ui, exceptions).
13 target modules total; largest is write.py at ~210 lines. Zero public API
changes, zero test changes. Single atomic commit migration; rollback is a
single git revert.
Three pure-ASGI middlewares close completeness-review gaps §3.1.4
(no body/rate limits) and §3.1.25 (no security headers):
- BodySizeLimitMiddleware — rejects oversized uploads (50 MB
default, CYCLONE_MAX_BODY_BYTES override). 413 on over-cap
Content-Length and on chunked reads that cross the cap.
- RateLimitMiddleware — sliding-window per-IP limiter (300/min
default, CYCLONE_RATE_LIMIT_PER_MIN override). 429 over the
window. /api/health is exempt.
- SecurityHeadersMiddleware — stamps X-Content-Type-Options,
X-Frame-Options, Referrer-Policy, Permissions-Policy, and a
strict Content-Security-Policy on every response.
Every 413/429 also writes a tamper-evident api.request_rejected
event into the SP11 audit chain so an operator can correlate
rejections with the SP18 JSON logs.
GET /api/health is rewritten to return a subsystem snapshot:
DB connectivity (SELECT 1), MFT scheduler state, backup scheduler
state, live pubsub subscriber counts, last batch id + timestamp.
Returns status='degraded' if any subsystem is unhappy; per-subsystem
errors surfaced in the respective dict.
Cyclone.pubsub.EventBus.stats() — new method for live subscriber
counts.
13 new tests (test_security.py) + 1 updated (test_api.py health
endpoint). All 883 backend tests pass.
Cyclone previously emitted stdlib default-formatted log lines that
operators couldn't parse with anything beyond grep. SP18 replaces
that with:
- JsonFormatter: newline-delimited JSON, ISO-8601 ms timestamps,
structured "extra" dict, exception tracebacks serialized as a
single string.
- CycloneDevFormatter: tabular format for tail -f in dev.
- PiiScrubber: logging.Filter that redacts NPIs, SSNs, DOBs,
patient names — both inline in the message ("npi 1881068062")
and via PHI-keyed extras ({"dob": "1980-04-12"}).
- setup_logging(): idempotent entry point used by the API lifespan
and CLI main; respects CYCLONE_LOG_LEVEL/FILE/JSON/NO_PII_SCRUB.
- CLI --log-format=json|dev + --log-file=… flags.
- Migrate 9 highest-value log sites in scheduler/backup_scheduler/
backup_service to use extra={...} (input_filename, claims, parser,
backup_id, db_fingerprint, etc.).
34 new tests (test_logging_formatter + test_logging_scrubber +
test_logging_setup). All 867 backend tests pass.
Adds automated encrypted backups of the live SQLite file. Closes the
'no backup automation' gap called out in the completeness review
(docs/reviews/2026-06-20-cyclone-completeness-review.md §3.1 #3) and
gives the SP16 MFT scheduler a recovery path when the MFT pipeline
loses days of inbound 999/277CA work in a single crash.
Architecture
------------
- AES-256-GCM with PBKDF2-HMAC-SHA256 (200,000 iters, 16-byte salt)
- Online backups via SQLite's .backup() API — no app downtime
- Salt + passphrase persisted to macOS Keychain (separate accounts
backup.passphrase + backup.salt) so the key is reproducible across
processes
- Two-step restore (initiate → confirm) with a one-shot 64-char hex
token; the second call disposes + rebuilds the engine only if the
token matches within a 5-minute TTL
- Tamper-evident audit chain (SP11) — db.backup_created,
db.backup_failed, db.backup_pruned, db.backup_restored,
db.backup_passphrase_set
- BackupService + BackupScheduler + module-level singletons
- 8 admin endpoints + 6 CLI subcommands
- Auto-start opt-in via CYCLONE_BACKUP_AUTOSTART=true; default
interval 24h, default retention 30 days
- Fallback posture: if no separate passphrase is set and SQLCipher
is enabled, the key is derived from the SQLCipher DB key with a
fixed salt + WARNING log (degraded but never plaintext)
New modules
-----------
- cyclone.backup — PBKDF2, AES-GCM, sidecar format
- cyclone.backup_service — create_now / list / verify / restore / prune / status
- cyclone.backup_scheduler — async tick loop with audit hooks
New surface
-----------
- 8 admin endpoints under /api/admin/backup/*
- 6 CLI subcommands under cyclone backup (init-passphrase, create,
list, verify, restore, prune, status)
- Migration 0012_backups.sql + DbBackup ORM
- store.add_backup_pending()
Tests
-----
- 14 unit tests in test_backup_crypto.py (key derivation, encrypt/
decrypt round-trip, tampered ciphertext, wrong passphrase, sidecar
round-trip, filename format)
- 19 tests in test_backup_service.py (create/list/verify/restore/
prune/status, error handling, fallback key, module singleton)
- 14 API tests in test_api_backup.py (all 8 endpoints + scheduler
endpoints, two-step restore, error responses)
- 10 tests in test_backup_scheduler.py (tick / start / stop /
audit / coalescing / module singleton)
- 5 CLI tests in test_cli_backup.py (create / list / verify /
restore confirm prompt / prune confirm prompt /
init-passphrase minimum-length check)
Total new tests: 62. All pass. Full backend suite: 833 passed,
9 skipped (gitignored prodfiles), 1 warning.
Design doc: docs/superpowers/specs/2026-06-21-cyclone-encrypted-backup-design.md
README: new 'Encrypted Backups (SP17)' section, SP17 entry in
Roadmap, retention default documented in §Project layout.
Adds an asyncio-based background scheduler that polls the Gainwell
MFT inbound path, downloads new files, and routes them through the
appropriate parser (999 / 835 / 277CA / TA1). Idempotent (re-ticks
and restarts skip already-processed files via the new
processed_inbound_files table). Crash-safe (per-file try/except so
one bad file doesn't stop the loop).
Lifespan auto-configures from the seeded dzinesco clearhouse's SFTP
block; auto-start is opt-in via CYCLONE_SCHEDULER_AUTOSTART.
Five admin endpoints added:
GET /api/admin/scheduler/status
POST /api/admin/scheduler/start
POST /api/admin/scheduler/stop
POST /api/admin/scheduler/tick
GET /api/admin/scheduler/processed-files?status=&limit=
20 new tests (15 unit + 5 API).
Self-review nits from the router-split commit:
- All four new files lacked a trailing newline (PEP 8 / POSIX).
- acks.py was lazily importing ParseResult999 / serialize_999 inside
get_ack_endpoint. Hoist to module-level — there's no import cycle
(acks.py does not import from cyclone.api), so the imports are safe
to do once.
No behavior change. Targeted tests (test_acks + test_health + test_api_gets)
still pass 41/41.
Step 2 (first half) of the architecture satisfaction loop. api.py
shrank from 2595 to 2452 lines (-143) by extracting three read-only
resource groups into cyclone.api_routers:
- health.py: GET /api/health (1 endpoint)
- acks.py: GET /api/acks, GET /api/acks/{ack_id} (2 endpoints)
- ta1_acks.py: GET /api/ta1-acks, GET /api/ta1-acks/{ack_id} (2 endpoints)
Each router owns its endpoint bodies + the small UI-shape helper that
goes with them (_ack_to_ui, _ta1_to_ui, _serialize_ta1_from_row). The
helpers stay in the router file rather than being shared because each
is only used by its own endpoints.
api.py now ends the app-wiring section with three include_router()
calls. The new package is named cyclone.api_routers (not
cyclone.api.routers) to avoid the Python package-vs-same-named-module
ambiguity that would shadow the existing cyclone.api module.
Verifies: 41 targeted tests (test_acks, test_health, test_api_gets)
pass, full pytest is 8 failed / 735 passed / 16 skipped — identical
to clean main baseline. Live curl against the running server:
GET /api/health -> 200, GET /api/acks -> 200, GET /api/ta1-acks -> 200.
See /tmp/refactor-cyclone.md for the full plan.
Adds in-place key rotation for the encrypted DB at rest (HIPAA
sec.164.308(a)(4) - periodic key rotation).
- db_crypto.rotate_db_key(): opens with old key, issues PRAGMA rekey,
reopens with new key, verifies schema survived (table-count sanity).
- db_crypto.generate_db_key(): fresh 256-bit CSPRNG hex key.
- db_crypto.fingerprint(): SHA-256[:8] of a key, for the operator to
compare across rotations.
- db.dispose_engine() + db.reinit_engine(): SP15 plumbing. The
rotation endpoint disposes the pooled connections (SQLCipher
refuses to rekey while another connection holds the file), runs
the rekey, then rebuilds the engine with the new key from the
Keychain.
- API: POST /api/admin/db/rotate-key with module-level threading.Lock
to serialize rotations. 400 when encryption not enabled, 409 when
a rotation is already in flight, 503 on rekey or Keychain failure
with a reason that tells the operator what to do next.
- Engine uses NullPool when SQLCipher is enabled: the default
QueuePool returns connections to a shared queue that any thread
can pull from, which breaks SQLCipher's thread affinity. NullPool
trades connection reuse for thread safety, the only correct
behavior under FastAPI's per-request threadpool.
- Audit event db.key_rotated with old/new fingerprints and
table_count, written after the engine is rebuilt so the new key
proves it can take new writes.
- previous key is stashed to a second Keychain account so the
operator can roll back if the new key turns out to be broken.
Tests:
- test_db_crypto.py: 12 new tests for generate/fingerprint/rekey
mechanics (5 require SQLCipher at runtime; skipped otherwise).
- test_api_rotate_key.py: 6 new tests for endpoint wiring
(encryption-required, Keychain update, audit event, rekey-failure
rollback, Keychain-write-failure 503, concurrent-rotation 409).
First checkpoint of the architecture satisfaction loop. Cyclone's api.py
was a 2281-line god-module with 14 cross-cutting helpers inlined next
to the @app route declarations. This commit moves them to a dedicated
cyclone.api_helpers module:
- NDJSON wire format: ndjson_line, ndjson_stream_837, ndjson_stream_835,
ndjson_stream_list.
- Content negotiation: client_wants_json, wants_ndjson.
- Strict / raw_segments rewrites: strict_rewrite_837, strict_rewrite_835,
drop_raw_segments_837, drop_raw_segments_835.
- Validation probes: has_claim_validation_errors, has_835_validation_errors.
- Live-tail generator: tail_events, heartbeat_seconds, utcnow.
api.py re-imports them under the original underscore-prefixed names so
every route call site stays unchanged. claims_stream, remittances_stream,
and activity_stream remain exposed at cyclone.api (test_api_stream_live
imports them directly).
Verifies byte-identical NDJSON wire format, content negotiation rules,
and the tail_events async-generator semantics (deliberately polls the
EventBus queue rather than awaiting its async iterator, so heartbeats
don't poison the bus subscription).
Live-tested: GET /api/health, /api/claims, /api/remittances,
/api/activity, /api/acks, /api/providers, /api/inbox/lanes,
/api/inbox/payer-rejected/acknowledge, and the /api/claims/stream
NDJSON tail all return expected codes / payload.
Backend pytest: 29 failures identical to baseline (pre-existing
secrets env, serialize_837, db_crypto env, prodfile env failures),
700 passed. Frontend npm test: 357/357 passing.
See /tmp/refactor-cyclone.md for the full checkpoint log and the plan
for the next step (splitting api.py routes into FastAPI APIRouters).
Autoreview: /tmp/grok-review-local.md (0 bugs, 1 suggestion, 4 nits —
all addressed: dead asyncio/os imports removed, dead Any import
removed, duplicate utcnow import dropped, trailing newline added).
Closes the gap between the SP10 backend (5 lanes) and the SP6
frontend (4 lanes). The Payer-Rejected lane (277CA STC A4/A6/A7)
is now rendered alongside Rejected/Candidates/Unmatched/Done,
with an Acknowledge bulk action that drops claims from the
working surface without erasing the original 277CA rejection
event (audit log stays intact, SP11).
Backend:
* Migration 0010: add payer_rejected_acknowledged_at +
payer_rejected_acknowledged_actor columns + partial index.
* db.py: surface the two new columns on the Claim model.
* inbox_lanes.py: filter acknowledged claims out of the
payer_rejected lane; expose the new fields on the row payload
for forward-compat (e.g. a future 'Recently acknowledged' view).
* api.py:
- POST /api/inbox/payer-rejected/acknowledge
Bulk-acknowledge. Idempotent. Returns transitioned /
already_acked / not_found / not_rejected counts so the UI
can show '3 of 5 were already acknowledged' on a noop bulk.
Writes a 'claim.payer_rejected_acknowledged' event to the
SP11 hash-chained audit log.
- GET /api/inbox/export.csv: accept 'payer_rejected' lane.
* test_acks.py: bump user_version assertion to 10.
* test_lane_filter_acknowledged.py: 4 tests for the lane filter
and forward-compat row payload.
* test_payer_rejected_acknowledge.py: 6 tests for the endpoint
(happy path, idempotency, no-op on non-rejected, missing
ids, 400 on empty, audit-log wiring + chain integrity).
Frontend:
* lib/inbox-api.ts: add payer_rejected to InboxLanes, add
acknowledgePayerRejected(), update exportInboxCsvUrl union.
* hooks/useInboxLanes.ts: add payer_rejected to initial state.
* hooks/useInboxLanes.test.ts: add payer_rejected to mocks.
* components/inbox/BulkBar.tsx: add 'payer_rejected' lane with
Acknowledge action (no Resubmit, no Dismiss — payer-rejected
is not eligible for either).
* components/inbox/BulkBar.test.tsx: add payer_rejected test.
* pages/Inbox.tsx: render the 5th lane, hook up onAcknowledge,
include payer_rejected in the needEyes count.
* pages/Inbox.test.tsx: 3 new tests (5-lane render, need-eyes
count, acknowledge action hits the right endpoint).
* components/inbox/InboxHeader.tsx: doc comment now explains
why payer_rejected rolls up into need-eyes.
Pre-existing typecheck warnings in BulkBar.test.tsx / InboxRow
.test.tsx / Lane.tsx / download.test.ts are unchanged from
main — not touched here.
Test counts: backend 724 -> 734 (+10). Frontend 350 -> 354 (+4).
Replace SftpClient stub write_file/list_inbound/read_file
implementations with real paramiko SSHClient + SFTPClient
calls. The public API (SftpClient.write_file, list_inbound,
read_file, get_secret) is unchanged from SP9 — same signature,
same return types — so the API layer needs no changes.
Real-mode behavior:
* _connect() returns a context manager yielding (ssh, sftp);
closes both on exit. Lazy-imports paramiko so the stub-only
test path doesn't need the dependency.
* Auth resolves from SftpBlock.auth: password_keychain_account
(MFT model) or key_file + optional key_passphrase_keychain_account.
Missing Keychain entries fail loud (RuntimeError) rather than
silently attempting empty-password auth.
* write_file: opens sftp.open(remote, 'wb') and writes bytes;
mkdirs the parent dir (idempotent — MFT pre-creates FromHPE/ToHPE).
* list_inbound: listdir_attr + per-file download into local
staging cache; skips directory entries (0o040000 mask).
* read_file: download via shutil.copyfileobj into BytesIO.
Stub mode is unchanged. AutoAddPolicy for first-time MFT host
fingerprint; operator should pin the key for production.
Adds tests/test_sftp_paramiko.py: 9 tests covering
* stub still works
* real-mode connect builds correct paramiko call from
password_keychain_account, raises on missing Keychain,
raises on missing auth config, raises on STUB_SECRET
* write_file opens 'wb' on the right path and writes bytes
* list_inbound translates attrs into InboundFile records and
caches files locally; skips dirs
Removes 2 obsolete tests in test_sftp_stub.py that expected
SP13-mode to raise NotImplementedError.
pyproject.toml: new optional 'sftp' extra (paramiko>=3.4,<6).
- New cyclone.db_crypto module:
* is_sqlcipher_available() — capability check
* is_encryption_enabled() — Keychain key + sqlcipher3 present
* get_db_key() — reads 'cyclone.db.key' from Keychain
* make_sqlcipher_connect_creator(url, key) — SQLAlchemy creator
- db._make_engine() now switches to SQLCipher when key is present
- pyproject.toml: optional 'sqlcipher' extra (sqlcipher3>=0.6,<1)
- Fallback: without Keychain key, DB stays plain SQLite (no surprise
behavior for operators who haven't set up encryption yet)
- Verified: encrypted file is unreadable as plain SQLite, wrong key
raises on first query, migrations + ORM work transparently
- HIPAA §164.312(a)(2)(iv) compliance note in docs
Tests: 705 -> 717 (12 new for SQLCipher). All 717 backend tests pass.
Tyler
sp15-bot