Address the three concerns raised by the spec reviewer on commit 534130e:
1. The 0014 data-preservation test was degraded: it only verified
post-0014 INSERTs round-trip, never proving that 0014's INSERT INTO
claims_new SELECT * FROM claims actually carries pre-existing rows
through the table recreation. Add migrate_to_version() helper to
db_migrate.py and rewrite the test to seed at v=13, apply 0014, and
assert the rows survive. Add a complementary test exercising the
JOIN-FK tables (matches / cas_adjustments / service_line_payments /
line_reconciliations) whose batch_id columns get populated by
0014's JOIN against the parent tables.
2. The four SQLAlchemy ORM before_insert events on Match /
CasAdjustment / ServiceLinePayment / LineReconciliation are
undocumented in the codebase. Add a clear module-level docstring
explaining what the events do, why they exist (composite-FK
NOT NULL columns require the batch side which most call sites
don't have readily available), when they fail (parent id not
findable in session.new / identity_map / DB -> misleading
'NOT NULL constraint failed' IntegrityError), and the workarounds.
3. The plan Task 1.3 Step 8 said 'do not modify production code to
make old tests pass'. In practice production code DID need to
change because composite-PK semantics make single-id lookups
ambiguous. Amend the plan to acknowledge this and cite the
s.get(Claim, X) -> s.query(Claim).filter().first() refactor as
the example. Also document the before_insert events in Step 5.
Brings in docs/superpowers/plans/2026-06-21-cyclone-parse-decide-workflow.md
from main (this branch was created before the plan file was committed)
with Task 1.3 Step 5 and Step 8 amended per the reviewer feedback.
The implementation shipped with 10 inline bug fixes that weren't reflected
in the original plan document. This commit updates the plan so it
matches the as-built code and adds a summary table near the top so a
future reader of the plan knows what was adjusted and why.
- Bug 1a: AckTimeoutError.__init__ signature widened from int to float
plus adds self.phase attribute for report remediation hints.
- Bug 1b: wait_for passes timeout_s through without int() round-trip.
- Bug 2: Markdown table assertion updated to match the actual format.
- Bug 3: Added ## Remediation section to write_report_md for soft/hard fails.
- Bug 4: 835 expected-by text now says 'typically the following Monday'.
- Bug 5: Removed bogus HealthSnapshot import in test_pipeline.py.
- Bug 6: All phase tests wrapped in 'async with CyclonePipeline(...) as p:'.
- Bug 7: Playwright + UploadPage imports moved to module level in pipeline.py.
- Bug 8: Added _phase_records_to_results() helper for PhaseRecord→PhaseResult.
- Bug 9: Resume test uses _make_stub factory so stubs append to completed_phases.
- Bug 10: CLI uses context_settings={'allow_interspersed_args': True}.
Update _get_typed in the plan to:
- raise immediately on 4xx (terminal)
- retry on 5xx and RequestError (transient)
- use BACKOFF_S = (1.0, 2.0) instead of (1.0, 2.0, 4.0)
- use RuntimeError instead of assert for the context-manager guard
Also add test_4xx_is_terminal_not_retried to the plan's test list.
The verbatim plan returned an un-entered CycloneClient from the fixture,
causing all 3 tests to fail with 'use async with CycloneClient(...)'.
Convert fixture to async generator that enters the context. Verified by
running pytest tests/test_api_client.py -v → 3 passed.
The verbatim plan's test asserted isinstance(log, structlog.stdlib.BoundLogger),
but cache_logger_on_first_use=True makes get_logger return a lazy proxy on
first call. Test now uses duck-typing (hasattr('bind') + callable). Also
dropped the misleading caplog assertion and unused logging/structlog imports.
- conftest.py example now uses SAMPLE_837P constant so sample_837p_bytes
reads the file (not the directory).
- Note that Python 3.13 satisfies requires-python >= 3.11 if 3.11 isn't
installed.
Adds pure local validators for the 10-digit NPI Luhn checksum (CMS-
published algorithm with the '80840' NPPES prefix) and 9-digit EIN
format (rejects reserved prefixes 00/07/80-89). No NPPES round-trip,
no IRS e-file lookup — catches the 99% typo case at parse time.
Surface:
- cyclone.npi.is_valid_npi / is_valid_tax_id / normalize_tax_id
- CLI: 'cyclone validate-npi <npi>' and 'cyclone validate-tax-id <ein>'
- API: GET /api/admin/validate-provider?npi=&tax_id=
- Parser validator: new R021_npi_checksum rule (warning, not error,
to keep test fixtures with placeholder NPIs ingestible)
- minimal_837p.txt fixture NPI updated from '1234567890' to the
Luhn-valid '1993999998' so strict-mode CLI parses still pass
Tests:
- test_npi.py — 27 cases (Luhn math, valid/invalid NPIs, EIN cases,
normalize_tax_id edge cases)
- test_api_validate_provider.py — 4 cases (both valid, both invalid,
omitted NPI, omitted tax_id)
- test_cli_validate.py — 8 cases (valid/invalid for both subcommands,
exit codes, malformed inputs)
- test_validator.py — 4 new R021 cases (valid Luhn silent, bad Luhn
warning, skipped when format bad, skipped when NPI missing)
Total: 923 tests pass.
13 tasks implementing the spec: pre-flight baseline, atomic-move of
store.py into store/__init__.py, 10 module extractions (exceptions,
records, orm_builders, ui, write, batches, claim_detail, acks, backups,
inbox, providers), and final verification + single atomic commit.
Working tree stays dirty through Tasks 1-11; final commit happens at
end of Task 12 per the spec's single-atomic-commit requirement.
Each task ends with a focused test-suite verification (not full baseline
— that's Task 12 Step 1's job) so regressions are caught early.
Step 4 of the file-split refactor series. Targets backend/src/cyclone/store.py
(2,412 lines) which sits on the hot path of parse-999, parse-277ca,
reconciliation, and inbox match/unmatch.
Decisions locked:
- Public surface: cyclone/store/ subpackage + thin facade (__init__.py
re-exports every currently-importable name).
- Class shape: module functions for bodies, CycloneStore keeps its current
method signatures as 1-line delegations.
- Helper distribution: dedicated utility modules (records, orm_builders,
ui, exceptions).
13 target modules total; largest is write.py at ~210 lines. Zero public API
changes, zero test changes. Single atomic commit migration; rollback is a
single git revert.
Three pure-ASGI middlewares close completeness-review gaps §3.1.4
(no body/rate limits) and §3.1.25 (no security headers):
- BodySizeLimitMiddleware — rejects oversized uploads (50 MB
default, CYCLONE_MAX_BODY_BYTES override). 413 on over-cap
Content-Length and on chunked reads that cross the cap.
- RateLimitMiddleware — sliding-window per-IP limiter (300/min
default, CYCLONE_RATE_LIMIT_PER_MIN override). 429 over the
window. /api/health is exempt.
- SecurityHeadersMiddleware — stamps X-Content-Type-Options,
X-Frame-Options, Referrer-Policy, Permissions-Policy, and a
strict Content-Security-Policy on every response.
Every 413/429 also writes a tamper-evident api.request_rejected
event into the SP11 audit chain so an operator can correlate
rejections with the SP18 JSON logs.
GET /api/health is rewritten to return a subsystem snapshot:
DB connectivity (SELECT 1), MFT scheduler state, backup scheduler
state, live pubsub subscriber counts, last batch id + timestamp.
Returns status='degraded' if any subsystem is unhappy; per-subsystem
errors surfaced in the respective dict.
Cyclone.pubsub.EventBus.stats() — new method for live subscriber
counts.
13 new tests (test_security.py) + 1 updated (test_api.py health
endpoint). All 883 backend tests pass.
Cyclone previously emitted stdlib default-formatted log lines that
operators couldn't parse with anything beyond grep. SP18 replaces
that with:
- JsonFormatter: newline-delimited JSON, ISO-8601 ms timestamps,
structured "extra" dict, exception tracebacks serialized as a
single string.
- CycloneDevFormatter: tabular format for tail -f in dev.
- PiiScrubber: logging.Filter that redacts NPIs, SSNs, DOBs,
patient names — both inline in the message ("npi 1881068062")
and via PHI-keyed extras ({"dob": "1980-04-12"}).
- setup_logging(): idempotent entry point used by the API lifespan
and CLI main; respects CYCLONE_LOG_LEVEL/FILE/JSON/NO_PII_SCRUB.
- CLI --log-format=json|dev + --log-file=… flags.
- Migrate 9 highest-value log sites in scheduler/backup_scheduler/
backup_service to use extra={...} (input_filename, claims, parser,
backup_id, db_fingerprint, etc.).
34 new tests (test_logging_formatter + test_logging_scrubber +
test_logging_setup). All 867 backend tests pass.
Adds automated encrypted backups of the live SQLite file. Closes the
'no backup automation' gap called out in the completeness review
(docs/reviews/2026-06-20-cyclone-completeness-review.md §3.1 #3) and
gives the SP16 MFT scheduler a recovery path when the MFT pipeline
loses days of inbound 999/277CA work in a single crash.
Architecture
------------
- AES-256-GCM with PBKDF2-HMAC-SHA256 (200,000 iters, 16-byte salt)
- Online backups via SQLite's .backup() API — no app downtime
- Salt + passphrase persisted to macOS Keychain (separate accounts
backup.passphrase + backup.salt) so the key is reproducible across
processes
- Two-step restore (initiate → confirm) with a one-shot 64-char hex
token; the second call disposes + rebuilds the engine only if the
token matches within a 5-minute TTL
- Tamper-evident audit chain (SP11) — db.backup_created,
db.backup_failed, db.backup_pruned, db.backup_restored,
db.backup_passphrase_set
- BackupService + BackupScheduler + module-level singletons
- 8 admin endpoints + 6 CLI subcommands
- Auto-start opt-in via CYCLONE_BACKUP_AUTOSTART=true; default
interval 24h, default retention 30 days
- Fallback posture: if no separate passphrase is set and SQLCipher
is enabled, the key is derived from the SQLCipher DB key with a
fixed salt + WARNING log (degraded but never plaintext)
New modules
-----------
- cyclone.backup — PBKDF2, AES-GCM, sidecar format
- cyclone.backup_service — create_now / list / verify / restore / prune / status
- cyclone.backup_scheduler — async tick loop with audit hooks
New surface
-----------
- 8 admin endpoints under /api/admin/backup/*
- 6 CLI subcommands under cyclone backup (init-passphrase, create,
list, verify, restore, prune, status)
- Migration 0012_backups.sql + DbBackup ORM
- store.add_backup_pending()
Tests
-----
- 14 unit tests in test_backup_crypto.py (key derivation, encrypt/
decrypt round-trip, tampered ciphertext, wrong passphrase, sidecar
round-trip, filename format)
- 19 tests in test_backup_service.py (create/list/verify/restore/
prune/status, error handling, fallback key, module singleton)
- 14 API tests in test_api_backup.py (all 8 endpoints + scheduler
endpoints, two-step restore, error responses)
- 10 tests in test_backup_scheduler.py (tick / start / stop /
audit / coalescing / module singleton)
- 5 CLI tests in test_cli_backup.py (create / list / verify /
restore confirm prompt / prune confirm prompt /
init-passphrase minimum-length check)
Total new tests: 62. All pass. Full backend suite: 833 passed,
9 skipped (gitignored prodfiles), 1 warning.
Design doc: docs/superpowers/specs/2026-06-21-cyclone-encrypted-backup-design.md
README: new 'Encrypted Backups (SP17)' section, SP17 entry in
Roadmap, retention default documented in §Project layout.
- backend/src/cyclone/parsers/serialize_837.py — full-rebuild 837P serializer.
Emits envelope (ISA/GS/ST/SE/GE/IEA + BHT) + submitter/receiver/billing
provider/subscriber/payer hierarchy + editable segments (CLM/REF*G1/HI)
+ per-service-line LX/SV1/DTP*472/REF*6R — all from canonical
ClaimOutput fields.
Pivoted from spec §3.1 hybrid to full rebuild because ClaimOutput.raw_segments
only captures post-CLM segments (CLM, REF*G1, HI, LX, SV1 pairs) — not the
envelope or hierarchies. A pass-through approach cannot regenerate those
without expanding raw_segments in parse_837.py (out of scope for this SP).
- backend/tests/test_serialize_837.py — 36 tests covering envelope shape,
hierarchy segments, claim-level builders, service-line builders, edited-field
propagation, round-trip, custom sender/receiver IDs, and resubmit helper.
- backend/tests/test_prodfiles_smoke.py::test_claims_prodfile_round_trip —
every file in docs/prodfiles/claims/ (113 files) round-trips through
serialize_837 → parse_837_text with deep-equal ClaimOutput (modulo
validation, which is recomputed by the parser).
- docs/superpowers/plans/2026-06-20-cyclone-serialize-837.md — full plan
with amendment note documenting the Approach A pivot.
Per session convention, plan note about unrelated modifications to
parse_835.py / fixtures stashed separately.
- Mark SP4 as fully shipped (batch diff, search, CSV, a11y all landed)
- Add SP6 (Inbox) and SP7 (Per-line reconciliation) sections
- Add SP6 + SP7 endpoint inventories
- Note next up: outbound 837P serializer
The workflow-automation plan was authored but never committed; the
features it specified have shipped, so commit it for the historical
record alongside this README refresh.
Sub-project 7 adds serialize_837.py so a parsed ClaimOutput can be
regenerated as a complete X12 837P file (the shape of
docs/prodfiles/claims/). Unblocks the SP6 resubmit lane end-to-end
and adds a 'Download 837' affordance on the claim drawer.
Approach: hybrid (fresh envelope + selective body rebuild). Stable
segments (provider/subscriber/payer hierarchies) pass through from
claim.raw_segments; editable segments (CLM, REF*G1, HI, SV1, DTP*472)
are rebuilt from canonical ClaimOutput fields so post-parse edits
propagate to the output.
Out of scope: 997 ACK, claim <-> TA1/999 linking, in-app claim
editing UI, multi-CLM envelope batching, Inbox/drawer redesign.
Brings the SP5 live-tail implementation into main:
Backend
* EventBus (cyclone.pubsub): per-kind fan-out with drop-oldest overflow
* FastAPI lifespan initializes the bus + db once per process
* store.add() publishes claim_written / remittance_written /
activity_recorded events on every batch write
* GET /api/{claims,remittances,activity}/stream: NDJSON snapshot +
live subscription + 15s idle heartbeat
* EventBus.unsubscribe() lets the tail loop release its queue on
client disconnect (no queue leak per open stream)
Frontend
* src/lib/tail-stream.ts: streamTail() async-generator over fetch
* src/store/tail-store.ts: zustand with FIFO cap 10k per slice
* src/hooks/useTailStream.ts: connecting/live/reconnecting/stalled/error/closed
state machine with 1→2→4→8→16→30s backoff
* src/hooks/useMergedTail.ts: base + tail merge with filter
* src/components/TailStatusPill.tsx: badge + Reconnect button
* Claims, Remittances, ActivityLog pages wired to the tail
Tests
* 437 backend tests pass (was 418 before SP5)
* 154 frontend tests pass (was 124)
* npm run typecheck clean
* end-to-end smoke: open /api/claims/stream, POST 837, see new claims
arrive in real time without refresh
# Conflicts:
# src/pages/ActivityLog.tsx
# src/pages/Remittances.test.tsx
# src/pages/Remittances.tsx
Replaces the in-memory store with SQLite via SQLAlchemy 2.0; adds
automatic 837P ↔ 835 reconciliation on every 835 parse; ships a
manual /reconciliation page; extends the claim lifecycle to a
7-state model with reversal handling.
Key decisions (locked from brainstorming):
- SQLite at ~/.local/share/cyclone/cyclone.db, overridable via
CYCLONE_DB_URL (Postgres escape hatch)
- SQLAlchemy 2.0 sync ORM; PRAGMA user_version migration runner
- Strict auto-match on patient_control_number (CLM01 == CLP01)
within +/-7 days of service_date
- 7-state claim model with reversal that sets Claim.state=REVERSED
and preserves prior state on Match.prior_claim_state
- Reconciliation triggered inside store.add() after ERA persist;
fail-soft (logged + activity event, never blocks parse)
- 3 new API endpoints; new /reconciliation page; same palette
and aesthetic voice as sub-project 1
Spec is approved-section-by-section and self-reviewed.
Tyler