Step 2 (first half) of the architecture satisfaction loop. api.py
shrank from 2595 to 2452 lines (-143) by extracting three read-only
resource groups into cyclone.api_routers:
- health.py: GET /api/health (1 endpoint)
- acks.py: GET /api/acks, GET /api/acks/{ack_id} (2 endpoints)
- ta1_acks.py: GET /api/ta1-acks, GET /api/ta1-acks/{ack_id} (2 endpoints)
Each router owns its endpoint bodies + the small UI-shape helper that
goes with them (_ack_to_ui, _ta1_to_ui, _serialize_ta1_from_row). The
helpers stay in the router file rather than being shared because each
is only used by its own endpoints.
api.py now ends the app-wiring section with three include_router()
calls. The new package is named cyclone.api_routers (not
cyclone.api.routers) to avoid the Python package-vs-same-named-module
ambiguity that would shadow the existing cyclone.api module.
Verifies: 41 targeted tests (test_acks, test_health, test_api_gets)
pass, full pytest is 8 failed / 735 passed / 16 skipped — identical
to clean main baseline. Live curl against the running server:
GET /api/health -> 200, GET /api/acks -> 200, GET /api/ta1-acks -> 200.
See /tmp/refactor-cyclone.md for the full plan.
Adds in-place key rotation for the encrypted DB at rest (HIPAA
sec.164.308(a)(4) - periodic key rotation).
- db_crypto.rotate_db_key(): opens with old key, issues PRAGMA rekey,
reopens with new key, verifies schema survived (table-count sanity).
- db_crypto.generate_db_key(): fresh 256-bit CSPRNG hex key.
- db_crypto.fingerprint(): SHA-256[:8] of a key, for the operator to
compare across rotations.
- db.dispose_engine() + db.reinit_engine(): SP15 plumbing. The
rotation endpoint disposes the pooled connections (SQLCipher
refuses to rekey while another connection holds the file), runs
the rekey, then rebuilds the engine with the new key from the
Keychain.
- API: POST /api/admin/db/rotate-key with module-level threading.Lock
to serialize rotations. 400 when encryption not enabled, 409 when
a rotation is already in flight, 503 on rekey or Keychain failure
with a reason that tells the operator what to do next.
- Engine uses NullPool when SQLCipher is enabled: the default
QueuePool returns connections to a shared queue that any thread
can pull from, which breaks SQLCipher's thread affinity. NullPool
trades connection reuse for thread safety, the only correct
behavior under FastAPI's per-request threadpool.
- Audit event db.key_rotated with old/new fingerprints and
table_count, written after the engine is rebuilt so the new key
proves it can take new writes.
- previous key is stashed to a second Keychain account so the
operator can roll back if the new key turns out to be broken.
Tests:
- test_db_crypto.py: 12 new tests for generate/fingerprint/rekey
mechanics (5 require SQLCipher at runtime; skipped otherwise).
- test_api_rotate_key.py: 6 new tests for endpoint wiring
(encryption-required, Keychain update, audit event, rekey-failure
rollback, Keychain-write-failure 503, concurrent-rotation 409).
First checkpoint of the architecture satisfaction loop. Cyclone's api.py
was a 2281-line god-module with 14 cross-cutting helpers inlined next
to the @app route declarations. This commit moves them to a dedicated
cyclone.api_helpers module:
- NDJSON wire format: ndjson_line, ndjson_stream_837, ndjson_stream_835,
ndjson_stream_list.
- Content negotiation: client_wants_json, wants_ndjson.
- Strict / raw_segments rewrites: strict_rewrite_837, strict_rewrite_835,
drop_raw_segments_837, drop_raw_segments_835.
- Validation probes: has_claim_validation_errors, has_835_validation_errors.
- Live-tail generator: tail_events, heartbeat_seconds, utcnow.
api.py re-imports them under the original underscore-prefixed names so
every route call site stays unchanged. claims_stream, remittances_stream,
and activity_stream remain exposed at cyclone.api (test_api_stream_live
imports them directly).
Verifies byte-identical NDJSON wire format, content negotiation rules,
and the tail_events async-generator semantics (deliberately polls the
EventBus queue rather than awaiting its async iterator, so heartbeats
don't poison the bus subscription).
Live-tested: GET /api/health, /api/claims, /api/remittances,
/api/activity, /api/acks, /api/providers, /api/inbox/lanes,
/api/inbox/payer-rejected/acknowledge, and the /api/claims/stream
NDJSON tail all return expected codes / payload.
Backend pytest: 29 failures identical to baseline (pre-existing
secrets env, serialize_837, db_crypto env, prodfile env failures),
700 passed. Frontend npm test: 357/357 passing.
See /tmp/refactor-cyclone.md for the full checkpoint log and the plan
for the next step (splitting api.py routes into FastAPI APIRouters).
Autoreview: /tmp/grok-review-local.md (0 bugs, 1 suggestion, 4 nits —
all addressed: dead asyncio/os imports removed, dead Any import
removed, duplicate utcnow import dropped, trailing newline added).
Closes the gap between the SP10 backend (5 lanes) and the SP6
frontend (4 lanes). The Payer-Rejected lane (277CA STC A4/A6/A7)
is now rendered alongside Rejected/Candidates/Unmatched/Done,
with an Acknowledge bulk action that drops claims from the
working surface without erasing the original 277CA rejection
event (audit log stays intact, SP11).
Backend:
* Migration 0010: add payer_rejected_acknowledged_at +
payer_rejected_acknowledged_actor columns + partial index.
* db.py: surface the two new columns on the Claim model.
* inbox_lanes.py: filter acknowledged claims out of the
payer_rejected lane; expose the new fields on the row payload
for forward-compat (e.g. a future 'Recently acknowledged' view).
* api.py:
- POST /api/inbox/payer-rejected/acknowledge
Bulk-acknowledge. Idempotent. Returns transitioned /
already_acked / not_found / not_rejected counts so the UI
can show '3 of 5 were already acknowledged' on a noop bulk.
Writes a 'claim.payer_rejected_acknowledged' event to the
SP11 hash-chained audit log.
- GET /api/inbox/export.csv: accept 'payer_rejected' lane.
* test_acks.py: bump user_version assertion to 10.
* test_lane_filter_acknowledged.py: 4 tests for the lane filter
and forward-compat row payload.
* test_payer_rejected_acknowledge.py: 6 tests for the endpoint
(happy path, idempotency, no-op on non-rejected, missing
ids, 400 on empty, audit-log wiring + chain integrity).
Frontend:
* lib/inbox-api.ts: add payer_rejected to InboxLanes, add
acknowledgePayerRejected(), update exportInboxCsvUrl union.
* hooks/useInboxLanes.ts: add payer_rejected to initial state.
* hooks/useInboxLanes.test.ts: add payer_rejected to mocks.
* components/inbox/BulkBar.tsx: add 'payer_rejected' lane with
Acknowledge action (no Resubmit, no Dismiss — payer-rejected
is not eligible for either).
* components/inbox/BulkBar.test.tsx: add payer_rejected test.
* pages/Inbox.tsx: render the 5th lane, hook up onAcknowledge,
include payer_rejected in the needEyes count.
* pages/Inbox.test.tsx: 3 new tests (5-lane render, need-eyes
count, acknowledge action hits the right endpoint).
* components/inbox/InboxHeader.tsx: doc comment now explains
why payer_rejected rolls up into need-eyes.
Pre-existing typecheck warnings in BulkBar.test.tsx / InboxRow
.test.tsx / Lane.tsx / download.test.ts are unchanged from
main — not touched here.
Test counts: backend 724 -> 734 (+10). Frontend 350 -> 354 (+4).
Replace SftpClient stub write_file/list_inbound/read_file
implementations with real paramiko SSHClient + SFTPClient
calls. The public API (SftpClient.write_file, list_inbound,
read_file, get_secret) is unchanged from SP9 — same signature,
same return types — so the API layer needs no changes.
Real-mode behavior:
* _connect() returns a context manager yielding (ssh, sftp);
closes both on exit. Lazy-imports paramiko so the stub-only
test path doesn't need the dependency.
* Auth resolves from SftpBlock.auth: password_keychain_account
(MFT model) or key_file + optional key_passphrase_keychain_account.
Missing Keychain entries fail loud (RuntimeError) rather than
silently attempting empty-password auth.
* write_file: opens sftp.open(remote, 'wb') and writes bytes;
mkdirs the parent dir (idempotent — MFT pre-creates FromHPE/ToHPE).
* list_inbound: listdir_attr + per-file download into local
staging cache; skips directory entries (0o040000 mask).
* read_file: download via shutil.copyfileobj into BytesIO.
Stub mode is unchanged. AutoAddPolicy for first-time MFT host
fingerprint; operator should pin the key for production.
Adds tests/test_sftp_paramiko.py: 9 tests covering
* stub still works
* real-mode connect builds correct paramiko call from
password_keychain_account, raises on missing Keychain,
raises on missing auth config, raises on STUB_SECRET
* write_file opens 'wb' on the right path and writes bytes
* list_inbound translates attrs into InboundFile records and
caches files locally; skips dirs
Removes 2 obsolete tests in test_sftp_stub.py that expected
SP13-mode to raise NotImplementedError.
pyproject.toml: new optional 'sftp' extra (paramiko>=3.4,<6).
- New cyclone.db_crypto module:
* is_sqlcipher_available() — capability check
* is_encryption_enabled() — Keychain key + sqlcipher3 present
* get_db_key() — reads 'cyclone.db.key' from Keychain
* make_sqlcipher_connect_creator(url, key) — SQLAlchemy creator
- db._make_engine() now switches to SQLCipher when key is present
- pyproject.toml: optional 'sqlcipher' extra (sqlcipher3>=0.6,<1)
- Fallback: without Keychain key, DB stays plain SQLite (no surprise
behavior for operators who haven't set up encryption yet)
- Verified: encrypted file is unreadable as plain SQLite, wrong key
raises on first query, migrations + ORM work transparently
- HIPAA §164.312(a)(2)(iv) compliance note in docs
Tests: 705 -> 717 (12 new for SQLCipher). All 717 backend tests pass.
- Mark SP8 as shipped (Sub-projects 2 through 8)
- Drop the 'Next up: outbound 837P serializer' trailer
- New 'Outbound 837 Serializer' section above 'Per-Line Adjustment
Audit': design rationale (Approach A full rebuild vs spec §3.1
hybrid), two-surface UX (single-claim drawer download + multi-claim
inbox ZIP bundle), pointers to the serializer module, the two API
endpoints, the frontend helpers, and the UI entry points.
- New 'SP8 endpoints' subsection in the endpoint inventory.
- inbox-api.test.ts: pin ?download=true POST contract; blob + filename +
X-Cyclone-Serialize-Errors parsing; non-2xx error surfacing.
- Inbox.test.tsx: end-to-end multi-select → Resubmit + Download path
verifies api call args and downloadBlob wiring.
- inbox-api.ts: drop redundant isConfigured short-circuit in download
variant (backend has its own auth gate) and switch error reading to
res.text() to match Blob response shape.
Wire the new ``?download=true`` resubmit endpoint into the Inbox
page. Operators can now ask the backend for a ZIP of regenerated
837s straight from the rejected-claims bulk action, with the
``X-Cyclone-Serialize-Errors`` header surfaced as a non-blocking
warning so partial successes don't swallow per-claim failures.
* ``src/lib/inbox-api.ts``: new ``resubmitRejectedWithDownload``
helper returning ``{blob, filename, serializeErrors}`` so callers
can hand the bundle to the new ``downloadBlob`` utility without
re-parsing headers.
* ``src/lib/download.ts``: new ``downloadBlob(blob, filename)`` plus
a test covering the extension/content-type mapping and the
"use the suggested filename when present" rule.
* ``src/pages/Inbox.tsx``: rejected-claims bulk action now exposes
a "Resubmit & download" button next to the existing JSON path,
wired through the helper. Conflicts and per-claim serialize
errors render in the existing toast/result surface.
Tests: 4 new download.ts tests, 5 inbox-api tests (including
serialize-errors header parsing).
Add an optional ``?download=true`` query param to
``POST /api/inbox/rejected/resubmit`` that returns the same operation
result as a ZIP archive of regenerated 837P files (one
``claim-{id}.x12`` per successfully resubmitted claim) rather than the
JSON envelope.
Why: operators who mass-resubmit rejected claims want to hand the
files straight to their clearinghouse; round-tripping through copy-
paste is error-prone.
Implementation notes:
* Uses ``serialize_837_for_resubmit`` so each X12 file in the bundle
gets a unique interchange/group control number (back-to-back
resubmits would otherwise collide on ISA13/GS06 = "000000001").
* Conflicts and missing ids are deliberately excluded from the ZIP
— the user already saw them in the JSON path on prior calls; the
download is the "give me the files I asked for" payload.
* Empty resubmit + download returns 200 with an empty ZIP so the UI
can still hand the user a downloadable artifact.
Tests (test_inbox_endpoints.py): 2 new tests covering the success
shape (one .x12 per accepted claim) and the conflict-exclusion
contract.
Three pieces:
- src/lib/download.ts: generic downloadTextFile(filename, mime, text)
helper. Mirrors csv.ts:downloadCsv but takes an explicit MIME type and
drops the BOM prepend (which would corrupt the ISA segment).
- src/lib/api.ts: serializeClaim837(id) → {text, filename}. Fetches
GET /api/claims/{id}/serialize-837, pulls the suggested filename from
Content-Disposition (falls back to claim-{id}.x12 if the header is
missing). Throws ApiError on non-2xx so callers can branch on .status.
- ClaimDrawerHeader: Download icon button between the amount and the
close button. Click → api.serializeClaim837 → downloadTextFile.
Disabled + 'Downloading 837 file' aria-label while the fetch is in
flight so the click feels responsive. Optional onError prop surfaces
fetch failures; defaults to a no-op so existing callers stay clean.
Tests: 3 download.test.ts, 3 api.test.ts, 2 header.test.ts (happy
path + error path). Frontend: 350 passing (+8 from 342).
X12 835 SVC segment:
SVC01 = composite procedure
SVC02 = charge
SVC03 = payment
SVC04 = Unit or Basis for Measurement Code (UN, MJ, DA, ...)
SVC05 = Service Unit Count
The parser previously read SVC04 as the units count and SVC05 as the
unit type — backwards. On real 835s (and the canonical minimal
fixture), SVC04 carries the code 'UN' which fails Decimal parsing, so
the units always came out as None and the code string was assigned to
unit_type. SP7's line-level matcher couldn't compare units on the SVC
side against the claim side because of this.
- _consume_service_payment: SVC04 → unit_type, SVC05 → units count
- Default unit_type to 'UN' when only the count is present
- minimal_835.txt + unbalanced_835.txt: swap positions to match spec
- Add 2 regression tests (units-and-unit-type, default-unit-type-to-UN)
Returns the persisted ClaimOutput as a regenerated X12 837P file via
the new outbound serializer (Approach A — full rebuild from canonical
fields). 404 on missing claim, 422 if the stored raw_json cannot be
validated as a ClaimOutput. text/x12 content-type, attachment
disposition with the claim id as the filename.
3 tests:
- endpoint returns text/x12 attachment starting with ISA*
- 404 for missing claim id
- regenerated text round-trips back through parse()
- backend/src/cyclone/parsers/serialize_837.py — full-rebuild 837P serializer.
Emits envelope (ISA/GS/ST/SE/GE/IEA + BHT) + submitter/receiver/billing
provider/subscriber/payer hierarchy + editable segments (CLM/REF*G1/HI)
+ per-service-line LX/SV1/DTP*472/REF*6R — all from canonical
ClaimOutput fields.
Pivoted from spec §3.1 hybrid to full rebuild because ClaimOutput.raw_segments
only captures post-CLM segments (CLM, REF*G1, HI, LX, SV1 pairs) — not the
envelope or hierarchies. A pass-through approach cannot regenerate those
without expanding raw_segments in parse_837.py (out of scope for this SP).
- backend/tests/test_serialize_837.py — 36 tests covering envelope shape,
hierarchy segments, claim-level builders, service-line builders, edited-field
propagation, round-trip, custom sender/receiver IDs, and resubmit helper.
- backend/tests/test_prodfiles_smoke.py::test_claims_prodfile_round_trip —
every file in docs/prodfiles/claims/ (113 files) round-trips through
serialize_837 → parse_837_text with deep-equal ClaimOutput (modulo
validation, which is recomputed by the parser).
- docs/superpowers/plans/2026-06-20-cyclone-serialize-837.md — full plan
with amendment note documenting the Approach A pivot.
Per session convention, plan note about unrelated modifications to
parse_835.py / fixtures stashed separately.
Existing test fixtures (ClaimDrawer, ClaimDrawerHeader, Claims pages)
construct partial ClaimDetail objects without the new slim
lineReconciliation projection. Other SP7 fields follow the optional
pattern (matchedLines?, serviceLinePayments?, claimLevelAdjustments?),
so apply the same to lineReconciliation.
Backend always emits it for matched claims; UI treats absent as
"no per-line audit yet".
manual_match previously only flipped the claim↔remit FK and the
claim state — it never ran line-level reconciliation, so manually-
paired claims surfaced empty line-reconciliation rows to the UI
and skipped CLP-level CAS aggregate recompute.
Refactor reconcile.run() to call a new per-pair helper
_reconcile_pair(session, claim, remittance) that:
- clears any existing LineReconciliation rows for the claim
(idempotent re-run; safe across manual_unmatch + manual_rematch
cycles that may pair the claim with a different remittance),
- reads 837 SV1 lines from Claim.raw_json and 835 SVC rows from
ServiceLinePayment,
- runs match_service_lines() and persists a LineReconciliation row
per side,
- recomputes Remittance.claim_level_adjustment_amount (CLP-level
CAS) and Remittance.adjustment_amount (total CAS).
manual_match now calls this helper after the FK is set and before
commit, mirroring the auto-match path. Reversals are skipped (they
don't have SV1↔SVC line pairs; per §7.3).
Tests (test_store_reconcile.py):
- test_manual_match_populates_line_reconciliation_rows: end-to-end
check that a manual pair writes the expected matched +
unmatched_837_only rows plus zero CAS aggregates.
- test_manual_match_idempotent_line_reconciliation: after
manual_unmatch + manual_rematch + inserting a CLP-level CAS row
directly, the claim has exactly two fresh LineReconciliation rows
(no duplicates) and the remittance aggregate reflects the new CAS.
Smoke tested end-to-end via TestClient: parse co_medicaid_837p.txt
+ co_medicaid_835.txt, auto-matcher skips (PCNs differ), manual
match via POST /api/reconciliation/match, then
GET /api/claims/{id}/line-reconciliation returns 2 rows and
GET /api/inbox/lanes shows matched_remittance.total_lines=2.
Note: matched_lines may be 0 on real 835 fixtures due to a
pre-existing parser bug in _consume_service_payment that swaps
SVC04 (units) and SVC05 (unit-of-measure) when reading units from
the segment. The SP7 strict-match criterion requires units parity,
so the misread produces None on the SVC side and strict-match
never succeeds. Surfaced as a follow-up; the SP7 work itself is
correct (line counts are computed and persisted).
- Mark SP4 as fully shipped (batch diff, search, CSV, a11y all landed)
- Add SP6 (Inbox) and SP7 (Per-line reconciliation) sections
- Add SP6 + SP7 endpoint inventories
- Note next up: outbound 837P serializer
The workflow-automation plan was authored but never committed; the
features it specified have shipped, so commit it for the historical
record alongside this README refresh.
Migration renumbered from 0005 to 0006 because 0005_create_ta1_acks.sql
(another agent's WIP, unmerged) already claims version 5. Bumping our
migration to version 6 avoids stepping on it; user_version will land
at 6 once both migrations run.
Tyler
sp15-bot