Replace SftpClient stub write_file/list_inbound/read_file
implementations with real paramiko SSHClient + SFTPClient
calls. The public API (SftpClient.write_file, list_inbound,
read_file, get_secret) is unchanged from SP9 — same signature,
same return types — so the API layer needs no changes.
Real-mode behavior:
* _connect() returns a context manager yielding (ssh, sftp);
closes both on exit. Lazy-imports paramiko so the stub-only
test path doesn't need the dependency.
* Auth resolves from SftpBlock.auth: password_keychain_account
(MFT model) or key_file + optional key_passphrase_keychain_account.
Missing Keychain entries fail loud (RuntimeError) rather than
silently attempting empty-password auth.
* write_file: opens sftp.open(remote, 'wb') and writes bytes;
mkdirs the parent dir (idempotent — MFT pre-creates FromHPE/ToHPE).
* list_inbound: listdir_attr + per-file download into local
staging cache; skips directory entries (0o040000 mask).
* read_file: download via shutil.copyfileobj into BytesIO.
Stub mode is unchanged. AutoAddPolicy for first-time MFT host
fingerprint; operator should pin the key for production.
Adds tests/test_sftp_paramiko.py: 9 tests covering
* stub still works
* real-mode connect builds correct paramiko call from
password_keychain_account, raises on missing Keychain,
raises on missing auth config, raises on STUB_SECRET
* write_file opens 'wb' on the right path and writes bytes
* list_inbound translates attrs into InboundFile records and
caches files locally; skips dirs
Removes 2 obsolete tests in test_sftp_stub.py that expected
SP13-mode to raise NotImplementedError.
pyproject.toml: new optional 'sftp' extra (paramiko>=3.4,<6).
- New cyclone.db_crypto module:
* is_sqlcipher_available() — capability check
* is_encryption_enabled() — Keychain key + sqlcipher3 present
* get_db_key() — reads 'cyclone.db.key' from Keychain
* make_sqlcipher_connect_creator(url, key) — SQLAlchemy creator
- db._make_engine() now switches to SQLCipher when key is present
- pyproject.toml: optional 'sqlcipher' extra (sqlcipher3>=0.6,<1)
- Fallback: without Keychain key, DB stays plain SQLite (no surprise
behavior for operators who haven't set up encryption yet)
- Verified: encrypted file is unreadable as plain SQLite, wrong key
raises on first query, migrations + ORM work transparently
- HIPAA §164.312(a)(2)(iv) compliance note in docs
Tests: 705 -> 717 (12 new for SQLCipher). All 717 backend tests pass.
Add an optional ``?download=true`` query param to
``POST /api/inbox/rejected/resubmit`` that returns the same operation
result as a ZIP archive of regenerated 837P files (one
``claim-{id}.x12`` per successfully resubmitted claim) rather than the
JSON envelope.
Why: operators who mass-resubmit rejected claims want to hand the
files straight to their clearinghouse; round-tripping through copy-
paste is error-prone.
Implementation notes:
* Uses ``serialize_837_for_resubmit`` so each X12 file in the bundle
gets a unique interchange/group control number (back-to-back
resubmits would otherwise collide on ISA13/GS06 = "000000001").
* Conflicts and missing ids are deliberately excluded from the ZIP
— the user already saw them in the JSON path on prior calls; the
download is the "give me the files I asked for" payload.
* Empty resubmit + download returns 200 with an empty ZIP so the UI
can still hand the user a downloadable artifact.
Tests (test_inbox_endpoints.py): 2 new tests covering the success
shape (one .x12 per accepted claim) and the conflict-exclusion
contract.
X12 835 SVC segment:
SVC01 = composite procedure
SVC02 = charge
SVC03 = payment
SVC04 = Unit or Basis for Measurement Code (UN, MJ, DA, ...)
SVC05 = Service Unit Count
The parser previously read SVC04 as the units count and SVC05 as the
unit type — backwards. On real 835s (and the canonical minimal
fixture), SVC04 carries the code 'UN' which fails Decimal parsing, so
the units always came out as None and the code string was assigned to
unit_type. SP7's line-level matcher couldn't compare units on the SVC
side against the claim side because of this.
- _consume_service_payment: SVC04 → unit_type, SVC05 → units count
- Default unit_type to 'UN' when only the count is present
- minimal_835.txt + unbalanced_835.txt: swap positions to match spec
- Add 2 regression tests (units-and-unit-type, default-unit-type-to-UN)
Returns the persisted ClaimOutput as a regenerated X12 837P file via
the new outbound serializer (Approach A — full rebuild from canonical
fields). 404 on missing claim, 422 if the stored raw_json cannot be
validated as a ClaimOutput. text/x12 content-type, attachment
disposition with the claim id as the filename.
3 tests:
- endpoint returns text/x12 attachment starting with ISA*
- 404 for missing claim id
- regenerated text round-trips back through parse()
- backend/src/cyclone/parsers/serialize_837.py — full-rebuild 837P serializer.
Emits envelope (ISA/GS/ST/SE/GE/IEA + BHT) + submitter/receiver/billing
provider/subscriber/payer hierarchy + editable segments (CLM/REF*G1/HI)
+ per-service-line LX/SV1/DTP*472/REF*6R — all from canonical
ClaimOutput fields.
Pivoted from spec §3.1 hybrid to full rebuild because ClaimOutput.raw_segments
only captures post-CLM segments (CLM, REF*G1, HI, LX, SV1 pairs) — not the
envelope or hierarchies. A pass-through approach cannot regenerate those
without expanding raw_segments in parse_837.py (out of scope for this SP).
- backend/tests/test_serialize_837.py — 36 tests covering envelope shape,
hierarchy segments, claim-level builders, service-line builders, edited-field
propagation, round-trip, custom sender/receiver IDs, and resubmit helper.
- backend/tests/test_prodfiles_smoke.py::test_claims_prodfile_round_trip —
every file in docs/prodfiles/claims/ (113 files) round-trips through
serialize_837 → parse_837_text with deep-equal ClaimOutput (modulo
validation, which is recomputed by the parser).
- docs/superpowers/plans/2026-06-20-cyclone-serialize-837.md — full plan
with amendment note documenting the Approach A pivot.
Per session convention, plan note about unrelated modifications to
parse_835.py / fixtures stashed separately.
manual_match previously only flipped the claim↔remit FK and the
claim state — it never ran line-level reconciliation, so manually-
paired claims surfaced empty line-reconciliation rows to the UI
and skipped CLP-level CAS aggregate recompute.
Refactor reconcile.run() to call a new per-pair helper
_reconcile_pair(session, claim, remittance) that:
- clears any existing LineReconciliation rows for the claim
(idempotent re-run; safe across manual_unmatch + manual_rematch
cycles that may pair the claim with a different remittance),
- reads 837 SV1 lines from Claim.raw_json and 835 SVC rows from
ServiceLinePayment,
- runs match_service_lines() and persists a LineReconciliation row
per side,
- recomputes Remittance.claim_level_adjustment_amount (CLP-level
CAS) and Remittance.adjustment_amount (total CAS).
manual_match now calls this helper after the FK is set and before
commit, mirroring the auto-match path. Reversals are skipped (they
don't have SV1↔SVC line pairs; per §7.3).
Tests (test_store_reconcile.py):
- test_manual_match_populates_line_reconciliation_rows: end-to-end
check that a manual pair writes the expected matched +
unmatched_837_only rows plus zero CAS aggregates.
- test_manual_match_idempotent_line_reconciliation: after
manual_unmatch + manual_rematch + inserting a CLP-level CAS row
directly, the claim has exactly two fresh LineReconciliation rows
(no duplicates) and the remittance aggregate reflects the new CAS.
Smoke tested end-to-end via TestClient: parse co_medicaid_837p.txt
+ co_medicaid_835.txt, auto-matcher skips (PCNs differ), manual
match via POST /api/reconciliation/match, then
GET /api/claims/{id}/line-reconciliation returns 2 rows and
GET /api/inbox/lanes shows matched_remittance.total_lines=2.
Note: matched_lines may be 0 on real 835 fixtures due to a
pre-existing parser bug in _consume_service_payment that swaps
SVC04 (units) and SVC05 (unit-of-measure) when reading units from
the segment. The SP7 strict-match criterion requires units parity,
so the misread produces None on the SVC side and strict-match
never succeeds. Surfaced as a follow-up; the SP7 work itself is
correct (line counts are computed and persisted).
T7-T10 combined (single edit: all endpoints share a section).
- GET /api/inbox/lanes — four lanes in one call
- POST /api/inbox/candidates/{remit_id}/match — 409 on conflict
- POST /api/inbox/candidates/dismiss — session-scoped dismissed set
- POST /api/inbox/rejected/resubmit — 200 with conflicts list
- GET /api/inbox/export.csv?lane=<lane> — streams CSV
Also adds module-level imports (db, Claim, ClaimState, Remittance,
csv, io, datetime) that the new endpoints need; cleans up the
duplicated local imports in the parse-999 SP6 T4 block.
T4 — calls apply_999_rejections inside /api/parse-999, then
publishes a claim.rejected event for each transitioned claim so
the Inbox live-tail refetches.
T3 — moves claims to ClaimState.REJECTED on 999 AK5 R/E/X.
- New module: src/cyclone/inbox_state.py
- New tests: tests/test_inbox_state.py (4 cases)
- Adds ORM mappings for rejection_reason / rejected_at / resubmit_count
/ state_changed_at on Claim (db.py) — these were added to the
schema by migration 0004 but not yet exposed to the ORM.
Adds:
- claims.rejection_reason, claims.rejected_at, claims.resubmit_count
- claims.state_changed_at (was missing, needed for Done-today lane)
- ix_claims_state_changed_at composite index
Also fixes the ClaimState count assertion in test_db_models.py
(7 → 8) to match the REJECTED enum value added in the previous
commit, and bumps the user_version expectation in test_acks.py
(3 → 4) for the same reason.
The store dedupes remittances by PCN. Refine the assertions:
- total_clps counts the raw CLP segments across all files (3374)
- unique_pcns counts distinct PCNs (persisted row count)
- Read claimId via the API response shape, not the SQLAlchemy attribute
Three related changes for real CO Medicaid data:
1. Drop UNIQUE(batch_id, patient_control_number) on claims and
UNIQUE(batch_id, payer_claim_control_number) on remittances. The X12
spec allows multiple CLM segments per 2000B subscriber loop and 835
ERAs can repeat a payer_claim_control_number for reversals. Claim/
remittance identity is provided by the primary key (claims.id = CLM01,
remittances.id = CLP01).
2. Add validator rule R835_MULTI_BPR warning for files with multiple BPR
segments (CO Medicaid split-payment pattern). The parser already sums
BPR02 paid_amounts; this surfaces the non-standard data to operators.
3. Skip R835_BAL_BPR_vs_CLP04 when BPR01='I' (Information Only 835).
In that mode BPR02 is informational and the per-claim CLP04 totals
are authoritative — a diff is expected, not an error.
Migration 0003 handles the drop with IF EXISTS so fresh DBs skip cleanly.
Updates affected tests to reflect new schema (no UNIQUE constraint on
batch_id + patient_control_number / payer_claim_control_number).
Fixes test_api_835::test_prodfile_round_trip_persists_separately which
was failing on real production data.
Adds three live-tail streaming endpoints that emit an NDJSON snapshot
then forward new event-bus events as they arrive, with a 15s idle
heartbeat (overridable via CYCLONE_TAIL_HEARTBEAT_S for tests).
Each endpoint:
1. yields a snapshot of existing rows as {"type":"item","data":<row>}
2. terminates the snapshot with {"type":"snapshot_end","data":{"count":N}}
3. subscribes to its event kind and forwards each new event as an
{"type":"item","data":<event>} line
4. emits a {"type":"heartbeat","data":{"ts":<iso>}} line every
CYCLONE_TAIL_HEARTBEAT_S seconds when idle
5. checks request.is_disconnected() before each yield and unsubscribes
from the bus on cleanup so a closed stream releases its queue
The shared tail loop lives in api._tail_events, which polls
bus.subscribe_raw()'s queue directly instead of using the bus's
async-iterator wrapper — wait_for on an async generator cancels the
inner future on timeout, which poisons subsequent __anext__ calls with
StopAsyncIteration. Queue.get() is idempotent under cancellation, so
heartbeats don't break the subscription.
EventBus gains an unsubscribe(queue, kinds) method (idempotent) so
the tail loop can release its queue in a try/finally. The disconnect
test asserts the subscriber list is empty after the body iterator is
closed, validating no queue leak per open stream.
Tests in test_api_stream_live.py: 8 tests covering snapshot shape,
post-snapshot publish, heartbeat timing, multi-item snapshots, and
client disconnect cleanup. Plus 2 tests in test_pubsub.py for the
new unsubscribe method.
- api.parse_837 / parse_835: pass request.app.state.event_bus into store.add()
- conftest: autouse fixture wires a fresh EventBus onto app.state for every
test, since TestClient does not invoke the FastAPI lifespan handler
unless used as a context manager
- test_pubsub: split get_event_bus coverage into a raises-when-missing test
and a returns-attached-bus test, both save/restore app.state.event_bus
around the assertion so the autouse fixture's bus is preserved
Phase 2 complete: db init moved to lifespan, EventBus is the process-wide
publish point, and the two ingest endpoints publish claim_written /
remittance_written / activity_recorded events on every store.add().
The P3 subagent shipped a Download button on /acks that fell back to
'no raw text' because the detail endpoint returned raw_json (the parsed
model dump) but never the regenerated X12.
Fix: detail endpoint now re-serializes raw_json via serialize_999 and
returns it as raw_999_text, so the operator can actually download the
999 file. List endpoint unchanged (keeps payload small).
Nora
Tyler