Sub-project 25 — Orphan Data Recovery: Design Spec

Date: 2026-07-07 Status: Draft, awaiting user sign-off Branch: sp25-orphan-data-recovery Aesthetic direction: No new UI; one CLI command + a small read-side defensive fallback.

1. Scope

This increment fixes three concrete data-tracking gaps in the production DB discovered during the 2026-07-07 dashboard postmortem:

A — Defensive rehydration of synthetic Batch rows. Four rows in the batches table (created by CycloneStore.reconcile_orphan_st02s(), kind 837p, input_filename='<synthetic:orphan-reconcile>', SCN 991102986–989) were inserted with raw_result_json=NULL. Any read-side path that deserializes a BatchRecord (currently _row_to_record() in store/batches.py) fails them with the Pydantic error summary: Field required, so GET /api/batches?limit=5 returns HTTP 500. Fix: _row_to_record returns a typed stub BatchRecord with empty claims / remittances lists and a minimal BatchSummary when raw_result_json is None or empty. The 999 ACK tracking these rows anchor remains intact — Batch.transaction_set_control_number and the acks.source_batch_id index are unchanged.
B — Recovery of stranded source files in ingest/. Four tp11525703-837P-20260701...txt claim files (338 claims, $57,986 billed, SCN 991102991–994) and one tp11525703-835_M019771179...x12 remittance file (1,148 payments) were never loaded into the DB. The 837Ps were SFTP-shipped to Gainwell but not recorded (because parse-837 CLI only writes JSON, and submit-batch was never called). The 835 came back from the payer and was dropped into ingest/ manually. New capability: cyclone recover-ingest CLI that parses each file, runs the canonical store write path (CycloneStore.add() → Batch + Claim for 837P; Batch + Remittance + service_line_payments for 835), and records the file in processed_inbound_files for idempotency. Does NOT SFTP-upload (files have already left this host).
C — Auto line-level reconciliation. Running B's 835 load already invokes reconcile.run() (per the submit-batch / store-add contract), which auto-matches the 1,148 service-line payments to their corresponding claims by patient_control_number ↔ payer_claim_control_number. No new code — verification step confirms the line-payment match count after B finishes.

Out of scope

SFTP-upload path stays unchanged. recover-ingest records what was already sent. Re-submitting these 837Ps to the payer would create duplicate billings; the operator explicitly does NOT want that.
999 ACK ingestion for SCN 991102991–994. Those ACK files never arrived in ingest/ (likely lost between Gainwell and the operator's SFTP client, or never generated because the envelopes never reached the payer — dashboard text "999 AK5=R … ak2=0" suggests Gainwell pre-rejected at the interchange level and never emitted AK2/AK5 segments). Recovery is the source 837 + the back 835; the missing 999s explain why the dashboard "0/N accepted" look is wrong even after B.
Re-rendering the Dashboard tiles: a pending 837 creates a SUBMITTED claim, which IS counted in _DASHBOARD_PENDING_STATES ({"submitted", "rejected"}) — so the "Pending AR" tile will light up automatically. The "Billed" tile depends on service_date_from, which the parser already populates.
Audit-log replay or hash-chain repair. The 999 acks in acks for SCN 991102986–989 were recorded by pull-inbound after reconcile_orphan_st02s() seeded the synthetic batches, so their audit chain is intact post-fix.

2. Threats

The auth boundary is HTTP (login required, bcrypt + HttpOnly session cookie); file-system threats remain the local-only threat model (SQLCipher at rest, macOS Keychain). This increment is data-recovery only — no new HTTP endpoints, no new auth surface.

The CLI cyclone recover-ingest runs in the operator's shell with the live DB. Risks: writing duplicate batches if dedup is bypassed; recording claims that weren't actually billed. Mitigation: dedup via processed_inbound_files (block name manual-recover, filename basename) so a re-run is a no-op. The dedup index is ux_processed_inbound_files_block_name UNIQUE (sftp_block_name, name).

The cyclone serve daemon is unaffected; this increment does not restart it. _row_to_record is read-path-only.

3. Decisions

D1. The stub BatchRecord for synthetic rows carries result.summary.total_claims=0, passed=0, failed=0, control_number=<ST02>, transaction_date=parsed_at.date(). UI surfaces see "0 claims, 0 charged" which is correct — these batches deliberately have no claim rows.
D2. cyclone recover-ingest parses locally, calls store.add(record, event_bus=app.state.event_bus) after FastAPI bootstrap so events reach the live-tail pages immediately. Outside of FastAPI: constructs a fresh EventBus() and runs _publish_events_sync against the local subscriber list (same pattern submit-batch uses).
D3. Exit codes: 0 = at least one file ingested OK; 2 = a file failed to parse or DB-write; 1 = unexpected exception. Mirrors submit-batch convention.
D4. Operator passes explicit --file PATH (repeatable) rather than scanning ingest/. The recovery targets known files; auto-scanning would risk pulling in 999 acks that have already been processed.
D5. recover-ingest does NOT inject a synthetic batch row even if the input is a 999 ACK file without a matching source — that's reconcile-orphan-st02s's job and it ran successfully on 7/7 for the SCNs it had visibility into.

4. Test impact

Backend: backend/tests/test_store_batches_synthetic.py — assert _row_to_record tolerates None/empty raw_result_json and returns a stub BatchRecord with empty claims[] and the SCN preserved on result.summary.control_number. Same shape for BatchRecord835.
Backend: backend/tests/test_api_batches_synthetic.py — assert GET /api/batches?limit=5 returns 200 when the table has the 4 synthetic rows.
Backend: backend/tests/test_cli_recover_ingest.py — three cases: ingest a synthetic 837P fixture, ingest the real tp11525703-835_M019771179-20260706005516577-1of1.x12 fixture (copied to backend/tests/fixtures/), idempotency (re-run is a no-op).
Recovery verification: a small inline script under backend/scripts/verify_dashboard_recovery.py that hits /api/dashboard/kpis after running the recovery and asserts count=339+ (≥338), billed ≥ $57,986, received ≥ <post-reconcile-paid>. Runs once during operator verification; not part of pytest.

5. Doc impact

docs/superpowers/specs/2026-07-07-cyclone-dashboard-mess-postmortem.md is the operator-facing writeup of the deep-dive (separate from this design spec). Its purpose is to record what was wrong with the dashboard on 2026-07-07, what was done to fix it, and what to watch for in future runs.

6. Risks

R1. Replaying an 837P that was already billed could re-submit if the operator later runs submit-batch on it. Mitigation: recover-ingest does NOT upload; the file remains unmodified in ingest/.
R2. Auto-reconciliation may not match every 835 line if the underlying claim's patient_control_number is empty or differs between the 837P CLM01 and the 835's CLP01. The verifier script reports matches=0 separately so the operator can decide whether to manually pair.
R3. The "999 AK5=R" dashboard badge for SCN 991102994 will still display as 0/N — there is no 999 to ingest. That's correct behavior, but the operator should expect a dashboard question.

7. Non-goals

Replaying audit log events for the recovered batches. Audit entries are best-effort; the recovery increments log new claim_submitted / remittance_received events with the recovery time.
Rebuilding the top-providers tile. With 4 NPIs still in providers and 338 claims about to land, the tile will recompute on the next KPI call.
Adjusting cyclone serve's in-memory SFTP block. The recovery is offline.

7.9 KiB Raw Blame History Unescape Escape