cyclone/docs/superpowers/specs/2026-07-07-cyclone-orphan-data-recovery-design.md

# Sub-project 25 — Orphan Data Recovery: Design Spec

 **Date:** 2026-07-07
 **Status:** Draft, awaiting user sign-off
 **Branch:** `sp25-orphan-data-recovery`
 **Aesthetic direction:** No new UI; one CLI command + a small read-side defensive fallback.

## 1. Scope

This increment fixes three concrete data-tracking gaps in the production DB discovered during the 2026-07-07 dashboard postmortem:

1. **A — Defensive rehydration of synthetic `Batch` rows.** Four rows in the `batches` table (created by `CycloneStore.reconcile_orphan_st02s()`, kind `837p`, `input_filename='<synthetic:orphan-reconcile>'`, SCN 991102986–989) were inserted with `raw_result_json=NULL`. Any read-side path that deserializes a `BatchRecord` (currently `_row_to_record()` in `store/batches.py`) fails them with the Pydantic error `summary: Field required`, so `GET /api/batches?limit=5` returns HTTP 500. Fix: `_row_to_record` returns a typed stub `BatchRecord` with empty `claims` / `remittances` lists and a minimal `BatchSummary` when `raw_result_json` is None or empty. The 999 ACK tracking these rows anchor remains intact — `Batch.transaction_set_control_number` and the `acks.source_batch_id` index are unchanged.

2. **B — Recovery of stranded source files in `ingest/`.** Four `tp11525703-837P-20260701...txt` claim files (338 claims, $57,986 billed, SCN 991102991–994) and one `tp11525703-835_M019771179...x12` remittance file (1,148 payments) were never loaded into the DB. The 837Ps were SFTP-shipped to Gainwell but not recorded (because `parse-837` CLI only writes JSON, and `submit-batch` was never called). The 835 came back from the payer and was dropped into `ingest/` manually. New capability: `cyclone recover-ingest` CLI that parses each file, runs the canonical store write path (`CycloneStore.add()` → `Batch` + `Claim` for 837P; `Batch` + `Remittance` + `service_line_payments` for 835), and records the file in `processed_inbound_files` for idempotency. Does NOT SFTP-upload (files have already left this host).

3. **C — Auto line-level reconciliation.** Running B's 835 load already invokes `reconcile.run()` (per the `submit-batch` / store-add contract), which auto-matches the 1,148 service-line payments to their corresponding claims by patient_control_number ↔ payer_claim_control_number. No new code — verification step confirms the line-payment match count after B finishes.

### Out of scope

- SFTP-upload path stays unchanged. `recover-ingest` records what was already sent. Re-submitting these 837Ps to the payer would create duplicate billings; the operator explicitly does NOT want that.
- 999 ACK ingestion for SCN 991102991–994. Those ACK files never arrived in `ingest/` (likely lost between Gainwell and the operator's SFTP client, or never generated because the envelopes never reached the payer — dashboard text "999 AK5=R … ak2=0" suggests Gainwell pre-rejected at the interchange level and never emitted AK2/AK5 segments). Recovery is the source 837 + the back 835; the missing 999s explain why the dashboard "0/N accepted" look is wrong even after B.
- Re-rendering the Dashboard tiles: a `pending` 837 creates a `SUBMITTED` claim, which IS counted in `_DASHBOARD_PENDING_STATES` (`{"submitted", "rejected"}`) — so the "Pending AR" tile will light up automatically. The "Billed" tile depends on `service_date_from`, which the parser already populates.
- Audit-log replay or hash-chain repair. The 999 acks in `acks` for SCN 991102986–989 were recorded by `pull-inbound` after `reconcile_orphan_st02s()` seeded the synthetic batches, so their audit chain is intact post-fix.

## 2. Threats

The auth boundary is HTTP (login required, bcrypt + HttpOnly session cookie); file-system threats remain the local-only threat model (SQLCipher at rest, macOS Keychain). This increment is data-recovery only — no new HTTP endpoints, no new auth surface.

The CLI `cyclone recover-ingest` runs in the operator's shell with the live DB. Risks: writing duplicate batches if dedup is bypassed; recording claims that weren't actually billed. Mitigation: dedup via `processed_inbound_files` (block name `manual-recover`, filename basename) so a re-run is a no-op. The dedup index is `ux_processed_inbound_files_block_name UNIQUE (sftp_block_name, name)`.

The `cyclone serve` daemon is unaffected; this increment does not restart it. `_row_to_record` is read-path-only.

## 3. Decisions

- **D1.** The stub `BatchRecord` for synthetic rows carries `result.summary.total_claims=0`, `passed=0`, `failed=0`, `control_number=<ST02>`, `transaction_date=parsed_at.date()`. UI surfaces see "0 claims, 0 charged" which is correct — these batches deliberately have no claim rows.
- **D2.** `cyclone recover-ingest` parses locally, calls `store.add(record, event_bus=app.state.event_bus)` after FastAPI bootstrap so events reach the live-tail pages immediately. Outside of FastAPI: constructs a fresh `EventBus()` and runs `_publish_events_sync` against the local subscriber list (same pattern `submit-batch` uses).
- **D3.** Exit codes: 0 = at least one file ingested OK; 2 = a file failed to parse or DB-write; 1 = unexpected exception. Mirrors `submit-batch` convention.
- **D4.** Operator passes explicit `--file PATH` (repeatable) rather than scanning `ingest/`. The recovery targets known files; auto-scanning would risk pulling in 999 acks that have already been processed.
- **D5.** `recover-ingest` does NOT inject a synthetic batch row even if the input is a 999 ACK file without a matching source — that's `reconcile-orphan-st02s`'s job and it ran successfully on 7/7 for the SCNs it had visibility into.

## 4. Test impact

- Backend: `backend/tests/test_store_batches_synthetic.py` — assert `_row_to_record` tolerates None/empty `raw_result_json` and returns a stub `BatchRecord` with empty `claims[]` and the SCN preserved on `result.summary.control_number`. Same shape for `BatchRecord835`.
- Backend: `backend/tests/test_api_batches_synthetic.py` — assert `GET /api/batches?limit=5` returns 200 when the table has the 4 synthetic rows.
- Backend: `backend/tests/test_cli_recover_ingest.py` — three cases: ingest a synthetic 837P fixture, ingest the real `tp11525703-835_M019771179-20260706005516577-1of1.x12` fixture (copied to `backend/tests/fixtures/`), idempotency (re-run is a no-op).
- Recovery verification: a small inline script under `backend/scripts/verify_dashboard_recovery.py` that hits `/api/dashboard/kpis` after running the recovery and asserts `count=339+ (≥338), billed ≥ $57,986, received ≥ <post-reconcile-paid>`. Runs once during operator verification; not part of pytest.

## 5. Doc impact

`docs/superpowers/specs/2026-07-07-cyclone-dashboard-mess-postmortem.md` is the operator-facing writeup of the deep-dive (separate from this design spec). Its purpose is to record what was wrong with the dashboard on 2026-07-07, what was done to fix it, and what to watch for in future runs.

## 6. Risks

- **R1.** Replaying an 837P that was already billed could re-submit if the operator later runs `submit-batch` on it. Mitigation: `recover-ingest` does NOT upload; the file remains unmodified in `ingest/`.
- **R2.** Auto-reconciliation may not match every 835 line if the underlying claim's `patient_control_number` is empty or differs between the 837P CLM01 and the 835's CLP01. The verifier script reports `matches=0` separately so the operator can decide whether to manually pair.
- **R3.** The "999 AK5=R" dashboard badge for SCN 991102994 will still display as 0/N — there is no 999 to ingest. That's correct behavior, but the operator should expect a dashboard question.

## 7. Non-goals

- Replaying audit log events for the recovered batches. Audit entries are best-effort; the recovery increments log new `claim_submitted` / `remittance_received` events with the recovery time.
- Rebuilding the top-providers tile. With 4 NPIs still in providers and 338 claims about to land, the tile will recompute on the next KPI call.
- Adjusting `cyclone serve`'s in-memory SFTP block. The recovery is offline.