tyler/cyclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(plan): implementation plan for SP25 orphan data recovery (Step 1)

Browse Source
This commit is contained in:
Nora
2026-07-07 14:18:22 -06:00
parent fa034b1cd7
commit c8f5af3ba5
1 changed files with 119 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/plans/2026-07-07-cyclone-orphan-data-recovery.md
+119
View File
@@ -0,0 +1,119 @@
# Orphan Data Recovery Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use
> superpowers:subagent-driven-development (recommended) or
> superpowers:executing-plans to implement this plan task-by-task. Steps
> use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Recover stranded billing data from `ingest/`, fix the synthetic-batch read-side bomb, and verify the dashboard KPI tiles come back to life after the recovery run.
**Architecture:** Three surgical changes — one defensive read-side fallback in `store/batches.py`, one new offline CLI `cyclone recover-ingest`, one verification script. No new HTTP endpoints, no schema changes, no SFTP interaction. Uses `CycloneStore.add()` as the canonical write path so live-tail pages light up immediately.
**Tech Stack:** Python 3.11+, FastAPI/uvicorn (for FastAPI bootstrap helper to share `EventBus`), SQLAlchemy (existing models), Click (existing CLI), Pydantic v2 (existing models).
**Spec:** [`docs/superpowers/specs/2026-07-07-cyclone-orphan-data-recovery-design.md`](../specs/2026-07-07-cyclone-orphan-data-recovery-design.md)
---
## File structure
```
backend/src/cyclone/
store/batches.py # A: defensive stub in _row_to_record
cli.py # B: add `cyclone recover-ingest`
submission/
recover.py # B: shared parse-then-store logic (new, small)
backend/tests/
test_store_batches_synthetic.py # A: unit
test_api_batches_synthetic.py # A: integration
test_cli_recover_ingest.py # B: cli + integration
fixtures/
recover_837p_minimal.txt # B: 2-claim 837P fixture (synthetic)
recover_835_minimal.x12 # B: 2-payment 835 fixture (synthetic)
backend/scripts/
verify_dashboard_recovery.py # post-recovery sanity check (operator-only)
docs/superpowers/specs/
2026-07-07-cyclone-dashboard-mess-postmortem.md # deep-dive writeup
```
## Task 0: Setup — branch + verify
- [ ] Confirm `main` is the working branch and is clean: `git status -sb`.
- [ ] Create the branch: `git checkout -b sp25-orphan-data-recovery`.
- [ ] Confirm DB file location: `sqlite3 ~/.local/share/cyclone/cyclone.db ".tables"` returns the 22 tables.
## Task 1: A — failing test for `_row_to_record` defensive stub
- [ ] Add `backend/tests/test_store_batches_synthetic.py` with two cases:
1. `test_row_to_record_handles_null_raw_json` — insert a Batch with `raw_result_json=None, kind='837p', input_filename='<synthetic:orphan-reconcile>'`, call `store.list_batches()`, assert the stub returns a `BatchRecord` with `result.summary.total_claims==0` and empty `result.claims`.
2. `test_row_to_record_handles_null_raw_json_835` — same but `kind='835'`, plus assert `result.claims==[]`.
- [ ] Run `cd backend && .venv/bin/pytest tests/test_store_batches_synthetic.py -v` — both tests must fail (`ValidationError` on `summary`).
## Task 2: A — implement stub in `_row_to_record`
- [ ] In `backend/src/cyclone/store/batches.py`, replace the body of `_row_to_record` (lines ~47-79) with a defensive version:
- If `row.raw_result_json` is None or `{}`, build a stub `ParseResult` (837p) with `envelope=None, claims=[], summary=BatchSummary(input_file=row.input_filename, control_number=row.transaction_set_control_number, total_claims=0, passed=0, failed=0)` and a `Date.today()`-based `transaction_date`, **or** a stub `ParseResult835` (835) with `envelope=Envelope(sender_id="",receiver_id="",control_number=row.transaction_set_control_number,transaction_date=date.today()), financial_info=FinancialInfo(... zeros ...), trace=ReassociationTrace(...), payer=Payer835(...), payee=Payee835(...), claims=[], summary=<minimal>`.
- Otherwise behave as today.
- [ ] Re-run `tests/test_store_batches_synthetic.py` — both tests now pass.
- [ ] Run the full backend suite: `cd backend && .venv/bin/pytest -q`. Must stay green.
## Task 3: A — verify against the live DB
- [ ] Restart-free smoke: `curl -sS 'http://127.0.0.1:8000/api/batches?limit=5' | python -m json.tool` — expect 200, `items[].claimCount==0` for the 4 synthetic rows, `inputFilename=="<synthetic:orphan-reconcile>"`.
## Task 4: B — copy prodfiles to fixtures + failing test for `recover-ingest`
- [ ] Copy the source 835 into fixtures (operator-only, but commit it as a fixture for hermetic recovery verification):
- `cp ingest/tp11525703-835_M019771179-20260706005516577-1of1.x12 backend/tests/fixtures/recover_835_real.x12`.
- `cp ingest/tp11525703-837P-20260701162932052-1of1.txt backend/tests/fixtures/recover_837p_real_145claims.txt`.
- [ ] Add `backend/tests/test_cli_recover_ingest.py` with three cases:
1. `test_ingest_835_persists_remittance` — invoke the `recover-ingest` CLI via Click's `CliRunner` with the real 835 fixture; assert `len(store.list_remittances())==1+1`, the row's `total_paid > 0`, and a `processed_inbound_files` row with `sftp_block_name='manual-recover'` was created.
2. `test_ingest_837p_persists_claims` — invoke with the 145-claim 837P fixture; assert `claims_total_count >= 145` via `SessionLocal().query(Claim).count()` and the corresponding `Batch` row's `claim_count` matches.
3. `test_ingest_idempotent` — invoke twice with the same file; second run is a no-op (skip messages logged, no extra rows).
- [ ] Run the test file: must fail (no `recover-ingest` command yet).
## Task 5: B — implement `cyclone recover-ingest`
- [ ] Add `backend/src/cyclone/submission/recover.py` with one function:
- `def recover_file(path: Path, *, sftp_block_name: str = "manual-recover") -> dict` — detects kind from `ST01*` (837 → '837p'; 835 → '835'), calls the matching parser, constructs a `BatchRecord`, calls `CycloneStore.add(record, event_bus=local_bus)`, returns `{"file": str(path), "status": "ok"|"duplicate"|"failed", "batch_id": str|None, "error": str|None}`.
- Wraps the dedup check around `processed_inbound_files` `(sftp_block_name, path.name)`.
- [ ] In `backend/src/cyclone/cli.py`, add the Click command `recover-ingest`:
- Options: `--file PATH` (repeatable), `--sftp-block-name` (default `manual-recover`).
- Validates that each `--file` exists.
- Calls `recover_file` per file; exits 0 if at least one ingested OK, 2 if a file parse failed, 1 otherwise.
- [ ] Re-run `tests/test_cli_recover_ingest.py` — all three pass.
- [ ] Run the full backend suite: `cd backend && .venv/bin/pytest -q`. Stays green.
## Task 6: C — line-reconciliation is automatic; verify with the live DB
- [ ] Capture the pre-recovery `/api/dashboard/kpis` snapshot into a file:
- `curl -sS 'http://127.0.0.1:8000/api/dashboard/kpis' > /tmp/recovery_pre.json`.
- [ ] Run the recovery against the 5 real files:
- `.venv/bin/python -m cyclone.cli recover-ingest --file ingest/tp11525703-837P-20260701162932052-1of1.txt --file ingest/tp11525703-837P-20260701162935524-1of1.txt --file ingest/tp11525703-837P-20260701162938977-1of1.txt --file ingest/tp11525703-837P-20260701162942746-1of1.txt --file ingest/tp11525703-835_M019771179-20260706005516577-1of1.x12`.
- Expect: 4× 837P → 4 new `Batch` rows + 338 `Claim` rows; 1× 835 → 1 new `Batch` + 1 `Remittance` + ~1,148 `service_line_payments` rows + N `Match` rows (auto-reconciled).
- [ ] Capture the post-recovery `/api/dashboard/kpis` snapshot: `curl ... > /tmp/recovery_post.json`.
- [ ] Open `backend/scripts/verify_dashboard_recovery.py` (write the script during this step):
- Parses both JSON files; asserts `count_post >= count_pre + 338`, `billed_post >= $57,986`, `pending_post >= 339`.
- Optionally prints the top-provisioners delta so the operator eyeballs the result.
## Task 7: write the postmortem doc
- [ ] Create `docs/superpowers/specs/2026-07-07-cyclone-dashboard-mess-postmortem.md`:
- Header: `# 2026-07-07 Cyclone Dashboard postmortem`.
- Sections: `## What we saw / ## Root cause / ## What we fixed / ## KPI values before & after / ## Open watch-items`.
- Embed the before/after KPI JSON snapshots as fenced code blocks under `## KPI values before & after`.
## Task 8: final verification + merge
- [ ] `cd backend && .venv/bin/pytest -q` — green.
- [ ] `npm test` in repo root — green.
- [ ] `git status -sb` — clean working tree.
- [ ] `git log --oneline -10` — at minimum: docs(spec), docs(plan), `feat(sp25):`, postmortem commit.
- [ ] Commit the chain (the merge commit is the last one):
- `git commit -m 'docs(spec): design for SP25 orphan data recovery (Step 1)'`
- `git commit -m 'docs(plan): implementation plan for SP25 orphan data recovery (Step 1)'`
- `git commit -am 'feat(sp25): defensive read-side stub for synthetic-batch rows + recover-ingest CLI (Steps 2-5)'`
- `git commit -m 'docs: dashboard mess postmortem (Step 7)'`
- [ ] Merge into `main` with `--no-ff`:
- `git checkout main && git merge --no-ff sp25-orphan-data-recovery -m 'merge: SP25 orphan data recovery into main'`
- Confirm a single atomic merge commit with all the above in its `git log`.