diff --git a/docs/superpowers/specs/2026-06-19-cyclone-production-readiness-design.md b/docs/superpowers/specs/2026-06-19-cyclone-production-readiness-design.md new file mode 100644 index 0000000..97146de --- /dev/null +++ b/docs/superpowers/specs/2026-06-19-cyclone-production-readiness-design.md @@ -0,0 +1,377 @@ +# Cyclone Production-Readiness (local-only) — Design + +**Date:** 2026-06-19 +**Status:** Approved (pending user review of this doc) +**Scope:** First sub-project of the four-part Cyclone roadmap. Local-only deployment (no auth, `127.0.0.1`-bound). Adds a backend in-memory batch store, GET endpoints, react-query wiring on the frontend, fresh reference notes, and a new root README. Out of scope: DB persistence, reconciliation, additional X12 transaction types, additional 837P/835 validation rules. + +--- + +## 1. Overview + +The Cyclone EDI suite already has working 837P and 835 parsers, a FastAPI surface for parsing uploads, and a React frontend with an Upload page. What it does not have is a way to *browse* the parsed data — once a file is uploaded, the user sees it in the Upload page, but a refresh or page navigation loses the result, and the existing Dashboard / Claims / Remittances / Providers / Activity pages still read static sample data. + +This sub-project closes that loop. We add a process-local in-memory store on the backend, expose GET endpoints that match the resource shape the existing UI already expects, wire react-query into the frontend so pages fetch live data and auto-refresh after a parse, and replace the lost reference notes with a clean new docs tree. + +After this, a user can: +1. Start the backend (`python -m cyclone serve`). +2. Start the Vite dev server (`npm run dev`). +3. Open `http://localhost:5173/upload`, drop in a `.txt` file, watch the claims stream in. +4. Navigate to `/claims`, `/remittances`, `/providers`, `/activity` and see the data they just parsed. +5. The whole stack is bound to `127.0.0.1`; no auth, no internet exposure. + +## 2. Goals + +1. **Persist parsed data for the session.** A new `InMemoryStore` on the backend keeps every successful parse and serves it to GET requests. Lost on backend restart — that is acceptable for a local-only tool and avoids a DB in this sub-project. +2. **Expose GET endpoints that match the UI's resource model.** `GET /api/claims`, `/api/remittances`, `/api/providers`, `/api/activity` plus `/api/batches` and `/api/batches/{id}`. The list endpoints accept filter / sort / pagination query params and support both JSON and NDJSON streaming. +3. **Wire the existing 4 UI pages to the live API** via `@tanstack/react-query` v5, with loading skeletons, error states, and automatic invalidation after a successful parse. Fall back to the in-memory sample store when `VITE_API_BASE_URL` is empty (the existing `data` adapter pattern stays). +4. **Replace the lost reference notes** with 4 fresh, short notes under `docs/reference/` (~50–80 lines each) covering 837P, 835, X12 naming, and CO Medicaid specifics. Also rewrite the root `README.md`. +5. **Tighten the local-only deploy posture.** `python -m cyclone serve` binds `127.0.0.1:8000` (not `0.0.0.0`); CORS allowlist stays `http://localhost:5173`. No auth, no API key. The README documents the exact command. + +## 3. Non-goals (this sub-project) + +- **Database persistence.** SQLite/Postgres is sub-project 2. +- **837P ↔ 835 reconciliation.** Sub-project 2. +- **More 837P/835 validation rules** (REF*G1 enforcement, BHT06, CAS deep-parsing). Sub-project 3. +- **999 ACK, 270/271, or other X12 transaction types.** Sub-project 3. +- **Auth of any kind.** Explicitly out — local-only, single-user, `127.0.0.1`-bound. +- **Structured logging, health-check enhancements, 12-factor env config, Dockerfile.** Deferred. +- **Dev tooling polish** (pre-commit, Makefile, CONTRIBUTING.md, .editorconfig). Deferred. + +## 4. Stack + +**Backend additions:** +- New module `cyclone.store` (`InMemoryStore`, `BatchRecord`, mappers). +- FastAPI (already in use). New routes added to `cyclone.api`. +- `threading.Lock` around the store's batch list (single-process, but FastAPI may run request handlers in a threadpool). +- `pytest` + `fastapi.testclient.TestClient` for new tests. + +**Frontend additions:** +- `@tanstack/react-query` v5 added to `package.json` dependencies. +- `QueryClientProvider` in `src/main.tsx`. +- `useQuery` / `useMutation` hooks under `src/hooks/`. +- `Skeleton` UI primitive added under `src/components/ui/` (re-uses `cn` helper). +- No build-tool changes; the existing Vite + TS + Tailwind stack stays. + +**Docs:** +- Plain Markdown under `docs/reference/`. No doc generator, no linter, no CI check. + +## 5. Architecture + +``` +┌──────────────────────┐ POST /api/parse-{837,835} ┌─────────────────────────┐ +│ Vite/React UI │ ────────────────────────────▶ │ FastAPI backend │ +│ │ GET /api/{batches,claims, │ (uvicorn 127.0.0.1) │ +│ ┌────────────────┐ │ remittances,providers, │ ┌────────────────────┐ │ +│ │ react-query v5 │ │ activity,batches/{id}} │ │ InMemoryStore │ │ +│ │ QueryClient │ │ ◀──────────────────────────── │ │ - batches: list │ │ +│ │ useQuery / │ │ (JSON or NDJSON stream) │ │ - lock │ │ +│ │ useMutation │ │ │ │ - mapper funcs │ │ +│ └────────────────┘ │ │ └────────────────────┘ │ +│ ┌────────────────┐ │ │ ┌────────────────────┐ │ +│ │ zustand store │ │ │ │ parse-837/835 │ │ +│ │ (sample data) │ │ │ │ (already shipped) │ │ +│ └────────────────┘ │ │ └────────────────────┘ │ +└──────────────────────┘ └─────────────────────────┘ + ▲ + │ CORS allow + │ http://localhost:5173 + │ only +``` + +**Data flow on parse:** +1. User drops a file in `/upload`, clicks Parse. +2. Frontend `useMutation` POSTs to `/api/parse-837` (or 835) with `onProgress` (NDJSON) or `Accept: application/json` for small files. +3. Backend's existing parse route appends the resulting `ParseResult` to `InMemoryStore` on success. Failures are *not* stored. +4. `useMutation.onSuccess` calls `queryClient.invalidateQueries(['batches'], ['claims'], ['remittances'], ['activity'])` (and 837/835-specific keys). +5. Any open page reactively re-fetches via `useQuery` and re-renders with the new data. + +**Data flow on browse:** +1. Page mount → `useQuery` fires the GET with the page's filter / sort / pagination params. +2. First render: skeleton. Success: data. Error: error block with retry. +3. Background: `refetchOnWindowFocus: true` (default in v5) so navigating back updates the view. + +## 6. Backend changes + +### 6.1 New module `backend/src/cyclone/store.py` + +```python +class BatchRecord(BaseModel): + model_config = ConfigDict(extra="ignore") + id: str # uuid4 hex + kind: Literal["837p", "835"] + input_filename: str + parsed_at: datetime # tz-aware UTC + result: ParseResult | ParseResult835 # union discriminated by .kind + +class InMemoryStore: + def __init__(self) -> None: + self._batches: list[BatchRecord] = [] + self._lock = threading.Lock() + + def add(self, record: BatchRecord) -> None: ... + def list(self, *, limit: int = 100) -> list[BatchRecord]: ... + def get(self, batch_id: str) -> BatchRecord | None: ... + def iter_claims(self, *, batch_id: str | None = None, ...) -> Iterable[Claim]: ... + def iter_remittances(self, *, batch_id: str | None = None, ...) -> Iterable[Remittance]: ... + def distinct_providers(self) -> list[Provider]: ... + def recent_activity(self, *, limit: int = 200) -> list[Activity]: ... + +store = InMemoryStore() # module-level singleton +``` + +**Mappers** (`to_ui_claim`, `to_ui_remittance`, `to_ui_provider`, `to_activity_event`) live in the same file. They translate the rich backend models (Pydantic, raw segments) to the simpler UI types the GET endpoints return. + +**Filter / sort API:** each `iter_*` method takes the relevant filter kwargs and applies them in Python. There is no index — the in-memory list is small (single-session, dozens of batches at most). Sorting uses `sorted(..., key=..., reverse=order == "desc")`. + +**Threading:** every public method acquires `self._lock` (RLock, since `list` may call `get` internally) and releases it. + +### 6.2 New routes in `backend/src/cyclone/api.py` + +| Method | Path | Behavior | +|---|---|---| +| GET | `/api/batches` | `store.list(limit=100)`, mapped to a `BatchSummary`-shaped response. Newest first. | +| GET | `/api/batches/{id}` | `store.get(id)`. 404 if missing. Returns the full `ParseResult` / `ParseResult835`. | +| GET | `/api/claims` | `store.iter_claims(...)` mapped to UI `Claim[]`. Supports `batch_id`, `status`, `payer`, `provider_npi`, `date_from`, `date_to`, `sort`, `order`, `limit` (≤ 1000), `offset`. | +| GET | `/api/remittances` | `store.iter_remittances(...)` mapped to UI `Remittance[]`. Supports `batch_id`, `payer`, `claim_id`, `date_from`, `date_to`, `sort`, `order`, `limit`, `offset`. | +| GET | `/api/providers` | `store.distinct_providers()`. Supports `npi`, `state`, `limit`, `offset`. | +| GET | `/api/activity` | `store.recent_activity(limit=200)`. Supports `kind`, `since`, `limit` (≤ 500). | +| GET | `/api/health` | (unchanged) | + +**Streaming:** every list endpoint above accepts `Accept: application/x-ndjson`. When set, the response is a `StreamingResponse` with one JSON object per line: `{type: "item", data: {...}}` for each result, then a final `{type: "summary", data: {total: N, returned: M, has_more: bool}}`. Default JSON response wraps the same data in a `{items: [...], total: N, returned: M, has_more: bool}` envelope so the frontend can paginate uniformly. + +**Status mapping (837P):** each parsed `ClaimOutput` has a `validation.passed` boolean. The UI's `Claim.status` is one of `draft|submitted|accepted|denied|paid|pending`. The mapper does: +- `validation.passed` and `frequency_code == "1"` → `submitted` +- `validation.passed` and any `validation.warnings` → `pending` +- `!validation.passed` and no error mentions `R050_diagnosis_present` → `denied` +- otherwise → `draft` (rare; means validation broke somewhere) + +**Status mapping (835):** 835 `CLP02` status codes map to `Remittance.status` (`received|posted|reconciled`): +- 1, 2, 19, 20 (paid, primary/secondary) → `received` +- 4 (denied) → `received` with a `denialReason` populated from the CAS segments +- 21, 22 (reversal of previous payment) → `reconciled` (out-of-scope: full reversal handling; the data is in `service_payments[*].adjustments`) +- all other valid codes → `received` (default) +- any code outside `cfg.allowed_status_codes` → still stored, mapped to `received`, and surfaced as a validation warning + +### 6.3 Existing parse routes — wire to the store + +- `POST /api/parse-837` — on success, build a `BatchRecord(kind="837p", result=result)` and call `store.add(...)` *before* returning the response. On `CycloneParseError` or any unhandled exception, do *not* add. +- `POST /api/parse-835` — same pattern with `kind="835"`. +- Both routes also stamp `parsed_at` and generate `id = uuid.uuid4().hex`. + +### 6.4 Uvicorn invocation + +`cyclone/__main__.py`'s `serve` branch: +```python +sys.argv = [sys.argv[0], "cyclone.api:app", "--host", "127.0.0.1", "--port", "8000"] +``` +- `--host 127.0.0.1` (not `0.0.0.0`). +- `--port 8000` (configurable via `CYCLONE_PORT` env, default `8000`). +- `--reload` only when `CYCLONE_RELOAD=1` (dev convenience). + +CORS in `api.py` stays as-is: `allow_origins=["http://localhost:5173"]`. + +## 7. Frontend changes + +### 7.1 Dependencies + +Add to `package.json`: +```json +"dependencies": { + ... + "@tanstack/react-query": "^5.59.0" +} +``` + +### 7.2 `src/main.tsx` + +Wrap the existing `` in `` where `queryClient` is a module-level `new QueryClient({ defaultOptions: { queries: { staleTime: 30_000, refetchOnWindowFocus: true } } })`. + +### 7.3 `src/lib/api.ts` — add GET methods + +Mirror the existing `request` helper. New methods (all return `Promise`): +```ts +api.listBatches(): Promise +api.getBatch(id: string): Promise +api.listClaims(params: ListClaimsParams): Promise> +api.listRemittances(params: ListRemittancesParams): Promise> +api.listProviders(params?: ListProvidersParams): Promise> +api.listActivity(params?: ListActivityParams): Promise> +``` + +NDJSON streaming is *not* required for the initial frontend wiring (the existing Upload page already streams from the POST); GET is fine as JSON. The streaming option on GET is for future large-batch scenarios and is a backend-side capability only. + +### 7.4 `src/hooks/` — new query / mutation hooks + +```ts +// src/hooks/useBatches.ts +export function useBatches() { + return useQuery({ queryKey: ['batches'], queryFn: () => api.listBatches() }); +} + +// src/hooks/useClaims.ts +export function useClaims(params: ListClaimsParams) { + return useQuery({ queryKey: ['claims', params], queryFn: () => api.listClaims(params) }); +} + +// ... useRemittances, useProviders, useActivity similarly ... + +// src/hooks/useParse.ts +export function useParse(kind: '837p' | '835') { + const queryClient = useQueryClient(); + return useMutation({ + mutationFn: ({ file, onProgress }) => + kind === '837p' ? api.parse837(file, { onProgress }) : api.parse835(file, { onProgress }), + onSuccess: () => { + queryClient.invalidateQueries({ queryKey: ['batches'] }); + queryClient.invalidateQueries({ queryKey: ['claims'] }); + queryClient.invalidateQueries({ queryKey: ['remittances'] }); + queryClient.invalidateQueries({ queryKey: ['providers'] }); + queryClient.invalidateQueries({ queryKey: ['activity'] }); + }, + }); +} +``` + +### 7.5 Page refactors + +| Page | Current | New | +|---|---|---| +| `src/pages/Claims.tsx` | `useAppStore((s) => s.claims)` | `useClaims({ status, search })`; falls back to store data when `!api.isConfigured` (handled in the hook) | +| `src/pages/Remittances.tsx` | store-backed | `useRemittances({})` | +| `src/pages/Providers.tsx` | store-backed | `useProviders({})` | +| `src/pages/ActivityLog.tsx` | store-backed | `useActivity({ since: now - 7d })`, `refetchInterval: 30_000` | +| `src/pages/Upload.tsx` | uses `useMutation`-less `addParsedBatch` | replace mutation with `useParse(kind)`; keep `addParsedBatch` for the "recent batches" list, but the live-data pages will refresh from the server | + +Each page gains: `` rows during `isLoading`, an error block with "Retry" button on `isError`, and a small footer showing `{returned} of {total}` when `has_more` is true. + +**Fallback when `VITE_API_BASE_URL` is empty:** the existing `data` adapter in `src/lib/api.ts` continues to provide a path that reads from the in-memory zustand store. Each new hook checks `api.isConfigured` and either runs the react-query path (default) or returns a synchronous result from the store via `useSyncExternalStore`. Page components don't need to know which path is active. The `data` adapter is removed in a follow-up sub-project once no caller depends on it. + +### 7.6 UI primitive: `src/components/ui/skeleton.tsx` + +~10 lines, shadcn-style: `
`. No new dep. + +## 8. Docs + +### 8.1 `docs/reference/837p.md` (~80 lines) + +Sections: +- File extension: `.txt`. Encoding: ASCII or UTF-8. Delimiters: `*` element, `:` component, `~` segment, `^` repetition. +- Envelope: `ISA/GS/ST` headers, `SE/GE/IEA` footers. +- Loops: 2000A (billing provider), 2000B (subscriber), 2300 (claim), 2400 (service line). +- Segments Cyclone parses: NM1, N3/N4, REF, CLM, HI, LX, SV1, DTP, BHT. +- Segments Cyclone skips (preserved in `raw_segments` but not modeled): too many to enumerate; see the parser source for the full list. +- CO Medicaid specifics: `REF*G1` prior-auth, no patient loop, `CLM05` shape `12:B:1` etc. + +### 8.2 `docs/reference/835.md` (~80 lines) + +- Envelope: same as 837P. +- Header: `BPR` (financial), `TRN` (trace), `DTM` (dates). +- Loops: 1000A (payer), 1000B (payee), 2100 (CLP claim payment), 2110 (SVC service payment). +- Critical balancing rules: `BPR02 == sum(CLP04)`, `CLP04 == sum(SVC03)`. +- CO Medicaid specifics: `BPR10` 81-1725341 (TXIX) or 84-0644739 (BHA), `N104` 7912900843, `N1*PR` name `CO_TXIX` or `CO_BHA`. + +### 8.3 `docs/reference/x12naming.md` (~50 lines) + +- Segments: 2–3 letter codes (`CLM`, `NM1`, `BPR`). +- Elements: `*`-separated, 1-indexed. +- Composite elements: `:`-separated sub-elements, indexed `01`, `02`, ... +- Loops: 4-digit IDs, hierarchical (2000 contains 2300 contains 2400). +- Common qualifiers we care about: `B` (POS qualifier), `ABK` (ICD-10 principal diagnosis), `MC` (Medicaid claim filing). + +### 8.4 `docs/reference/co-medicaid.md` (~80 lines) + +- Trading partner ID (TPID) — sender / receiver roles in 837P and 835. +- Payer IDs: + - 837P: `NM1*PR N104 = "SKCO0"` (COHCPF). + - 835: `BPR10 = "81-1725341"` (TXIX) or `"84-0644739"` (BHA); `N1*PR N104 = "7912900843"`. +- Allowed frequency codes (837P `CLM05-3`): `{1, 7, 8}`. +- Allowed status codes (835 `CLP02`): `{1, 2, 3, 4, 19, 20, 21, 22, 23, 25}`. +- POS codes: full CMS set (89 codes; canonical list in `cyclone.parsers.payer.CMS_PLACE_OF_SERVICE_CODES`). + +### 8.5 `README.md` (root, rewrite) + +Sections: +- **What is Cyclone?** — 1 paragraph. EDI claim parser + browser UI. +- **Install** — `git clone`, `cd backend && python -m venv .venv && .venv/bin/pip install -e '.[dev]'`, `cd .. && npm install`. +- **Dev** — two terminals: `cd backend && .venv/bin/python -m cyclone serve` and `npm run dev`. Open `http://localhost:5173`. +- **Test** — `cd backend && .venv/bin/pytest` and `npm run build`. +- **Project layout** — `backend/` (Python API + parsers), `src/` (React UI), `docs/reference/` (companion notes), `docs/prodfiles/` (sample EDI files). +- **Roadmap** — short list of sub-projects 2–4 with one-line descriptions and a note that they're not in this build. +- **License** — no license file yet; add a `LICENSE` file when one is decided. The project is internal-use until then. + +## 9. Error handling + +- **Backend store errors:** none expected (in-memory, no I/O). If a `BatchRecord.result` doesn't conform (e.g. we tried to store a result from a code path that returns `None`), the model validator fails loudly at insertion time. +- **GET route errors:** `404` for missing `batch_id`; `400` for invalid query params (FastAPI's default validation handles this); `500` only for unexpected bugs. +- **NDJSON streaming:** on a serialization error mid-stream, the response is already partially written; we log the error and end the stream. The client sees a truncated response and the final summary line is missing. The frontend falls back to "show what we have" and shows a toast. +- **Frontend:** `useQuery` exposes `isError` + `error`. Each page renders an error block with a retry button that calls `refetch()`. Network errors show a sonner toast on top of the inline error. +- **Parse failure:** the parse route already returns 400 / 422 with a clear message. The Upload page shows a destructive toast. The store is unchanged (failed parses are not persisted). + +## 10. Testing + +### 10.1 Backend (`backend/tests/`) + +- `test_store.py` (~8 tests) — add/list/get; concurrent adds under a thread; mapper functions; filter / sort / paginate; distinct providers; activity derivation. +- `test_api_gets.py` (~12 tests) — one happy-path + one filter test per new GET endpoint (6 endpoints × 2 = 12); plus one 404 test for `/api/batches/{id}`. +- `test_api_streaming.py` (1 test) — `Accept: application/x-ndjson` on `/api/claims` returns parseable lines. +- `test_api_parse_persists.py` (2 tests) — parse-837 followed by `GET /api/claims` returns the claim; failed parse does *not* create a batch. + +Total new backend tests: **~23**. + +### 10.2 Frontend (`src/`) + +- `src/lib/api.test.ts` — extend with 3 tests: `listClaims` builds the right URL with params, `listBatches` hits `/api/batches`, `getBatch` returns `ParseResult835` when `kind === "835"`. +- No DOM / component tests in this sub-project. The page refactors are small and the smoke test below is the integration check. + +### 10.3 End-to-end smoke test (manual, documented in the plan) + +``` +# Terminal 1 +cd backend && .venv/bin/python -m cyclone serve + +# Terminal 2 +npm run dev + +# In a third terminal +cd backend && .venv/bin/python -m cyclone.cli parse-837 tests/fixtures/co_medicaid_837p.txt --output-dir /tmp/co-837 +# (this populates a real production-like batch; we don't need the JSON files for the test, just the upload flow) + +# In a browser: open http://localhost:5173/upload, drop tests/fixtures/co_medicaid_837p.txt +# Verify: claims stream in, summary toast appears, navigate to /claims and see them +``` + +## 11. Migration / rollout + +- No DB, so no migration. +- The frontend's existing `data` adapter stays during the transition. We delete it in a follow-up once no one calls it. +- The new `InMemoryStore` is empty on first run. Existing parse-837/parse-835 callers see no behavior change (they still get the parse result; we just *also* add to the store now). +- A uvicorn restart wipes the store. That is by design and called out in the README. + +## 12. Out of scope (explicitly) + +- DB persistence, Alembic migrations, transactional safety. +- 837P ↔ 835 reconciliation (matching payouts to original claims). +- 999 ACK, 270/271, or other X12 transaction types. +- More 837P validation rules beyond what already ships. +- 835 CAS deep-parsing with reason-code explanations. +- Auth (any kind: API key, JWT, session). +- Rate limiting, request size limits beyond FastAPI defaults. +- Structured logging, log levels via env, JSON logs. +- Health-check enhancements (uptime, parser state, last-batch timestamp). +- 12-factor env config beyond `CYCLONE_PORT` and `CYCLONE_RELOAD`. +- Docker / docker-compose / any deploy artifact. +- Pre-commit hooks, Makefile, .editorconfig, CONTRIBUTING.md, lint config. +- Component tests for the React pages. +- E2E browser tests (Playwright/Cypress). + +## 13. Acceptance checklist + +- [ ] `pytest` passes with the new store + GET tests; total ≥ 121 + 23 = 144. +- [ ] `npm run build` still passes. +- [ ] `python -m cyclone serve` binds to `127.0.0.1:8000` and refuses connections from non-loopback. +- [ ] Upload page streams a 2-claim CO fixture; navigating to `/claims` shows the same 2 claims. +- [ ] Filtering by `status=submitted` on `/api/claims` returns the right subset; UI shows the same. +- [ ] `GET /api/claims` with `Accept: application/x-ndjson` returns valid NDJSON. +- [ ] Restarting the backend wipes the in-memory store; the UI shows "no claims yet" on first refresh. +- [ ] All 4 reference notes exist under `docs/reference/` and the root `README.md` is rewritten. +- [ ] The frontend still works without a backend (`VITE_API_BASE_URL` empty) — pages fall back to the zustand sample store via the existing `data` adapter.