Files
cyclone/docs/superpowers/specs/2026-06-19-cyclone-production-readiness-design.md
T

378 lines
23 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Cyclone Production-Readiness (local-only) — Design
**Date:** 2026-06-19
**Status:** Approved (pending user review of this doc)
**Scope:** First sub-project of the four-part Cyclone roadmap. Local-only deployment (no auth, `127.0.0.1`-bound). Adds a backend in-memory batch store, GET endpoints, react-query wiring on the frontend, fresh reference notes, and a new root README. Out of scope: DB persistence, reconciliation, additional X12 transaction types, additional 837P/835 validation rules.
---
## 1. Overview
The Cyclone EDI suite already has working 837P and 835 parsers, a FastAPI surface for parsing uploads, and a React frontend with an Upload page. What it does not have is a way to *browse* the parsed data — once a file is uploaded, the user sees it in the Upload page, but a refresh or page navigation loses the result, and the existing Dashboard / Claims / Remittances / Providers / Activity pages still read static sample data.
This sub-project closes that loop. We add a process-local in-memory store on the backend, expose GET endpoints that match the resource shape the existing UI already expects, wire react-query into the frontend so pages fetch live data and auto-refresh after a parse, and replace the lost reference notes with a clean new docs tree.
After this, a user can:
1. Start the backend (`python -m cyclone serve`).
2. Start the Vite dev server (`npm run dev`).
3. Open `http://localhost:5173/upload`, drop in a `.txt` file, watch the claims stream in.
4. Navigate to `/claims`, `/remittances`, `/providers`, `/activity` and see the data they just parsed.
5. The whole stack is bound to `127.0.0.1`; no auth, no internet exposure.
## 2. Goals
1. **Persist parsed data for the session.** A new `InMemoryStore` on the backend keeps every successful parse and serves it to GET requests. Lost on backend restart — that is acceptable for a local-only tool and avoids a DB in this sub-project.
2. **Expose GET endpoints that match the UI's resource model.** `GET /api/claims`, `/api/remittances`, `/api/providers`, `/api/activity` plus `/api/batches` and `/api/batches/{id}`. The list endpoints accept filter / sort / pagination query params and support both JSON and NDJSON streaming.
3. **Wire the existing 4 UI pages to the live API** via `@tanstack/react-query` v5, with loading skeletons, error states, and automatic invalidation after a successful parse. Fall back to the in-memory sample store when `VITE_API_BASE_URL` is empty (the existing `data` adapter pattern stays).
4. **Replace the lost reference notes** with 4 fresh, short notes under `docs/reference/` (~5080 lines each) covering 837P, 835, X12 naming, and CO Medicaid specifics. Also rewrite the root `README.md`.
5. **Tighten the local-only deploy posture.** `python -m cyclone serve` binds `127.0.0.1:8000` (not `0.0.0.0`); CORS allowlist stays `http://localhost:5173`. No auth, no API key. The README documents the exact command.
## 3. Non-goals (this sub-project)
- **Database persistence.** SQLite/Postgres is sub-project 2.
- **837P ↔ 835 reconciliation.** Sub-project 2.
- **More 837P/835 validation rules** (REF*G1 enforcement, BHT06, CAS deep-parsing). Sub-project 3.
- **999 ACK, 270/271, or other X12 transaction types.** Sub-project 3.
- **Auth of any kind.** Explicitly out — local-only, single-user, `127.0.0.1`-bound.
- **Structured logging, health-check enhancements, 12-factor env config, Dockerfile.** Deferred.
- **Dev tooling polish** (pre-commit, Makefile, CONTRIBUTING.md, .editorconfig). Deferred.
## 4. Stack
**Backend additions:**
- New module `cyclone.store` (`InMemoryStore`, `BatchRecord`, mappers).
- FastAPI (already in use). New routes added to `cyclone.api`.
- `threading.Lock` around the store's batch list (single-process, but FastAPI may run request handlers in a threadpool).
- `pytest` + `fastapi.testclient.TestClient` for new tests.
**Frontend additions:**
- `@tanstack/react-query` v5 added to `package.json` dependencies.
- `QueryClientProvider` in `src/main.tsx`.
- `useQuery` / `useMutation` hooks under `src/hooks/`.
- `Skeleton` UI primitive added under `src/components/ui/` (re-uses `cn` helper).
- No build-tool changes; the existing Vite + TS + Tailwind stack stays.
**Docs:**
- Plain Markdown under `docs/reference/`. No doc generator, no linter, no CI check.
## 5. Architecture
```
┌──────────────────────┐ POST /api/parse-{837,835} ┌─────────────────────────┐
│ Vite/React UI │ ────────────────────────────▶ │ FastAPI backend │
│ │ GET /api/{batches,claims, │ (uvicorn 127.0.0.1) │
│ ┌────────────────┐ │ remittances,providers, │ ┌────────────────────┐ │
│ │ react-query v5 │ │ activity,batches/{id}} │ │ InMemoryStore │ │
│ │ QueryClient │ │ ◀──────────────────────────── │ │ - batches: list │ │
│ │ useQuery / │ │ (JSON or NDJSON stream) │ │ - lock │ │
│ │ useMutation │ │ │ │ - mapper funcs │ │
│ └────────────────┘ │ │ └────────────────────┘ │
│ ┌────────────────┐ │ │ ┌────────────────────┐ │
│ │ zustand store │ │ │ │ parse-837/835 │ │
│ │ (sample data) │ │ │ │ (already shipped) │ │
│ └────────────────┘ │ │ └────────────────────┘ │
└──────────────────────┘ └─────────────────────────┘
│ CORS allow
│ http://localhost:5173
│ only
```
**Data flow on parse:**
1. User drops a file in `/upload`, clicks Parse.
2. Frontend `useMutation` POSTs to `/api/parse-837` (or 835) with `onProgress` (NDJSON) or `Accept: application/json` for small files.
3. Backend's existing parse route appends the resulting `ParseResult` to `InMemoryStore` on success. Failures are *not* stored.
4. `useMutation.onSuccess` calls `queryClient.invalidateQueries(['batches'], ['claims'], ['remittances'], ['activity'])` (and 837/835-specific keys).
5. Any open page reactively re-fetches via `useQuery` and re-renders with the new data.
**Data flow on browse:**
1. Page mount → `useQuery` fires the GET with the page's filter / sort / pagination params.
2. First render: skeleton. Success: data. Error: error block with retry.
3. Background: `refetchOnWindowFocus: true` (default in v5) so navigating back updates the view.
## 6. Backend changes
### 6.1 New module `backend/src/cyclone/store.py`
```python
class BatchRecord(BaseModel):
model_config = ConfigDict(extra="ignore")
id: str # uuid4 hex
kind: Literal["837p", "835"]
input_filename: str
parsed_at: datetime # tz-aware UTC
result: ParseResult | ParseResult835 # union discriminated by .kind
class InMemoryStore:
def __init__(self) -> None:
self._batches: list[BatchRecord] = []
self._lock = threading.Lock()
def add(self, record: BatchRecord) -> None: ...
def list(self, *, limit: int = 100) -> list[BatchRecord]: ...
def get(self, batch_id: str) -> BatchRecord | None: ...
def iter_claims(self, *, batch_id: str | None = None, ...) -> Iterable[Claim]: ...
def iter_remittances(self, *, batch_id: str | None = None, ...) -> Iterable[Remittance]: ...
def distinct_providers(self) -> list[Provider]: ...
def recent_activity(self, *, limit: int = 200) -> list[Activity]: ...
store = InMemoryStore() # module-level singleton
```
**Mappers** (`to_ui_claim`, `to_ui_remittance`, `to_ui_provider`, `to_activity_event`) live in the same file. They translate the rich backend models (Pydantic, raw segments) to the simpler UI types the GET endpoints return.
**Filter / sort API:** each `iter_*` method takes the relevant filter kwargs and applies them in Python. There is no index — the in-memory list is small (single-session, dozens of batches at most). Sorting uses `sorted(..., key=..., reverse=order == "desc")`.
**Threading:** every public method acquires `self._lock` (RLock, since `list` may call `get` internally) and releases it.
### 6.2 New routes in `backend/src/cyclone/api.py`
| Method | Path | Behavior |
|---|---|---|
| GET | `/api/batches` | `store.list(limit=100)`, mapped to a `BatchSummary`-shaped response. Newest first. |
| GET | `/api/batches/{id}` | `store.get(id)`. 404 if missing. Returns the full `ParseResult` / `ParseResult835`. |
| GET | `/api/claims` | `store.iter_claims(...)` mapped to UI `Claim[]`. Supports `batch_id`, `status`, `payer`, `provider_npi`, `date_from`, `date_to`, `sort`, `order`, `limit` (≤ 1000), `offset`. |
| GET | `/api/remittances` | `store.iter_remittances(...)` mapped to UI `Remittance[]`. Supports `batch_id`, `payer`, `claim_id`, `date_from`, `date_to`, `sort`, `order`, `limit`, `offset`. |
| GET | `/api/providers` | `store.distinct_providers()`. Supports `npi`, `state`, `limit`, `offset`. |
| GET | `/api/activity` | `store.recent_activity(limit=200)`. Supports `kind`, `since`, `limit` (≤ 500). |
| GET | `/api/health` | (unchanged) |
**Streaming:** every list endpoint above accepts `Accept: application/x-ndjson`. When set, the response is a `StreamingResponse` with one JSON object per line: `{type: "item", data: {...}}` for each result, then a final `{type: "summary", data: {total: N, returned: M, has_more: bool}}`. Default JSON response wraps the same data in a `{items: [...], total: N, returned: M, has_more: bool}` envelope so the frontend can paginate uniformly.
**Status mapping (837P):** each parsed `ClaimOutput` has a `validation.passed` boolean. The UI's `Claim.status` is one of `draft|submitted|accepted|denied|paid|pending`. The mapper does:
- `validation.passed` and `frequency_code == "1"``submitted`
- `validation.passed` and any `validation.warnings``pending`
- `!validation.passed` and no error mentions `R050_diagnosis_present``denied`
- otherwise → `draft` (rare; means validation broke somewhere)
**Status mapping (835):** 835 `CLP02` status codes map to `Remittance.status` (`received|posted|reconciled`):
- 1, 2, 19, 20 (paid, primary/secondary) → `received`
- 4 (denied) → `received` with a `denialReason` populated from the CAS segments
- 21, 22 (reversal of previous payment) → `reconciled` (out-of-scope: full reversal handling; the data is in `service_payments[*].adjustments`)
- all other valid codes → `received` (default)
- any code outside `cfg.allowed_status_codes` → still stored, mapped to `received`, and surfaced as a validation warning
### 6.3 Existing parse routes — wire to the store
- `POST /api/parse-837` — on success, build a `BatchRecord(kind="837p", result=result)` and call `store.add(...)` *before* returning the response. On `CycloneParseError` or any unhandled exception, do *not* add.
- `POST /api/parse-835` — same pattern with `kind="835"`.
- Both routes also stamp `parsed_at` and generate `id = uuid.uuid4().hex`.
### 6.4 Uvicorn invocation
`cyclone/__main__.py`'s `serve` branch:
```python
sys.argv = [sys.argv[0], "cyclone.api:app", "--host", "127.0.0.1", "--port", "8000"]
```
- `--host 127.0.0.1` (not `0.0.0.0`).
- `--port 8000` (configurable via `CYCLONE_PORT` env, default `8000`).
- `--reload` only when `CYCLONE_RELOAD=1` (dev convenience).
CORS in `api.py` stays as-is: `allow_origins=["http://localhost:5173"]`.
## 7. Frontend changes
### 7.1 Dependencies
Add to `package.json`:
```json
"dependencies": {
...
"@tanstack/react-query": "^5.59.0"
}
```
### 7.2 `src/main.tsx`
Wrap the existing `<App />` in `<QueryClientProvider client={queryClient}>` where `queryClient` is a module-level `new QueryClient({ defaultOptions: { queries: { staleTime: 30_000, refetchOnWindowFocus: true } } })`.
### 7.3 `src/lib/api.ts` — add GET methods
Mirror the existing `request<T>` helper. New methods (all return `Promise<T>`):
```ts
api.listBatches(): Promise<BatchSummary[]>
api.getBatch(id: string): Promise<ParseResult837 | ParseResult835>
api.listClaims(params: ListClaimsParams): Promise<PaginatedResponse<Claim>>
api.listRemittances(params: ListRemittancesParams): Promise<PaginatedResponse<Remittance>>
api.listProviders(params?: ListProvidersParams): Promise<PaginatedResponse<Provider>>
api.listActivity(params?: ListActivityParams): Promise<PaginatedResponse<Activity>>
```
NDJSON streaming is *not* required for the initial frontend wiring (the existing Upload page already streams from the POST); GET is fine as JSON. The streaming option on GET is for future large-batch scenarios and is a backend-side capability only.
### 7.4 `src/hooks/` — new query / mutation hooks
```ts
// src/hooks/useBatches.ts
export function useBatches() {
return useQuery({ queryKey: ['batches'], queryFn: () => api.listBatches() });
}
// src/hooks/useClaims.ts
export function useClaims(params: ListClaimsParams) {
return useQuery({ queryKey: ['claims', params], queryFn: () => api.listClaims(params) });
}
// ... useRemittances, useProviders, useActivity similarly ...
// src/hooks/useParse.ts
export function useParse(kind: '837p' | '835') {
const queryClient = useQueryClient();
return useMutation({
mutationFn: ({ file, onProgress }) =>
kind === '837p' ? api.parse837(file, { onProgress }) : api.parse835(file, { onProgress }),
onSuccess: () => {
queryClient.invalidateQueries({ queryKey: ['batches'] });
queryClient.invalidateQueries({ queryKey: ['claims'] });
queryClient.invalidateQueries({ queryKey: ['remittances'] });
queryClient.invalidateQueries({ queryKey: ['providers'] });
queryClient.invalidateQueries({ queryKey: ['activity'] });
},
});
}
```
### 7.5 Page refactors
| Page | Current | New |
|---|---|---|
| `src/pages/Claims.tsx` | `useAppStore((s) => s.claims)` | `useClaims({ status, search })`; falls back to store data when `!api.isConfigured` (handled in the hook) |
| `src/pages/Remittances.tsx` | store-backed | `useRemittances({})` |
| `src/pages/Providers.tsx` | store-backed | `useProviders({})` |
| `src/pages/ActivityLog.tsx` | store-backed | `useActivity({ since: now - 7d })`, `refetchInterval: 30_000` |
| `src/pages/Upload.tsx` | uses `useMutation`-less `addParsedBatch` | replace mutation with `useParse(kind)`; keep `addParsedBatch` for the "recent batches" list, but the live-data pages will refresh from the server |
Each page gains: `<Skeleton />` rows during `isLoading`, an error block with "Retry" button on `isError`, and a small footer showing `{returned} of {total}` when `has_more` is true.
**Fallback when `VITE_API_BASE_URL` is empty:** the existing `data` adapter in `src/lib/api.ts` continues to provide a path that reads from the in-memory zustand store. Each new hook checks `api.isConfigured` and either runs the react-query path (default) or returns a synchronous result from the store via `useSyncExternalStore`. Page components don't need to know which path is active. The `data` adapter is removed in a follow-up sub-project once no caller depends on it.
### 7.6 UI primitive: `src/components/ui/skeleton.tsx`
~10 lines, shadcn-style: `<div className={cn('animate-pulse rounded-md bg-muted', className)} />`. No new dep.
## 8. Docs
### 8.1 `docs/reference/837p.md` (~80 lines)
Sections:
- File extension: `.txt`. Encoding: ASCII or UTF-8. Delimiters: `*` element, `:` component, `~` segment, `^` repetition.
- Envelope: `ISA/GS/ST` headers, `SE/GE/IEA` footers.
- Loops: 2000A (billing provider), 2000B (subscriber), 2300 (claim), 2400 (service line).
- Segments Cyclone parses: NM1, N3/N4, REF, CLM, HI, LX, SV1, DTP, BHT.
- Segments Cyclone skips (preserved in `raw_segments` but not modeled): too many to enumerate; see the parser source for the full list.
- CO Medicaid specifics: `REF*G1` prior-auth, no patient loop, `CLM05` shape `12:B:1` etc.
### 8.2 `docs/reference/835.md` (~80 lines)
- Envelope: same as 837P.
- Header: `BPR` (financial), `TRN` (trace), `DTM` (dates).
- Loops: 1000A (payer), 1000B (payee), 2100 (CLP claim payment), 2110 (SVC service payment).
- Critical balancing rules: `BPR02 == sum(CLP04)`, `CLP04 == sum(SVC03)`.
- CO Medicaid specifics: `BPR10` 81-1725341 (TXIX) or 84-0644739 (BHA), `N104` 7912900843, `N1*PR` name `CO_TXIX` or `CO_BHA`.
### 8.3 `docs/reference/x12naming.md` (~50 lines)
- Segments: 23 letter codes (`CLM`, `NM1`, `BPR`).
- Elements: `*`-separated, 1-indexed.
- Composite elements: `:`-separated sub-elements, indexed `01`, `02`, ...
- Loops: 4-digit IDs, hierarchical (2000 contains 2300 contains 2400).
- Common qualifiers we care about: `B` (POS qualifier), `ABK` (ICD-10 principal diagnosis), `MC` (Medicaid claim filing).
### 8.4 `docs/reference/co-medicaid.md` (~80 lines)
- Trading partner ID (TPID) — sender / receiver roles in 837P and 835.
- Payer IDs:
- 837P: `NM1*PR N104 = "SKCO0"` (COHCPF).
- 835: `BPR10 = "81-1725341"` (TXIX) or `"84-0644739"` (BHA); `N1*PR N104 = "7912900843"`.
- Allowed frequency codes (837P `CLM05-3`): `{1, 7, 8}`.
- Allowed status codes (835 `CLP02`): `{1, 2, 3, 4, 19, 20, 21, 22, 23, 25}`.
- POS codes: full CMS set (89 codes; canonical list in `cyclone.parsers.payer.CMS_PLACE_OF_SERVICE_CODES`).
### 8.5 `README.md` (root, rewrite)
Sections:
- **What is Cyclone?** — 1 paragraph. EDI claim parser + browser UI.
- **Install** — `git clone`, `cd backend && python -m venv .venv && .venv/bin/pip install -e '.[dev]'`, `cd .. && npm install`.
- **Dev** — two terminals: `cd backend && .venv/bin/python -m cyclone serve` and `npm run dev`. Open `http://localhost:5173`.
- **Test** — `cd backend && .venv/bin/pytest` and `npm run build`.
- **Project layout** — `backend/` (Python API + parsers), `src/` (React UI), `docs/reference/` (companion notes), `docs/prodfiles/` (sample EDI files).
- **Roadmap** — short list of sub-projects 24 with one-line descriptions and a note that they're not in this build.
- **License** — no license file yet; add a `LICENSE` file when one is decided. The project is internal-use until then.
## 9. Error handling
- **Backend store errors:** none expected (in-memory, no I/O). If a `BatchRecord.result` doesn't conform (e.g. we tried to store a result from a code path that returns `None`), the model validator fails loudly at insertion time.
- **GET route errors:** `404` for missing `batch_id`; `400` for invalid query params (FastAPI's default validation handles this); `500` only for unexpected bugs.
- **NDJSON streaming:** on a serialization error mid-stream, the response is already partially written; we log the error and end the stream. The client sees a truncated response and the final summary line is missing. The frontend falls back to "show what we have" and shows a toast.
- **Frontend:** `useQuery` exposes `isError` + `error`. Each page renders an error block with a retry button that calls `refetch()`. Network errors show a sonner toast on top of the inline error.
- **Parse failure:** the parse route already returns 400 / 422 with a clear message. The Upload page shows a destructive toast. The store is unchanged (failed parses are not persisted).
## 10. Testing
### 10.1 Backend (`backend/tests/`)
- `test_store.py` (~8 tests) — add/list/get; concurrent adds under a thread; mapper functions; filter / sort / paginate; distinct providers; activity derivation.
- `test_api_gets.py` (~12 tests) — one happy-path + one filter test per new GET endpoint (6 endpoints × 2 = 12); plus one 404 test for `/api/batches/{id}`.
- `test_api_streaming.py` (1 test) — `Accept: application/x-ndjson` on `/api/claims` returns parseable lines.
- `test_api_parse_persists.py` (2 tests) — parse-837 followed by `GET /api/claims` returns the claim; failed parse does *not* create a batch.
Total new backend tests: **~23**.
### 10.2 Frontend (`src/`)
- `src/lib/api.test.ts` — extend with 3 tests: `listClaims` builds the right URL with params, `listBatches` hits `/api/batches`, `getBatch` returns `ParseResult835` when `kind === "835"`.
- No DOM / component tests in this sub-project. The page refactors are small and the smoke test below is the integration check.
### 10.3 End-to-end smoke test (manual, documented in the plan)
```
# Terminal 1
cd backend && .venv/bin/python -m cyclone serve
# Terminal 2
npm run dev
# In a third terminal
cd backend && .venv/bin/python -m cyclone.cli parse-837 tests/fixtures/co_medicaid_837p.txt --output-dir /tmp/co-837
# (this populates a real production-like batch; we don't need the JSON files for the test, just the upload flow)
# In a browser: open http://localhost:5173/upload, drop tests/fixtures/co_medicaid_837p.txt
# Verify: claims stream in, summary toast appears, navigate to /claims and see them
```
## 11. Migration / rollout
- No DB, so no migration.
- The frontend's existing `data` adapter stays during the transition. We delete it in a follow-up once no one calls it.
- The new `InMemoryStore` is empty on first run. Existing parse-837/parse-835 callers see no behavior change (they still get the parse result; we just *also* add to the store now).
- A uvicorn restart wipes the store. That is by design and called out in the README.
## 12. Out of scope (explicitly)
- DB persistence, Alembic migrations, transactional safety.
- 837P ↔ 835 reconciliation (matching payouts to original claims).
- 999 ACK, 270/271, or other X12 transaction types.
- More 837P validation rules beyond what already ships.
- 835 CAS deep-parsing with reason-code explanations.
- Auth (any kind: API key, JWT, session).
- Rate limiting, request size limits beyond FastAPI defaults.
- Structured logging, log levels via env, JSON logs.
- Health-check enhancements (uptime, parser state, last-batch timestamp).
- 12-factor env config beyond `CYCLONE_PORT` and `CYCLONE_RELOAD`.
- Docker / docker-compose / any deploy artifact.
- Pre-commit hooks, Makefile, .editorconfig, CONTRIBUTING.md, lint config.
- Component tests for the React pages.
- E2E browser tests (Playwright/Cypress).
## 13. Acceptance checklist
- [ ] `pytest` passes with the new store + GET tests; total ≥ 121 + 23 = 144.
- [ ] `npm run build` still passes.
- [ ] `python -m cyclone serve` binds to `127.0.0.1:8000` and refuses connections from non-loopback.
- [ ] Upload page streams a 2-claim CO fixture; navigating to `/claims` shows the same 2 claims.
- [ ] Filtering by `status=submitted` on `/api/claims` returns the right subset; UI shows the same.
- [ ] `GET /api/claims` with `Accept: application/x-ndjson` returns valid NDJSON.
- [ ] Restarting the backend wipes the in-memory store; the UI shows "no claims yet" on first refresh.
- [ ] All 4 reference notes exist under `docs/reference/` and the root `README.md` is rewritten.
- [ ] The frontend still works without a backend (`VITE_API_BASE_URL` empty) — pages fall back to the zustand sample store via the existing `data` adapter.