Files

T

Tyler 9e34f22cb3 docs: add production-readiness design spec (local-only, sub-project 1 of 4)

2026-06-19 18:31:40 -06:00

23 KiB

Raw Blame History

Cyclone Production-Readiness (local-only) — Design

Date: 2026-06-19 Status: Approved (pending user review of this doc) Scope: First sub-project of the four-part Cyclone roadmap. Local-only deployment (no auth, 127.0.0.1-bound). Adds a backend in-memory batch store, GET endpoints, react-query wiring on the frontend, fresh reference notes, and a new root README. Out of scope: DB persistence, reconciliation, additional X12 transaction types, additional 837P/835 validation rules.

1. Overview

The Cyclone EDI suite already has working 837P and 835 parsers, a FastAPI surface for parsing uploads, and a React frontend with an Upload page. What it does not have is a way to browse the parsed data — once a file is uploaded, the user sees it in the Upload page, but a refresh or page navigation loses the result, and the existing Dashboard / Claims / Remittances / Providers / Activity pages still read static sample data.

This sub-project closes that loop. We add a process-local in-memory store on the backend, expose GET endpoints that match the resource shape the existing UI already expects, wire react-query into the frontend so pages fetch live data and auto-refresh after a parse, and replace the lost reference notes with a clean new docs tree.

After this, a user can:

Start the backend (python -m cyclone serve).
Start the Vite dev server (npm run dev).
Open http://localhost:5173/upload, drop in a .txt file, watch the claims stream in.
Navigate to /claims, /remittances, /providers, /activity and see the data they just parsed.
The whole stack is bound to 127.0.0.1; no auth, no internet exposure.

2. Goals

Persist parsed data for the session. A new InMemoryStore on the backend keeps every successful parse and serves it to GET requests. Lost on backend restart — that is acceptable for a local-only tool and avoids a DB in this sub-project.
Expose GET endpoints that match the UI's resource model. GET /api/claims, /api/remittances, /api/providers, /api/activity plus /api/batches and /api/batches/{id}. The list endpoints accept filter / sort / pagination query params and support both JSON and NDJSON streaming.
Wire the existing 4 UI pages to the live API via @tanstack/react-query v5, with loading skeletons, error states, and automatic invalidation after a successful parse. Fall back to the in-memory sample store when VITE_API_BASE_URL is empty (the existing data adapter pattern stays).
Replace the lost reference notes with 4 fresh, short notes under docs/reference/ (~50–80 lines each) covering 837P, 835, X12 naming, and CO Medicaid specifics. Also rewrite the root README.md.
Tighten the local-only deploy posture. python -m cyclone serve binds 127.0.0.1:8000 (not 0.0.0.0); CORS allowlist stays http://localhost:5173. No auth, no API key. The README documents the exact command.

3. Non-goals (this sub-project)

Database persistence. SQLite/Postgres is sub-project 2.
837P ↔ 835 reconciliation. Sub-project 2.
More 837P/835 validation rules (REF*G1 enforcement, BHT06, CAS deep-parsing). Sub-project 3.
999 ACK, 270/271, or other X12 transaction types. Sub-project 3.
Auth of any kind. Explicitly out — local-only, single-user, 127.0.0.1-bound.
Structured logging, health-check enhancements, 12-factor env config, Dockerfile. Deferred.
Dev tooling polish (pre-commit, Makefile, CONTRIBUTING.md, .editorconfig). Deferred.

4. Stack

Backend additions:

New module cyclone.store (InMemoryStore, BatchRecord, mappers).
FastAPI (already in use). New routes added to cyclone.api.
threading.Lock around the store's batch list (single-process, but FastAPI may run request handlers in a threadpool).
pytest + fastapi.testclient.TestClient for new tests.

Frontend additions:

@tanstack/react-query v5 added to package.json dependencies.
QueryClientProvider in src/main.tsx.
useQuery / useMutation hooks under src/hooks/.
Skeleton UI primitive added under src/components/ui/ (re-uses cn helper).
No build-tool changes; the existing Vite + TS + Tailwind stack stays.

Docs:

Plain Markdown under docs/reference/. No doc generator, no linter, no CI check.

5. Architecture

┌──────────────────────┐  POST /api/parse-{837,835}    ┌─────────────────────────┐
│  Vite/React UI       │ ────────────────────────────▶ │  FastAPI backend        │
│                      │  GET  /api/{batches,claims,   │  (uvicorn 127.0.0.1)    │
│  ┌────────────────┐  │       remittances,providers,  │  ┌────────────────────┐ │
│  │ react-query v5 │  │       activity,batches/{id}}  │  │ InMemoryStore      │ │
│  │ QueryClient    │  │ ◀──────────────────────────── │  │ - batches: list    │ │
│  │ useQuery /     │  │   (JSON or NDJSON stream)     │  │ - lock             │ │
│  │ useMutation    │  │                               │  │ - mapper funcs     │ │
│  └────────────────┘  │                               │  └────────────────────┘ │
│  ┌────────────────┐  │                               │  ┌────────────────────┐ │
│  │ zustand store  │  │                               │  │ parse-837/835      │ │
│  │ (sample data)  │  │                               │  │ (already shipped)  │ │
│  └────────────────┘  │                               │  └────────────────────┘ │
└──────────────────────┘                               └─────────────────────────┘
                                                          ▲
                                                          │ CORS allow
                                                          │ http://localhost:5173
                                                          │ only

Data flow on parse:

User drops a file in /upload, clicks Parse.
Frontend useMutation POSTs to /api/parse-837 (or 835) with onProgress (NDJSON) or Accept: application/json for small files.
Backend's existing parse route appends the resulting ParseResult to InMemoryStore on success. Failures are not stored.
useMutation.onSuccess calls queryClient.invalidateQueries(['batches'], ['claims'], ['remittances'], ['activity']) (and 837/835-specific keys).
Any open page reactively re-fetches via useQuery and re-renders with the new data.

Data flow on browse:

Page mount → useQuery fires the GET with the page's filter / sort / pagination params.
First render: skeleton. Success: data. Error: error block with retry.
Background: refetchOnWindowFocus: true (default in v5) so navigating back updates the view.

6. Backend changes

6.1 New module `backend/src/cyclone/store.py`

class BatchRecord(BaseModel):
    model_config = ConfigDict(extra="ignore")
    id: str                                # uuid4 hex
    kind: Literal["837p", "835"]
    input_filename: str
    parsed_at: datetime                    # tz-aware UTC
    result: ParseResult | ParseResult835   # union discriminated by .kind

class InMemoryStore:
    def __init__(self) -> None:
        self._batches: list[BatchRecord] = []
        self._lock = threading.Lock()

    def add(self, record: BatchRecord) -> None: ...
    def list(self, *, limit: int = 100) -> list[BatchRecord]: ...
    def get(self, batch_id: str) -> BatchRecord | None: ...
    def iter_claims(self, *, batch_id: str | None = None, ...) -> Iterable[Claim]: ...
    def iter_remittances(self, *, batch_id: str | None = None, ...) -> Iterable[Remittance]: ...
    def distinct_providers(self) -> list[Provider]: ...
    def recent_activity(self, *, limit: int = 200) -> list[Activity]: ...

store = InMemoryStore()   # module-level singleton

Mappers (to_ui_claim, to_ui_remittance, to_ui_provider, to_activity_event) live in the same file. They translate the rich backend models (Pydantic, raw segments) to the simpler UI types the GET endpoints return.

Filter / sort API: each iter_* method takes the relevant filter kwargs and applies them in Python. There is no index — the in-memory list is small (single-session, dozens of batches at most). Sorting uses sorted(..., key=..., reverse=order == "desc").

Threading: every public method acquires self._lock (RLock, since list may call get internally) and releases it.

6.2 New routes in `backend/src/cyclone/api.py`

Method	Path	Behavior
GET	`/api/batches`	`store.list(limit=100)`, mapped to a `BatchSummary`-shaped response. Newest first.
GET	`/api/batches/{id}`	`store.get(id)`. 404 if missing. Returns the full `ParseResult` / `ParseResult835`.
GET	`/api/claims`	`store.iter_claims(...)` mapped to UI `Claim[]`. Supports `batch_id`, `status`, `payer`, `provider_npi`, `date_from`, `date_to`, `sort`, `order`, `limit` (≤ 1000), `offset`.
GET	`/api/remittances`	`store.iter_remittances(...)` mapped to UI `Remittance[]`. Supports `batch_id`, `payer`, `claim_id`, `date_from`, `date_to`, `sort`, `order`, `limit`, `offset`.
GET	`/api/providers`	`store.distinct_providers()`. Supports `npi`, `state`, `limit`, `offset`.
GET	`/api/activity`	`store.recent_activity(limit=200)`. Supports `kind`, `since`, `limit` (≤ 500).
GET	`/api/health`	(unchanged)

Streaming: every list endpoint above accepts Accept: application/x-ndjson. When set, the response is a StreamingResponse with one JSON object per line: {type: "item", data: {...}} for each result, then a final {type: "summary", data: {total: N, returned: M, has_more: bool}}. Default JSON response wraps the same data in a {items: [...], total: N, returned: M, has_more: bool} envelope so the frontend can paginate uniformly.

validation.passed and frequency_code == "1" → submitted
validation.passed and any validation.warnings → pending
!validation.passed and no error mentions R050_diagnosis_present → denied
otherwise → draft (rare; means validation broke somewhere)

Status mapping (835): 835 CLP02 status codes map to Remittance.status (received|posted|reconciled):

1, 2, 19, 20 (paid, primary/secondary) → received
4 (denied) → received with a denialReason populated from the CAS segments
21, 22 (reversal of previous payment) → reconciled (out-of-scope: full reversal handling; the data is in service_payments[*].adjustments)
all other valid codes → received (default)
any code outside cfg.allowed_status_codes → still stored, mapped to received, and surfaced as a validation warning

6.3 Existing parse routes — wire to the store

POST /api/parse-837 — on success, build a BatchRecord(kind="837p", result=result) and call store.add(...) before returning the response. On CycloneParseError or any unhandled exception, do not add.
POST /api/parse-835 — same pattern with kind="835".
Both routes also stamp parsed_at and generate id = uuid.uuid4().hex.

6.4 Uvicorn invocation

cyclone/__main__.py's serve branch:

sys.argv = [sys.argv[0], "cyclone.api:app", "--host", "127.0.0.1", "--port", "8000"]

--host 127.0.0.1 (not 0.0.0.0).
--port 8000 (configurable via CYCLONE_PORT env, default 8000).
--reload only when CYCLONE_RELOAD=1 (dev convenience).

CORS in api.py stays as-is: allow_origins=["http://localhost:5173"].

7. Frontend changes

7.1 Dependencies

Add to package.json:

"dependencies": {
  ...
  "@tanstack/react-query": "^5.59.0"
}

7.2 `src/main.tsx`

Wrap the existing <App /> in <QueryClientProvider client={queryClient}> where queryClient is a module-level new QueryClient({ defaultOptions: { queries: { staleTime: 30_000, refetchOnWindowFocus: true } } }).

7.3 `src/lib/api.ts` — add GET methods

Mirror the existing request<T> helper. New methods (all return Promise<T>):

api.listBatches(): Promise<BatchSummary[]>
api.getBatch(id: string): Promise<ParseResult837 | ParseResult835>
api.listClaims(params: ListClaimsParams): Promise<PaginatedResponse<Claim>>
api.listRemittances(params: ListRemittancesParams): Promise<PaginatedResponse<Remittance>>
api.listProviders(params?: ListProvidersParams): Promise<PaginatedResponse<Provider>>
api.listActivity(params?: ListActivityParams): Promise<PaginatedResponse<Activity>>

NDJSON streaming is not required for the initial frontend wiring (the existing Upload page already streams from the POST); GET is fine as JSON. The streaming option on GET is for future large-batch scenarios and is a backend-side capability only.

7.4 `src/hooks/` — new query / mutation hooks

// src/hooks/useBatches.ts
export function useBatches() {
  return useQuery({ queryKey: ['batches'], queryFn: () => api.listBatches() });
}

// src/hooks/useClaims.ts
export function useClaims(params: ListClaimsParams) {
  return useQuery({ queryKey: ['claims', params], queryFn: () => api.listClaims(params) });
}

// ... useRemittances, useProviders, useActivity similarly ...

// src/hooks/useParse.ts
export function useParse(kind: '837p' | '835') {
  const queryClient = useQueryClient();
  return useMutation({
    mutationFn: ({ file, onProgress }) =>
      kind === '837p' ? api.parse837(file, { onProgress }) : api.parse835(file, { onProgress }),
    onSuccess: () => {
      queryClient.invalidateQueries({ queryKey: ['batches'] });
      queryClient.invalidateQueries({ queryKey: ['claims'] });
      queryClient.invalidateQueries({ queryKey: ['remittances'] });
      queryClient.invalidateQueries({ queryKey: ['providers'] });
      queryClient.invalidateQueries({ queryKey: ['activity'] });
    },
  });
}

7.5 Page refactors

Page	Current	New
`src/pages/Claims.tsx`	`useAppStore((s) => s.claims)`	`useClaims({ status, search })`; falls back to store data when `!api.isConfigured` (handled in the hook)
`src/pages/Remittances.tsx`	store-backed	`useRemittances({})`
`src/pages/Providers.tsx`	store-backed	`useProviders({})`
`src/pages/ActivityLog.tsx`	store-backed	`useActivity({ since: now - 7d })`, `refetchInterval: 30_000`
`src/pages/Upload.tsx`	uses `useMutation`-less `addParsedBatch`	replace mutation with `useParse(kind)`; keep `addParsedBatch` for the "recent batches" list, but the live-data pages will refresh from the server

Each page gains: <Skeleton /> rows during isLoading, an error block with "Retry" button on isError, and a small footer showing {returned} of {total} when has_more is true.

Fallback when VITE_API_BASE_URL is empty: the existing data adapter in src/lib/api.ts continues to provide a path that reads from the in-memory zustand store. Each new hook checks api.isConfigured and either runs the react-query path (default) or returns a synchronous result from the store via useSyncExternalStore. Page components don't need to know which path is active. The data adapter is removed in a follow-up sub-project once no caller depends on it.

7.6 UI primitive: `src/components/ui/skeleton.tsx`

~10 lines, shadcn-style: <div className={cn('animate-pulse rounded-md bg-muted', className)} />. No new dep.

8. Docs

8.1 `docs/reference/837p.md` (~80 lines)

Sections:

File extension: .txt. Encoding: ASCII or UTF-8. Delimiters: * element, : component, ~ segment, ^ repetition.
Envelope: ISA/GS/ST headers, SE/GE/IEA footers.
Loops: 2000A (billing provider), 2000B (subscriber), 2300 (claim), 2400 (service line).
Segments Cyclone parses: NM1, N3/N4, REF, CLM, HI, LX, SV1, DTP, BHT.
Segments Cyclone skips (preserved in raw_segments but not modeled): too many to enumerate; see the parser source for the full list.
CO Medicaid specifics: REF*G1 prior-auth, no patient loop, CLM05 shape 12:B:1 etc.

8.2 `docs/reference/835.md` (~80 lines)

Envelope: same as 837P.
Header: BPR (financial), TRN (trace), DTM (dates).
Loops: 1000A (payer), 1000B (payee), 2100 (CLP claim payment), 2110 (SVC service payment).
Critical balancing rules: BPR02 == sum(CLP04), CLP04 == sum(SVC03).
CO Medicaid specifics: BPR10 81-1725341 (TXIX) or 84-0644739 (BHA), N104 7912900843, N1*PR name CO_TXIX or CO_BHA.

8.3 `docs/reference/x12naming.md` (~50 lines)

Segments: 2–3 letter codes (CLM, NM1, BPR).
Elements: *-separated, 1-indexed.
Composite elements: :-separated sub-elements, indexed 01, 02, ...
Loops: 4-digit IDs, hierarchical (2000 contains 2300 contains 2400).
Common qualifiers we care about: B (POS qualifier), ABK (ICD-10 principal diagnosis), MC (Medicaid claim filing).

8.4 `docs/reference/co-medicaid.md` (~80 lines)

Trading partner ID (TPID) — sender / receiver roles in 837P and 835.
Payer IDs:
- 837P: NM1*PR N104 = "SKCO0" (COHCPF).
- 835: BPR10 = "81-1725341" (TXIX) or "84-0644739" (BHA); N1*PR N104 = "7912900843".
Allowed frequency codes (837P CLM05-3): {1, 7, 8}.
Allowed status codes (835 CLP02): {1, 2, 3, 4, 19, 20, 21, 22, 23, 25}.
POS codes: full CMS set (89 codes; canonical list in cyclone.parsers.payer.CMS_PLACE_OF_SERVICE_CODES).

8.5 `README.md` (root, rewrite)

Sections:

What is Cyclone? — 1 paragraph. EDI claim parser + browser UI.
Install — git clone, cd backend && python -m venv .venv && .venv/bin/pip install -e '.[dev]', cd .. && npm install.
Dev — two terminals: cd backend && .venv/bin/python -m cyclone serve and npm run dev. Open http://localhost:5173.
Test — cd backend && .venv/bin/pytest and npm run build.
Project layout — backend/ (Python API + parsers), src/ (React UI), docs/reference/ (companion notes), docs/prodfiles/ (sample EDI files).
Roadmap — short list of sub-projects 2–4 with one-line descriptions and a note that they're not in this build.
License — no license file yet; add a LICENSE file when one is decided. The project is internal-use until then.

9. Error handling

Backend store errors: none expected (in-memory, no I/O). If a BatchRecord.result doesn't conform (e.g. we tried to store a result from a code path that returns None), the model validator fails loudly at insertion time.
GET route errors: 404 for missing batch_id; 400 for invalid query params (FastAPI's default validation handles this); 500 only for unexpected bugs.
NDJSON streaming: on a serialization error mid-stream, the response is already partially written; we log the error and end the stream. The client sees a truncated response and the final summary line is missing. The frontend falls back to "show what we have" and shows a toast.
Frontend: useQuery exposes isError + error. Each page renders an error block with a retry button that calls refetch(). Network errors show a sonner toast on top of the inline error.
Parse failure: the parse route already returns 400 / 422 with a clear message. The Upload page shows a destructive toast. The store is unchanged (failed parses are not persisted).

10. Testing

10.1 Backend (`backend/tests/`)

test_store.py (~8 tests) — add/list/get; concurrent adds under a thread; mapper functions; filter / sort / paginate; distinct providers; activity derivation.
test_api_gets.py (~12 tests) — one happy-path + one filter test per new GET endpoint (6 endpoints × 2 = 12); plus one 404 test for /api/batches/{id}.
test_api_streaming.py (1 test) — Accept: application/x-ndjson on /api/claims returns parseable lines.
test_api_parse_persists.py (2 tests) — parse-837 followed by GET /api/claims returns the claim; failed parse does not create a batch.

Total new backend tests: ~23.

10.2 Frontend (`src/`)

src/lib/api.test.ts — extend with 3 tests: listClaims builds the right URL with params, listBatches hits /api/batches, getBatch returns ParseResult835 when kind === "835".
No DOM / component tests in this sub-project. The page refactors are small and the smoke test below is the integration check.

10.3 End-to-end smoke test (manual, documented in the plan)

# Terminal 1
cd backend && .venv/bin/python -m cyclone serve

# Terminal 2
npm run dev

# In a third terminal
cd backend && .venv/bin/python -m cyclone.cli parse-837 tests/fixtures/co_medicaid_837p.txt --output-dir /tmp/co-837
# (this populates a real production-like batch; we don't need the JSON files for the test, just the upload flow)

# In a browser: open http://localhost:5173/upload, drop tests/fixtures/co_medicaid_837p.txt
# Verify: claims stream in, summary toast appears, navigate to /claims and see them

11. Migration / rollout

No DB, so no migration.
The frontend's existing data adapter stays during the transition. We delete it in a follow-up once no one calls it.
The new InMemoryStore is empty on first run. Existing parse-837/parse-835 callers see no behavior change (they still get the parse result; we just also add to the store now).
A uvicorn restart wipes the store. That is by design and called out in the README.

12. Out of scope (explicitly)

DB persistence, Alembic migrations, transactional safety.
837P ↔ 835 reconciliation (matching payouts to original claims).
999 ACK, 270/271, or other X12 transaction types.
More 837P validation rules beyond what already ships.
835 CAS deep-parsing with reason-code explanations.
Auth (any kind: API key, JWT, session).
Rate limiting, request size limits beyond FastAPI defaults.
Structured logging, log levels via env, JSON logs.
Health-check enhancements (uptime, parser state, last-batch timestamp).
12-factor env config beyond CYCLONE_PORT and CYCLONE_RELOAD.
Docker / docker-compose / any deploy artifact.
Pre-commit hooks, Makefile, .editorconfig, CONTRIBUTING.md, lint config.
Component tests for the React pages.
E2E browser tests (Playwright/Cypress).

13. Acceptance checklist

pytest passes with the new store + GET tests; total ≥ 121 + 23 = 144.
npm run build still passes.
python -m cyclone serve binds to 127.0.0.1:8000 and refuses connections from non-loopback.
Upload page streams a 2-claim CO fixture; navigating to /claims shows the same 2 claims.
Filtering by status=submitted on /api/claims returns the right subset; UI shows the same.
GET /api/claims with Accept: application/x-ndjson returns valid NDJSON.
Restarting the backend wipes the in-memory store; the UI shows "no claims yet" on first refresh.
All 4 reference notes exist under docs/reference/ and the root README.md is rewritten.
The frontend still works without a backend (VITE_API_BASE_URL empty) — pages fall back to the zustand sample store via the existing data adapter.

23 KiB Raw Blame History Unescape Escape