cyclone/README.md

# Cyclone

A self-hosted EDI claims management suite for a single billing office.
Parses 837P professional claims and 835 ERA remittances (X12 005010X222A1
and 005010X221A1), with a local-only FastAPI backend and a React UI for
browsing, filtering, and inspecting the parsed data.

Local-only on purpose: binds to `127.0.0.1`, no auth, no internet exposure.
Built for one operator, one machine, one trading partner (Colorado
Medicaid, currently).

## Install

```bash
# Backend (Python 3.11+)
cd backend
python -m venv .venv
.venv/bin/pip install -e '.[dev]'

# Frontend (Node 20+)
cd ..
npm install
```

## Dev

Two terminals:

```bash
# Terminal 1 — backend
cd backend
.venv/bin/python -m cyclone serve
# (defaults to 127.0.0.1:8000; override with CYCLONE_PORT=...; reload with CYCLONE_RELOAD=1)

# Terminal 2 — frontend
npm run dev
# (Vite on http://localhost:5173)
```

Then open `http://localhost:5173`. Drop an `.txt` 837P or 835 file on the
Upload page; navigate to Claims, Remittances, Providers, or Activity to
see the parsed data.

The frontend reads its backend URL from `VITE_API_BASE_URL` (default
empty). Create a `.env.local` at the repo root with:

```
VITE_API_BASE_URL=http://127.0.0.1:8000
```

Without that, the UI falls back to its in-memory sample store via the
existing `data` adapter (parses are disabled).

## Test

```bash
# Backend
cd backend && .venv/bin/pytest

# Frontend type-check + build
npm run typecheck
npm run build

# Frontend unit tests
npm test
```

## Live updates

The Claims, Remittances, and Activity pages stay current without
manual refresh. The backend publishes an internal event on every
store write, the page opens a streaming HTTP connection to the
matching `/api/<resource>/stream` endpoint, and new rows are
appended to the table the moment they hit the database.

### Wire format

Each stream endpoint emits newline-delimited JSON. The first batch
is the **snapshot** of currently-known rows; after that comes
**`snapshot_end`** with the count, then the **live** events.

```json
{"type":"item","data":{"id":"CLM-1", "...":"..."}}
{"type":"item","data":{"id":"CLM-2", "...":"..."}}
{"type":"snapshot_end","data":{"count":2}}
{"type":"item","data":{"id":"CLM-3", "...":"..."}}   ← live
{"type":"heartbeat","data":{"ts":"2026-06-20T23:17:09Z"}}   ← idle keep-alive
```

Lines are `{"type": ..., "data": ...}`; known types are `item`,
`snapshot_end`, `heartbeat`, and (rare) `item_dropped` /
`error`. Heartbeats keep the connection alive when nothing is
happening — clients flip to `stalled` after 30s of total silence
(heartbeat or otherwise) and surface a **↻ Reconnect** button.

### Endpoints

| Method | Path                       | Subscribes to       | Default sort        |
| ------ | -------------------------- | ------------------- | ------------------- |
| GET    | `/api/claims/stream`       | `claim_written`     | `-submission_date`  |
| GET    | `/api/remittances/stream`  | `remittance_written`| `-received_date`    |
| GET    | `/api/activity/stream`     | `activity_recorded` | `-timestamp` (limit 50) |

All three accept the same query params as their non-streaming
counterparts (`status`, `payer`, `date_from`, …) so a frontend can
swap a one-shot fetch for a tail with no URL surgery. Responses
are `Content-Type: application/x-ndjson`.

### Status pill

The pill in each toolbar reflects the current connection state:

| Status        | Badge variant | What it means                                                      |
| ------------- | ------------- | ------------------------------------------------------------------ |
| `live`        | success       | snapshot received, listening for new events                        |
| `connecting`  | warning       | opening the stream (initial mount)                                 |
| `reconnecting`| warning       | previous attempt failed, backing off before retry                  |
| `stalled`     | destructive   | no event (including heartbeat) for 30s — click **↻ Reconnect**     |
| `error`       | destructive   | stream errored — click **↻ Reconnect**                             |
| `closed`      | destructive   | page unmounted                                                     |

The reconnect button is only shown on `stalled` and `error`. The
backoff schedule on error is `1s → 2s → 4s → 8s → 16s → 30s` capped.

### Knobs

| Env var                      | Default | Effect                                                  |
| ---------------------------- | ------- | ------------------------------------------------------- |
| `CYCLONE_TAIL_HEARTBEAT_S`   | `15`    | Idle heartbeat interval (seconds). Lower it for tests.  |

## Inbox

`/inbox` is the working surface. Four lanes, dark by default (Ticker Tape
aesthetic):

- **Rejected** — claims whose 999 set-level response was R or E. Re-submit in bulk.
- **Candidates** — remits whose CLP-claim-id didn't match exactly; each one shows its top scored claim. One-click manual match or dismiss.
- **Unmatched** — claims still waiting for a remit, and remits with no candidates above the threshold.
- **Done today** — terminal state transitions in the last 24 hours.

The page subscribes to the SP5 claim and remittance tail streams
(`/api/claims/stream` and `/api/remittances/stream`); a new `item`
event refetches the lane payload (debounced ~250ms). When a 999
parses and rejects claims, the inbox reflects the new
**Rejected** rows within a fraction of a second.

### Inbox endpoints

| Method | Path                                          | Notes                                                            |
| ------ | --------------------------------------------- | ---------------------------------------------------------------- |
| GET    | `/api/inbox/lanes`                            | All four lanes in one call.                                      |
| POST   | `/api/inbox/candidates/{remit_id}/match`      | Manual match. `409` if the claim state moved out from under us.  |
| POST   | `/api/inbox/candidates/dismiss`               | `{pairs: [{claim_id, remit_id}]}`. Session-scoped.               |
| POST   | `/api/inbox/rejected/resubmit`                | `{claim_ids: [...]}`. `200` with `conflicts` for non-rejected.   |
| GET    | `/api/inbox/export.csv?lane=<lane>`           | Streams CSV of the lane's rows.                                  |

## Outbound 837 Serializer

The same canonical `ClaimOutput` model that powers 837 ingestion also
drives **outbound** 837P regeneration. Operators can take a claim
that's been parsed, edited, or rejected and emit a byte-faithful X12
837P file — closing the resubmit loop without leaving the app.

Two surfaces:

- **Single claim → .x12 download** from the ClaimDrawer header. Click
  the **Download 837** icon, get a `claim-{id}.x12` attachment. The
  file is a complete ISA/GS/ST envelope + hierarchy + CLM/SV1 lines,
  regenerated from the canonical `ClaimOutput` fields (not by patching
  `raw_segments`).
- **Multi-claim → .zip bundle** from the Inbox rejected lane. Select
  N>1 rejected claims, hit **Resubmit**, choose **Resubmit + Download**
  in the confirm modal, and the backend hands you
  `resubmit-{N}-claims.zip` with one `.x12` per successfully
  resubmitted claim. Each file gets a unique ISA13/GS06 so back-to-back
  claims don't collide on control numbers.

Design spec at `docs/superpowers/specs/2026-06-20-cyclone-serialize-837-design.md`.
Implementation note: the spec proposed a hybrid "patch `raw_segments`"
approach, but `raw_segments` only captures post-CLM segments (CLM,
REF*G1, HI, LX, SV1 pairs) — the envelope and hierarchy segments
(NM1, N3, N4, HL, PRV, SBR, DMG) are not captured, so a pass-through
serializer cannot regenerate them byte-for-byte. The shipped
implementation rebuilds the entire file from canonical fields
("Approach A" in the plan). The round-trip guarantee still holds:
all 113 prodfiles in `docs/prodfiles/claims/*.x12` parse → serialize →
parse back to the same claim id.

### Where to find it

- Backend serializer: `backend/src/cyclone/parsers/serialize_837.py`
- Single-claim download API: `GET /api/claims/{id}/serialize-837`
- Bundle API: `POST /api/inbox/rejected/resubmit?download=true`
- Frontend helper: `src/lib/api.ts:serializeClaim837`,
  `src/lib/inbox-api.ts:resubmitRejectedWithDownload`
- Browser download utility: `src/lib/download.ts` (`downloadTextFile`,
  `downloadBlob`)
- UI:
  - `src/components/ClaimDrawer/ClaimDrawerHeader.tsx` — Download icon
  - `src/pages/Inbox.tsx` — Resubmit bundle modal + progress overlay

## Per-Line Adjustment Audit

Every 835 CAS segment is tied back to the specific 837 service line it
adjudicates, instead of being aggregated to the claim level. The match
runs **eagerly** at 835 ingest time so the audit trail is stable across
re-reads (a re-migration is required if the algorithm changes).

### Where to find it

- **ClaimDrawer → "Line Reconciliation" tab** — side-by-side view of the
  837 SV1 lines and 835 SVC composites with per-line CAS reasons.
- **ClaimDrawer → ServiceLinesTable** — every billed line now shows its
  paid amount and adjustment sum on the right.
- **RemitDrawer → CasAdjustmentsPanel** — each CAS row carries a
  `proc · #N` reference; claim-level CAS clusters separately.
- **ClaimDrawer → MatchedRemitCard** — a `lines: N/M matched` badge
  lights up amber when at least one line is unmatched.

### Match criteria (strict)

A 835 SVC composite matches a 837 SV1 line iff all four criteria align:

| Field         | Rule                                            |
| ------------- | ----------------------------------------------- |
| Procedure     | Exact match (case-insensitive, uppercased).     |
| Modifiers     | Set-equal (order-independent).                  |
| Service date  | Exact match (both null counts as a match).      |
| Units         | Exact match (both null counts as a match).      |

Unmatched lines surface as a **soft warning** — the claim still posts to
PAID/PARTIAL/RECEIVED, but the Inbox shows the unmatched count and the
drawer surfaces a per-line "no 837 line matched" note.

### Line-level endpoints

| Method | Path                                                | Notes                                                                                  |
| ------ | --------------------------------------------------- | -------------------------------------------------------------------------------------- |
| GET    | `/api/claims/{claim_id}/line-reconciliation`        | Dedicated view for the drawer tab.                                                     |
| GET    | `/api/claims/{claim_id}`                            | Now includes a slim `lineReconciliation[]`.                                            |
| GET    | `/api/remittances/{remittance_id}`                  | Now includes `serviceLinePayments[]` + `claimLevelAdjustments[]`.                      |
| GET    | `/api/inbox/lanes`                                  | `matched_remittance` payload gains `matched_lines` + `total_lines`.                    |

### Scoring

4-field weighted (sum = 100):

| Field                   | Weight | Rule                                                          |
| ----------------------- | ------ | ------------------------------------------------------------- |
| Patient control number  | 40     | Exact match, normalized (case + leading zeros).              |
| Service date            | 25     | Linear decay over ±3 days.                                    |
| Charge amount           | 20     | Linear decay over ±10%.                                       |
| Provider NPI            | 15     | Exact match. Missing → 0.                                     |

Tiers: **strong** (≥75, full opacity, Match enabled), **weak**
(50–74, dimmed), **hidden** (<50, not surfaced).

## Persistence

Parsed batches, claims, remittances, matches, and activity events are
stored in a SQLite file at `~/.local/share/cyclone/cyclone.db` by
default. The directory is auto-created on first run.

To use a different location, set `CYCLONE_DB_URL`:

```bash
export CYCLONE_DB_URL=sqlite:///path/to/cyclone.db
# or
export CYCLONE_DB_URL=postgresql://user:pass@host:5432/cyclone
```

### Backup

```bash
sqlite3 ~/.local/share/cyclone/cyclone.db ".backup /path/to/backup.db"
```

This is safe to run while the backend is running (uses SQLite's online
backup API).

## Project layout

```
.
├── backend/
│   ├── src/cyclone/
│   │   ├── api.py             # FastAPI app, GET + parse routes, /api/{resource}/stream
│   │   ├── pubsub.py          # in-process EventBus (drop-oldest, per-kind fan-out)
│   │   ├── store.py           # InMemoryStore, mappers, publish-on-write
│   │   ├── __main__.py        # `python -m cyclone serve`
│   │   ├── cli.py             # click CLI
│   │   └── parsers/           # X12 tokenizer, models, validator, writers
│   └── tests/
│       ├── fixtures/          # co_medicaid_*.txt, minimal_*.txt
│       ├── test_api.py        # parse-837/835 round-trip
│       ├── test_api_gets.py   # 6 GET endpoints
│       ├── test_store.py      # store + mappers + iterators
│       ├── test_api_streaming.py
│       ├── test_api_stream_live.py  # 3 live-tail endpoints + disconnect cleanup
│       ├── test_pubsub.py     # EventBus + subscribe/unsubscribe
│       └── test_api_parse_persists.py
├── src/                       # React + Vite + TypeScript UI
│   ├── components/
│   │   ├── ui/                # Skeleton, EmptyState, ErrorState, FilterChips, Pagination, …
│   │   └── TailStatusPill.tsx # live-tail status badge + reconnect button
│   ├── pages/                 # Claims, Remittances, Providers, Activity, Upload
│   ├── hooks/                 # useBatches, useClaims, useRemittances, useProviders, useActivity, useParse
│   │                          # + useTailStream, useMergedTail (live tail)
│   ├── lib/                   # api.ts (6 GET + parse837/parse835/health), format.ts, utils.ts
│   │                          # + tail-stream.ts (NDJSON parser)
│   ├── store/                 # zustand sample-data + parsed-batches store
│   │                          # + tail-store.ts (FIFO-capped live tail slices)
│   └── types/                 # shared TS types
├── docs/
│   ├── reference/             # condensed 837P/835/X12/CO Medicaid notes
│   └── superpowers/plans/     # implementation plan
├── tailwind.config.js         # shimmer, scan, row-flash keyframes
└── package.json
```

## Roadmap

Sub-projects 2 through 8 are **shipped**. Next up:

- **Sub-project 8 (shipped) — Outbound 837P serializer.** Closes the
  resubmit loop: rejected claims can be regenerated back to an X12 837P
  file (single download from the claim drawer, or ZIP bundle from the
  inbox rejected lane). See the "Outbound 837 Serializer" section
  below for details.
- **Sub-project 3 (shipped) — More 837P/835 features.**
  - **837P validation rules:** R034 enforces `REF*G1` on frequency-code
    7/8 claims; R035 enforces `BHT06` transaction-type-code is in the
    allowed set per payer config.
  - **835 CAS deep-parsing:** every CAS adjustment now surfaces with its
    CARC reason code + label (e.g. `CO-29: The time limit for filing has
    expired`), surfaced via `GET /api/remittances/{id}` and rendered as
    an expansion row on the Remittances page.
  - **999 ACK transaction set:** full inbound parser + outbound
    serializer. Auto-generated on 837 ingest when `?ack=true` is passed
    to `POST /api/parse-837`. Persisted to the `acks` table, browsable
    on a new `/acks` page, and downloadable as the regenerated raw
    999 text.
  - **270/271 eligibility (API-only):** `POST /api/eligibility/request`
    builds a 270 from a JSON payload (subscriber, provider, payer,
    service type); `POST /api/eligibility/parse-271` ingests the
    response and returns structured `coverage_benefits`.
  - **TA1 interchange ACK:** parser + persistence + API for the
    envelope-level TA1 acknowledgment (separate from the 999
    transaction-set ACK).
- **Sub-project 4 (shipped) — Frontend features.**
  - **Per-claim detail drawer:** click any row on the Claims page to
    open a side-panel drawer with the full claim context — state +
    amount header, validation panel, service lines, diagnoses, parties
    (billing provider, subscriber, payer), raw X12 segments, matched
    remittance summary, and a vertical state-history timeline. URL is
    synced (`?claim=...`) so links and back-button restore the drawer.
    Keyboard nav: `j`/`k` move between claims, `esc` closes, `?` opens
    the cheatsheet overlay. Skeleton + error + not-found (404) states
    are all distinct. Powered by a new backend endpoint
    `GET /api/claims/{claim_id}` that returns the full drawer payload
    in a single round-trip.
  - **Remit drawer:** mirror of the claim drawer for remittances —
    payer + provider header, financial summary, claim-payments table,
    CAS adjustments panel, parties grid, raw X12 segments. URL is
    synced (`?remit=...`).
  - **Batch diff view:** side-by-side comparison of two batches
    (added / removed / changed claims) reachable from the Batches
    page. Powers the `BatchDetail` and `BatchDiff` views.
  - **Global Cmd-K search:** cross-resource search across claims,
    remittances, and activity; opens from anywhere via the keyboard
    shortcut.
  - **CSV export:** per-page export buttons (claims, remittances, lanes)
    that stream CSV directly from the backend.
  - **Accessibility + print styles:** keyboard navigation focus rings,
    `aria` roles, and `@media print` stylesheets for paper-friendly
    output.
- **Sub-project 5 (shipped) — Live updates.**
  - The Claims, Remittances, and Activity pages stream new rows in
    real time over Server-Sent Events encoded as newline-delimited
    JSON. The backend publishes a `claim_written` /
    `remittance_written` / `activity_recorded` event on every store
    write; the frontend opens an HTTP stream and dispatches each
    event into a per-resource Zustand store that the page reads on
    top of its base query results.
  - A small status pill in each toolbar surfaces the connection
    state — `live` / `connecting` / `reconnecting` / `stalled` /
    `error` / `closed` — and offers a manual **↻ Reconnect** button
    on `stalled` or `error`. Stale connections (no event for 30s,
    heartbeat included) flip to `stalled` automatically; the
    back-off ladder on errors is `1s → 2s → 4s → 8s → 16s → 30s`
    capped. See the "Live updates" section below for details.
- **Sub-project 6 (shipped) — Inbox workflow automation.**
  - **Ticker-Tape inbox (`/inbox`):** four lanes ordered by urgency —
    **Rejected** (claims whose 999 rejected them), **Candidates**
    (remits that didn't auto-match a claim), **Unmatched** (claims
    still waiting for a remit), and **Done today** (terminal
    transitions in the last 24 hours). The page subscribes to the
    claim and remittance tail streams so new rejected claims land in
    the Rejected lane within a fraction of a second.
  - **4-field weighted scoring:** every candidate remit is scored on
    patient control number (40), service date (25), charge amount
    (20), and provider NPI (15). Tiers: strong (≥75, Match enabled),
    weak (50–74, dimmed), hidden (<50).
  - **Bulk actions:** multi-select rows in any lane to dismiss
    candidates, resubmit rejected claims, or export the lane as CSV
    in a single round-trip.
- **Sub-project 7 (shipped) — Per-line adjustment audit.**
  - Every 835 CAS segment is tied back to the specific 837 service
    line it adjudicates, instead of being aggregated to the claim
    level. The match runs **eagerly** at 835 ingest time so the audit
    trail is stable across re-reads.
  - New endpoints surface the per-line projection: `GET
    /api/claims/{id}/line-reconciliation` for the dedicated tab in
    the ClaimDrawer; the existing claim + remit detail endpoints
    gained slim `lineReconciliation[]` / `serviceLinePayments[]` /
    `claimLevelAdjustments[]` fields; the inbox lanes include
    `matched_lines` / `total_lines` badges on matched remittances.
  - The ClaimDrawer gains a **Line Reconciliation** tab (alongside
    Details) that pairs every 837 SV1 line with its 835 SVC composite
    and surfaces per-line CAS reasons + unmatched-line warnings.
  - See the "Per-Line Adjustment Audit" section below for details.

### SP3 endpoints

- `POST /api/parse-837?ack=true` — existing 837 parse, plus optional
  auto-generated 999 ACK persisted alongside the batch.
- `POST /api/parse-999` — parse an inbound 999 ACK and persist it.
- `GET /api/acks` — list ACKs.
- `GET /api/acks/{id}` — ACK detail, including the regenerated
  `raw_999_text`.
- `POST /api/eligibility/request` — build a 270 from JSON.
- `POST /api/eligibility/parse-271` — ingest a 271 and return parsed
  coverage benefits.

The UI gains a new **Acks** page (sidebar entry) that lists persisted
ACKs and lets you download the regenerated 999 text.

### SP4 endpoints (claim drawer)

- `GET /api/claims/{claim_id}` — full claim context for the drawer
  payload: header fields, diagnoses, service lines, parties
  (billing provider / subscriber / payer), validation result, raw
  X12 segments, matched remittance summary (or `null`), and the
  claim's state history (ordered newest-first). 404 with a structured
  `{ "error": "Not found", "detail": "Claim {id} not found" }` body
  when the claim doesn't exist — distinct from a transient fetch
  failure so the UI can render a dedicated not-found state.

### SP5 endpoints (live updates)

- `GET /api/claims/stream` — NDJSON stream of claim events. Emits a
  snapshot (filtered by the same query params as `GET /api/claims`),
  then `snapshot_end`, then live `claim_written` events as they
  arrive, with a 15s idle heartbeat.
- `GET /api/remittances/stream` — same shape for remittances
  (subscribes to `remittance_written`, default sort `-received_date`).
- `GET /api/activity/stream` — same shape for activity events
  (subscribes to `activity_recorded`, default `limit=50`).

### SP6 endpoints (inbox)

- `GET /api/inbox/lanes` — all four lanes (rejected, candidates,
  unmatched, done_today) in a single round-trip, with row-level
  scoring and matched-remit context.
- `POST /api/inbox/candidates/{remit_id}/match` — manual match of a
  candidate remit to one of its scored claims; `409` if the claim
  state moved out from under us.
- `POST /api/inbox/candidates/dismiss` — `{pairs: [{claim_id,
  remit_id}]}`. Session-scoped, no DB persistence.
- `POST /api/inbox/rejected/resubmit` — `{claim_ids: [...]}`. `200`
  with `conflicts` for non-rejected claims in the batch.
- `GET /api/inbox/export.csv?lane=<lane>` — streams CSV of the
  lane's rows.

### SP7 endpoints (per-line reconciliation)

- `GET /api/claims/{id}/line-reconciliation` — dedicated view for
  the ClaimDrawer's Line Reconciliation tab; returns the slim
  per-line projection (`summary` + `lines[]`).
- `GET /api/claims/{id}` — now includes a slim `lineReconciliation[]`
  so the ServiceLinesTable can show paid + adjustments columns
  without a second fetch.
- `GET /api/remittances/{id}` — now includes `serviceLinePayments[]`
  + `claimLevelAdjustments[]`; the CAS panel renders per-line + claim
  level separately.
- `GET /api/inbox/lanes` — `matched_remittance` payload gained
  `matched_lines` + `total_lines` so the MatchedRemitCard can badge
  partial matches.

### SP8 endpoints (outbound 837P)

- `GET /api/claims/{id}/serialize-837` — regenerate the persisted
  claim as a byte-faithful X12 837P file (`text/x12` body,
  `Content-Disposition: attachment; filename="claim-{id}.x12"`).
  404 if the claim doesn't exist, 422 if the stored `raw_json`
  can't be revalidated as a `ClaimOutput`. Drives the **Download 837**
  icon in the ClaimDrawer header.
- `POST /api/inbox/rejected/resubmit?download=true` — same mass-resubmit
  endpoint as SP6, but the response is a ZIP archive (`application/zip`)
  containing one `claim-{id}.x12` per successfully resubmitted claim.
  Each file uses `serialize_837_for_resubmit` so back-to-back files in
  the bundle get unique ISA13/GS06 control numbers (back-to-back
  resubmits would otherwise all share `000000001`). Conflicts and
  missing ids are deliberately omitted from the ZIP — the user already
  saw them in the JSON response on prior calls. Per-claim regenerate
  failures (rare — usually means `raw_json` is corrupted for one claim)
  are surfaced via the `X-Cyclone-Serialize-Errors` response header
  (JSON-encoded array) so the UI can show "10 resubmitted, 2 couldn't
  be regenerated" without parsing the binary. Drives the **Resubmit +
  Download** button in the Inbox rejected-lane BulkBar (N>1 modal
  prompt).

## License

No license file yet; this is internal-use software. Add a `LICENSE` file
when one is decided.