cyclone/README.md

# Cyclone

A self-hosted EDI claims management suite for a single billing office.
Parses 837P professional claims and 835 ERA remittances (X12 005010X222A1
and 005010X221A1), with a local-only FastAPI backend and a React UI for
browsing, filtering, and inspecting the parsed data.

Local-only on purpose: binds to `127.0.0.1`, no auth, no internet exposure.
Built for one operator, one machine, one trading partner (Colorado
Medicaid, currently).

## Install

```bash
# Backend (Python 3.11+)
cd backend
python -m venv .venv
.venv/bin/pip install -e '.[dev]'

# Frontend (Node 20+)
cd ..
npm install
```

## Dev

Two terminals:

```bash
# Terminal 1 — backend
cd backend
.venv/bin/python -m cyclone serve
# (defaults to 127.0.0.1:8000; override with CYCLONE_PORT=...; reload with CYCLONE_RELOAD=1)

# Terminal 2 — frontend
npm run dev
# (Vite on http://localhost:5173)
```

Then open `http://localhost:5173`. Drop an `.txt` 837P or 835 file on the
Upload page; navigate to Claims, Remittances, Providers, or Activity to
see the parsed data.

The frontend reads its backend URL from `VITE_API_BASE_URL` (default
empty). Create a `.env.local` at the repo root with:

```
VITE_API_BASE_URL=http://127.0.0.1:8000
```

Without that, the UI falls back to its in-memory sample store via the
existing `data` adapter (parses are disabled).

## Test

```bash
# Backend
cd backend && .venv/bin/pytest

# Frontend type-check + build
npm run typecheck
npm run build

# Frontend unit tests
npm test
```

## Live updates

The Claims, Remittances, and Activity pages stay current without
manual refresh. The backend publishes an internal event on every
store write, the page opens a streaming HTTP connection to the
matching `/api/<resource>/stream` endpoint, and new rows are
appended to the table the moment they hit the database.

### Wire format

Each stream endpoint emits newline-delimited JSON. The first batch
is the **snapshot** of currently-known rows; after that comes
**`snapshot_end`** with the count, then the **live** events.

```json
{"type":"item","data":{"id":"CLM-1", "...":"..."}}
{"type":"item","data":{"id":"CLM-2", "...":"..."}}
{"type":"snapshot_end","data":{"count":2}}
{"type":"item","data":{"id":"CLM-3", "...":"..."}}   ← live
{"type":"heartbeat","data":{"ts":"2026-06-20T23:17:09Z"}}   ← idle keep-alive
```

Lines are `{"type": ..., "data": ...}`; known types are `item`,
`snapshot_end`, `heartbeat`, and (rare) `item_dropped` /
`error`. Heartbeats keep the connection alive when nothing is
happening — clients flip to `stalled` after 30s of total silence
(heartbeat or otherwise) and surface a **↻ Reconnect** button.

### Endpoints

| Method | Path                       | Subscribes to       | Default sort        |
| ------ | -------------------------- | ------------------- | ------------------- |
| GET    | `/api/claims/stream`       | `claim_written`     | `-submission_date`  |
| GET    | `/api/remittances/stream`  | `remittance_written`| `-received_date`    |
| GET    | `/api/activity/stream`     | `activity_recorded` | `-timestamp` (limit 50) |

All three accept the same query params as their non-streaming
counterparts (`status`, `payer`, `date_from`, …) so a frontend can
swap a one-shot fetch for a tail with no URL surgery. Responses
are `Content-Type: application/x-ndjson`.

### Status pill

The pill in each toolbar reflects the current connection state:

| Status        | Badge variant | What it means                                                      |
| ------------- | ------------- | ------------------------------------------------------------------ |
| `live`        | success       | snapshot received, listening for new events                        |
| `connecting`  | warning       | opening the stream (initial mount)                                 |
| `reconnecting`| warning       | previous attempt failed, backing off before retry                  |
| `stalled`     | destructive   | no event (including heartbeat) for 30s — click **↻ Reconnect**     |
| `error`       | destructive   | stream errored — click **↻ Reconnect**                             |
| `closed`      | destructive   | page unmounted                                                     |

The reconnect button is only shown on `stalled` and `error`. The
backoff schedule on error is `1s → 2s → 4s → 8s → 16s → 30s` capped.

### Knobs

| Env var                      | Default | Effect                                                  |
| ---------------------------- | ------- | ------------------------------------------------------- |
| `CYCLONE_TAIL_HEARTBEAT_S`   | `15`    | Idle heartbeat interval (seconds). Lower it for tests.  |

## Inbox

`/inbox` is the working surface. Four lanes, dark by default (Ticker Tape
aesthetic):

- **Rejected** — claims whose 999 set-level response was R or E. Re-submit in bulk.
- **Candidates** — remits whose CLP-claim-id didn't match exactly; each one shows its top scored claim. One-click manual match or dismiss.
- **Unmatched** — claims still waiting for a remit, and remits with no candidates above the threshold.
- **Done today** — terminal state transitions in the last 24 hours.

The page subscribes to the SP5 claim and remittance tail streams
(`/api/claims/stream` and `/api/remittances/stream`); a new `item`
event refetches the lane payload (debounced ~250ms). When a 999
parses and rejects claims, the inbox reflects the new
**Rejected** rows within a fraction of a second.

### Inbox endpoints

| Method | Path                                          | Notes                                                            |
| ------ | --------------------------------------------- | ---------------------------------------------------------------- |
| GET    | `/api/inbox/lanes`                            | All four lanes in one call.                                      |
| POST   | `/api/inbox/candidates/{remit_id}/match`      | Manual match. `409` if the claim state moved out from under us.  |
| POST   | `/api/inbox/candidates/dismiss`               | `{pairs: [{claim_id, remit_id}]}`. Session-scoped.               |
| POST   | `/api/inbox/rejected/resubmit`                | `{claim_ids: [...]}`. `200` with `conflicts` for non-rejected.   |
| GET    | `/api/inbox/export.csv?lane=<lane>`           | Streams CSV of the lane's rows.                                  |

## Per-Line Adjustment Audit

Every 835 CAS segment is tied back to the specific 837 service line it
adjudicates, instead of being aggregated to the claim level. The match
runs **eagerly** at 835 ingest time so the audit trail is stable across
re-reads (a re-migration is required if the algorithm changes).

### Where to find it

- **ClaimDrawer → "Line Reconciliation" tab** — side-by-side view of the
  837 SV1 lines and 835 SVC composites with per-line CAS reasons.
- **ClaimDrawer → ServiceLinesTable** — every billed line now shows its
  paid amount and adjustment sum on the right.
- **RemitDrawer → CasAdjustmentsPanel** — each CAS row carries a
  `proc · #N` reference; claim-level CAS clusters separately.
- **ClaimDrawer → MatchedRemitCard** — a `lines: N/M matched` badge
  lights up amber when at least one line is unmatched.

### Match criteria (strict)

A 835 SVC composite matches a 837 SV1 line iff all four criteria align:

| Field         | Rule                                            |
| ------------- | ----------------------------------------------- |
| Procedure     | Exact match (case-insensitive, uppercased).     |
| Modifiers     | Set-equal (order-independent).                  |
| Service date  | Exact match (both null counts as a match).      |
| Units         | Exact match (both null counts as a match).      |

Unmatched lines surface as a **soft warning** — the claim still posts to
PAID/PARTIAL/RECEIVED, but the Inbox shows the unmatched count and the
drawer surfaces a per-line "no 837 line matched" note.

### Line-level endpoints

| Method | Path                                                | Notes                                                                                  |
| ------ | --------------------------------------------------- | -------------------------------------------------------------------------------------- |
| GET    | `/api/claims/{claim_id}/line-reconciliation`        | Dedicated view for the drawer tab.                                                     |
| GET    | `/api/claims/{claim_id}`                            | Now includes a slim `lineReconciliation[]`.                                            |
| GET    | `/api/remittances/{remittance_id}`                  | Now includes `serviceLinePayments[]` + `claimLevelAdjustments[]`.                      |
| GET    | `/api/inbox/lanes`                                  | `matched_remittance` payload gains `matched_lines` + `total_lines`.                    |

### Scoring

4-field weighted (sum = 100):

| Field                   | Weight | Rule                                                          |
| ----------------------- | ------ | ------------------------------------------------------------- |
| Patient control number  | 40     | Exact match, normalized (case + leading zeros).              |
| Service date            | 25     | Linear decay over ±3 days.                                    |
| Charge amount           | 20     | Linear decay over ±10%.                                       |
| Provider NPI            | 15     | Exact match. Missing → 0.                                     |

Tiers: **strong** (≥75, full opacity, Match enabled), **weak**
(50–74, dimmed), **hidden** (<50, not surfaced).

## Persistence

Parsed batches, claims, remittances, matches, and activity events are
stored in a SQLite file at `~/.local/share/cyclone/cyclone.db` by
default. The directory is auto-created on first run.

To use a different location, set `CYCLONE_DB_URL`:

```bash
export CYCLONE_DB_URL=sqlite:///path/to/cyclone.db
# or
export CYCLONE_DB_URL=postgresql://user:pass@host:5432/cyclone
```

### Backup

```bash
sqlite3 ~/.local/share/cyclone/cyclone.db ".backup /path/to/backup.db"
```

This is safe to run while the backend is running (uses SQLite's online
backup API).

## Project layout

```
.
├── backend/
│   ├── src/cyclone/
│   │   ├── api.py             # FastAPI app, GET + parse routes, /api/{resource}/stream
│   │   ├── pubsub.py          # in-process EventBus (drop-oldest, per-kind fan-out)
│   │   ├── store.py           # InMemoryStore, mappers, publish-on-write
│   │   ├── __main__.py        # `python -m cyclone serve`
│   │   ├── cli.py             # click CLI
│   │   └── parsers/           # X12 tokenizer, models, validator, writers
│   └── tests/
│       ├── fixtures/          # co_medicaid_*.txt, minimal_*.txt
│       ├── test_api.py        # parse-837/835 round-trip
│       ├── test_api_gets.py   # 6 GET endpoints
│       ├── test_store.py      # store + mappers + iterators
│       ├── test_api_streaming.py
│       ├── test_api_stream_live.py  # 3 live-tail endpoints + disconnect cleanup
│       ├── test_pubsub.py     # EventBus + subscribe/unsubscribe
│       └── test_api_parse_persists.py
├── src/                       # React + Vite + TypeScript UI
│   ├── components/
│   │   ├── ui/                # Skeleton, EmptyState, ErrorState, FilterChips, Pagination, …
│   │   └── TailStatusPill.tsx # live-tail status badge + reconnect button
│   ├── pages/                 # Claims, Remittances, Providers, Activity, Upload
│   ├── hooks/                 # useBatches, useClaims, useRemittances, useProviders, useActivity, useParse
│   │                          # + useTailStream, useMergedTail (live tail)
│   ├── lib/                   # api.ts (6 GET + parse837/parse835/health), format.ts, utils.ts
│   │                          # + tail-stream.ts (NDJSON parser)
│   ├── store/                 # zustand sample-data + parsed-batches store
│   │                          # + tail-store.ts (FIFO-capped live tail slices)
│   └── types/                 # shared TS types
├── docs/
│   ├── reference/             # condensed 837P/835/X12/CO Medicaid notes
│   └── superpowers/plans/     # implementation plan
├── tailwind.config.js         # shimmer, scan, row-flash keyframes
└── package.json
```

## Roadmap

Sub-projects 2 through 7 are **shipped**. Next up:

- **Sub-project 3 (shipped) — More 837P/835 features.**
  - **837P validation rules:** R034 enforces `REF*G1` on frequency-code
    7/8 claims; R035 enforces `BHT06` transaction-type-code is in the
    allowed set per payer config.
  - **835 CAS deep-parsing:** every CAS adjustment now surfaces with its
    CARC reason code + label (e.g. `CO-29: The time limit for filing has
    expired`), surfaced via `GET /api/remittances/{id}` and rendered as
    an expansion row on the Remittances page.
  - **999 ACK transaction set:** full inbound parser + outbound
    serializer. Auto-generated on 837 ingest when `?ack=true` is passed
    to `POST /api/parse-837`. Persisted to the `acks` table, browsable
    on a new `/acks` page, and downloadable as the regenerated raw
    999 text.
  - **270/271 eligibility (API-only):** `POST /api/eligibility/request`
    builds a 270 from a JSON payload (subscriber, provider, payer,
    service type); `POST /api/eligibility/parse-271` ingests the
    response and returns structured `coverage_benefits`.
  - **TA1 interchange ACK:** parser + persistence + API for the
    envelope-level TA1 acknowledgment (separate from the 999
    transaction-set ACK).
- **Sub-project 4 (shipped) — Frontend features.**
  - **Per-claim detail drawer:** click any row on the Claims page to
    open a side-panel drawer with the full claim context — state +
    amount header, validation panel, service lines, diagnoses, parties
    (billing provider, subscriber, payer), raw X12 segments, matched
    remittance summary, and a vertical state-history timeline. URL is
    synced (`?claim=...`) so links and back-button restore the drawer.
    Keyboard nav: `j`/`k` move between claims, `esc` closes, `?` opens
    the cheatsheet overlay. Skeleton + error + not-found (404) states
    are all distinct. Powered by a new backend endpoint
    `GET /api/claims/{claim_id}` that returns the full drawer payload
    in a single round-trip.
  - **Remit drawer:** mirror of the claim drawer for remittances —
    payer + provider header, financial summary, claim-payments table,
    CAS adjustments panel, parties grid, raw X12 segments. URL is
    synced (`?remit=...`).
  - **Batch diff view:** side-by-side comparison of two batches
    (added / removed / changed claims) reachable from the Batches
    page. Powers the `BatchDetail` and `BatchDiff` views.
  - **Global Cmd-K search:** cross-resource search across claims,
    remittances, and activity; opens from anywhere via the keyboard
    shortcut.
  - **CSV export:** per-page export buttons (claims, remittances, lanes)
    that stream CSV directly from the backend.
  - **Accessibility + print styles:** keyboard navigation focus rings,
    `aria` roles, and `@media print` stylesheets for paper-friendly
    output.
- **Sub-project 5 (shipped) — Live updates.**
  - The Claims, Remittances, and Activity pages stream new rows in
    real time over Server-Sent Events encoded as newline-delimited
    JSON. The backend publishes a `claim_written` /
    `remittance_written` / `activity_recorded` event on every store
    write; the frontend opens an HTTP stream and dispatches each
    event into a per-resource Zustand store that the page reads on
    top of its base query results.
  - A small status pill in each toolbar surfaces the connection
    state — `live` / `connecting` / `reconnecting` / `stalled` /
    `error` / `closed` — and offers a manual **↻ Reconnect** button
    on `stalled` or `error`. Stale connections (no event for 30s,
    heartbeat included) flip to `stalled` automatically; the
    back-off ladder on errors is `1s → 2s → 4s → 8s → 16s → 30s`
    capped. See the "Live updates" section below for details.
- **Sub-project 6 (shipped) — Inbox workflow automation.**
  - **Ticker-Tape inbox (`/inbox`):** four lanes ordered by urgency —
    **Rejected** (claims whose 999 rejected them), **Candidates**
    (remits that didn't auto-match a claim), **Unmatched** (claims
    still waiting for a remit), and **Done today** (terminal
    transitions in the last 24 hours). The page subscribes to the
    claim and remittance tail streams so new rejected claims land in
    the Rejected lane within a fraction of a second.
  - **4-field weighted scoring:** every candidate remit is scored on
    patient control number (40), service date (25), charge amount
    (20), and provider NPI (15). Tiers: strong (≥75, Match enabled),
    weak (50–74, dimmed), hidden (<50).
  - **Bulk actions:** multi-select rows in any lane to dismiss
    candidates, resubmit rejected claims, or export the lane as CSV
    in a single round-trip.
- **Sub-project 7 (shipped) — Per-line adjustment audit.**
  - Every 835 CAS segment is tied back to the specific 837 service
    line it adjudicates, instead of being aggregated to the claim
    level. The match runs **eagerly** at 835 ingest time so the audit
    trail is stable across re-reads.
  - New endpoints surface the per-line projection: `GET
    /api/claims/{id}/line-reconciliation` for the dedicated tab in
    the ClaimDrawer; the existing claim + remit detail endpoints
    gained slim `lineReconciliation[]` / `serviceLinePayments[]` /
    `claimLevelAdjustments[]` fields; the inbox lanes include
    `matched_lines` / `total_lines` badges on matched remittances.
  - The ClaimDrawer gains a **Line Reconciliation** tab (alongside
    Details) that pairs every 837 SV1 line with its 835 SVC composite
    and surfaces per-line CAS reasons + unmatched-line warnings.
  - See the "Per-Line Adjustment Audit" section below for details.

**Next up:** outbound 837P serializer — closes the resubmit loop
(rejected claim → corrected 837P back to payer). Design spec at
`docs/superpowers/specs/2026-06-20-cyclone-serialize-837-design.md`.

### SP3 endpoints

- `POST /api/parse-837?ack=true` — existing 837 parse, plus optional
  auto-generated 999 ACK persisted alongside the batch.
- `POST /api/parse-999` — parse an inbound 999 ACK and persist it.
- `GET /api/acks` — list ACKs.
- `GET /api/acks/{id}` — ACK detail, including the regenerated
  `raw_999_text`.
- `POST /api/eligibility/request` — build a 270 from JSON.
- `POST /api/eligibility/parse-271` — ingest a 271 and return parsed
  coverage benefits.

The UI gains a new **Acks** page (sidebar entry) that lists persisted
ACKs and lets you download the regenerated 999 text.

### SP4 endpoints (claim drawer)

- `GET /api/claims/{claim_id}` — full claim context for the drawer
  payload: header fields, diagnoses, service lines, parties
  (billing provider / subscriber / payer), validation result, raw
  X12 segments, matched remittance summary (or `null`), and the
  claim's state history (ordered newest-first). 404 with a structured
  `{ "error": "Not found", "detail": "Claim {id} not found" }` body
  when the claim doesn't exist — distinct from a transient fetch
  failure so the UI can render a dedicated not-found state.

### SP5 endpoints (live updates)

- `GET /api/claims/stream` — NDJSON stream of claim events. Emits a
  snapshot (filtered by the same query params as `GET /api/claims`),
  then `snapshot_end`, then live `claim_written` events as they
  arrive, with a 15s idle heartbeat.
- `GET /api/remittances/stream` — same shape for remittances
  (subscribes to `remittance_written`, default sort `-received_date`).
- `GET /api/activity/stream` — same shape for activity events
  (subscribes to `activity_recorded`, default `limit=50`).

### SP6 endpoints (inbox)

- `GET /api/inbox/lanes` — all four lanes (rejected, candidates,
  unmatched, done_today) in a single round-trip, with row-level
  scoring and matched-remit context.
- `POST /api/inbox/candidates/{remit_id}/match` — manual match of a
  candidate remit to one of its scored claims; `409` if the claim
  state moved out from under us.
- `POST /api/inbox/candidates/dismiss` — `{pairs: [{claim_id,
  remit_id}]}`. Session-scoped, no DB persistence.
- `POST /api/inbox/rejected/resubmit` — `{claim_ids: [...]}`. `200`
  with `conflicts` for non-rejected claims in the batch.
- `GET /api/inbox/export.csv?lane=<lane>` — streams CSV of the
  lane's rows.

### SP7 endpoints (per-line reconciliation)

- `GET /api/claims/{id}/line-reconciliation` — dedicated view for
  the ClaimDrawer's Line Reconciliation tab; returns the slim
  per-line projection (`summary` + `lines[]`).
- `GET /api/claims/{id}` — now includes a slim `lineReconciliation[]`
  so the ServiceLinesTable can show paid + adjustments columns
  without a second fetch.
- `GET /api/remittances/{id}` — now includes `serviceLinePayments[]`
  + `claimLevelAdjustments[]`; the CAS panel renders per-line + claim
  level separately.
- `GET /api/inbox/lanes` — `matched_remittance` payload gained
  `matched_lines` + `total_lines` so the MatchedRemitCard can badge
  partial matches.

## License

No license file yet; this is internal-use software. Add a `LICENSE` file
when one is decided.