From d4f6fdd49cd2bdc85c1ca00fa034122cad1dd96a Mon Sep 17 00:00:00 2001 From: Tyler Date: Sun, 21 Jun 2026 00:36:33 -0600 Subject: [PATCH] docs: sync READMEs with SP14 (Payer-Rejected lane) + endpoint inventory --- README.md | 278 +++++++++++++++++++++++++++++++++++++++++++--- backend/README.md | 24 +++- 2 files changed, 279 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index 2614c71..ccdbf04 100644 --- a/README.md +++ b/README.md @@ -130,10 +130,13 @@ backoff schedule on error is `1s → 2s → 4s → 8s → 16s → 30s` capped. ## Inbox -`/inbox` is the working surface. Four lanes, dark by default (Ticker Tape +`/inbox` is the working surface. Five lanes, dark by default (Ticker Tape aesthetic): - **Rejected** — claims whose 999 set-level response was R or E. Re-submit in bulk. +- **Payer-rejected** — claims whose 277CA STC category is A4, A6, or A7 (the payer + accepted the file but denied the claim). Stamped at 277CA ingest time and never + overwritten by a looser later 277CA. - **Candidates** — remits whose CLP-claim-id didn't match exactly; each one shows its top scored claim. One-click manual match or dismiss. - **Unmatched** — claims still waiting for a remit, and remits with no candidates above the threshold. - **Done today** — terminal state transitions in the last 24 hours. @@ -148,7 +151,7 @@ parses and rejects claims, the inbox reflects the new | Method | Path | Notes | | ------ | --------------------------------------------- | ---------------------------------------------------------------- | -| GET | `/api/inbox/lanes` | All four lanes in one call. | +| GET | `/api/inbox/lanes` | All five lanes in one call. | | POST | `/api/inbox/candidates/{remit_id}/match` | Manual match. `409` if the claim state moved out from under us. | | POST | `/api/inbox/candidates/dismiss` | `{pairs: [{claim_id, remit_id}]}`. Session-scoped. | | POST | `/api/inbox/rejected/resubmit` | `{claim_ids: [...]}`. `200` with `conflicts` for non-rejected. | @@ -255,11 +258,139 @@ drawer surfaces a per-line "no 837 line matched" note. Tiers: **strong** (≥75, full opacity, Match enabled), **weak** (50–74, dimmed), **hidden** (<50, not surfaced). +## Multi-Payer, Multi-NPI & Clearhouse + +The payer and provider identity that used to live as a single hard-coded +`PayerConfig` dict in the backend is now data, not code. Three new tables +plus a YAML file drive the entire configuration: + +- **`providers` table** — one row per billing-provider NPI (Montrose + `1881068062`, Delta `1851446637`, Salida `1467507269`). All three share + the same `TOC, Inc.` legal name, tax ID `721587149`, and taxonomy + `251E00000X`. Outbound 837 files pick the right `BillingProvider` by + NPI; `claim.party.npi` is now a foreign key into `providers`. +- **`payers` table** — one row per payer (`CO_TXIX`, …) with its + receiver identity (NM1*40 / ISA08 / GS03). +- **`payer_configs` join table** — one row per `(payer_id, transaction_type)` + pair. 837P and 835 can carry different `BHT06`, SBR defaults, and + allowed status codes per payer. +- **`clearhouse` single-row config** — dzinesco's identity: TPID + `11525703`, submitter name, MT-clock file-naming block, SFTP block. +- **`config/payers.yaml`** — the on-disk source for everything above, + schema-validated at boot against a Pydantic model. A typo or missing + field fails the boot with a precise error. The original in-code + `PAYER_FACTORIES` dict is kept as a fallback for ad-hoc testing. + +### Config + clearhouse endpoints + +| Method | Path | Notes | +| ------ | --------------------------------------------- | ---------------------------------------------------------------- | +| GET | `/api/clearhouse` | The `clearhouse` singleton (name, TPID, file/SFTP blocks). | +| POST | `/api/clearhouse/submit` | Push a batch of generated 837 files via SFTP (see SFTP section). | +| GET | `/api/config/providers` | All providers. | +| GET | `/api/config/providers/{npi}` | One provider. | +| GET | `/api/config/payers` | All payers. | +| GET | `/api/config/payers/{payer_id}/configs` | All `(payer_id, transaction_type)` configs for one payer. | +| POST | `/api/admin/reload-config` | Re-read `config/payers.yaml` and refresh the in-process cache. | + +## 277CA Claim Acknowledgment + +A 277CA (`005010X214`) is the per-claim acknowledgment CMS and +Colorado Medicaid rely on: the file was syntactically valid *and* each +named claim was accepted, pended, or rejected by the payer at the claim +level. It is distinct from a 999 (file-level) and a TA1 (envelope-level). + +Cyclone ingests 277CA files the same way it ingests 999 / 835 — drop the +file on the Upload page or `POST /api/parse-277ca` — and stamps every +claim whose `STC` category is `A4`, `A6`, or `A7` with a non-null +`payer_rejected_at` + `payer_rejected_reason` + originating 277CA row id. + +| Method | Path | Notes | +| ------ | -------------------------- | ---------------------------------------------------------------- | +| POST | `/api/parse-277ca` | Upload a 277CA, persist the parsed status rows. | +| GET | `/api/277ca-acks` | List 277CA acks (filterable by date / payer). | +| GET | `/api/277ca-acks/{id}` | One 277CA ack with its per-claim status rows + regenerated text. | + +The `payer_rejected` stamp is **monotonic**: a later 277CA with a looser +status set cannot clear a previous rejection. The Payer-Rejected inbox +lane surfaces every claim with a non-null `payer_rejected_at` — it is +distinct from the 999 `Rejected` lane (envelope reject) and they can +both be true for the same claim. + +## Tamper-Evident Audit Log + +The `audit_log` table is the canonical record of every state transition +the system has ever observed — claim lifecycle, reconciliation +decisions, config reloads, SFTP submissions, 277CA rejects. SP11 made +it tamper-evident: every row carries a SHA-256 hash of +`(prev_hash || row_payload)`, forming a chain back to a genesis row. +Any `INSERT`, `UPDATE`, or `DELETE` that breaks the chain is detectable +in a single walk. + +| Method | Path | Notes | +| ------ | --------------------------------- | -------------------------------------------------------------- | +| GET | `/api/admin/audit-log` | Paginated audit log (filterable by event type / actor / date). | +| GET | `/api/admin/audit-log/verify` | Walk the chain; return the first broken link, or `{ok: true}`. | + +`verify_chain` is the integrity check that backs the audit promise — +it is intentionally cheap (one indexed walk) and intentionally +side-effect-free so a scheduler can run it on a cron and alert on any +non-`{ok: true}` result. Chain verification is **not** access-gated +beyond the same `127.0.0.1` bind the rest of the API uses; for a +hostile multi-operator deployment, wrap the route in your reverse proxy. + +## Encryption at Rest + +When the macOS Keychain carries an entry at service `cyclone`, account +`cyclone.db.key`, and the optional `sqlcipher3` Python package is +installed, the SQLite file at `~/.local/share/cyclone/cyclone.db` is +opened with SQLCipher (AES-256). The key is read from the Keychain +once at process start, applied via a SQLAlchemy `connect` event so +every connection — including migrations and tests — gets the same +`PRAGMA key`. The key is never written to disk or to a Python global. + +When the Keychain entry is missing **or** `sqlcipher3` is not +installed, the DB falls back to plain SQLite. The intent is a graceful +default for developers and CI; the production posture is that every +operator has created the Keychain entry on first run. See +[docs/reference/co-medicaid.md §Keychain setup](docs/reference/co-medicaid.md) +for the one-time setup recipe and the HIPAA Security Rule §164.312(a)(2)(iv) +mapping. + +## SFTP Wire-Up (paramiko) + +The `clearhouse.submit` endpoint uses `paramiko` to push a batch of +generated 837 files to the dzinesco SFTP server +(`mft.gainwelltechnologies.com:22`, path +`/CO XIX/PROD/coxix_prod_11525703/FromHPE`). The SFTP credential is +fetched from the macOS Keychain at call time — never read from YAML, +never logged, never written to disk. The wire-up honors the file-naming +template stored in the `clearhouse` config: + +``` +outbound: {tpid}-{tx}-{ts_mt}-1of1.{ext} e.g. 11525703-837P-20260620181814559-1of1.txt +inbound: TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12 +``` + +where `{ts_mt}` is a 17-digit `yyyymmddhhmmssSSS` Mountain Time stamp. +Inbound filenames are routed by `` and `` to the +matching parser (`999`, `TA1`, `271`, `277`, `277CA`, `835`). + +The `SftpClient` interface is the same one the SP9 stub used — the swap +was a one-file change (`sftp_paramiko.py` replacing `sftp_stub.py`). +`paramiko` is an optional dependency; the stub remains the default when +the `paramiko` extras aren't installed so the test suite stays green on +Linux dev boxes. + ## Persistence -Parsed batches, claims, remittances, matches, and activity events are -stored in a SQLite file at `~/.local/share/cyclone/cyclone.db` by -default. The directory is auto-created on first run. +Parsed batches, claims, remittances, matches, 277CA rejections, +hash-chained audit log entries, and SFTP submission history are stored +in a SQLite file at `~/.local/share/cyclone/cyclone.db` by default. +The directory is auto-created on first run. The DB is optionally +encrypted with SQLCipher — see +[Encryption at Rest](#encryption-at-rest) for the Keychain-driven +setup. To use a different location, set `CYCLONE_DB_URL`: @@ -285,11 +416,23 @@ backup API). ├── backend/ │ ├── src/cyclone/ │ │ ├── api.py # FastAPI app, GET + parse routes, /api/{resource}/stream +│ │ ├── api_helpers.py # NDJSON / content-negotiation / live-tail helpers │ │ ├── pubsub.py # in-process EventBus (drop-oldest, per-kind fan-out) -│ │ ├── store.py # InMemoryStore, mappers, publish-on-write +│ │ ├── store.py # CycloneStore, mappers, publish-on-write +│ │ ├── db.py # SQLAlchemy engine, session factory, ORM models +│ │ ├── db_migrate.py # PRAGMA user_version migration runner +│ │ ├── db_crypto.py # optional SQLCipher encryption at rest (SP12) +│ │ ├── audit_log.py # tamper-evident hash-chained audit_log (SP11) +│ │ ├── inbox_lanes.py # rejected / payer_rejected / candidates / unmatched / done_today +│ │ ├── inbox_state.py # 999 envelope reject → claim state transitions +│ │ ├── inbox_state_277ca.py # 277CA STC A4/A6/A7 → payer_rejected stamp (SP10) +│ │ ├── providers.py # multi-NPI provider lookups (SP9) +│ │ ├── payers.py # payer / payer_config lookups (SP9) +│ │ ├── secrets.py # macOS Keychain-backed secret fetcher +│ │ ├── reconcile.py # pure-function 835→claim match + line-level match │ │ ├── __main__.py # `python -m cyclone serve` │ │ ├── cli.py # click CLI -│ │ └── parsers/ # X12 tokenizer, models, validator, writers +│ │ └── parsers/ # X12 tokenizer, models, validator, writers, 277CA, 999, TA1, 270, 271 │ └── tests/ │ ├── fixtures/ # co_medicaid_*.txt, minimal_*.txt │ ├── test_api.py # parse-837/835 round-trip @@ -298,29 +441,71 @@ backup API). │ ├── test_api_streaming.py │ ├── test_api_stream_live.py # 3 live-tail endpoints + disconnect cleanup │ ├── test_pubsub.py # EventBus + subscribe/unsubscribe -│ └── test_api_parse_persists.py +│ ├── test_api_parse_persists.py +│ ├── test_db.py / test_db_crypto.py / test_db_migrate.py +│ ├── test_audit_log.py +│ ├── test_inbox_lanes.py / test_inbox_state.py +│ ├── test_apply_277ca_rejections.py +│ ├── test_sftp_stub.py / test_sftp_paramiko.py +│ └── test_providers_seed.py / test_payer_config_loading.py ├── src/ # React + Vite + TypeScript UI │ ├── components/ │ │ ├── ui/ # Skeleton, EmptyState, ErrorState, FilterChips, Pagination, … │ │ └── TailStatusPill.tsx # live-tail status badge + reconnect button -│ ├── pages/ # Claims, Remittances, Providers, Activity, Upload +│ ├── pages/ # Claims, Remittances, Providers, Acks, Activity, Upload, Inbox, … │ ├── hooks/ # useBatches, useClaims, useRemittances, useProviders, useActivity, useParse │ │ # + useTailStream, useMergedTail (live tail) -│ ├── lib/ # api.ts (6 GET + parse837/parse835/health), format.ts, utils.ts +│ ├── lib/ # api.ts, format.ts, utils.ts │ │ # + tail-stream.ts (NDJSON parser) │ ├── store/ # zustand sample-data + parsed-batches store │ │ # + tail-store.ts (FIFO-capped live tail slices) │ └── types/ # shared TS types +├── config/ +│ └── payers.yaml # YAML-driven payer + clearhouse config (SP9) ├── docs/ -│ ├── reference/ # condensed 837P/835/X12/CO Medicaid notes -│ └── superpowers/plans/ # implementation plan +│ ├── reference/ # condensed 837P/835/X12/CO Medicaid notes (incl. Keychain setup) +│ ├── reviews/ # post-SP completeness reviews +│ ├── superpowers/plans/ # implementation plans +│ └── superpowers/specs/ # design specs (incl. SP9-SP13) ├── tailwind.config.js # shimmer, scan, row-flash keyframes └── package.json ``` ## Roadmap -Sub-projects 2 through 8 are **shipped**. Next up: +Sub-projects 2 through 13 are **shipped**. See the [completeness +review](docs/reviews/2026-06-20-cyclone-completeness-review.md) for +the honest gap analysis against the industry definition of a HIPAA +clearinghouse — the short version is that the local-only, +single-operator, single-payer design contract is honored, and the +items that would be needed to expand that contract (AS2/AS4, SNIP 1–7, +HITRUST, 276/277 status, 278 referrals, COB) are intentionally out of +scope. + +Shipped sub-projects (most recent first): + +- **Sub-project 13 (shipped) — SFTP wire-up.** `paramiko`-backed + `SftpClient` replaces the SP9 stub. The clearhouse.submit endpoint + actually pushes to + `mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`. + SFTP credentials are read from the macOS Keychain at call time. +- **Sub-project 12 (shipped) — Encryption at rest.** Optional + SQLCipher AES-256 encryption of the SQLite file, with the key + fetched from the macOS Keychain. Falls back to plain SQLite when + the Keychain entry is missing or `sqlcipher3` isn't installed. +- **Sub-project 11 (shipped) — Tamper-evident audit log.** Every + `audit_log` row carries a SHA-256 hash chained to the previous row; + a single walk via `GET /api/admin/audit-log/verify` detects any + break. +- **Sub-project 10 (shipped) — 277CA + Payer-Rejected lane.** Inbound + 277CA parser + a new Payer-Rejected inbox lane distinct from the + 999-envelope Rejected lane. The rejection stamp is monotonic. +- **Sub-project 9 (shipped) — Multi-payer, multi-NPI, SFTP stub.** The + in-code `PAYER_FACTORIES` dict is replaced by a `config/payers.yaml` + + 3 new DB tables (`providers`, `payers`, `payer_configs`) + + `clearhouse` singleton. Added a `POST /api/clearhouse/submit` stub + that writes to a local `staging_dir` — swapped for real `paramiko` + in SP13. - **Sub-project 8 (shipped) — Outbound 837P serializer.** Closes the resubmit loop: rejected claims can be regenerated back to an X12 837P @@ -390,8 +575,9 @@ Sub-projects 2 through 8 are **shipped**. Next up: back-off ladder on errors is `1s → 2s → 4s → 8s → 16s → 30s` capped. See the "Live updates" section below for details. - **Sub-project 6 (shipped) — Inbox workflow automation.** - - **Ticker-Tape inbox (`/inbox`):** four lanes ordered by urgency — - **Rejected** (claims whose 999 rejected them), **Candidates** + - **Ticker-Tape inbox (`/inbox`):** five lanes ordered by urgency — + **Rejected** (claims whose 999 rejected them), **Payer-rejected** + (claims whose 277CA denied them — added in SP10), **Candidates** (remits that didn't auto-match a claim), **Unmatched** (claims still waiting for a remit), and **Done today** (terminal transitions in the last 24 hours). The page subscribes to the @@ -428,6 +614,10 @@ Sub-projects 2 through 8 are **shipped**. Next up: - `GET /api/acks` — list ACKs. - `GET /api/acks/{id}` — ACK detail, including the regenerated `raw_999_text`. +- `POST /api/parse-ta1` — parse an inbound TA1 envelope ACK and persist it. +- `GET /api/ta1-acks` — list TA1 acks. +- `GET /api/ta1-acks/{id}` — TA1 ack detail (envelope control segments + + the parser's accept/reject verdict). - `POST /api/eligibility/request` — build a 270 from JSON. - `POST /api/eligibility/parse-271` — ingest a 271 and return parsed coverage benefits. @@ -459,9 +649,9 @@ ACKs and lets you download the regenerated 999 text. ### SP6 endpoints (inbox) -- `GET /api/inbox/lanes` — all four lanes (rejected, candidates, - unmatched, done_today) in a single round-trip, with row-level - scoring and matched-remit context. +- `GET /api/inbox/lanes` — all five lanes (rejected, payer_rejected, + candidates, unmatched, done_today) in a single round-trip, with + row-level scoring and matched-remit context. - `POST /api/inbox/candidates/{remit_id}/match` — manual match of a candidate remit to one of its scored claims; `409` if the claim state moved out from under us. @@ -510,6 +700,58 @@ ACKs and lets you download the regenerated 999 text. Download** button in the Inbox rejected-lane BulkBar (N>1 modal prompt). +### SP9 endpoints (multi-payer, multi-NPI, SFTP stub) + +- `GET /api/clearhouse` — the `clearhouse` singleton (name, TPID, + file-naming block, SFTP block). +- `POST /api/clearhouse/submit` — push a batch of generated 837 files. + The SP9 implementation writes to a local `staging_dir`; the SP13 + swap replaces the write with a real `paramiko` SFTP push without + changing the route shape. +- `GET /api/config/providers` and `GET /api/config/providers/{npi}` — + list / fetch providers from the new `providers` table. +- `GET /api/config/payers` and + `GET /api/config/payers/{payer_id}/configs` — list payers; for a + given payer, return the per-transaction-type `payer_configs` rows. +- `POST /api/admin/reload-config` — re-read `config/payers.yaml` and + refresh the in-process cache without a server restart. + +### SP10 endpoints (277CA + Payer-Rejected lane) + +- `POST /api/parse-277ca` — upload a 277CA file; persist the parsed + `ClaimStatus` rows and stamp the matching claims with + `payer_rejected_at` (monotonic, never overwritten by `NULL`). +- `GET /api/277ca-acks` — list 277CA acks. +- `GET /api/277ca-acks/{id}` — one 277CA ack with its per-claim + `ClaimStatus` rows + regenerated text. +- `GET /api/inbox/lanes` — the response now also carries a + `payer_rejected` lane populated from + `Claim.payer_rejected_at IS NOT NULL`. + +### SP11 endpoints (tamper-evident audit log) + +- `GET /api/admin/audit-log` — paginated audit log. Each row carries + `(id, prev_hash, row_hash, event_type, actor, payload_json, + created_at)` where `row_hash = sha256(prev_hash || canonical_json(payload))`. +- `GET /api/admin/audit-log/verify` — walk the chain in insertion + order; return `{ok: true}` or the first `{id, expected, got}` + mismatch. The walk is O(n) with one indexed lookup per row. + +### SP12 (encryption at rest — no new routes) + +SP12 introduces no API routes. The `cyclone.db.key` Keychain entry + +optional `sqlcipher3` dependency enable AES-256 encryption transparently +on the next connection. See [Encryption at Rest](#encryption-at-rest) +and [docs/reference/co-medicaid.md](docs/reference/co-medicaid.md) for +the one-time setup recipe. + +### SP13 endpoints (paramiko SFTP) + +- `POST /api/clearhouse/submit` — same endpoint as SP9; the + implementation is now a real `paramiko` `SftpClient.write` to + `mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`. + SFTP credentials are fetched from the macOS Keychain at call time. + ## License No license file yet; this is internal-use software. Add a `LICENSE` file diff --git a/backend/README.md b/backend/README.md index 46f9d32..40d7dc0 100644 --- a/backend/README.md +++ b/backend/README.md @@ -78,11 +78,25 @@ python -m cyclone serve ### Endpoints -| Method | Path | Purpose | -| ------ | ---------------- | -------------------------------------------------- | -| GET | `/api/health` | Liveness probe: `{"status": "ok", "version": ...}` | -| POST | `/api/parse-837` | Upload an X12 837P file, get parsed claims back | -| POST | `/api/parse-835` | Upload an X12 835 ERA file, get parsed payouts back | +| Method | Path | Purpose | +| ------ | --------------------- | ------------------------------------------------------------------ | +| GET | `/api/health` | Liveness probe: `{"status": "ok", "version": ...}` | +| POST | `/api/parse-837` | Upload an X12 837P file, get parsed claims back | +| POST | `/api/parse-835` | Upload an X12 835 ERA file, get parsed payouts back | +| POST | `/api/parse-999` | Upload an inbound 999 ACK and persist it | +| POST | `/api/parse-ta1` | Upload an inbound TA1 envelope ACK and persist it | +| POST | `/api/parse-277ca` | Upload a 277CA claim acknowledgment and stamp payer_rejected claims | +| POST | `/api/eligibility/request` | Build a 270 from JSON (subscriber / provider / payer) | +| POST | `/api/eligibility/parse-271` | Ingest a 271 and return structured `coverage_benefits` | +| GET | `/api/clearhouse` | The `clearhouse` singleton (SP9) | +| POST | `/api/clearhouse/submit` | Push a batch of generated 837 files via SFTP (SP9 stub, SP13 real) | +| POST | `/api/admin/reload-config` | Re-read `config/payers.yaml` and refresh the in-process cache | +| GET | `/api/admin/audit-log` | Paginated tamper-evident audit log (SP11) | +| GET | `/api/admin/audit-log/verify` | Walk the audit_log hash chain (SP11) | + +The full surface — claim / remittance / batch / inbox / stream +endpoints, config lookups, and the 270/271 builder — is enumerated in +the root [README](../README.md#multi-payer-multi-npi--clearhouse). `POST /api/parse-837` accepts `multipart/form-data` with a single `file` field. Optional query parameters: