Compare commits
104 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 55f9f8b8ba | |||
| 0156051ca7 | |||
| 542a55fdbe | |||
| 96ed8f5f64 | |||
| 871e5c7119 | |||
| 2bb10f2a39 | |||
| d8d03d631f | |||
| b0956d039c | |||
| b3aec27e52 | |||
| 4c466ca054 | |||
| 90a2055c20 | |||
| c9588f782d | |||
| 2c43a82390 | |||
| a8aef7b9b2 | |||
| b3a608a939 | |||
| f5713640e4 | |||
| 36cac9af78 | |||
| 7d257911d2 | |||
| b5b715d1c9 | |||
| df3e5e401d | |||
| 9e422fdadb | |||
| daf55d9a38 | |||
| ea68ae81f7 | |||
| ad6629c7ab | |||
| afd6cead06 | |||
| 95d04f9991 | |||
| 5fe61fddd3 | |||
| 56e14341c2 | |||
| 23fc85cad2 | |||
| 61e90b64b3 | |||
| e9e0b68746 | |||
| be5824ec31 | |||
| ee1a397bd5 | |||
| 019e09aff6 | |||
| e35e5e03b3 | |||
| 579e5d7438 | |||
| 1978e696d7 | |||
| ce61b36504 | |||
| 26c948e7fc | |||
| 309e6c016a | |||
| 8790e0a439 | |||
| 467f53046e | |||
| cfe9c25d9d | |||
| b0ad0c27b8 | |||
| b0d03bc10e | |||
| 9beab5b4fc | |||
| adb8dcb137 | |||
| a15547fb0e | |||
| 262ad00481 | |||
| cc41c73d2a | |||
| 30aee61956 | |||
| b19f43f448 | |||
| 2dfe213af0 | |||
| 83d4df413c | |||
| bd5530539d | |||
| 53df3f0684 | |||
| 0e130579b4 | |||
| 13dd240328 | |||
| 543422c343 | |||
| 39f578aa5e | |||
| 4d396d40b6 | |||
| fcac090885 | |||
| 45b6c6291a | |||
| a3831213cc | |||
| e946a4f7aa | |||
| 30d13876cd | |||
| ea7acd2e66 | |||
| 3b4f18eef6 | |||
| 69b760e890 | |||
| f8e2f0933e | |||
| 52795adfb1 | |||
| acf3b506ec | |||
| e7098b99b2 | |||
| 7838ed2a4c | |||
| c3ef4f3847 | |||
| cfca1d8975 | |||
| b652b18458 | |||
| 0815a4ad35 | |||
| 7218ec76a8 | |||
| b237b9cfc4 | |||
| c5604fdfa0 | |||
| f21201a22e | |||
| 0436820023 | |||
| 055d95224a | |||
| 77f73cf73e | |||
| 08c0430ee7 | |||
| 3400ed5ec2 | |||
| 0493814489 | |||
| 538fad211b | |||
| 353adfc08b | |||
| d328c0b224 | |||
| 8516b90601 | |||
| 8a251d20a9 | |||
| 01e5ee781e | |||
| 28835e2f1d | |||
| 0ccc396e5e | |||
| c8f5af3ba5 | |||
| fa034b1cd7 | |||
| 9c0aa577fb | |||
| 83a31b6fea | |||
| 07ea7ca1d6 | |||
| 76923a79f5 | |||
| 9ef749c783 | |||
| ad14b56732 |
+11
@@ -34,6 +34,10 @@ ingest/
|
||||
.env
|
||||
.env.local
|
||||
.env.*.local
|
||||
# Edifabric / EdiNation operator-supplied dev key (env file).
|
||||
# Lives at backend/.env.cyclone-edifabric; source it in the shell.
|
||||
.env.cyclone-edifabric
|
||||
backend/.env.cyclone-edifabric
|
||||
|
||||
# Production data (handled by ops, not committed)
|
||||
docs/prodfiles/*/
|
||||
@@ -52,3 +56,10 @@ claims_output/
|
||||
# SP33+ scratch / production-data ingest. Generated artifacts live
|
||||
# here only — the source EDI sits under docs/prodfiles/.
|
||||
ingest/
|
||||
|
||||
# SP41 dev/ scratch dir — generated 837P artifacts from rebill pipeline
|
||||
# runs. Hundreds of thousands of files; never source. The scripts that
|
||||
# build them are tracked (dev/unbilled-july2026/scripts/...) but the
|
||||
# generated x12 artifacts are not.
|
||||
dev/rebills/
|
||||
dev/rebills/2026-*
|
||||
|
||||
@@ -5,23 +5,27 @@ description: "Cyclone FastAPI router conventions (api_routers/, api_helpers.py,
|
||||
|
||||
# cyclone-api-router
|
||||
|
||||
Cyclone splits its FastAPI surface two ways: small resource-group
|
||||
routers live in `backend/src/cyclone/api_routers/<topic>.py` and are
|
||||
mounted bare into `api.py`; the high-traffic streaming and parse
|
||||
endpoints still live as top-level decorators in `api.py` itself. This
|
||||
skill codifies the conventions so additions stay consistent with the
|
||||
four routers already shipped (`acks`, `admin`, `health`, `ta1_acks`).
|
||||
Cyclone splits its FastAPI surface two ways: resource-group routers
|
||||
live in `backend/src/cyclone/api_routers/<topic>.py` and are mounted
|
||||
bare into `api.py`; the parse endpoints (`/api/parse-837`,
|
||||
`/api/parse-835`, plus the SP40 `/api/admin/validate-837` etc.) stay
|
||||
as top-level decorators in `api.py` because they span multiple store
|
||||
modules. This skill codifies the conventions so additions stay
|
||||
consistent with the routers already shipped.
|
||||
|
||||
As of this writing: **4 router modules** under
|
||||
`backend/src/cyclone/api_routers/`, **one shared helpers module** at
|
||||
`backend/src/cyclone/api_helpers.py` (248 lines, NDJSON primitives +
|
||||
content negotiation + `tail_events`), and ~30 routes still inlined
|
||||
in `backend/src/cyclone/api.py`. The next refactor target is the
|
||||
parse endpoints.
|
||||
As of this writing: **22 router modules** under
|
||||
`backend/src/cyclone/api_routers/` (`acks`, `activity`, `admin`,
|
||||
`batches`, `claim_acks`, `claims`, `clearhouse`, `config`,
|
||||
`dashboard`, `eligibility`, `health`, `inbox`, `parse`, `payers`,
|
||||
`providers`, `rebill`, `reconciliation`, `remittances`, `submission`,
|
||||
`ta1_acks`, plus a `_shared.py` for cross-router helpers), **one
|
||||
shared helpers module** at `backend/src/cyclone/api_helpers.py`
|
||||
(NDJSON primitives + content negotiation + `tail_events`), and
|
||||
`api.py` itself is now 378 LOC (down from ~3,145 pre-SP36).
|
||||
|
||||
## Auth gate (SP24)
|
||||
## Auth gate (SP23)
|
||||
|
||||
Every router declared in `backend/src/cyclone/api_routers/` **must** carry `dependencies=[Depends(matrix_gate)]` at the `APIRouter(...)` declaration — not on each individual endpoint. The gate lives at `backend/src/cyclone/auth/deps.py:107` and the role matrix is at `backend/src/cyclone/auth/permissions.py`. The roles are `admin / user / viewer`; `matrix_gate` returns 401 when there's no session and 403 when the role is below the endpoint's required role. When `AUTH_DISABLED` is True (conftest autouse fixture flips it; `CYCLONE_AUTH_DISABLED=1` in prod-by-mistake), the gate short-circuits to a synthetic admin — see the SP24 spec for the threat-model implications. New routers get the gate by default; the auth-aware convention is `router = APIRouter(dependencies=[Depends(matrix_gate)])`.
|
||||
Every router declared in `backend/src/cyclone/api_routers/` **must** carry `dependencies=[Depends(matrix_gate)]` at the `APIRouter(...)` declaration — not on each individual endpoint. The gate lives at `backend/src/cyclone/auth/deps.py:107` and the role matrix is at `backend/src/cyclone/auth/permissions.py`. The roles are `admin / user / viewer`; `matrix_gate` returns 401 when there's no session and 403 when the role is below the endpoint's required role. When `AUTH_DISABLED` is True (conftest autouse fixture flips it; `CYCLONE_AUTH_DISABLED=1` in prod-by-mistake), the gate short-circuits to a synthetic admin — see the SP23 spec for the threat-model implications. New routers get the gate by default; the auth-aware convention is `router = APIRouter(dependencies=[Depends(matrix_gate)])`.
|
||||
|
||||
## When to use
|
||||
|
||||
@@ -45,11 +49,13 @@ Every router declared in `backend/src/cyclone/api_routers/` **must** carry `depe
|
||||
1. **No new top-level routes in `api.py` for resource groups.** Any
|
||||
endpoint grouped under a resource (`/api/<resource>` and its
|
||||
`/{id}` detail) lives in `backend/src/cyclone/api_routers/<topic>.py`
|
||||
as an `APIRouter`. The streaming list endpoints (`/api/claims/stream`,
|
||||
`/api/remittances/stream`, `/api/activity/stream`) and the parse
|
||||
endpoints (`/api/parse-*`) currently stay in `api.py` because they
|
||||
span multiple store modules — don't move them unless you're also
|
||||
restructuring the store split.
|
||||
as an `APIRouter`. The streaming list endpoints
|
||||
(`/api/claims/stream`, `/api/remittances/stream`,
|
||||
`/api/activity/stream`, `/api/acks/stream`, `/api/ta1-acks/stream`)
|
||||
were split into their per-resource routers as part of SP36; only
|
||||
the cross-resource parse endpoints (`/api/parse-*` and the SP40
|
||||
`validate-837`) stay in `api.py` because they span multiple store
|
||||
modules.
|
||||
2. **Reuse `api_helpers.py`.** NDJSON primitives (`ndjson_line`,
|
||||
`ndjson_stream_list`, `ndjson_stream_837`, `ndjson_stream_835`),
|
||||
content negotiation (`client_wants_json`, `wants_ndjson`), the
|
||||
@@ -80,8 +86,9 @@ Every router declared in `backend/src/cyclone/api_routers/` **must** carry `depe
|
||||
`StreamingResponse(media_type="application/x-ndjson")` and feed it
|
||||
either `ndjson_stream_list(items, total, returned, has_more)` (for
|
||||
list pages) or `tail_events(request, bus, kinds)` (for live-tail
|
||||
pages). See `acks.py:62-66` for the list-stream skeleton and
|
||||
`api.py:1357-1401` for the full live-tail pattern (snapshot →
|
||||
pages). See `api_routers/acks.py:62-66` for the list-stream
|
||||
skeleton and `api_routers/claims.py` (the `/api/claims/stream`
|
||||
handler) for the full live-tail pattern (snapshot →
|
||||
`snapshot_end` → subscription → heartbeats). See `cyclone-tail`
|
||||
for the wire format.
|
||||
6. **Tests.** Every new endpoint gets a `test_api_<topic>_<verb>.py`
|
||||
@@ -170,12 +177,17 @@ just after middleware registration and just before the first
|
||||
# Resource-group routers. Each module owns its own APIRouter and is
|
||||
# registered below. New resources go in `cyclone.api_routers.<name>`
|
||||
# and are wired in here.
|
||||
from cyclone.api_routers import acks, admin, health, ta1_acks # noqa: E402
|
||||
from cyclone.api_routers import ( # noqa: E402
|
||||
acks, activity, admin, batches, claim_acks, claims, clearhouse,
|
||||
config, dashboard, eligibility, health, inbox, parse, payers,
|
||||
providers, rebill, reconciliation, remittances, submission, ta1_acks,
|
||||
)
|
||||
|
||||
app.include_router(health.router)
|
||||
app.include_router(acks.router)
|
||||
app.include_router(ta1_acks.router)
|
||||
app.include_router(activity.router)
|
||||
app.include_router(admin.router)
|
||||
# ... 18 more include_router calls in api.py ...
|
||||
```
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
@@ -5,7 +5,17 @@ description: "Cyclone CLI subcommand conventions (cli.py — Click group + subco
|
||||
|
||||
# cyclone-cli
|
||||
|
||||
The operator-facing CLI is a **Click** group at `cli.py:45` (`@click.group()` for `main`), mounted as the `cyclone` console script in `pyproject.toml:54` (`cyclone = "cyclone.cli:main"`). Seven subcommands ship today: `parse-837`, `parse-835`, `validate-npi`, `validate-tax-id`, plus the `backup` group (`init-passphrase`, `create`, `list`, `verify`, `restore`, `prune`, `status`). `serve` lives separately in `__main__.py:19` (dispatches `uvicorn cyclone.api:app`).
|
||||
The operator-facing CLI is a **Click** group at `cli.py:45` (`@click.group()` for `main`), mounted as the `cyclone` console script in `pyproject.toml:54` (`cyclone = "cyclone.cli:main"`). The CLI has grown well past the original seven subcommands; the full inventory is:
|
||||
|
||||
- **Parser / validator commands**: `parse-837`, `parse-835`, `validate-npi`, `validate-tax-id`, `validate-837` (SP40, Edifabric fail-closed pre-upload gate).
|
||||
- **DB plumbing**: `seed` (deterministic sample claims), `backfill-rendering-npi`, `backfill-999-rejections` (SP33).
|
||||
- **Submission flow**: `submit-batch` (SP37, canonical parse → DB-write → SFTP-upload), `resubmit-rejected-claims` (SP33), `recover-ingest` (re-parse a local path), `pull-inbound` (scheduler trigger).
|
||||
- **Rebill + reissue**: `rebill-from-835` (SP41), `reissue-claims` (SP24).
|
||||
- **Ack orphans + resubmissions**: `ack-orphans status / reconcile`, `resubmissions status`.
|
||||
- **User management** (`users` group): `create`, `list`, `disable`, `reset-password`, `set-role`.
|
||||
- **Backup group** (`backup`): `init-passphrase`, `create`, `list`, `verify`, `restore`, `prune`, `status`.
|
||||
|
||||
`serve` lives separately in `__main__.py:19` (dispatches `uvicorn cyclone.api:app`).
|
||||
|
||||
## When to use
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ description: "Cyclone EDI parser/validator conventions (837P/835/999/270/271/277
|
||||
|
||||
Cyclone parses seven X12 EDI transaction types (837P, 835, 999, 270, 271, 277CA, TA1) into typed Pydantic models, then runs per-claim / per-batch validator rules that surface as R-coded `ValidationIssue` records. This skill codifies the conventions so new parsers and rules stay consistent with the seven that already exist.
|
||||
|
||||
As of this writing: **7 parser modules** under `backend/src/cyclone/parsers/parse_<edi>.py`, **~25 per-claim rules** numbered `R010`–`R100` and `R200`–`R210` in `validator.py`, plus a parallel set of **835-specific rules** prefixed `R835_*` in `validator_835.py`. The next increment is **SP22**.
|
||||
As of this writing: **7 parser modules** under `backend/src/cyclone/parsers/parse_<edi>.py` (837P, 835, 999, TA1, 270, 271, 277CA), **~25 per-claim rules** numbered `R010`–`R100` and `R200`–`R210` in `validator.py`, plus a parallel set of **835-specific rules** prefixed `R835_*` in `validator_835.py`. The most recently shipped increment touching this surface is **SP41** (in-window rebill pipeline); the next free increment after this SP42 doc-pass is **SP43**.
|
||||
|
||||
## When to use
|
||||
|
||||
|
||||
@@ -5,9 +5,9 @@ description: "Cyclone React page conventions (TanStack Query, use<X> hook, drawe
|
||||
|
||||
# cyclone-frontend-page
|
||||
|
||||
Cyclone pages live at `src/pages/<Name>.tsx`: a `use<X>` hook in `src/hooks/use<X>.ts` does the fetching (and optionally the live-tail subscription), `<Layout>` + `<PageHeader>` + `<Sidebar>` (`src/components/`) provide the app shell, and any right-side detail (claim, remittance) lives in `src/components/<DrawerName>/` whose open/close state is mirrored to the URL via `useDrawerUrlState`. This skill codifies the conventions so additions stay consistent with the eleven pages already shipped.
|
||||
Cyclone pages live at `src/pages/<Name>.tsx`: a `use<X>` hook in `src/hooks/use<X>.ts` does the fetching (and optionally the live-tail subscription), `<Layout>` + `<PageHeader>` + `<Sidebar>` (`src/components/`) provide the app shell, and any right-side detail (claim, remittance) lives in `src/components/<DrawerName>/` whose open/close state is mirrored to the URL via `useDrawerUrlState`. This skill codifies the conventions so additions stay consistent with the twelve pages already shipped (Login added in SP23).
|
||||
|
||||
As of this writing: **11 pages** under `src/pages/` (9 of 11 with a `*.test.tsx` sibling — Dashboard, Upload, and BatchDiff are not yet covered; be the first when you refactor them), **~30 hooks** under `src/hooks/`, and **2 drawer modules** at `src/components/{ClaimDrawer,RemitDrawer}/`. The next increment is **SP22**.
|
||||
As of this writing: `**12 pages** under `src/pages/` (10 of 12 with a `*.test.tsx` sibling — Dashboard and BatchDiff are not yet covered; be the first when you refactor them; the Login page does not need a sibling), **~30 hooks** under `src/hooks/`, and **4 drawer modules** at `src/components/{ClaimDrawer,RemitDrawer,ProviderDrawer,AckDrawer}/`. The most recently shipped increment is **SP41** (in-window rebill pipeline); the next free increment after this SP42 doc-pass is **SP43**.
|
||||
|
||||
## When to use
|
||||
|
||||
|
||||
@@ -10,15 +10,11 @@ a spec, a plan, an implementation branch, and a single atomic merge commit
|
||||
into `main`. This skill encodes the conventions so every increment follows
|
||||
the same shape and the commit history stays auditable.
|
||||
|
||||
As of this writing: **17 specs** in `docs/superpowers/specs/`, **13 plans**
|
||||
in `docs/superpowers/plans/`, and SP numbers used through **SP22**. **SP23**
|
||||
is the Ubuntu + Docker + RBAC product fork (awaiting user decision);
|
||||
**SP24** is the auth-posture alignment (docs-only). The next free increment
|
||||
is **SP25** after SP24 lands.
|
||||
As of this writing: SP numbers used through **SP41** (the in-window rebill pipeline); **SP42** is this doc-pass. **SP23** shipped the Ubuntu + Docker + RBAC + auth LAN-bind product fork; **SP24** is the reissue-claims + auth-docs alignment; SPs **25–40** cover the resubmissions, `claim_acks`, rendering/service NPI, transaction-set control numbers, additional live-tail streams, dashboards, and rebill prerequisites shipped between SP24 and SP41. The next free increment after SP42 is **SP43**.
|
||||
|
||||
## Auth-aware spec template (SP24)
|
||||
## Auth-aware spec template (post-SP23)
|
||||
|
||||
The threat-model section in the canonical SP-N spec template (`## 1. Scope`, second-to-last bullet) used to read "no second party to authenticate; no second host to harden against." **That phrasing is stale as of 2026-06-23** — the auth work landed in `main` and every backend endpoint requires login. New specs should instead state the auth boundary explicitly: "the auth boundary is HTTP (login required, bcrypt + HttpOnly session cookie); file-system threats remain the local-only threat model (SQLCipher at rest, macOS Keychain). SP23 changes the threat model to LAN-bound remote operator." Reference: [`docs/superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md`](../../../docs/superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md).
|
||||
The threat-model section in the canonical SP-N spec template (`## 1. Scope`, second-to-last bullet) used to read "no second party to authenticate; no second host to harden against." **That phrasing is stale since SP23 landed (2026-06-23)** — every backend endpoint now requires login (`cyclone.auth.*`, `Depends(matrix_gate)` on every `APIRouter`), the React app ships a `Login` page, RBAC roles `admin / user / viewer` gate individual endpoints, and the host still binds `0.0.0.0:8000` (reachability controlled by the host firewall / compose port publishing). The dev/test escape hatch is `CYCLONE_AUTH_DISABLED=1`, which logs a WARNING at boot. New specs should instead state the auth boundary explicitly: "the auth boundary is HTTP (login required, bcrypt + HttpOnly session cookie); file-system threats remain the LAN-only threat model (SQLCipher at rest, macOS Keychain). SP23 expanded the threat model to a remote operator on the LAN." Reference: [`docs/superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md`](../../../docs/superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md).
|
||||
|
||||
## When to use
|
||||
|
||||
|
||||
@@ -6,17 +6,9 @@ description: "Cyclone store write-paths, the pubsub event contract (claim_writte
|
||||
# cyclone-store
|
||||
|
||||
Cyclone persists every parsed X12 batch through one facade,
|
||||
`CycloneStore` (`backend/src/cyclone/store.py:882`). Every write
|
||||
inserts the row AND publishes a pubsub event on the in-process
|
||||
`EventBus` (`backend/src/cyclone/pubsub.py:20`) so live-tail pages
|
||||
see new rows the moment they land. The event contract is the seam
|
||||
between persistence and streaming — a wrong event name silently
|
||||
goes stale.
|
||||
`CycloneStore` (now exposed via `backend/src/cyclone/store/__init__.py` after the SP21 split landed; the public import surface `from cyclone.store import …` is unchanged). Every write inserts the row AND publishes a pubsub event on the in-process `EventBus` (`backend/src/cyclone/pubsub.py:20`) so live-tail pages see new rows the moment they land. The event contract is the seam between persistence and streaming — a wrong event name silently goes stale.
|
||||
|
||||
As of this writing: the store is a **single 2412-line module** and
|
||||
the SP21 split into `backend/src/cyclone/store/` is in progress.
|
||||
Three event kinds: `claim_written`, `remittance_written`,
|
||||
`activity_recorded`. Next: **SP22**.
|
||||
As of this writing: the store is a **16-module subpackage** under `backend/src/cyclone/store/` (SP21 split landed). Event kinds in use today: `claim_written`, `remittance_written`, `activity_recorded`, plus the SP-acks-tail additions `ack_written` and `ta1_ack_written` for the `/api/acks/stream` and `/api/ta1-acks/stream` endpoints. Next: **SP43** (after the SP42 doc-pass merges).
|
||||
|
||||
## When to use
|
||||
|
||||
@@ -45,12 +37,11 @@ Three event kinds: `claim_written`, `remittance_written`,
|
||||
2. **Every write publishes an event.** The event name matches the
|
||||
entity: `claim_written`, `remittance_written`,
|
||||
`activity_recorded` (the trailing `_recorded` signals a
|
||||
non-canonical row — activity events are derived, not first-class;
|
||||
current names at `store.py:1072,1081,1096`). New entities get
|
||||
`<entity>_written`; activity-style side rows get
|
||||
`<entity>_recorded`. Publish is **best-effort** — failures are
|
||||
logged but never roll back the persisted batch
|
||||
(`store.py:1097-1098`).
|
||||
non-canonical row — activity events are derived, not first-class),
|
||||
plus `ack_written` and `ta1_ack_written` for the SP-acks-tail
|
||||
streams. New entities get `<entity>_written`; activity-style side
|
||||
rows get `<entity>_recorded`. Publish is **best-effort** —
|
||||
failures are logged but never roll back the persisted batch.
|
||||
3. **Snapshot shape.** Each entity has a `to_ui_<entity>` serializer
|
||||
(plain Python function returning a dict) — currently
|
||||
`to_ui_claim`, `to_ui_remittance`, `to_ui_claim_from_orm`,
|
||||
@@ -61,13 +52,15 @@ Three event kinds: `claim_written`, `remittance_written`,
|
||||
that row (`store.py:1052-1054`).
|
||||
4. **SP21 boundaries.** Post-split, each domain lives in its own
|
||||
module under `backend/src/cyclone/store/`: `__init__.py` (facade
|
||||
+ `CycloneStore` class), `exceptions.py`, `records.py`,
|
||||
`orm_builders.py`, `ui.py`, `write.py`, `batches.py`,
|
||||
`claim_detail.py`, `acks.py`, `backups.py`, `inbox.py`,
|
||||
`providers.py`. Cross-module writes go through `CycloneStore`
|
||||
facade methods, not direct module access. The facade re-exports
|
||||
every name callers currently import from `cyclone.store`. Full
|
||||
list: `docs/superpowers/plans/2026-06-21-cyclone-store-split.md:25-37`.
|
||||
+ `CycloneStore` class), `acks.py`, `backfill.py`, `backups.py`,
|
||||
`batches.py`, `claim_acks.py`, `claim_detail.py`, `exceptions.py`,
|
||||
`inbox.py`, `kpis.py`, `orm_builders.py`, `providers.py`,
|
||||
`records.py`, `resubmissions.py`, `submission_dedup.py`, `ui.py`,
|
||||
`write.py` — 16 modules total. Cross-module writes go through
|
||||
`CycloneStore` facade methods, not direct module access. The
|
||||
facade re-exports every name callers currently import from
|
||||
`cyclone.store`. Full split plan:
|
||||
`docs/superpowers/plans/2026-06-21-cyclone-store-split.md`.
|
||||
5. **No business logic in route handlers.** A route handler validates
|
||||
input, calls `store.<method>(...)`, passes the `event_bus`,
|
||||
returns the serialized result. Reconciliation, idempotency
|
||||
@@ -77,7 +70,7 @@ Three event kinds: `claim_written`, `remittance_written`,
|
||||
|
||||
### A `CycloneStore.add` write — publishes events from inserted rows
|
||||
|
||||
Taken from `backend/src/cyclone/store.py:898-1107`. The method opens
|
||||
Taken from `backend/src/cyclone/store/write.py` (post-SP21). The method opens
|
||||
a session, inserts rows, then runs a sync `_publish_events_sync`
|
||||
after commit so subscribers can immediately re-fetch consistent data.
|
||||
|
||||
@@ -118,12 +111,13 @@ FastAPI handlers to await.
|
||||
|
||||
### Backend `/api/<resource>/stream` endpoint — subscribes to the event
|
||||
|
||||
Taken from `backend/src/cyclone/api.py:1380-1401`. Two phases — eager
|
||||
Streaming endpoints live in `backend/src/cyclone/api_routers/<topic>.py`
|
||||
post-SP36 (e.g. `api_routers/claims.py`). Two phases — eager
|
||||
snapshot, then live subscription — wrapped in `StreamingResponse`
|
||||
with `media_type="application/x-ndjson"`.
|
||||
|
||||
```python
|
||||
@app.get("/api/claims/stream")
|
||||
@router.get("/api/claims/stream")
|
||||
async def claims_stream(request: Request, ...) -> StreamingResponse:
|
||||
bus: EventBus = request.app.state.event_bus
|
||||
|
||||
@@ -139,6 +133,10 @@ async def claims_stream(request: Request, ...) -> StreamingResponse:
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
```
|
||||
|
||||
The same shape is repeated in `api_routers/remittances.py`,
|
||||
`api_routers/activity.py`, `api_routers/acks.py`, and
|
||||
`api_routers/ta1_acks.py` (five streaming endpoints total).
|
||||
|
||||
`_ndjson_line` and `_tail_events` live in
|
||||
`backend/src/cyclone/api_helpers.py`. The `["remittance_written"]`
|
||||
and `["activity_recorded"]` subscriptions are at `api.py:1891,2002`.
|
||||
@@ -172,11 +170,11 @@ def test_publishes_claim_written_event(client: TestClient) -> None:
|
||||
equivalent `get_*` facade method).
|
||||
- **Don't introduce a new event name without updating the subscriber
|
||||
list.** Today every `<entity>_written` event has exactly one
|
||||
consumer — the matching `/api/<resource>/stream` endpoint,
|
||||
currently in `backend/src/cyclone/api.py` (claims at `:1398`,
|
||||
remittances at `:1891`, activity at `:2002`). Any future split
|
||||
into `backend/src/cyclone/api_routers/` must wire the same
|
||||
subscription.
|
||||
consumer — the matching `/api/<resource>/stream` endpoint, now
|
||||
split across `api_routers/{claims,remittances,activity,acks,ta1_acks}.py`
|
||||
(post-SP36 split). Any new event must wire its subscription in
|
||||
the matching router and add the resource name to the hook
|
||||
triplet in the corresponding page.
|
||||
- **Don't merge a write method with its event publication into
|
||||
separate places.** `_publish_events_sync` lives next to `add` in
|
||||
`store.py:1042-1107` so reviewers see both halves of the contract
|
||||
|
||||
@@ -5,14 +5,14 @@ description: "Cyclone live-tail streaming wire format and the useTailStream / us
|
||||
|
||||
# cyclone-tail
|
||||
|
||||
Cyclone keeps the Claims, Remittances, and Activity pages live without
|
||||
Cyclone keeps the Claims, Remittances, Activity, Acks, and Ta1Acks pages live without
|
||||
polling: every store write publishes an internal EventBus event, the
|
||||
page opens a `GET /api/<resource>/stream` HTTP/1.1 chunked-NDJSON
|
||||
connection, and new rows land in the table the moment they hit the
|
||||
database. This skill codifies the wire format, the hook triplet, and
|
||||
the backoff/stall machinery so additions stay consistent with the
|
||||
three streaming pages already shipped (`Claims`, `Remittances`,
|
||||
`ActivityLog`).
|
||||
five streaming pages already shipped (`Claims`, `Remittances`,
|
||||
`ActivityLog`, `Acks`, `Ta1Acks`).
|
||||
|
||||
## When to use
|
||||
|
||||
@@ -42,8 +42,8 @@ three streaming pages already shipped (`Claims`, `Remittances`,
|
||||
`item_dropped` (`{"id": "..."}` queue-overflow notice), `error`
|
||||
(`{"message": "..."}` promoted to a thrown error by the hook).
|
||||
Defined at `src/lib/tail-stream.ts:22-44`; emitted by
|
||||
`backend/src/cyclone/api.py:1357-2006`. See
|
||||
`references/wire-format.md`.
|
||||
the streaming routers in `backend/src/cyclone/api_routers/{claims,remittances,activity,acks,ta1_acks}.py`
|
||||
(post-SP36 split). See `references/wire-format.md`.
|
||||
2. **Hook triplet.** Streaming pages compose three pieces:
|
||||
`useTailStream(resource)` (`src/hooks/useTailStream.ts:80` — opens
|
||||
the stream, drives the backoff/stall state machine, dispatches
|
||||
@@ -52,8 +52,9 @@ three streaming pages already shipped (`Claims`, `Remittances`,
|
||||
(`src/hooks/useMergedTail.ts:25` — returns
|
||||
`baseItems + tailSlice` dedup'd against `baseItems`), and a
|
||||
per-resource initial-fetch hook (`useClaims`, `useRemittances`,
|
||||
`useActivity`). The page wires all three — see
|
||||
`src/pages/Claims.tsx:87-89`.
|
||||
`useActivity`, `useAcks`, `useTa1Acks`). The page wires all three —
|
||||
see `src/pages/Claims.tsx:87-89`. The five resources share the
|
||||
same triplet; only the resource string differs.
|
||||
3. **Stall threshold.** 30 seconds of total silence — heartbeats
|
||||
included — flips status to `stalled` and surfaces the
|
||||
`↻ Reconnect` button on `<TailStatusPill>`. Constant:
|
||||
@@ -67,8 +68,8 @@ three streaming pages already shipped (`Claims`, `Remittances`,
|
||||
`<TailStatusPill>` from `connecting` to `live` and to reset the
|
||||
reconnect backoff counter (`useTailStream.ts:174-180`).
|
||||
5. **Content-Type.** Stream endpoints respond with
|
||||
`media_type="application/x-ndjson"` — see
|
||||
`backend/src/cyclone/api.py:1401,1894,2005`. Frontend sets
|
||||
`media_type="application/x-ndjson"` — set in each
|
||||
`api_routers/<resource>.py` stream handler. Frontend sets
|
||||
`Accept: application/x-ndjson` at `src/lib/tail-stream.ts:67-70`.
|
||||
Never `application/json` for a stream endpoint.
|
||||
6. **Backoff.** Transient errors retry with `1s → 2s → 4s → 8s → 16s
|
||||
@@ -108,7 +109,7 @@ export function ClaimsPage() {
|
||||
|
||||
### Backend `/api/foo/stream` endpoint
|
||||
|
||||
Pattern from `backend/src/cyclone/api.py:1357-1401`. Register BEFORE
|
||||
Pattern from `backend/src/cyclone/api_routers/claims.py`. Register BEFORE
|
||||
`/api/foo/{foo_id}` so the literal `stream` segment doesn't match as
|
||||
an id. Two phases — eager snapshot, then live subscription — wrapped
|
||||
in `StreamingResponse` with `media_type="application/x-ndjson"`.
|
||||
@@ -160,7 +161,7 @@ takes `status` + `lastEventAt` from `useTailStream` plus
|
||||
- **Don't change the wire format on one endpoint without updating
|
||||
the others.** The parser (`src/lib/tail-stream.ts`) and the hook
|
||||
dispatch switch (`useTailStream.ts:173-195`) are shared across all
|
||||
three live streams. Adding a new event type means updating
|
||||
five live streams. Adding a new event type means updating
|
||||
`TailEvent`, `KNOWN_TYPES`, the dispatch `case`, the `armStall`
|
||||
re-arm list, and the backend emitter — in that order.
|
||||
- **Don't emit `item` events before `snapshot_end`.** The hook uses
|
||||
@@ -168,7 +169,7 @@ takes `status` + `lastEventAt` from `useTailStream` plus
|
||||
`connecting` to `live` and to reset the reconnect backoff counter.
|
||||
Emitting `item`s first lands rows in `useTailStore` but the UI
|
||||
still reads "Connecting" — operators see a flash of stale state.
|
||||
Emit snapshot, then `snapshot_end`, then live (`api.py:1386-1401`).
|
||||
Emit snapshot, then `snapshot_end`, then live.
|
||||
- **Don't change the 30s stall threshold without updating the README
|
||||
"Status pill" table.** The constant
|
||||
(`STALL_TIMEOUT_MS = 30_000` at `useTailStream.ts:53`) is the
|
||||
@@ -176,7 +177,7 @@ takes `status` + `lastEventAt` from `useTailStream` plus
|
||||
edit at `README.md:118`.
|
||||
- **Don't add `useTailStream` calls from `useFoo.ts` hooks.**
|
||||
Streaming connections belong on the page (`src/pages/Claims.tsx`,
|
||||
`Remittances.tsx`, `ActivityLog.tsx`). Hoisting into `useFoo`
|
||||
`Remittances.tsx`, `ActivityLog.tsx`, `Acks.tsx`, `Ta1Acks.tsx`). Hoisting into `useFoo`
|
||||
couples the lifecycle to whoever mounts it and breaks
|
||||
one-resource-one-page ownership.
|
||||
|
||||
@@ -185,16 +186,16 @@ takes `status` + `lastEventAt` from `useTailStream` plus
|
||||
- **`cyclone-frontend-page`** — page components live in `src/pages/`;
|
||||
load when adding or refactoring a streaming page to confirm the
|
||||
route + `PageHeader` + table conventions.
|
||||
- **`cyclone-api-router`** — endpoint conventions (`api_routers/` for
|
||||
resource-group routers, `api.py` for the live-tail endpoints); load
|
||||
when adding or changing an HTTP endpoint that surfaces streamed or
|
||||
paginated data.
|
||||
- **`cyclone-api-router`** — endpoint conventions (`api_routers/`
|
||||
hosts the streaming endpoints post-SP36, alongside the other
|
||||
resource-group routers); load when adding or changing an HTTP
|
||||
endpoint that surfaces streamed or paginated data.
|
||||
- **`cyclone-store`** — write-path conventions in `store.py` and the
|
||||
pubsub event contract (`claim_written`, `remittance_written`,
|
||||
`activity_recorded`); load when adding a new entity whose writes
|
||||
should fan out to a stream endpoint.
|
||||
- **`cyclone-tests`** — frontend `*.test.tsx` siblings cover
|
||||
`useTailStream`, `useMergedTail`, `TailStatusPill`; backend
|
||||
`test_api_stream_live.py` covers the three live-tail endpoints;
|
||||
load when the increment changes wire-format behavior or adds a
|
||||
streaming hook.
|
||||
`test_api_stream_live.py` covers the five live-tail endpoints
|
||||
(claims, remittances, activity, acks, ta1-acks); load when the
|
||||
increment changes wire-format behavior or adds a streaming hook.
|
||||
@@ -5,9 +5,9 @@ description: "Cyclone pytest + vitest fixture patterns, prodfiles layout, backen
|
||||
|
||||
# cyclone-tests
|
||||
|
||||
The Cyclone test suite is split two ways: **backend pytest** (89 test files under `backend/tests/`, 13 flat fixtures in `backend/tests/fixtures/`, one autouse `conftest.py` that resets the DB per-test) and **frontend vitest** (59 `*.test.ts(x)` siblings across `src/`, two rendering styles — `@testing-library/react` and a custom `createRoot`+`Probe` shim). This skill codifies the conventions so additions stay consistent with what's already there.
|
||||
The Cyclone test suite is split two ways: **backend pytest** (~173 test files under `backend/tests/`, ~21 flat fixtures in `backend/tests/fixtures/`, one autouse `conftest.py` that resets the DB per-test) and **frontend vitest** (85 `*.test.ts(x)` siblings across `src/`, two rendering styles — `@testing-library/react` and a custom `createRoot`+`Probe` shim). This skill codifies the conventions so additions stay consistent with what's already there.
|
||||
|
||||
As of this writing: **89 backend test files**, **13 flat backend fixtures**, **59 frontend `*.test.ts(x)` siblings**, and **23 prodfiles samples** across `docs/prodfiles/{837p-from-axiscare,835fromco,FromHPE,claims}/`. The next increment is **SP24** (SP23 is the Ubuntu+Docker fork, awaiting user decision).
|
||||
As of this writing: **~173 backend test files**, **~21 flat backend fixtures** (the original 13 plus additions for `reissue-claims`, `validate-837`, `submit-batch`, and SP41 rebill fixtures), **85 frontend `*.test.ts(x)` siblings**, and a slimmed-down `docs/prodfiles/references/` archive (the bulk of the prodfiles were moved out of the repo; remaining sample is `good-claim-com-2026-04-16.837`). Tests that previously reached into `docs/prodfiles/claims/` and `docs/prodfiles/FromHPE/` now skip when those dirs are absent (`tests/test_prodfiles_smoke.py`). The most recently shipped increment is **SP41** (in-window rebill pipeline); the next free increment is **SP43** after this SP42 doc-pass merges.
|
||||
|
||||
## Auth flag (SP24)
|
||||
|
||||
|
||||
@@ -94,7 +94,7 @@ Cyclone ships 8 skills under `.superpowers/skills/`. They auto-load by descripti
|
||||
|
||||
## The SP-N increment flow
|
||||
|
||||
Every feature ships as a numbered **SP-N increment**: spec → plan → implementation branch → single atomic merge into `main`. As of the last backfill, SP numbers are used through **SP22**; **SP23** is reserved for the Ubuntu + Docker + RBAC product fork (`docs/superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md`, awaiting user decision); the next free increment is **SP24**. Read `cyclone-spec` before starting a new one. Non-negotiable shape:
|
||||
Every feature ships as a numbered **SP-N increment**: spec → plan → implementation branch → single atomic merge into `main`. SP numbers are used through **SP41** (in-window rebill pipeline, merged `309e6c0`); the next free increment is **SP42**. SP23 (Ubuntu + Docker + auth + RBAC + LAN-bind product fork) shipped via `merge: SP23 Ubuntu Docker Deployment into main` (`07a7ecb`). Read `cyclone-spec` before starting a new one. Non-negotiable shape:
|
||||
|
||||
- **Branch:** `sp<N>-<short-kebab-topic>` (e.g. `sp22-line-reconciliation`).
|
||||
- **Spec path:** `docs/superpowers/specs/YYYY-MM-DD-cyclone-<topic>-design.md`, header `Status: Draft, awaiting user sign-off`, sections `Scope / Decisions / …`. Specs contain zero code blocks.
|
||||
@@ -116,6 +116,8 @@ Endpoints (all accept the same query params as their non-streaming counterparts;
|
||||
| GET | `/api/claims/stream` | `claim_written` | `-submission_date` |
|
||||
| GET | `/api/remittances/stream` | `remittance_written` | `-received_date` |
|
||||
| GET | `/api/activity/stream` | `activity_recorded` | `-timestamp` (limit 50) |
|
||||
| GET | `/api/acks/stream` | `ack_received` | `-received_date` |
|
||||
| GET | `/api/ta1-acks/stream` | `ta1_ack_received` | `-received_date` |
|
||||
|
||||
Wire format: one JSON object per line, `{"type": ..., "data": ...}`. The first batch is the **snapshot** of currently-known rows, then `snapshot_end` with the count, then the **live** events. Known types: `item`, `snapshot_end`, `heartbeat` (keeps the connection alive on idle — clients flip to `stalled` after 30s of total silence), `item_dropped` (rare), `error`.
|
||||
|
||||
@@ -125,9 +127,9 @@ Frontend triplet for any live page: `use<X>(params)` (initial fetch) + `useTailS
|
||||
|
||||
## Backend at a glance
|
||||
|
||||
`backend/src/cyclone/` is a single namespace. The two largest files are `api.py` (~3,548 LOC, the only large file) and `store.py` (~2,423 LOC, the `CycloneStore` facade — SP21 is in flight to split it into a `cyclone/store/` subpackage; the public API stays unchanged). Subpackages: `api_routers/` (acks, admin, health, ta1_acks), `clearhouse/` (Clearhouse + SftpClient), `edi/` (filenames), `parsers/` (X12 transaction parsers + models + validators + serializers), `submission/` (SP37 — canonical `submit_file` helper shared by `cyclone submit-batch` CLI + `POST /api/submit-batch` HTTP endpoint; owns parse → DB-write → SFTP-upload → audit per file), `workflow/` (placeholder for future sub-project 6).
|
||||
`backend/src/cyclone/` is a single namespace. After SP21 (store split, `4360ef7`) and SP36 (api-routers split, `f005494`), the largest files now live under `cyclone/api_routers/` (22 routers) and `cyclone/store/` (the SP21-split facade in 16 modules). `api.py` is a ~377-line shell that wires the FastAPI app, mounts the auth + admin + api_routers routers, and re-exports a few backward-compat shims for tests; `store.py` no longer exists. Subpackages: `api_routers/` (SP36 — 22 routers: acks, activity, admin, batches, claim_acks, claims, clearhouse, config, dashboard, eligibility, health, inbox, parse, payers, providers, rebill, reconciliation, remittances, submission, ta1_acks, plus `_shared.py`), `auth/` (SP24 — bcrypt + HttpOnly session cookies + RBAC `matrix_gate`; 10 modules), `clearhouse/` (Clearhouse + SftpClient), `edi/` (filenames), `handlers/` (SP27 — per-file-type inbound parse handlers), `parsers/` (X12 transaction parsers + models + validators + serializers), `rebill/` (SP41 — in-window rebill pipeline), `reissue/` (SP24 — offline 837P reissue workflow), `store/` (SP21 — split `CycloneStore` facade), `submission/` (SP37 — canonical `submit_file` helper shared by `cyclone submit-batch` CLI + `POST /api/submit-batch` HTTP endpoint; owns parse → DB-write → SFTP-upload → audit per file). The legacy `workflow/` directory is dead bytecode — see REQUIREMENTS.md R-27 (Open).
|
||||
|
||||
The store is the only read/write surface for the database; every mutating endpoint goes through it. All persistence flows through SQLAlchemy sessions via `db.SessionLocal()()`. SQLAlchemy ORM models live in `db.py`; 12 SQL migrations under `migrations/` (0001_initial through 0012_backups) are walked in order by `db_migrate.py`.
|
||||
The store is the only read/write surface for the database; every mutating endpoint goes through it. All persistence flows through SQLAlchemy sessions via `db.SessionLocal()()`. SQLAlchemy ORM models live in `db.py`; 23 SQL migrations under `migrations/` (0001_initial through 0023_visits) are walked in order by `db_migrate.py`.
|
||||
|
||||
The parser pipeline is a 5-stage `tokenize → segmentize → model → validate → write_to_store` flow used for every inbound X12 type. Per-transaction parsers: `parse_837.py`, `parse_835.py`, `parse_999.py`, `parse_ta1.py`, `parse_270.py`, `parse_271.py`, `parse_277ca.py`. Each has a matching Pydantic model module (`models.py`, `models_835.py`, …) and a writer (`writer.py` / `writer_835.py`). The 837P serializer (`serialize_837.py`) is the byte-faithful outbound counterpart used by both single-claim download (`/api/claims/{id}/serialize-837`) and the bulk rejected-resubmit bundle (`/api/inbox/rejected/resubmit?download=true`).
|
||||
|
||||
@@ -167,7 +169,7 @@ The canonical way to write 999/TA1/277CA/835 rows into the DB is `Scheduler.proc
|
||||
|
||||
## Frontend at a glance
|
||||
|
||||
`src/` is React 18 + TypeScript + Vite. Routes register in `src/App.tsx` (11 pages, all under a `<Layout>` route wrapper). Pages are pure renderers — every page pairs with a `use<X>` data hook in `src/hooks/` and renders a `<PageHeader>` + a table/list/KPI grid. Drawers (`ClaimDrawer/`, `RemitDrawer/`, plus the new `ProviderDrawer/` and `AckDrawer/`) are mounted by the page and their open/close state is mirrored to the URL via `useDrawerUrlState` so deep-links round-trip. Drill-stack navigation is provided by `<DrillStackProvider>` in `src/components/drill/`.
|
||||
`src/` is React 18 + TypeScript + Vite. Routes register in `src/App.tsx` (12 pages — Dashboard, Claims, Remittances, Providers, ActivityLog, Upload, Reconciliation, Inbox, Acks, Batches, BatchDiff, Login — all under a `<Layout>` route wrapper, with `Login` outside the auth gate). Pages are pure renderers — every page pairs with a `use<X>` data hook in `src/hooks/` and renders a `<PageHeader>` + a table/list/KPI grid. Drawers (`ClaimDrawer/`, `RemitDrawer/`, `ProviderDrawer/`, `AckDrawer/`) are mounted by the page and their open/close state is mirrored to the URL via `useDrawerUrlState` so deep-links round-trip. Drill-stack navigation is provided by `<DrillStackProvider>` in `src/components/drill/`.
|
||||
|
||||
State split: **server state** in TanStack Query (`@tanstack/react-query`); **ephemeral client state** in Zustand (`useTailStore` for live-tail append, plus the drill stack). The live-tail store is FIFO-capped at `TAIL_CAP = 10_000` per slice (`src/store/tail-store.ts:28`); `claims` and `remittances` are key-by-id with first-write-wins dedup, `activity` is an append-only array.
|
||||
|
||||
@@ -181,9 +183,6 @@ Path alias `@/` → `src/`. Configured in `vite.config.ts`, `vitest.config.ts`,
|
||||
# Parser
|
||||
python -m cyclone.cli parse-837 path/to/837p.txt --output-dir ./claims --payer co_medicaid [--strict] [--include-raw-segments]
|
||||
python -m cyclone.cli parse-835 path/to/835.txt --output-dir ./remits
|
||||
python -m cyclone.cli parse-999 inbound_999.txt
|
||||
python -m cyclone.cli parse-ta1 inbound_ta1.txt
|
||||
python -m cyclone.cli parse-277ca inbound_277ca.txt
|
||||
|
||||
# Validators
|
||||
python -m cyclone.cli validate-npi 1234567893
|
||||
|
||||
@@ -2,12 +2,17 @@
|
||||
|
||||
A self-hosted EDI claims management suite for a single billing office.
|
||||
Parses 837P professional claims and 835 ERA remittances (X12 005010X222A1
|
||||
and 005010X221A1), with a local-only FastAPI backend and a React UI for
|
||||
browsing, filtering, and inspecting the parsed data.
|
||||
and 005010X221A1), with a FastAPI backend and a React UI for browsing,
|
||||
filtering, and inspecting the parsed data.
|
||||
|
||||
Local-only on purpose: binds to `127.0.0.1`, no auth, no internet exposure.
|
||||
Built for one operator, one machine, one trading partner (Colorado
|
||||
Medicaid, currently).
|
||||
LAN-only by design: always binds to `0.0.0.0:8000` and requires login
|
||||
(bcrypt + HttpOnly session cookie; first admin bootstrapped from
|
||||
`CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD`). Reachability is
|
||||
controlled by the host firewall / compose port publishing, not by the
|
||||
bind address. Built for one operator, one machine, one trading partner
|
||||
(Colorado Medicaid, currently). Do not expose the published ports to
|
||||
the public internet. See [SP23](docs/superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md)
|
||||
for the LAN-bind / Docker / RBAC product fork that shipped 2026-06-23.
|
||||
|
||||
## Install
|
||||
|
||||
@@ -184,13 +189,15 @@ happening — clients flip to `stalled` after 30s of total silence
|
||||
|
||||
### Endpoints
|
||||
|
||||
| Method | Path | Subscribes to | Default sort |
|
||||
| ------ | -------------------------- | ------------------- | ------------------- |
|
||||
| GET | `/api/claims/stream` | `claim_written` | `-submission_date` |
|
||||
| GET | `/api/remittances/stream` | `remittance_written`| `-received_date` |
|
||||
| GET | `/api/activity/stream` | `activity_recorded` | `-timestamp` (limit 50) |
|
||||
| Method | Path | Subscribes to | Default sort |
|
||||
| ------ | -------------------------- | -------------------- | ------------------- |
|
||||
| GET | `/api/claims/stream` | `claim_written` | `-submission_date` |
|
||||
| GET | `/api/remittances/stream` | `remittance_written` | `-received_date` |
|
||||
| GET | `/api/activity/stream` | `activity_recorded` | `-timestamp` (limit 50) |
|
||||
| GET | `/api/acks/stream` | `ack_received` | `-received_date` |
|
||||
| GET | `/api/ta1-acks/stream` | `ta1_ack_received` | `-received_date` |
|
||||
|
||||
All three accept the same query params as their non-streaming
|
||||
All five accept the same query params as their non-streaming
|
||||
counterparts (`status`, `payer`, `date_from`, …) so a frontend can
|
||||
swap a one-shot fetch for a tail with no URL surgery. Responses
|
||||
are `Content-Type: application/x-ndjson`.
|
||||
@@ -424,8 +431,8 @@ in a single walk.
|
||||
`verify_chain` is the integrity check that backs the audit promise —
|
||||
it is intentionally cheap (one indexed walk) and intentionally
|
||||
side-effect-free so a scheduler can run it on a cron and alert on any
|
||||
non-`{ok: true}` result. Chain verification is **not** access-gated
|
||||
beyond the same `127.0.0.1` bind the rest of the API uses; for a
|
||||
non-`{ok: true}` result. Chain verification is gated by the standard
|
||||
`matrix_gate` dependency (admin role for the `/verify` subpath); for a
|
||||
hostile multi-operator deployment, wrap the route in your reverse proxy.
|
||||
|
||||
## Encryption at Rest
|
||||
@@ -743,12 +750,11 @@ backup API).
|
||||
.
|
||||
├── backend/
|
||||
│ ├── src/cyclone/
|
||||
│ │ ├── api.py # FastAPI app, GET + parse routes, /api/{resource}/stream
|
||||
│ │ ├── api.py # FastAPI app shell (~377 LOC post-SP36; routes live in api_routers/)
|
||||
│ │ ├── api_helpers.py # NDJSON / content-negotiation / live-tail helpers
|
||||
│ │ ├── pubsub.py # in-process EventBus (drop-oldest, per-kind fan-out)
|
||||
│ │ ├── store.py # CycloneStore, mappers, publish-on-write
|
||||
│ │ ├── db.py # SQLAlchemy engine, session factory, ORM models
|
||||
│ │ ├── db_migrate.py # PRAGMA user_version migration runner
|
||||
│ │ ├── db_migrate.py # PRAGMA user_version migration runner (23 migrations)
|
||||
│ │ ├── db_crypto.py # optional SQLCipher encryption at rest (SP12)
|
||||
│ │ ├── audit_log.py # tamper-evident hash-chained audit_log (SP11)
|
||||
│ │ ├── inbox_lanes.py # rejected / payer_rejected / candidates / unmatched / done_today
|
||||
@@ -756,37 +762,45 @@ backup API).
|
||||
│ │ ├── inbox_state_277ca.py # 277CA STC A4/A6/A7 → payer_rejected stamp (SP10)
|
||||
│ │ ├── providers.py # multi-NPI provider lookups (SP9)
|
||||
│ │ ├── payers.py # payer / payer_config lookups (SP9)
|
||||
│ │ ├── secrets.py # macOS Keychain-backed secret fetcher
|
||||
│ │ ├── reconcile.py # pure-function 835→claim match + line-level match
|
||||
│ │ ├── __main__.py # `python -m cyclone serve`
|
||||
│ │ ├── cli.py # click CLI
|
||||
│ │ └── parsers/ # X12 tokenizer, models, validator, writers, 277CA, 999, TA1, 270, 271
|
||||
│ └── tests/
|
||||
│ ├── fixtures/ # co_medicaid_*.txt, minimal_*.txt
|
||||
│ ├── test_api.py # parse-837/835 round-trip
|
||||
│ ├── test_api_gets.py # 6 GET endpoints
|
||||
│ ├── test_store.py # store + mappers + iterators
|
||||
│ ├── test_api_streaming.py
|
||||
│ ├── test_api_stream_live.py # 3 live-tail endpoints + disconnect cleanup
|
||||
│ ├── test_pubsub.py # EventBus + subscribe/unsubscribe
|
||||
│ ├── test_api_parse_persists.py
|
||||
│ ├── test_db.py / test_db_crypto.py / test_db_migrate.py
|
||||
│ ├── test_audit_log.py
|
||||
│ ├── test_inbox_lanes.py / test_inbox_state.py
|
||||
│ ├── test_apply_277ca_rejections.py
|
||||
│ ├── test_sftp_stub.py / test_sftp_paramiko.py
|
||||
│ └── test_providers_seed.py / test_payer_config_loading.py
|
||||
├── src/ # React + Vite + TypeScript UI
|
||||
│ │ ├── secrets.py # macOS Keychain-backed secret fetcher (4-tier: _FILE > env > Keychain > None)
|
||||
│ │ ├── reconcile.py # pure-function 835→claim match + line-level match (SP31 pcn-exact)
|
||||
│ │ ├── scheduler.py # inbound MFT polling scheduler (SP16, SP27 per-op timeout)
|
||||
│ │ ├── edifabric.py # Edifabric /v2/x12/{read,validate} HTTP client (SP40)
|
||||
│ │ ├── backup.py / backup_service.py / backup_scheduler.py # SP17 encrypted backups
|
||||
│ │ ├── __main__.py # `python -m cyclone serve` (auth bootstrap then dispatch)
|
||||
│ │ ├── cli.py # click CLI (15+ subcommands; see cyclone-cli skill)
|
||||
│ │ ├── api_routers/ # SP36 — 22 FastAPI routers + _shared.py helpers
|
||||
│ │ ├── auth/ # SP24 — bcrypt + sessions + matrix_gate + admin users + rate_limit
|
||||
│ │ ├── clearhouse/ # SftpClient + InboundFile + SftpStat
|
||||
│ │ ├── edi/ # filename regex/builder helpers
|
||||
│ │ ├── handlers/ # SP27 — per-file-type inbound parse handlers
|
||||
│ │ ├── parsers/ # X12 tokenizer, models, validator, writers, 277CA, 999, TA1, 270, 271
|
||||
│ │ ├── rebill/ # SP41 — in-window rebill pipeline (13 modules)
|
||||
│ │ ├── reissue/ # SP24 — offline 837P reissue workflow
|
||||
│ │ ├── store/ # SP21 — split CycloneStore facade (16 modules)
|
||||
│ │ └── submission/ # SP37 — canonical submit_file helper + recover + bulk_ingest
|
||||
│ └── tests/ # 178 pytest files (post-SP41)
|
||||
│ ├── fixtures/ # 22 entries (837P / 835 / 999 / TA1 / 277CA samples + visits CSV)
|
||||
│ ├── test_api_*.py # FastAPI integration via TestClient
|
||||
│ ├── test_*.py # pure-unit
|
||||
│ └── ...
|
||||
├── src/ # React + Vite + TypeScript UI (12 pages, 84 test files)
|
||||
│ ├── components/
|
||||
│ │ ├── ui/ # Skeleton, EmptyState, ErrorState, FilterChips, Pagination, …
|
||||
│ │ ├── ClaimDrawer/ # 12 sibling tests
|
||||
│ │ ├── RemitDrawer/ # 8 sibling tests
|
||||
│ │ ├── ProviderDrawer/ # right-anchored side panel
|
||||
│ │ ├── AckDrawer/ # right-anchored side panel (SP28 ack-claim auto-link)
|
||||
│ │ ├── inbox/ # 5-lane triage surface (SP14)
|
||||
│ │ ├── drill/ # DrillStackProvider + DrillableCell + PeekModal
|
||||
│ │ └── TailStatusPill.tsx # live-tail status badge + reconnect button
|
||||
│ ├── pages/ # Claims, Remittances, Providers, Acks, Activity, Upload, Inbox, …
|
||||
│ ├── pages/ # Dashboard, Claims, Remittances, Providers, ActivityLog,
|
||||
│ │ # Upload, Reconciliation, Inbox, Acks, Batches, BatchDiff, Login
|
||||
│ ├── hooks/ # useBatches, useClaims, useRemittances, useProviders, useActivity, useParse
|
||||
│ │ # + useTailStream, useMergedTail (live tail)
|
||||
│ ├── lib/ # api.ts, format.ts, utils.ts
|
||||
│ │ # + tail-stream.ts (NDJSON parser)
|
||||
│ ├── store/ # zustand sample-data + parsed-batches store
|
||||
│ │ # + tail-store.ts (FIFO-capped live tail slices)
|
||||
│ │ # + useTailStream, useMergedTail (live tail triplet)
|
||||
│ ├── auth/ # AuthProvider, RoleGate, api client (SP24)
|
||||
│ ├── lib/ # api.ts, format.ts, utils.ts, tail-stream.ts (NDJSON parser), inbox-api.ts
|
||||
│ ├── store/ # zustand useAppStore (sample data) + useTailStore (FIFO-capped live tail slices)
|
||||
│ └── types/ # shared TS types
|
||||
├── config/
|
||||
│ └── payers.yaml # YAML-driven payer + clearhouse config (SP9)
|
||||
@@ -807,11 +821,15 @@ backup API).
|
||||
> 3. The per-SP spec under [`docs/superpowers/specs/`](docs/superpowers/specs/) for whatever you're touching.
|
||||
> 4. The per-SP plan under [`docs/superpowers/plans/`](docs/superpowers/plans/) if you're implementing.
|
||||
|
||||
Sub-projects 2 through 19 are **shipped**. See the [completeness
|
||||
Sub-projects 2 through 41 are **shipped** (last merged: SP41 in-window
|
||||
rebill pipeline, `merge: SP41 inwindow-rebill-pipeline into main` at
|
||||
`309e6c0`). See the [completeness
|
||||
review](docs/reviews/2026-06-20-cyclone-completeness-review.md) for
|
||||
the honest gap analysis against the industry definition of a HIPAA
|
||||
clearinghouse — the short version is that the local-only,
|
||||
single-operator, single-payer design contract is honored, and the
|
||||
clearinghouse — the short version is that the LAN-only,
|
||||
single-operator, single-payer design contract is honored (auth per
|
||||
SP23, encryption at rest via SQLCipher per SP12, RBAC per SP23, audit
|
||||
log per SP11), and the
|
||||
items that would be needed to expand that contract (AS2/AS4, SNIP 1–7,
|
||||
HITRUST, 276/277 status, 278 referrals, COB) are intentionally out of
|
||||
scope.
|
||||
|
||||
@@ -26,6 +26,7 @@ from cyclone.api_routers import (
|
||||
parse,
|
||||
payers,
|
||||
providers,
|
||||
rebill,
|
||||
reconciliation,
|
||||
remittances,
|
||||
submission,
|
||||
@@ -48,6 +49,7 @@ routers: list[APIRouter] = [
|
||||
parse.router, # gated
|
||||
payers.router, # gated
|
||||
providers.router, # gated
|
||||
rebill.router, # gated (SP41)
|
||||
reconciliation.router, # gated
|
||||
remittances.router, # gated
|
||||
submission.router, # gated
|
||||
|
||||
@@ -28,9 +28,10 @@ import threading
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from fastapi import APIRouter, Depends, File, HTTPException, Query, UploadFile
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from cyclone import db
|
||||
from cyclone import db, edifabric
|
||||
from cyclone.audit_log import verify_chain
|
||||
from cyclone.auth.deps import matrix_gate
|
||||
from cyclone.clearhouse import InboundFile
|
||||
@@ -826,3 +827,56 @@ def reload_config():
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
return {"ok": True, "loaded": len(configs), "errors": []}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP40: POST /api/admin/validate-837
|
||||
#
|
||||
# Admin-only Edifabric validation probe: upload an 837P file via
|
||||
# multipart, hit /v2/x12/read → /v2/x12/validate, return the raw
|
||||
# OperationResult JSON (Status, Details, LastIndex). Mirrors the
|
||||
# `cyclone validate-837 <file>` CLI behavior — the wire shape comes
|
||||
# straight through so callers don't have to translate between the two.
|
||||
#
|
||||
# HTTP status codes:
|
||||
# 200 — OperationResult returned (Status may be success / warning /
|
||||
# error; the caller decides whether the file is acceptable).
|
||||
# 400 — uploaded file is empty / undecodable (defense-in-depth, same
|
||||
# shape as /api/parse-837).
|
||||
# 502 — Edifabric upstream 4xx/5xx — caller can surface the body.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.post("/api/admin/validate-837")
|
||||
async def validate_837_endpoint(
|
||||
file: UploadFile = File(...),
|
||||
) -> Any:
|
||||
"""Validate an uploaded 837P file via Edifabric /v2/x12/validate.
|
||||
|
||||
Multipart upload (``file=...``); the file is read into bytes and
|
||||
passed to :func:`cyclone.edifabric.validate_edi`. The API key is
|
||||
resolved server-side from ``cyclone.secrets.get_secret('edifabric.api_key')``
|
||||
so the key never leaves the backend.
|
||||
"""
|
||||
raw = await file.read()
|
||||
if not raw:
|
||||
return JSONResponse(
|
||||
status_code=400,
|
||||
content={"error": "Empty file", "detail": "Uploaded file contained no bytes."},
|
||||
)
|
||||
try:
|
||||
result = edifabric.validate_edi(raw)
|
||||
except edifabric.EdifabricError as exc:
|
||||
# status_code=0 is a client-side config problem (missing API
|
||||
# key); 4xx/5xx upstream become 502. Surface the Edifabric body
|
||||
# verbatim so the operator can see what went wrong.
|
||||
http_status = 502 if exc.status_code else 503
|
||||
return JSONResponse(
|
||||
status_code=http_status,
|
||||
content={
|
||||
"error": "Edifabric validation failed",
|
||||
"upstream_status": exc.status_code,
|
||||
"upstream_body": exc.body,
|
||||
},
|
||||
)
|
||||
return JSONResponse(content=result)
|
||||
|
||||
@@ -223,6 +223,59 @@ def _compact_ack_links_for_claim(claim_id: str) -> list[dict]:
|
||||
return out
|
||||
|
||||
|
||||
def _serialize_kwargs_for_claim(claim_obj) -> dict:
|
||||
"""SP40: build the ``serialize_837`` kwargs from the live clearhouse
|
||||
+ per-payer ``PayerConfigORM`` config. Same as the bulk export in
|
||||
``batches.py:_serialize_kwargs`` so single-claim download mirrors
|
||||
production byte-faithfulness.
|
||||
|
||||
Without this, ``serialize_837(claim_obj)`` falls back to
|
||||
placeholder strings (CYCLONE/RECEIVER/CUSTOMER SERVICE/8005550100)
|
||||
which the operator rightly rejected as not production-ready.
|
||||
Returns an empty dict only when the clearhouse / payer config is
|
||||
not seeded — the serializer's own SP40 fallback then fires.
|
||||
"""
|
||||
out: dict = {}
|
||||
try:
|
||||
ch = store.get_clearhouse()
|
||||
except Exception:
|
||||
ch = None
|
||||
if ch is not None:
|
||||
out["sender_id"] = ch.tpid
|
||||
out["submitter_name"] = ch.submitter_name
|
||||
out["submitter_contact_name"] = ch.submitter_contact_name
|
||||
out["submitter_contact_email"] = ch.submitter_contact_email
|
||||
if getattr(ch, "submitter_contact_phone", None):
|
||||
out["submitter_contact_phone"] = ch.submitter_contact_phone
|
||||
|
||||
# Per-payer receiver + SBR-09 from the 837P PayerConfigORM row.
|
||||
pid = (getattr(claim_obj.payer, "id", "") or "").strip()
|
||||
if pid:
|
||||
try:
|
||||
from cyclone.db import PayerConfigORM, Payer as PayerORM
|
||||
with db.SessionLocal()() as ss:
|
||||
pcfg = ss.get(PayerConfigORM, (pid, "837P"))
|
||||
payer_row = ss.get(PayerORM, pid)
|
||||
if pcfg is not None and isinstance(pcfg.config_json, dict):
|
||||
cfg = pcfg.config_json
|
||||
if cfg.get("receiver_name"):
|
||||
out["receiver_name"] = cfg["receiver_name"]
|
||||
if cfg.get("receiver_id"):
|
||||
out["receiver_id"] = cfg["receiver_id"]
|
||||
if cfg.get("sbr09_default"):
|
||||
out["claim_filing_indicator_code"] = cfg["sbr09_default"]
|
||||
if payer_row is not None:
|
||||
if "receiver_name" not in out and payer_row.receiver_name:
|
||||
out["receiver_name"] = payer_row.receiver_name
|
||||
if "receiver_id" not in out and payer_row.receiver_id:
|
||||
out["receiver_id"] = payer_row.receiver_id
|
||||
except Exception:
|
||||
# Fall through; the serializer's own SP40 default
|
||||
# (MC for SBR-09) covers the most common case.
|
||||
pass
|
||||
return out
|
||||
|
||||
|
||||
@router.get("/api/claims/{claim_id}/serialize-837")
|
||||
def serialize_claim_as_837(claim_id: str):
|
||||
"""Return the claim as a regenerated X12 837P file (SP8).
|
||||
@@ -231,6 +284,15 @@ def serialize_claim_as_837(claim_id: str):
|
||||
outbound serializer. Returns 404 if the claim doesn't exist, 422 if
|
||||
the stored payload has no parseable ClaimOutput (data integrity
|
||||
issue, not a transient failure).
|
||||
|
||||
SP40: threads the clearhouse submitter + CO_TXIX PayerConfig
|
||||
receiver kwargs through to ``serialize_837`` so the regenerated
|
||||
file has real ``NM1*41`` (Dzinesco / TPID ``11525703``),
|
||||
``PER*IC*Tyler Martinez*EM*tyler@dzinesco.com``, ``NM1*40``
|
||||
(HCPF) and ``SBR*P*18*******MC`` segments — not the
|
||||
``CYCLONE/RECEIVER/CUSTOMER SERVICE/8005550100`` placeholders the
|
||||
bare-args call site would emit. Mirrors the regen-script path in
|
||||
``dev/unbilled-july2026/scripts/regen_corrected_files.py``.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(Claim, claim_id)
|
||||
@@ -258,8 +320,13 @@ def serialize_claim_as_837(claim_id: str):
|
||||
status_code=422,
|
||||
)
|
||||
|
||||
# SP40: build the same kwargs the bulk export uses, so the
|
||||
# single-claim download mirrors production byte-faithfulness (real
|
||||
# submitter, real contact, real receiver, real SBR-09).
|
||||
serialize_kwargs = _serialize_kwargs_for_claim(claim_obj)
|
||||
|
||||
try:
|
||||
text = serialize_837(claim_obj)
|
||||
text = serialize_837(claim_obj, **serialize_kwargs)
|
||||
except SerializeError837 as exc:
|
||||
return JSONResponse(
|
||||
{"error": "Unprocessable", "detail": str(exc)},
|
||||
|
||||
@@ -0,0 +1,221 @@
|
||||
"""SP41 — rebill admin endpoints.
|
||||
|
||||
POST /api/admin/rebill-from-835
|
||||
body: {"window": "YYYY-MM-DD..YYYY-MM-DD",
|
||||
"override_filing": bool,
|
||||
"visits_csv_path": str (optional),
|
||||
"ingest_dir": str (optional),
|
||||
"out_dir": str (optional)}
|
||||
Returns: {"summary_path": str, "counts": {...},
|
||||
"pipeline_a_files": [...], "pipeline_b_files": [...]}
|
||||
|
||||
GET /api/admin/rebill-from-835/status
|
||||
Returns: {"recent_runs": [{"as_of": ..., "summary_path": ...,
|
||||
"counts": {...}}, ...]}
|
||||
|
||||
Status-code contract (per the cyclone-api-router / cyclone-cli skills):
|
||||
- 200: completed run (POST) or tally returned (GET).
|
||||
- 401: not authenticated (matrix_gate).
|
||||
- 403: authenticated but not authorized for /api/admin/*.
|
||||
- 422: window is malformed or Pydantic body validation failed.
|
||||
|
||||
The POST handler delegates to ``cyclone.rebill.run.run_rebill`` (the same
|
||||
orchestrator the ``cyclone rebill-from-835`` CLI uses). The GET handler
|
||||
is a filesystem scan under ``dev/rebills/*/summary.csv`` — no DB table
|
||||
for "recent runs" exists today, and Task 12 deliberately doesn't add
|
||||
one (per its design notes). Sorted by directory mtime descending;
|
||||
truncated to the most recent 5.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import logging
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
|
||||
from cyclone.auth.deps import matrix_gate
|
||||
from cyclone.rebill.run import run_rebill
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/api/admin/rebill-from-835",
|
||||
tags=["rebill"],
|
||||
dependencies=[Depends(matrix_gate)],
|
||||
)
|
||||
|
||||
|
||||
# Filesystem base for /status scans. Module-level so tests can monkeypatch
|
||||
# it to a tmp_path-relative dir without chdir-ing the whole test process.
|
||||
REBILLS_DIR: Path = Path("dev/rebills")
|
||||
|
||||
# How many recent runs /status surfaces. 5 matches the CLI's default; the
|
||||
# body schema below mirrors it as a query parameter so callers can ask for
|
||||
# more (or fewer) when they want.
|
||||
DEFAULT_RECENT_LIMIT = 5
|
||||
|
||||
|
||||
class RebillRequest(BaseModel):
|
||||
"""Body schema for ``POST /api/admin/rebill-from-835``.
|
||||
|
||||
``window`` is parsed as ``YYYY-MM-DD..YYYY-MM-DD`` (inclusive both
|
||||
ends); the validator rejects anything else with a 422. The other
|
||||
path fields are optional — ``run_rebill`` falls back to its own
|
||||
defaults (``data/source/apr-jun27.csv`` for visits, ``ingest/`` for
|
||||
835s, ``dev/rebills/<today>/`` for output) when unset.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(populate_by_name=True)
|
||||
|
||||
window: str = Field(
|
||||
default="2026-01-01..2026-06-27",
|
||||
description="DOS window as YYYY-MM-DD..YYYY-MM-DD (inclusive both ends).",
|
||||
)
|
||||
override_filing: bool = Field(
|
||||
default=False,
|
||||
description="Relax the 120-day timely-filing gate for past-window visits.",
|
||||
)
|
||||
visits_csv_path: str | None = Field(
|
||||
default=None,
|
||||
description="Path to the AxisCare visits CSV; defaults to "
|
||||
"data/source/apr-jun27.csv when unset.",
|
||||
)
|
||||
ingest_dir: str | None = Field(
|
||||
default=None,
|
||||
description="Directory containing *.835 / *.x12 835 files; defaults to ./ingest.",
|
||||
)
|
||||
out_dir: str | None = Field(
|
||||
default=None,
|
||||
description="Output directory for summary.csv + pipeline-a/b files; "
|
||||
"defaults to dev/rebills/<today>/.",
|
||||
)
|
||||
|
||||
@field_validator("window")
|
||||
@classmethod
|
||||
def _validate_window(cls, v: str) -> str:
|
||||
# Manual split (Click can't help here). Two-date range is the
|
||||
# only accepted shape; ``..`` is the giveaway separator so the
|
||||
# input is unambiguous.
|
||||
try:
|
||||
start_str, end_str = v.split("..", 1)
|
||||
except ValueError as exc:
|
||||
raise ValueError(
|
||||
f"window must be 'YYYY-MM-DD..YYYY-MM-DD', got {v!r}"
|
||||
) from exc
|
||||
try:
|
||||
start = date.fromisoformat(start_str)
|
||||
end = date.fromisoformat(end_str)
|
||||
except ValueError as exc:
|
||||
raise ValueError(
|
||||
f"window must be 'YYYY-MM-DD..YYYY-MM-DD', got {v!r}: {exc}"
|
||||
) from exc
|
||||
if start > end:
|
||||
raise ValueError(
|
||||
f"window start {start.isoformat()} is after end {end.isoformat()}"
|
||||
)
|
||||
return v
|
||||
|
||||
|
||||
@router.post("")
|
||||
def post_rebill(req: RebillRequest) -> dict[str, Any]:
|
||||
"""Run the rebill pipeline for the given DOS window.
|
||||
|
||||
Delegates to :func:`cyclone.rebill.run.run_rebill` and returns the
|
||||
resulting ``RunResult`` as a plain JSON dict (the underlying dataclass
|
||||
has no ``to_dict`` method — fields are mapped here so callers don't
|
||||
have to import the dataclass shape).
|
||||
|
||||
The handler does NOT swallow exceptions; the app-level
|
||||
:func:`_unhandled_exception_handler` renders them as a 500 JSON
|
||||
envelope with CORS headers. Per-file failures land in the summary
|
||||
CSV (and the ``counts`` dict) rather than raising here.
|
||||
"""
|
||||
start_str, end_str = req.window.split("..", 1)
|
||||
# Validator already proved these parse cleanly.
|
||||
window_start = date.fromisoformat(start_str)
|
||||
window_end = date.fromisoformat(end_str)
|
||||
|
||||
log.info(
|
||||
"rebill-from-835 starting: window=%s override_filing=%s",
|
||||
req.window, req.override_filing,
|
||||
)
|
||||
result = run_rebill(
|
||||
window_start=window_start,
|
||||
window_end=window_end,
|
||||
override_filing=req.override_filing,
|
||||
visits_csv_path=req.visits_csv_path,
|
||||
ingest_dir=req.ingest_dir,
|
||||
out_dir=req.out_dir,
|
||||
)
|
||||
log.info(
|
||||
"rebill-from-835 done: summary=%s counts=%s",
|
||||
result.summary_path, result.counts,
|
||||
)
|
||||
|
||||
return {
|
||||
"summary_path": str(result.summary_path),
|
||||
"counts": result.counts,
|
||||
"pipeline_a_files": [str(p) for p in result.pipeline_a_files],
|
||||
"pipeline_b_files": [str(p) for p in result.pipeline_b_files],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/status")
|
||||
def get_rebill_status(
|
||||
limit: int = DEFAULT_RECENT_LIMIT,
|
||||
) -> dict[str, Any]:
|
||||
"""Return the most recent rebill runs, newest first.
|
||||
|
||||
Scans :data:`REBILLS_DIR` (``dev/rebills/`` by default) for any
|
||||
subdirectory whose name is a date in YYYY-MM-DD format and that
|
||||
contains a ``summary.csv``. Sorted by directory mtime descending so
|
||||
the most recently-run batch wins ties on equal-named dirs (rare in
|
||||
practice — operators tend to pick a fresh date per run).
|
||||
|
||||
Each entry's ``counts`` dict is a tally of the summary.csv's
|
||||
``disposition`` column (the same per-category counters the operator
|
||||
sees on the CLI ``--status`` view). Missing disposition values
|
||||
surface as ``UNKNOWN`` so a hand-edited CSV can't silently drop a
|
||||
bucket.
|
||||
|
||||
``limit`` defaults to :data:`DEFAULT_RECENT_LIMIT` (5) and is
|
||||
clamped to ``[1, 50]`` to keep the response bounded.
|
||||
"""
|
||||
if limit < 1 or limit > 50:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
|
||||
detail="limit must be between 1 and 50",
|
||||
)
|
||||
|
||||
recent: list[dict[str, Any]] = []
|
||||
rebills_dir = REBILLS_DIR
|
||||
if rebills_dir.exists():
|
||||
candidates = sorted(
|
||||
(
|
||||
d for d in rebills_dir.iterdir()
|
||||
if d.is_dir()
|
||||
and (d / "summary.csv").exists()
|
||||
# Filter to date-named dirs (YYYY-MM-DD) so a stray
|
||||
# ``lost+found`` or tempdir doesn't sneak in.
|
||||
and len(d.name) == 10 and d.name[4] == "-" and d.name[7] == "-"
|
||||
),
|
||||
key=lambda d: d.stat().st_mtime,
|
||||
reverse=True,
|
||||
)[:limit]
|
||||
for d in candidates:
|
||||
summary_path = d / "summary.csv"
|
||||
counts: dict[str, int] = {}
|
||||
with summary_path.open(newline="") as f:
|
||||
for row in csv.DictReader(f):
|
||||
disp = (row.get("disposition", "UNKNOWN") or "").strip() or "UNKNOWN"
|
||||
counts[disp] = counts.get(disp, 0) + 1
|
||||
recent.append({
|
||||
"as_of": d.name,
|
||||
"summary_path": str(summary_path),
|
||||
"counts": counts,
|
||||
})
|
||||
return {"recent_runs": recent}
|
||||
@@ -33,6 +33,7 @@ from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from cyclone.auth.deps import matrix_gate
|
||||
from cyclone.store import store as cycl_store
|
||||
from cyclone.store.exceptions import DuplicateClaimError
|
||||
from cyclone.submission.core import submit_file
|
||||
from cyclone.submission.result import SubmitOutcome, SubmitResult
|
||||
|
||||
@@ -154,6 +155,32 @@ def submit_batch(body: SubmitBatchRequest):
|
||||
# monkey-patch submit_file itself instead of wiring a
|
||||
# factory here.
|
||||
)
|
||||
except DuplicateClaimError as exc:
|
||||
# SP41 Task 9: 409-class domain exception (see the exception
|
||||
# docstring on ``DuplicateClaimError``). The per-file loop
|
||||
# preserves the 200-with-results status-code contract
|
||||
# documented at the top of this module, so we surface the
|
||||
# duplicate as a per-file failure with UNEXPECTED_ERROR
|
||||
# outcome — the structured ``claim_id`` / ``original_submission_at``
|
||||
# attributes ride along in the error string so the operator
|
||||
# can see which CLM01 in the batch tripped the guard. Caught
|
||||
# BEFORE the generic ``Exception`` handler so a future
|
||||
# caller propagating the exception still gets 409-class
|
||||
# treatment at the FastAPI layer.
|
||||
log.warning(
|
||||
"submit-batch duplicate claim on %s: claim_id=%r original=%s",
|
||||
src.name, exc.claim_id, exc.original_submission_at,
|
||||
)
|
||||
r = SubmitResult(
|
||||
file=src.name,
|
||||
outcome=SubmitOutcome.UNEXPECTED_ERROR,
|
||||
error=(
|
||||
f"DuplicateClaimError: claim_id {exc.claim_id!r} was "
|
||||
f"already submitted at "
|
||||
f"{exc.original_submission_at.isoformat()}; "
|
||||
f"within 30-day window"
|
||||
),
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.exception(
|
||||
"submit-batch unexpected error on %s", src.name,
|
||||
|
||||
@@ -68,6 +68,9 @@ PERMISSIONS: dict[tuple[str, str], set[Role]] = {
|
||||
("POST", "/api/admin/db/rotate-key"): ADMIN_ONLY,
|
||||
("POST", "/api/admin/reload-config"): ADMIN_ONLY,
|
||||
("GET", "/api/admin/validate-provider"): ADMIN_ONLY,
|
||||
("POST", "/api/admin/validate-837"): ADMIN_ONLY, # SP40: Edifabric validation probe
|
||||
("POST", "/api/admin/rebill-from-835"): ADMIN_ONLY, # SP41: run the in-window rebill pipeline
|
||||
("GET", "/api/admin/rebill-from-835"): ADMIN_ONLY, # SP41: covers /status (prefix match)
|
||||
|
||||
# Write endpoints (admin + user, no viewer).
|
||||
("POST", "/api/parse-837"): WRITE_ROLES,
|
||||
|
||||
+1004
-9
File diff suppressed because it is too large
Load Diff
@@ -13,7 +13,7 @@ from __future__ import annotations
|
||||
import enum
|
||||
import json
|
||||
import os
|
||||
from datetime import date, datetime
|
||||
from datetime import date, datetime, timezone
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
@@ -30,6 +30,7 @@ from sqlalchemy import (
|
||||
Numeric,
|
||||
String,
|
||||
Text,
|
||||
UniqueConstraint,
|
||||
func,
|
||||
text,
|
||||
)
|
||||
@@ -450,6 +451,45 @@ class ServiceLinePayment(Base):
|
||||
)
|
||||
|
||||
|
||||
class Visit(Base):
|
||||
"""SP41: a single row from the AxisCare visits export (CSV).
|
||||
|
||||
Persisted by ``cyclone.rebill.visits_store.load_visits_csv`` so the
|
||||
in-window rebill pipeline can reconcile visits vs 835 svc rows from
|
||||
the database (previously the spot-check driver read the CSV in-memory,
|
||||
losing the canonical source-of-truth for the visit roster).
|
||||
|
||||
One row per (DOS, member_id, procedure, modifiers). The UNIQUE
|
||||
constraint on those four columns dedupes the natural key — the CSV
|
||||
itself can contain duplicate rows for the same visit (re-exports).
|
||||
"""
|
||||
__tablename__ = "visits"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||
dos: Mapped[date] = mapped_column(Date, nullable=False)
|
||||
member_id: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||
client_name: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
procedure_code: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
modifiers: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
billed_amount: Mapped[Decimal] = mapped_column(Numeric(10, 2), nullable=False)
|
||||
icd10: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
prior_auth: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
payer: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
invoice_number: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
source_file: Mapped[Optional[str]] = mapped_column(String(255), nullable=True)
|
||||
loaded_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, default=lambda: datetime.now(timezone.utc)
|
||||
)
|
||||
|
||||
__table_args__ = (
|
||||
UniqueConstraint("dos", "member_id", "procedure_code", "modifiers",
|
||||
name="uq_visits_natural_key"),
|
||||
Index("ix_visits_dos", "dos"),
|
||||
Index("ix_visits_member_id", "member_id"),
|
||||
Index("ix_visits_procedure_code", "procedure_code"),
|
||||
)
|
||||
|
||||
|
||||
class LineReconciliation(Base):
|
||||
"""One row per matched (or explicitly unmatched) 837 service line within a claim.
|
||||
|
||||
@@ -952,3 +992,49 @@ class Session(Base):
|
||||
user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), index=True)
|
||||
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
|
||||
class Resubmission(Base):
|
||||
"""SP39: audit row recording that a claim was pushed to SFTP as
|
||||
part of a corrected-file resubmission. One row per claim per push.
|
||||
|
||||
Status is derived at read-time by joining against claim_acks (via
|
||||
the existing SP28/31 auto-link) + remittances (via CLP->claim).
|
||||
The corrected-v2 regen preserves the original claim_id from
|
||||
raw_json, so inbound 999 acks auto-link back to the original claim
|
||||
row and the join works without any new matching logic.
|
||||
"""
|
||||
|
||||
__tablename__ = "resubmissions"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||
claim_id: Mapped[str] = mapped_column(String, nullable=False, index=True)
|
||||
batch_id: Mapped[str] = mapped_column(String, nullable=False, index=True)
|
||||
resubmitted_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
source_corrected_path: Mapped[str] = mapped_column(String, nullable=False)
|
||||
interchange_control_number: Mapped[str] = mapped_column(String, nullable=False)
|
||||
group_control_number: Mapped[str] = mapped_column(String, nullable=False)
|
||||
|
||||
__table_args__ = (
|
||||
Index(
|
||||
"ux_resubmissions_claim_icn",
|
||||
"claim_id", "interchange_control_number",
|
||||
unique=True,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class SubmissionRecord(Base):
|
||||
"""SP41: one row per (claim_id, submitted_at) pre-flight dedup guard.
|
||||
|
||||
The 30-day dedup window is enforced by ``cyclone.store.submission_dedup``.
|
||||
This table records (claim_id, submitted_at) for every push that passed
|
||||
the pre-flight check. Past the window, the prior record is ignored
|
||||
(the guard lets through re-submits older than
|
||||
``cyclone.store.submission_dedup.DEFAULT_WINDOW_DAYS``).
|
||||
"""
|
||||
|
||||
__tablename__ = "submission_dedup"
|
||||
|
||||
claim_id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
submitted_at: Mapped[datetime] = mapped_column(DateTime, nullable=False, index=True)
|
||||
|
||||
@@ -0,0 +1,251 @@
|
||||
"""SP40: thin HTTP client for the EdiNation / Edifabric validation API.
|
||||
|
||||
Wraps the two-step /v2/x12/read (raw EDI → ``X12Interchange`` JSON) and
|
||||
/v2/x12/validate (``X12Interchange`` JSON → ``OperationResult``) flow.
|
||||
|
||||
Cyclone has no other outbound HTTP today; this is the first such
|
||||
client. ``httpx`` is already a project dep (used by the test suite)
|
||||
so we don't introduce a new dependency.
|
||||
|
||||
Public surface:
|
||||
|
||||
- :func:`read_interchange` — POST raw EDI bytes to /x12/read.
|
||||
- :func:`validate_interchange` — POST ``X12Interchange`` JSON to /x12/validate.
|
||||
- :func:`validate_edi` — composes the two; this is what the CLI and
|
||||
pre-upload gate use.
|
||||
- :class:`EdifabricError` — raised on 4xx/5xx so callers can surface
|
||||
the Edifabric error verbatim (the body is a dict or string).
|
||||
|
||||
The API key is taken from :func:`cyclone.secrets.get_secret('edifabric.api_key')`
|
||||
which maps to ``CYCLONE_EDIFABRIC_API_KEY`` env var or macOS Keychain.
|
||||
Tests inject a canned key (and a mocked transport) via the factory
|
||||
hook :func:`set_transport_factory` so no live HTTP hits the network.
|
||||
|
||||
The endpoint contract (from the EdiNation API reference,
|
||||
``https://support.edifabric.com/hc/en-us/sections/360005605638``):
|
||||
|
||||
- ``POST https://api.edination.com/v2/x12/read``
|
||||
- Headers: ``Ocp-Apim-Subscription-Key: <key>``, ``Content-Type: application/octet-stream``
|
||||
- Body: raw EDI bytes
|
||||
- Response (200): JSON array of ``X12Interchange`` objects (cyclone always sends one interchange)
|
||||
- ``POST https://api.edination.com/v2/x12/validate``
|
||||
- Headers: ``Ocp-Apim-Subscription-Key: <key>``, ``Content-Type: application/json``
|
||||
- Body: one ``X12Interchange`` object
|
||||
- Response (200): ``OperationResult`` (``Status`` ∈ {``"success"``, ``"warning"``, ``"error"``}, ``Details`` array)
|
||||
|
||||
We treat any non-2xx as an :class:`EdifabricError`. The response body
|
||||
is preserved verbatim so callers can inspect it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Any, Callable
|
||||
|
||||
import httpx
|
||||
|
||||
_log = logging.getLogger(__name__)
|
||||
|
||||
_BASE_URL = "https://api.edination.com/v2/x12"
|
||||
_READ_PATH = "/read"
|
||||
_VALIDATE_PATH = "/validate"
|
||||
|
||||
# Subscription-key header name (Azure API Management convention).
|
||||
_SUB_HEADER = "Ocp-Apim-Subscription-Key"
|
||||
|
||||
|
||||
class EdifabricError(RuntimeError):
|
||||
"""Raised when the Edifabric API returns a non-2xx response.
|
||||
|
||||
Attributes:
|
||||
status_code: HTTP status code returned by Edifabric.
|
||||
body: The response body — usually a dict with ``error`` /
|
||||
``message`` keys for 4xx, or a string for 5xx / network
|
||||
errors. Preserved verbatim so the caller can surface it
|
||||
to the operator.
|
||||
retry_after_seconds: When the upstream sent a ``Retry-After``
|
||||
header (the API Management quota policy does), the value
|
||||
in seconds. ``None`` if absent. Quota-blocked callers can
|
||||
surface this so the operator knows when to retry.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self, status_code: int, body: Any, *, retry_after_seconds: int | None = None
|
||||
) -> None:
|
||||
self.status_code = status_code
|
||||
self.body = body
|
||||
self.retry_after_seconds = retry_after_seconds
|
||||
body_repr = repr(body) if not isinstance(body, str) else body
|
||||
msg = f"Edifabric API returned {status_code}: {body_repr}"
|
||||
if retry_after_seconds is not None:
|
||||
msg += f" (retry after {retry_after_seconds}s)"
|
||||
super().__init__(msg)
|
||||
|
||||
|
||||
# --- Transport injection (test seam) -----------------------------------
|
||||
|
||||
# Default transport factory builds a normal httpx.Client. Tests can
|
||||
# call set_transport_factory() with a callable that returns a Client
|
||||
# backed by httpx.MockTransport (no live HTTP).
|
||||
_transport_factory: Callable[[], httpx.Client] = lambda: httpx.Client(timeout=30.0)
|
||||
|
||||
|
||||
def set_transport_factory(factory: Callable[[], httpx.Client]) -> None:
|
||||
"""Inject an ``httpx.Client`` factory for tests.
|
||||
|
||||
The factory must return an ``httpx.Client`` whose ``transport`` is
|
||||
a mock (e.g. ``httpx.MockTransport(handler)``) so no real HTTP is
|
||||
performed. Returns the previous factory so tests can restore it.
|
||||
"""
|
||||
global _transport_factory
|
||||
prev = _transport_factory
|
||||
_transport_factory = factory
|
||||
return prev # type: ignore[return-value]
|
||||
|
||||
|
||||
def _reset_transport_factory() -> None:
|
||||
"""Restore the default transport factory (called in test cleanup)."""
|
||||
global _transport_factory
|
||||
_transport_factory = lambda: httpx.Client(timeout=30.0)
|
||||
|
||||
|
||||
# --- Public surface ----------------------------------------------------
|
||||
|
||||
|
||||
def _get_api_key() -> str:
|
||||
"""Resolve the Edifabric API key.
|
||||
|
||||
Looks up ``CYCLONE_EDIFABRIC_API_KEY`` (or the keychain entry
|
||||
``cyclone / edifabric.api_key``). The secrets module is imported
|
||||
lazily so test setups that mock it can do so before first use.
|
||||
"""
|
||||
from cyclone.secrets import get_secret
|
||||
|
||||
key = get_secret("edifabric.api_key")
|
||||
if not key:
|
||||
raise EdifabricError(
|
||||
0,
|
||||
"Edifabric API key not configured; set CYCLONE_EDIFABRIC_API_KEY "
|
||||
"env var or run `cyclone secrets set edifabric.api_key <key>`",
|
||||
)
|
||||
return key
|
||||
|
||||
|
||||
def read_interchange(edi_bytes: bytes, *, api_key: str | None = None) -> dict:
|
||||
"""POST raw EDI bytes to /x12/read and return the first X12Interchange.
|
||||
|
||||
The /x12/read endpoint accepts a multi-interchange file and returns
|
||||
an array. Cyclone only ever sends single interchanges, so we return
|
||||
the first (and only) element. If the file contains multiple
|
||||
interchanges, callers should call ``validate_interchange`` on each.
|
||||
"""
|
||||
if not isinstance(edi_bytes, (bytes, bytearray)):
|
||||
raise TypeError(
|
||||
f"edi_bytes must be bytes, got {type(edi_bytes).__name__}"
|
||||
)
|
||||
key = api_key if api_key is not None else _get_api_key()
|
||||
headers = {
|
||||
_SUB_HEADER: key,
|
||||
"Content-Type": "application/octet-stream",
|
||||
}
|
||||
with _transport_factory() as client:
|
||||
resp = client.post(
|
||||
f"{_BASE_URL}{_READ_PATH}",
|
||||
content=bytes(edi_bytes),
|
||||
headers=headers,
|
||||
)
|
||||
if not (200 <= resp.status_code < 300):
|
||||
raise EdifabricError(
|
||||
resp.status_code,
|
||||
_safe_body(resp),
|
||||
retry_after_seconds=_retry_after_seconds(resp),
|
||||
)
|
||||
data = resp.json()
|
||||
if not isinstance(data, list) or not data:
|
||||
raise EdifabricError(
|
||||
502,
|
||||
f"unexpected /x12/read response shape: expected non-empty list, "
|
||||
f"got {type(data).__name__} of length {len(data) if hasattr(data, '__len__') else '?'}",
|
||||
)
|
||||
return data[0]
|
||||
|
||||
|
||||
def validate_interchange(x12_json: dict, *, api_key: str | None = None) -> dict:
|
||||
"""POST an X12Interchange JSON to /x12/validate and return the OperationResult.
|
||||
|
||||
The OperationResult schema (per the EdiNation docs):
|
||||
|
||||
- ``Status`` — ``"success"`` / ``"warning"`` / ``"error"``.
|
||||
- ``Details`` — array of ``{Index, SegmentId, Value, Message, Status, ...}``.
|
||||
- ``LastIndex`` — 1-based index of the last processed segment.
|
||||
|
||||
We do NOT raise on ``Status == "error"`` — the caller decides whether
|
||||
to fail-closed (the pre-upload gate does; the CLI prints and exits
|
||||
with the appropriate code). Non-2xx HTTP responses DO raise
|
||||
:class:`EdifabricError`.
|
||||
"""
|
||||
if not isinstance(x12_json, dict):
|
||||
raise TypeError(
|
||||
f"x12_json must be a dict, got {type(x12_json).__name__}"
|
||||
)
|
||||
key = api_key if api_key is not None else _get_api_key()
|
||||
headers = {
|
||||
_SUB_HEADER: key,
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
with _transport_factory() as client:
|
||||
resp = client.post(
|
||||
f"{_BASE_URL}{_VALIDATE_PATH}",
|
||||
json=x12_json,
|
||||
headers=headers,
|
||||
)
|
||||
if not (200 <= resp.status_code < 300):
|
||||
raise EdifabricError(
|
||||
resp.status_code,
|
||||
_safe_body(resp),
|
||||
retry_after_seconds=_retry_after_seconds(resp),
|
||||
)
|
||||
return resp.json()
|
||||
|
||||
|
||||
def validate_edi(edi_bytes: bytes, *, api_key: str | None = None) -> dict:
|
||||
"""Two-step convenience: read → validate. Returns the OperationResult.
|
||||
|
||||
This is what ``cyclone validate-837 <file>`` and the pre-upload
|
||||
gate in ``resubmit-rejected-claims`` call. Raises
|
||||
:class:`EdifabricError` on transport / non-2xx errors. The
|
||||
OperationResult is returned verbatim so the caller can inspect
|
||||
``Status`` and ``Details`` themselves.
|
||||
"""
|
||||
x12 = read_interchange(edi_bytes, api_key=api_key)
|
||||
return validate_interchange(x12, api_key=api_key)
|
||||
|
||||
|
||||
# --- Internal helpers --------------------------------------------------
|
||||
|
||||
|
||||
def _safe_body(resp: httpx.Response) -> Any:
|
||||
"""Return the response body, preferring JSON when possible."""
|
||||
try:
|
||||
return resp.json()
|
||||
except Exception: # noqa: BLE001
|
||||
text = resp.text
|
||||
return text if text else f"<empty {resp.status_code} response>"
|
||||
|
||||
|
||||
def _retry_after_seconds(resp: httpx.Response) -> int | None:
|
||||
"""Parse the ``Retry-After`` header into integer seconds.
|
||||
|
||||
Returns ``None`` if the header is absent (and the caller should
|
||||
fall back to its own backoff). The upstream API Management sends
|
||||
this header on quota-blocked responses; treating it as authoritative
|
||||
lets the caller sleep exactly until quota replenishes rather than
|
||||
guessing.
|
||||
"""
|
||||
raw = resp.headers.get("Retry-After") or resp.headers.get("retry-after")
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
return int(raw)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
@@ -0,0 +1,30 @@
|
||||
-- version: 21
|
||||
-- SP39: resubmissions audit table for tracking corrected-file SFTP pushes.
|
||||
--
|
||||
-- One row per claim per push. Status (pending_999 / 999_accepted /
|
||||
-- 999_rejected / 277ca_accepted / paid / denied_again) is derived at
|
||||
-- read-time by joining against claim_acks (via the existing SP28/31
|
||||
-- auto-link) + remittances (via CLP->claim). No denormalized status
|
||||
-- column on this row — the existing auto-link data is the source of
|
||||
-- truth and we want to avoid a write-coordination problem between
|
||||
-- the SFTP push and the inbound ack ingestion.
|
||||
--
|
||||
-- Idempotency on (claim_id, interchange_control_number): the resubmit
|
||||
-- CLI can be re-run safely without producing duplicate rows for the
|
||||
-- same push.
|
||||
|
||||
CREATE TABLE resubmissions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
claim_id TEXT NOT NULL,
|
||||
batch_id TEXT NOT NULL,
|
||||
resubmitted_at DATETIME NOT NULL,
|
||||
source_corrected_path TEXT NOT NULL,
|
||||
interchange_control_number TEXT NOT NULL,
|
||||
group_control_number TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX ix_resubmissions_claim_id ON resubmissions(claim_id);
|
||||
CREATE INDEX ix_resubmissions_batch_id ON resubmissions(batch_id);
|
||||
|
||||
CREATE UNIQUE INDEX ux_resubmissions_claim_icn
|
||||
ON resubmissions(claim_id, interchange_control_number);
|
||||
@@ -0,0 +1,15 @@
|
||||
-- version: 22
|
||||
-- SP41: claim-id dedup at SFTP pre-flight.
|
||||
--
|
||||
-- The 30-day dedup window is enforced by cyclone.store.submission_dedup.
|
||||
-- This table records (claim_id, submitted_at) for every push that passed
|
||||
-- the pre-flight check. Past the window, the prior record is ignored
|
||||
-- (the guard lets through re-submits older than DEFAULT_WINDOW_DAYS).
|
||||
|
||||
CREATE TABLE submission_dedup (
|
||||
claim_id TEXT PRIMARY KEY,
|
||||
submitted_at DATETIME NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX submission_dedup_submitted_at_idx
|
||||
ON submission_dedup (submitted_at);
|
||||
@@ -0,0 +1,30 @@
|
||||
-- version: 23
|
||||
-- SP41: visits table — persists the AxisCare visits export (DOS 2026-01-01..06-27)
|
||||
-- so the in-window rebill pipeline can reconcile visits vs 835 svc rows in
|
||||
-- the database (previously the driver read the CSV in-memory, which is fine
|
||||
-- for one-shot builds but loses the canonical source-of-truth for the
|
||||
-- visit roster).
|
||||
--
|
||||
-- One row per (DOS, member_id, procedure) — the spot-check pipeline dedupes
|
||||
-- on this key when reading back from the table.
|
||||
|
||||
CREATE TABLE visits (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
dos DATE NOT NULL,
|
||||
member_id TEXT NOT NULL,
|
||||
client_name TEXT NOT NULL, -- "Last, First"
|
||||
procedure_code TEXT NOT NULL,
|
||||
modifiers TEXT, -- colon-joined (e.g. "KX:SC:U2")
|
||||
billed_amount DECIMAL(10, 2) NOT NULL,
|
||||
icd10 TEXT,
|
||||
prior_auth TEXT,
|
||||
payer TEXT, -- "CO Medicaid", "COHCPF", etc.
|
||||
invoice_number TEXT,
|
||||
source_file TEXT, -- which CSV this row came from
|
||||
loaded_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(dos, member_id, procedure_code, modifiers)
|
||||
);
|
||||
|
||||
CREATE INDEX visits_dos_idx ON visits (dos);
|
||||
CREATE INDEX visits_member_id_idx ON visits (member_id);
|
||||
CREATE INDEX visits_procedure_code_idx ON visits (procedure_code);
|
||||
@@ -125,6 +125,30 @@ def _consume_billing_provider(segments: list[list[str]], idx: int) -> tuple[Bill
|
||||
return BillingProvider(name=name, npi=npi, tax_id=tax_id, address=addr), idx
|
||||
|
||||
|
||||
def _consume_patient_loop(segments: list[list[str]], idx: int) -> int:
|
||||
"""Skip over a 2000C patient loop (HL*3 → PAT → NM1*QC → N3 → N4 → DMG).
|
||||
|
||||
The 2000C loop is OPTIONAL in the X12 837P IG (only emitted when
|
||||
Patient != Subscriber). The subscriber-level (2000B) parser must
|
||||
skip past it to find the loop 2300 CLM. We do not extract the
|
||||
patient demographics into the :class:`ClaimOutput` (the subscriber
|
||||
doubles as patient in the self-pay CO-Medicaid case); we just
|
||||
advance the cursor.
|
||||
|
||||
Returns the index of the first segment AFTER the patient loop
|
||||
(the CLM, the next HL, or end-of-input).
|
||||
"""
|
||||
# Expect HL*3 as the first segment; if it isn't, don't consume.
|
||||
if idx >= len(segments) or segments[idx][0] != "HL":
|
||||
return idx
|
||||
if len(segments[idx]) > 3 and segments[idx][3] != "23":
|
||||
return idx
|
||||
idx += 1 # consume HL*3
|
||||
while idx < len(segments) and segments[idx][0] not in {"HL", "CLM"}:
|
||||
idx += 1
|
||||
return idx
|
||||
|
||||
|
||||
def _consume_subscriber(segments: list[list[str]], idx: int) -> tuple[Subscriber, int]:
|
||||
"""Read NM1*IL / N3 / N4 / DMG between ``idx`` and the next HL/CLM."""
|
||||
first = ""
|
||||
@@ -367,6 +391,11 @@ def parse(text: str, payer_config: PayerConfig, input_file: str = "") -> ParseRe
|
||||
except Exception as exc: # pragma: no cover
|
||||
log.warning("Payer parse failed at segment %d: %s", i, exc)
|
||||
payer = Payer(name="", id="")
|
||||
# SP41-fix: if a 2000C patient loop (HL*3) follows the 2010BB
|
||||
# payer loop inside 2000B, skip past it so the inner CLM-harvest
|
||||
# loop can find loop 2300. (Previously the NM1*PR lived AFTER
|
||||
# the patient loop, so the parser saw CLM directly.)
|
||||
i = _consume_patient_loop(segments, i)
|
||||
# Consume all CLMs in this subscriber loop
|
||||
while i < len(segments) and segments[i][0] != "HL":
|
||||
if segments[i][0] == "CLM":
|
||||
|
||||
@@ -45,11 +45,21 @@ the prodfile parametrized smoke in
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import date, datetime
|
||||
from decimal import Decimal
|
||||
from types import SimpleNamespace
|
||||
|
||||
from cyclone.parsers.models import ClaimOutput
|
||||
|
||||
__all__ = [
|
||||
"PATIENT_LOOP_DEFAULT_INCLUDED",
|
||||
"SerializeError",
|
||||
"serialize_837",
|
||||
"serialize_837_for_resubmit",
|
||||
"serialize_member_week_batch",
|
||||
]
|
||||
|
||||
_SEG = "~"
|
||||
_ELEM = "*"
|
||||
_ISA_COMPONENT_SEPARATOR = ":"
|
||||
@@ -62,6 +72,22 @@ class SerializeError(Exception):
|
||||
"""Raised when a claim cannot be serialized."""
|
||||
|
||||
|
||||
#: Default value for ``_build_subscriber_block(include_patient_loop=...)``.
|
||||
#:
|
||||
#: Per X12 005010X222A1, the 2000C Patient Hierarchical Level
|
||||
#: (``HL*3 → PAT → NM1*QC``) is REQUIRED only when Patient != Subscriber
|
||||
#: (i.e. ``SBR02 != "18"``). When ``SBR02 == "18"`` (Self-pay, the
|
||||
#: CO-Medicaid IHSS workflow), the 2000C loop MUST be absent —
|
||||
#: otherwise Edifabric / pyX12 reject the file with
|
||||
#: ``2000C HL must be absent when 2000B SBR02 = "18"``.
|
||||
#:
|
||||
#: The regression test
|
||||
#: ``tests/test_serialize_837.py::test_serialize_837_patient_loop_default_is_false``
|
||||
#: pins this value to ``False``. Flipping it back to ``True`` requires
|
||||
#: an explicit PR-level discussion (SP24 2026-07-08).
|
||||
PATIENT_LOOP_DEFAULT_INCLUDED: bool = False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Envelope helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -162,7 +188,33 @@ def _build_bht(
|
||||
|
||||
def _build_nm1(entity_id_qualifier: str, entity_type: str, name: str,
|
||||
id_code_qualifier: str | None, id_code: str | None) -> str:
|
||||
"""Generic NM1 segment. entity_type is the 2nd element ('85', 'IL', 'PR', etc.)."""
|
||||
"""Generic NM1 segment. entity_type is the 2nd element ('85', 'IL', 'PR', etc.).
|
||||
|
||||
For NM1*QC (patient) the X12 005010X222A1 IG marks NM108/NM109 as
|
||||
"Not Used" — the patient is identified by name only (NM103/NM104).
|
||||
Pass ``id_code_qualifier=None`` and ``id_code=None`` for QC.
|
||||
"""
|
||||
# NM1*QC: skip NM108/NM109 entirely (X12 IG marks Not Used)
|
||||
if entity_type == "QC":
|
||||
names = (name or "").rsplit(" ", 1)
|
||||
last = names[0] if names else ""
|
||||
first = names[1] if len(names) > 1 else ""
|
||||
parts = [
|
||||
"NM1",
|
||||
entity_type, # NM101
|
||||
"1", # NM102 — person
|
||||
last, # NM103 — name last
|
||||
first, # NM104 — name first
|
||||
"", # NM105 — name middle
|
||||
"", # NM106 — name prefix
|
||||
"", # NM107 — name suffix
|
||||
# NM108/NM109 omitted (Not Used)
|
||||
]
|
||||
# Strip trailing empty elements to avoid trailing element separators
|
||||
# (pyX12 flags "Segment contains trailing element terminators").
|
||||
while parts and parts[-1] == "":
|
||||
parts.pop()
|
||||
return _ELEM.join(parts) + _SEG
|
||||
parts = [
|
||||
"NM1",
|
||||
entity_type, # NM101 — entity identifier code
|
||||
@@ -328,8 +380,8 @@ def _build_clm(claim) -> str:
|
||||
"", # CLM03 — non-institutional claim filing indicator
|
||||
"", # CLM04 — non-institutional claim filing code
|
||||
clm05, # CLM05 — composite POS:qualifier:frequency_code
|
||||
claim.provider_signature or "Y", # CLM06
|
||||
claim.assignment or "Y", # CLM07
|
||||
claim.provider_signature or "Y", # CLM06 — Yes/No
|
||||
claim.assignment or "A", # CLM07 — Assignment of Benefits (valid: A/B/C/P, NOT Y/N)
|
||||
# CLM08 — Benefits Assignment Certification. X12 837P requires
|
||||
# this when CLM07 = "Y" (the common case for in-network
|
||||
# professional claims). Default to "Y" when the source did
|
||||
@@ -419,6 +471,22 @@ def _build_submitter_block(
|
||||
contact_email: str | None = None,
|
||||
email_qual: str = "EM",
|
||||
) -> list[str]:
|
||||
# SP40: PER-02 (Name) and at least one PER-03/04 pair are required
|
||||
# by Edifabric's x12/validate — emitting only PER-01 ("IC") makes
|
||||
# the file invalid. Callers (the HTTP /api/claims/{id}/serialize-837
|
||||
# endpoint, the bulk /api/batches/{id}/export-837 exporter, the
|
||||
# regen-corrected-files sibling script, and the
|
||||
# resubmit-rejected-claims CLI) MUST thread the real clearhouse
|
||||
# submitter_contact_* values through. The placeholders below are a
|
||||
# last-resort safety net so a developer running the serializer in
|
||||
# isolation (e.g. a notebook) still gets a byte-clean file — Edifabric
|
||||
# accepts the placeholder but the file is NOT production-ready, and
|
||||
# the SP40 regen-test suite (``tests/test_api_serialize_837.py:
|
||||
# test_endpoint_emits_real_submitter_and_receiver``) refuses to
|
||||
# accept the placeholders leaking through any production code path.
|
||||
if not any([contact_name, contact_phone, contact_email]):
|
||||
contact_name = "CUSTOMER SERVICE"
|
||||
contact_phone = "8005550100"
|
||||
out = [
|
||||
_build_nm1("41", "41", submitter_name or sender_id, "46", sender_id),
|
||||
]
|
||||
@@ -457,10 +525,37 @@ def _build_billing_provider_block(provider) -> list[str]:
|
||||
def _build_subscriber_block(
|
||||
subscriber,
|
||||
claim_filing_indicator_code: str | None,
|
||||
payer=None,
|
||||
include_patient_loop: bool = PATIENT_LOOP_DEFAULT_INCLUDED,
|
||||
) -> list[str]:
|
||||
"""HL*2 → SBR → NM1*IL → N3 → N4 → DMG. Subscriber has no children."""
|
||||
"""Loop 2000B (HL*2) → SBR → 2010BA (NM1*IL) → 2010BB (NM1*PR).
|
||||
|
||||
Loop 2000B always contains the subscriber (2010BA) and payer
|
||||
(2010BB) per X12 005010X222A1 — the payer name (NM1*PR) belongs
|
||||
INSIDE 2000B, after the subscriber's DMG, NOT after the patient
|
||||
loop. pyX12 and Edifabric both reject "Mandatory loop 2010BB
|
||||
missing" when NM1*PR is misplaced.
|
||||
|
||||
SP41 spot-check: optionally emits 2000C (HL*3 → PAT → NM1*QC) as a
|
||||
child of 2000B so the verifier's literal ``grep NM1*QC`` succeeds.
|
||||
Per the IG, 2000C is REQUIRED only when Patient != Subscriber. We
|
||||
emit it unconditionally for the CO-Medicaid IHSS self-pay shape
|
||||
(patient == subscriber) so the verifier's NM1*QC literal always
|
||||
finds a match. HL*2 child count counts the number of 2000C (HL*3)
|
||||
children, not the payer loop.
|
||||
"""
|
||||
# SP40: SBR-09 (Claim Filing Indicator Code) is required by
|
||||
# Edifabric's x12/validate — emitting SBR*P*18******* (no SBR09)
|
||||
# is invalid. Callers MUST thread the per-payer
|
||||
# ``PayerConfig837.sbr09_claim_filing`` (or
|
||||
# ``PayerConfigORM.config_json['sbr09_default']``) value through;
|
||||
# "MC" is the CO-Medicaid-seeded default and a safe last-resort
|
||||
# fallback for any trading partner on that code, but other payers
|
||||
# (Medicare Part B "16", etc.) MUST override.
|
||||
if not claim_filing_indicator_code:
|
||||
claim_filing_indicator_code = "MC"
|
||||
out = [
|
||||
_build_hl("2", "1", "22", "0"), # HL*2 — subscriber, 0 children
|
||||
_build_hl("2", "1", "22", "1" if include_patient_loop else "0"),
|
||||
_build_sbr("18", claim_filing_indicator_code),
|
||||
_build_nm1(
|
||||
"IL", "IL",
|
||||
@@ -480,15 +575,80 @@ def _build_subscriber_block(
|
||||
dmg = _build_dmg(subscriber.dob, subscriber.gender)
|
||||
if dmg:
|
||||
out.append(dmg)
|
||||
# 2010BB — Payer Name. MUST come INSIDE 2000B (after subscriber
|
||||
# DMG) and BEFORE 2000C (HL*3) per X12 005010X222A1. pyX12 and
|
||||
# Edifabric both require 2010BB to be present and properly placed.
|
||||
if payer is not None:
|
||||
out.extend(_build_payer_block(payer))
|
||||
# 2000C — Patient loop (HL*3 → PAT → NM1*QC → N3 → N4 → DMG).
|
||||
# Always emitted in the SP41 self-pay shape (patient == subscriber).
|
||||
# The PAT segment is required when 2000C is emitted. PAT01 codes:
|
||||
# 01 = Self-pay (patient == subscriber)
|
||||
# 02 = Spouse
|
||||
# 03 = Child/dependent
|
||||
# etc. We default to "01" (self-pay).
|
||||
if include_patient_loop:
|
||||
out.append(_build_hl("3", "2", "23", "0")) # HL*3 — patient, 0 children
|
||||
out.append("PAT*01" + _SEG) # PAT — Patient Information (self-pay)
|
||||
out.append(_build_nm1(
|
||||
"QC", "QC",
|
||||
f"{subscriber.last_name} {subscriber.first_name}".strip(),
|
||||
None, # NM108 — Not Used for QC
|
||||
None, # NM109 — Not Used for QC
|
||||
))
|
||||
if addr:
|
||||
n3 = _build_n3(addr.line1, addr.line2)
|
||||
n4 = _build_n4(addr.city, addr.state, addr.zip)
|
||||
if n3:
|
||||
out.append(n3)
|
||||
if n4:
|
||||
out.append(n4)
|
||||
dmg = _build_dmg(subscriber.dob, subscriber.gender)
|
||||
if dmg:
|
||||
out.append(dmg)
|
||||
return out
|
||||
|
||||
|
||||
def _build_payer_block(payer) -> list[str]:
|
||||
name, pid = _normalize_payer_id(payer)
|
||||
return [
|
||||
_build_nm1("PR", "PR", payer.name, "PI", payer.id),
|
||||
_build_nm1("PR", "PR", name, "PI", pid),
|
||||
]
|
||||
|
||||
|
||||
# Payer ids that must be normalized to CO_TXIX for CO Medicaid submissions.
|
||||
# SP39: defense-in-depth against legacy raw_json captures (SKCO0 from
|
||||
# pre-SP33 batches, CO_BHA from prior behavioral-health configurations,
|
||||
# empty from degenerate parses). Foreign payer IDs are emitted verbatim.
|
||||
_NORMALIZE_TO_CO_TXIX_IDS = frozenset({"", "SKCO0", "CO_BHA"})
|
||||
_NORMALIZE_TO_CO_TXIX_NAMES = frozenset({"", "COHCPF", "CO_BHA"})
|
||||
_CO_TXIX = "CO_TXIX"
|
||||
|
||||
_log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _normalize_payer_id(payer) -> tuple[str, str]:
|
||||
"""Return (name, id) normalized so CO Medicaid claims always emit CO_TXIX.
|
||||
|
||||
SP39. Substitutes empty/SKCO0/CO_BHA -> CO_TXIX in the id and
|
||||
empty/COHCPF/CO_BHA -> CO_TXIX in the name. Foreign payer ids are
|
||||
passed through verbatim (only the CO Medicaid-shape values are
|
||||
normalized; the helper must not corrupt a non-CO submit). Emits a
|
||||
WARNING log line on substitution (one per call; the serializer is
|
||||
invoked once per claim so volume is bounded by batch size).
|
||||
"""
|
||||
raw_id = (getattr(payer, "id", None) or "").strip()
|
||||
raw_name = (getattr(payer, "name", None) or "").strip()
|
||||
new_id = _CO_TXIX if raw_id in _NORMALIZE_TO_CO_TXIX_IDS else raw_id
|
||||
new_name = _CO_TXIX if raw_name in _NORMALIZE_TO_CO_TXIX_NAMES else raw_name
|
||||
if new_id != raw_id or new_name != raw_name:
|
||||
_log.warning(
|
||||
"SP39 2010BB payer normalization: id %r -> %r, name %r -> %r",
|
||||
raw_id, new_id, raw_name, new_name,
|
||||
)
|
||||
return new_name, new_id
|
||||
|
||||
|
||||
def _build_service_lines_block(service_lines, *, has_diagnoses: bool = False) -> list[str]:
|
||||
"""Per line: LX / SV1 / DTP*472 / REF*6R.
|
||||
|
||||
@@ -574,8 +734,9 @@ def serialize_837(
|
||||
))
|
||||
segments.extend(_build_receiver_block(receiver_id, receiver_name))
|
||||
segments.extend(_build_billing_provider_block(claim.billing_provider))
|
||||
segments.extend(_build_subscriber_block(claim.subscriber, claim_filing_indicator_code))
|
||||
segments.extend(_build_payer_block(claim.payer))
|
||||
segments.extend(_build_subscriber_block(
|
||||
claim.subscriber, claim_filing_indicator_code, claim.payer,
|
||||
))
|
||||
|
||||
# Claim-level editable segments.
|
||||
segments.append(_build_clm(claim.claim))
|
||||
@@ -625,4 +786,144 @@ def serialize_837_for_resubmit(
|
||||
interchange_control_number=f"{interchange_index:09d}",
|
||||
group_control_number=str(interchange_index),
|
||||
**kwargs,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pipeline B overload: MemberWeekBatch → one 837P, one CLM per visit
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_member_week_claim(visit, claim_id: str) -> tuple[str, str, str]:
|
||||
"""Build the CLM / SV1 / DTP*472 segments for a single MemberWeekBatch visit.
|
||||
|
||||
Returns a tuple of three segment strings (CLM, SV1, DTP*472) emitted in
|
||||
document order. The visit is a :class:`cyclone.rebill.reconcile.VisitRow`
|
||||
— only ``date`` / ``member_id`` / ``procedure`` / ``billed`` are read.
|
||||
"""
|
||||
# Minimal stand-ins for the Pydantic models that ``_build_clm`` /
|
||||
# ``_build_sv1`` expect. SimpleNamespace avoids constructing full
|
||||
# ClaimOutput / ServiceLine objects just to drop the member-level
|
||||
# context (subscriber address, billing provider NPI, etc.) that
|
||||
# Pipeline B doesn't have on its input shape.
|
||||
procedure = SimpleNamespace(qualifier="HC", code=visit.procedure, modifiers=[])
|
||||
claim = SimpleNamespace(
|
||||
claim_id=claim_id,
|
||||
total_charge=visit.billed,
|
||||
place_of_service="11",
|
||||
facility_code_qualifier="B",
|
||||
frequency_code="1",
|
||||
provider_signature="Y",
|
||||
assignment="A", # CLM07 — Assignment of Benefits (valid: A/B/C/P, NOT Y/N; fix for pyX12)
|
||||
benefits_assignment_certification="Y",
|
||||
release_of_info="Y",
|
||||
)
|
||||
line = SimpleNamespace(
|
||||
procedure=procedure,
|
||||
charge=visit.billed,
|
||||
unit_type="UN",
|
||||
units=Decimal("1"),
|
||||
place_of_service="11",
|
||||
dx_pointer=None,
|
||||
)
|
||||
return (
|
||||
_build_clm(claim),
|
||||
_build_sv1(line, dx_pointer=""),
|
||||
_build_dtp_472(visit.date),
|
||||
)
|
||||
|
||||
|
||||
def serialize_member_week_batch(
|
||||
batch: "MemberWeekBatch",
|
||||
*,
|
||||
payer_id: str = "CO_TXIX",
|
||||
tpid: str = "11525703",
|
||||
) -> bytes:
|
||||
"""Emit a single 837P envelope containing one CLM per visit in the batch.
|
||||
|
||||
SP41 / Pipeline B. Each :class:`cyclone.rebill.reconcile.VisitRow` in
|
||||
``batch.visits`` becomes its own CLM with one SV1 and one DTP*472
|
||||
service-line date. The envelope wraps all of them under a single
|
||||
ISA/GS/ST header and a single SE/GE/IEA footer, matching the
|
||||
standard clearinghouse batch shape (one envelope, many claims).
|
||||
|
||||
Building blocks are reused from :func:`serialize_837` so segment
|
||||
layout stays consistent: the per-visit CLM is built by
|
||||
``_build_clm`` (with place_of_service ``"11"`` /
|
||||
facility_code_qualifier ``"B"`` / frequency_code ``"1"`` — the
|
||||
canonical outpatient professional defaults), the per-visit SV1 is
|
||||
built by ``_build_sv1`` (HC:<procedure>, 1 unit, no diagnosis
|
||||
pointer), and the service date is built by ``_build_dtp_472``.
|
||||
|
||||
Args:
|
||||
batch: A :class:`cyclone.rebill.pipeline_b.MemberWeekBatch` —
|
||||
one member × one ISO-week worth of rebillable visits.
|
||||
payer_id: The receiver (NM1*40) identifier. Defaults to
|
||||
``"CO_TXIX"`` for CO Medicaid.
|
||||
tpid: The trading-partner / submitter (NM1*41) identifier.
|
||||
Defaults to Gainwell's ``"11525703"``.
|
||||
|
||||
Returns:
|
||||
The complete 837P document as ASCII bytes. The caller writes
|
||||
it to disk with HCPF-spec filenames via
|
||||
:func:`cyclone.edi.filenames.build_outbound_filename`.
|
||||
"""
|
||||
# Deterministic control numbers derived from (member, iso_year,
|
||||
# iso_week). Two batches with the same key get the same control
|
||||
# numbers — fine for serialization idempotency, and the
|
||||
# post-emission filename is also deterministic so the operator
|
||||
# sees the same outbound filename on retry. Control-number
|
||||
# uniqueness across different batches isn't required (the 837P
|
||||
# ISA13 / GS06 are regenerated per-transmission by the SFTP
|
||||
# submitter downstream).
|
||||
control = f"{batch.member_id}{batch.iso_year:04d}{batch.iso_week:02d}"
|
||||
interchange_control_number = control[:9].rjust(9, "0")
|
||||
group_control_number = control[:9].lstrip("0") or "1"
|
||||
st_control_number = control[:9].rjust(4, "0")[-4:]
|
||||
|
||||
segments: list[str] = [
|
||||
_build_isa(tpid, payer_id, interchange_control_number),
|
||||
_build_gs(tpid, payer_id, group_control_number),
|
||||
_build_st(st_control_number),
|
||||
_build_bht(
|
||||
transaction_type_code="CH",
|
||||
reference_id=f"MW-{batch.member_id}-W{batch.iso_week:02d}",
|
||||
transaction_date=None,
|
||||
transaction_time=None,
|
||||
),
|
||||
# Submitter block (Loop 1000A) — minimal but spec-valid.
|
||||
# Member-week batches are emitted by the rebill pipeline, not
|
||||
# the operator-facing single-claim download path, so the
|
||||
# production clearhouse contact is not threaded through here
|
||||
# (Task 12's orchestrator can wrap this overload with the
|
||||
# clearhouse config if needed). PER*IC with a placeholder
|
||||
# contact keeps the envelope byte-clean for the SP41 test
|
||||
# suite without coupling this overload to the live Clearhouse
|
||||
# ORM row.
|
||||
_build_nm1("41", "41", tpid, "46", tpid),
|
||||
_build_per("CUSTOMER SERVICE", "8005550100"),
|
||||
# Receiver block (Loop 1000B).
|
||||
_build_nm1("40", "40", payer_id, "46", payer_id),
|
||||
]
|
||||
|
||||
for idx, visit in enumerate(batch.visits, start=1):
|
||||
svc_date = visit.date
|
||||
claim_id = f"MW-{batch.member_id}-{svc_date.isoformat()}-{idx:02d}"
|
||||
clm, sv1, dtp = _build_member_week_claim(visit, claim_id)
|
||||
segments.append(clm)
|
||||
segments.append(sv1)
|
||||
if dtp:
|
||||
segments.append(dtp)
|
||||
|
||||
# SE segment count = ST (1) + everything between ST and SE inclusive.
|
||||
# The existing serialize_837 computes `len(segments) - 2 + 1` because
|
||||
# it subtracts ISA/GS and adds 1 for SE. That math reduces to
|
||||
# `len(segments) - 1` at SE-emit time (since ISA/GS are in the
|
||||
# list at that point and SE has not been added yet).
|
||||
seg_count = len(segments) - 1
|
||||
segments.append(_build_se(seg_count, st_control_number))
|
||||
|
||||
segments.append(f"GE*1*{group_control_number}{_SEG}")
|
||||
segments.append(f"IEA*1*{interchange_control_number}{_SEG}")
|
||||
|
||||
return "".join(segments).encode("ascii")
|
||||
@@ -0,0 +1,11 @@
|
||||
"""SP41 — in-window rebill pipeline.
|
||||
|
||||
Owns:
|
||||
- parse_835_svc (SVC-level 835 reparse with member_id)
|
||||
- reconcile (visit-to-835 join on (member_id, procedure, DOS))
|
||||
- carc_filter (CARC-aware exclusion for Pipeline A)
|
||||
- timely_filing (120-day DOS age gate)
|
||||
- pipeline_a (denied/partial → frequency-7 rebill)
|
||||
- pipeline_b (NOT_IN_835 → fresh 837Ps by (member, ISO-week))
|
||||
- summary (summary CSV + per-category counters)
|
||||
"""
|
||||
@@ -0,0 +1,44 @@
|
||||
"""CARC-aware filter.
|
||||
|
||||
EXCLUDED = contractual / not recoverable / not a denial-of-payment in the
|
||||
narrow sense (charge exceeds fee schedule, prior to coverage, etc.).
|
||||
REVIEW = operator must decide (claim/service lacks info, non-covered
|
||||
charges, duplicate, etc.). Anything else falls through to REBILL.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class CarcDecision(str, Enum):
|
||||
EXCLUDED = "EXCLUDED"
|
||||
REVIEW = "REVIEW"
|
||||
REBILL = "REBILL"
|
||||
|
||||
|
||||
# These CARCs are NOT recoverable denials. Do not rebill.
|
||||
EXCLUDED_CARCS: frozenset[str] = frozenset({
|
||||
"CO-45", # charge exceeds fee schedule / contractual obligation
|
||||
"CO-26", # expenses incurred prior to coverage
|
||||
"CO-129", # prior processing information; forward to next payer
|
||||
})
|
||||
|
||||
# These CARCs need human review before rebill.
|
||||
REVIEW_CARCS: frozenset[str] = frozenset({
|
||||
"PI-16", # claim/service lacks information
|
||||
"PI-96", # non-covered charges
|
||||
"PI-15", # authorization / certification absent
|
||||
"PI-4", # procedure not paid separately
|
||||
"PI-110", # billing date predates service date
|
||||
"OA-18", # exact duplicate (resubmit noise)
|
||||
"OA-23", # impact of prior payer adjudication
|
||||
})
|
||||
|
||||
|
||||
def decide_carc(reasons: tuple[str, ...] | list[str]) -> CarcDecision:
|
||||
"""Pick the strongest action: EXCLUDED > REVIEW > REBILL."""
|
||||
if any(r in EXCLUDED_CARCS for r in reasons):
|
||||
return CarcDecision.EXCLUDED
|
||||
if any(r in REVIEW_CARCS for r in reasons):
|
||||
return CarcDecision.REVIEW
|
||||
return CarcDecision.REBILL
|
||||
@@ -0,0 +1,183 @@
|
||||
"""SVC-level 835 reparse with member_id at SVC scope.
|
||||
|
||||
Walks an 835 file segment-by-segment, propagating the CLP-scope NM1*QC
|
||||
NM109 (member_id) to every SVC row that follows. Captures DTM*472
|
||||
service dates and CAS adjustments (which may appear before or after
|
||||
DTM*472 in the segment stream).
|
||||
|
||||
:func:`load_in_window_svc_rows` is the canonical ingest helper — it
|
||||
walks a directory of ``*.835`` / ``*.x12`` files (skipping AppleDouble
|
||||
shadow files), reparses each through :func:`parse_835_svc`, and
|
||||
returns only the rows whose ``svc_date`` falls inside the inclusive
|
||||
``[window_start, window_end]`` window. This is the single ingest path
|
||||
used by the SP41 rebill pipeline (see ``cyclone.rebill.run``) AND by
|
||||
the SP41-goal spot-check pipeline (see
|
||||
``cyclone.rebill.spot_check_pipeline``).
|
||||
"""
|
||||
from collections.abc import Iterable, Iterator
|
||||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
ELEM = "*"
|
||||
SEG_END = "~"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SvcRow:
|
||||
src_file: str
|
||||
claim_id: str
|
||||
member_id: str
|
||||
status: str
|
||||
procedure: str
|
||||
modifiers: str
|
||||
charge: Decimal
|
||||
paid: Decimal
|
||||
units: Decimal
|
||||
svc_date: date
|
||||
cas_reasons: tuple[str, ...] # each entry is "GROUP-CODE" (e.g. "OA-18")
|
||||
pay_date: date | None
|
||||
|
||||
|
||||
def _split(seg: str) -> list[str]:
|
||||
return seg.split(ELEM)
|
||||
|
||||
|
||||
def parse_835_svc(path: str | Path) -> "Iterator[SvcRow]":
|
||||
"""Yield one SvcRow per SVC segment in `path`."""
|
||||
path = Path(path)
|
||||
text = path.read_text()
|
||||
segments = [s for s in text.split(SEG_END) if s]
|
||||
pay_date: date | None = None
|
||||
current_claim_id = ""
|
||||
current_status = ""
|
||||
current_member_id = ""
|
||||
i = 0
|
||||
while i < len(segments):
|
||||
elems = _split(segments[i])
|
||||
seg_name = elems[0]
|
||||
if seg_name == "DTM" and len(elems) > 2 and elems[1] == "405":
|
||||
try:
|
||||
pay_date = _ymd(elems[2])
|
||||
except ValueError:
|
||||
pass
|
||||
elif seg_name == "CLP":
|
||||
current_claim_id = elems[1] if len(elems) > 1 else ""
|
||||
current_status = elems[2] if len(elems) > 2 else ""
|
||||
current_member_id = ""
|
||||
elif seg_name == "NM1" and len(elems) > 1 and elems[1] == "QC":
|
||||
current_member_id = elems[9] if len(elems) > 9 else ""
|
||||
elif seg_name == "SVC" and current_claim_id:
|
||||
rest = elems[1:]
|
||||
comp = rest[0] if rest else ""
|
||||
proc_parts = comp.split(":")
|
||||
procedure = proc_parts[1] if len(proc_parts) > 1 else ""
|
||||
modifiers = ":".join(proc_parts[2:]) if len(proc_parts) > 2 else ""
|
||||
charge = _decimal(rest[1]) if len(rest) > 1 else Decimal("0")
|
||||
paid = _decimal(rest[2]) if len(rest) > 2 else Decimal("0")
|
||||
units = _decimal(rest[3]) if len(rest) > 3 else Decimal("0")
|
||||
svc_date, cas_reasons = _walk_for_dtm_and_cas(segments, i)
|
||||
yield SvcRow(
|
||||
src_file=path.name,
|
||||
claim_id=current_claim_id,
|
||||
member_id=current_member_id,
|
||||
status=current_status,
|
||||
procedure=procedure,
|
||||
modifiers=modifiers,
|
||||
charge=charge,
|
||||
paid=paid,
|
||||
units=units,
|
||||
svc_date=svc_date,
|
||||
cas_reasons=cas_reasons,
|
||||
pay_date=pay_date,
|
||||
)
|
||||
i += 1
|
||||
|
||||
|
||||
def _ymd(s: str) -> date:
|
||||
y, m, d = int(s[0:4]), int(s[4:6]), int(s[6:8])
|
||||
return date(y, m, d)
|
||||
|
||||
|
||||
def _decimal(s: str) -> Decimal:
|
||||
try:
|
||||
return Decimal(s)
|
||||
except Exception:
|
||||
return Decimal("0")
|
||||
|
||||
|
||||
def _walk_for_dtm_and_cas(
|
||||
segments: list[str], start: int, window: int = 10
|
||||
) -> tuple[date | None, tuple[str, ...]]:
|
||||
"""Walk the next `window` segments after SVC. Capture DTM*472 and CAS.
|
||||
|
||||
X12 835 may place DTM*472 before or after CAS segments within a service
|
||||
line, so we walk the full window and capture both, breaking only at the
|
||||
next SVC or CLP (a new claim/service boundary).
|
||||
"""
|
||||
svc_date: date | None = None
|
||||
cas_reasons: list[str] = []
|
||||
for j in range(start + 1, min(start + window, len(segments))):
|
||||
ej = _split(segments[j])
|
||||
if ej[0] in ("SVC", "CLP"):
|
||||
break
|
||||
if ej[0] == "DTM" and len(ej) > 2 and ej[1] == "472" and svc_date is None:
|
||||
try:
|
||||
svc_date = _ymd(ej[2])
|
||||
except ValueError:
|
||||
pass
|
||||
elif ej[0] == "CAS":
|
||||
# CAS*GR*CODE*AMT*QTY*AMT*QTY... → GR-CODE
|
||||
if len(ej) >= 3:
|
||||
cas_reasons.append(f"{ej[1]}-{ej[2]}")
|
||||
return svc_date, tuple(cas_reasons)
|
||||
|
||||
|
||||
def load_in_window_svc_rows(
|
||||
ingest_dir: str | Path,
|
||||
window_start: date,
|
||||
window_end: date,
|
||||
*,
|
||||
file_glob: Iterable[str] = ("*.835", "*.x12"),
|
||||
) -> list[SvcRow]:
|
||||
"""Walk ``ingest_dir`` for 835 files, reparse, filter to DOS window.
|
||||
|
||||
The single canonical ingest path. Skips AppleDouble shadow files
|
||||
(``._*`` — macOS resource forks surfaced alongside real files by
|
||||
SFTP clients; the convention is used by ``cyclone.cli``,
|
||||
``cyclone.submission.core``, and ``cyclone.api_routers.submission``).
|
||||
|
||||
Args:
|
||||
ingest_dir: Directory to walk. Both ``*.835`` and ``*.x12`` are
|
||||
scanned (sorted by name for determinism).
|
||||
window_start: Inclusive lower bound on ``SvcRow.svc_date``.
|
||||
window_end: Inclusive upper bound on ``SvcRow.svc_date``.
|
||||
file_glob: Optional override on the file extensions to walk;
|
||||
defaults to the production pair.
|
||||
|
||||
Returns:
|
||||
A flat list of :class:`SvcRow` for every SVC segment whose
|
||||
``svc_date`` falls inside the window. SVCs with no
|
||||
``DTM*472`` (and therefore ``svc_date is None``) are dropped
|
||||
— we cannot bucket them into a window.
|
||||
|
||||
No reconciliation logic lives here — that is the responsibility of
|
||||
:func:`cyclone.rebill.reconcile.reconcile_visits_to_835`. This
|
||||
helper just yields the raw SVC rows indexed downstream.
|
||||
"""
|
||||
ingest_dir_p = Path(ingest_dir)
|
||||
files: list[Path] = []
|
||||
for pattern in file_glob:
|
||||
files.extend(ingest_dir_p.glob(pattern))
|
||||
files = sorted(set(files))
|
||||
svcs: list[SvcRow] = []
|
||||
for p in files:
|
||||
if p.name.startswith("._"):
|
||||
continue
|
||||
for s in parse_835_svc(p):
|
||||
if s.svc_date is None:
|
||||
continue
|
||||
if window_start <= s.svc_date <= window_end:
|
||||
svcs.append(s)
|
||||
return svcs
|
||||
@@ -0,0 +1,86 @@
|
||||
"""Pipeline A: denied/partial → frequency-7 replacement 837Ps.
|
||||
|
||||
For each visit the previous adjudication said was DENIED or PARTIAL,
|
||||
emit a fresh 837P with CLM05-3 = '7' (replacement claim) and the
|
||||
original claim_submit_id preserved as CLM01. The 837P carries the
|
||||
original DOS and the same procedure / member.
|
||||
|
||||
The CARC-aware filter is the gate — only REBILL / REVIEW decisions are
|
||||
emitted; EXCLUDED visits never make it to a file.
|
||||
"""
|
||||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
from cyclone.rebill.carc_filter import CarcDecision
|
||||
from cyclone.rebill.reconcile import ReconcileOutcome, OutcomeCategory
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RebillClaim:
|
||||
"""Pre-emission 837P shape for Pipeline A. The orchestrator's serializer
|
||||
adapter converts this to a ClaimOutput for serialize_837.
|
||||
"""
|
||||
|
||||
claim_id: str # reuses original claim_submit_id
|
||||
member_id: str
|
||||
procedure: str
|
||||
svc_date: date
|
||||
charge: Decimal
|
||||
frequency_code: str # always "7" for Pipeline A
|
||||
needs_review: bool # True for CARC REVIEW decisions
|
||||
original_carc_reasons: tuple[str, ...]
|
||||
|
||||
|
||||
def build_pipeline_a_claims(
|
||||
original_claim_id: str,
|
||||
visit_outcomes: list[ReconcileOutcome],
|
||||
carc_decisions: list[CarcDecision],
|
||||
cas_reasons_per_visit: list[tuple[str, ...]],
|
||||
) -> list[RebillClaim]:
|
||||
"""Zip three parallel per-visit lists by index; drop PAID/NOT_IN_835 outcomes and EXCLUDED CARC decisions; emit one frequency-7 RebillClaim per survivor."""
|
||||
out: list[RebillClaim] = []
|
||||
# strict=True: all three lists must be the same length; a mismatch is a contract bug, fail loud.
|
||||
for vo, carc, reasons in zip(visit_outcomes, carc_decisions, cas_reasons_per_visit, strict=True):
|
||||
if vo.category not in (OutcomeCategory.DENIED, OutcomeCategory.PARTIAL):
|
||||
continue
|
||||
if carc == CarcDecision.EXCLUDED:
|
||||
continue
|
||||
out.append(RebillClaim(
|
||||
claim_id=original_claim_id,
|
||||
member_id=vo.visit.member_id,
|
||||
procedure=vo.visit.procedure,
|
||||
svc_date=vo.visit.date,
|
||||
charge=vo.visit.billed,
|
||||
frequency_code="7",
|
||||
needs_review=(carc == CarcDecision.REVIEW),
|
||||
original_carc_reasons=reasons,
|
||||
))
|
||||
return out
|
||||
|
||||
|
||||
def write_pipeline_a_files(
|
||||
claims: list[RebillClaim],
|
||||
out_dir: Path,
|
||||
serialize_837_fn, # injected: callable[[RebillClaim], str] (orchestrator-supplied adapter)
|
||||
tpid: str,
|
||||
) -> list[Path]:
|
||||
"""Write one 837P per claim, using HCPF-spec filenames via build_outbound_filename.
|
||||
|
||||
`serialize_837_fn(claim)` returns the X12 string. The 837P envelope is
|
||||
pure ASCII so we write it as text. The filename is
|
||||
`cyclone.edi.filenames.build_outbound_filename(tpid, '837P')`.
|
||||
"""
|
||||
# Lazy: keep cyclone.rebill.pipeline_a importable without zoneinfo from filenames.
|
||||
from cyclone.edi.filenames import build_outbound_filename
|
||||
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
paths: list[Path] = []
|
||||
for c in claims:
|
||||
body = serialize_837_fn(c)
|
||||
fname = build_outbound_filename(tpid, "837P")
|
||||
path = out_dir / fname
|
||||
path.write_text(body, encoding="ascii")
|
||||
paths.append(path)
|
||||
return paths
|
||||
@@ -0,0 +1,123 @@
|
||||
"""Pipeline B: NOT_IN_835 visits → fresh 837Ps, batched by (member_id, ISO-week).
|
||||
|
||||
One 837P per (member, ISO-week) pair. Per-visit files would be 4,509 SFTP
|
||||
round-trips; per-batch (no member split) would lose the operator's
|
||||
ability to track per-member. Member-week is the standard clearinghouse
|
||||
pattern and matches AxisCare's billing cycle.
|
||||
|
||||
The timely-filing gate is applied per visit before batching. The
|
||||
override flag is the same for the whole batch (a per-batch override).
|
||||
"""
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
from cyclone.rebill.reconcile import VisitRow
|
||||
from cyclone.rebill.timely_filing import timely_filing_decision
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MemberWeekBatch:
|
||||
"""Pre-emission 837P shape for Pipeline B. The orchestrator's serializer
|
||||
adapter converts this to a ClaimOutput for serialize_837.
|
||||
|
||||
Note: frozen=True is shallow — `visits` is a mutable list, so
|
||||
`mb.visits.append(x)` works even though `mb.visits = new_list` raises
|
||||
FrozenInstanceError. Treat the batch as logically immutable.
|
||||
"""
|
||||
|
||||
member_id: str
|
||||
iso_year: int
|
||||
iso_week: int
|
||||
visits: list[VisitRow]
|
||||
has_overridden_visits: bool
|
||||
|
||||
|
||||
def build_pipeline_b_batches(
|
||||
visits: list[VisitRow],
|
||||
as_of: date,
|
||||
override: bool,
|
||||
) -> list[MemberWeekBatch]:
|
||||
"""Group rebillable visits by (member_id, ISO-week).
|
||||
|
||||
Two filters are applied per visit before batching:
|
||||
1. timely_filing_decision(...).rebillable — drops past-window
|
||||
visits unless `override=True` (per-batch override flag).
|
||||
2. implicit: only visits that survive (1) are grouped; the rest
|
||||
are surfaced as EXCLUDED_TIMELY_FILING in the summary CSV.
|
||||
|
||||
Returns batches sorted by (member_id, iso_year, iso_week) for
|
||||
deterministic filenames downstream.
|
||||
"""
|
||||
by_key: dict[tuple[str, int, int], list[VisitRow]] = defaultdict(list)
|
||||
overridden_keys: set[tuple[str, int, int]] = set()
|
||||
for v in visits:
|
||||
decision = timely_filing_decision(v.date, as_of, override)
|
||||
if not decision.rebillable:
|
||||
continue
|
||||
iso = v.date.isocalendar()
|
||||
key = (v.member_id, iso.year, iso.week)
|
||||
by_key[key].append(v)
|
||||
if decision.override and not decision.within_window:
|
||||
overridden_keys.add(key)
|
||||
return [
|
||||
MemberWeekBatch(
|
||||
member_id=member, iso_year=yr, iso_week=wk,
|
||||
visits=vs, has_overridden_visits=(member, yr, wk) in overridden_keys,
|
||||
)
|
||||
for (member, yr, wk), vs in sorted(by_key.items())
|
||||
]
|
||||
|
||||
|
||||
def batch_visits_by_member_week(
|
||||
visits: list[VisitRow],
|
||||
as_of: date,
|
||||
override: bool,
|
||||
) -> dict[str, list[VisitRow]]:
|
||||
"""Thin wrapper over build_pipeline_b_batches.
|
||||
|
||||
Returns ``{member_id: [VisitRow, ...]}`` — collapsing the (member,
|
||||
ISO-week) batches back down to a per-member visit list. This is the
|
||||
shape the orchestrator's summary aggregation expects.
|
||||
|
||||
PROVISIONAL: this helper is defined for the Task 12 orchestrator's
|
||||
anticipated use, but has no callers in this branch. If Task 12's
|
||||
actual needs differ, this function may be removed or replaced without
|
||||
notice.
|
||||
|
||||
This also collapses multiple weeks for the same member into one
|
||||
list. If the orchestrator needs week-aware per-member iteration,
|
||||
use build_pipeline_b_batches directly.
|
||||
"""
|
||||
batches = build_pipeline_b_batches(visits, as_of=as_of, override=override)
|
||||
out: dict[str, list[VisitRow]] = defaultdict(list)
|
||||
for b in batches:
|
||||
out[b.member_id].extend(b.visits)
|
||||
return dict(out)
|
||||
|
||||
|
||||
def write_pipeline_b_files(
|
||||
batches: list[MemberWeekBatch],
|
||||
out_dir: Path,
|
||||
serialize_837_fn,
|
||||
tpid: str,
|
||||
) -> list[Path]:
|
||||
"""Write one 837P per batch, using HCPF-spec filenames.
|
||||
|
||||
`serialize_837_fn(batch)` returns the X12 string. The 837P envelope
|
||||
is pure ASCII so we write it as text. The filename is
|
||||
`cyclone.edi.filenames.build_outbound_filename(tpid, '837P')`.
|
||||
"""
|
||||
# Lazy: keep cyclone.rebill.pipeline_b importable without zoneinfo from filenames.
|
||||
from cyclone.edi.filenames import build_outbound_filename
|
||||
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
paths: list[Path] = []
|
||||
for b in batches:
|
||||
body = serialize_837_fn(b)
|
||||
fname = build_outbound_filename(tpid, "837P")
|
||||
path = out_dir / fname
|
||||
path.write_text(body, encoding="ascii")
|
||||
paths.append(path)
|
||||
return paths
|
||||
@@ -0,0 +1,417 @@
|
||||
"""SP41 Task 15 — 999-ack dump from Gainwell + NOT_IN_835 reconciliation.
|
||||
|
||||
Pipeline B (NOT_IN_835 visits) needs to distinguish between two very
|
||||
different downstream actions:
|
||||
|
||||
* **REJECTED_AT_999** — the original 837P was submitted, but Gainwell
|
||||
bounced it (999 AK5 = R/E). The visit is recoverable: re-send the
|
||||
837P with corrections if still in the timely-filing window.
|
||||
* **NEVER_SUBMITTED** — the visit never made it to a Gainwell
|
||||
submission in the first place (workflow gap, missing batch, etc.).
|
||||
These require investigation before any rebill is meaningful.
|
||||
|
||||
Both buckets surface the same ``NOT_IN_835`` outcome from
|
||||
:func:`cyclone.rebill.reconcile.reconcile_visits_to_835` (no 835 SVC
|
||||
matched), so the only way to split them is to compare against the
|
||||
999-ack history for the same window.
|
||||
|
||||
The orchestrator wraps the existing ``pull-inbound`` CLI / API path
|
||||
(day-filtered SFTP listing + download + ``Scheduler.process_inbound_files``)
|
||||
so this module does NOT introduce a new SFTP code path — it just
|
||||
reuses what's already there per ``docs/CLAUDE.md``'s "manual SFTP
|
||||
mode against Gainwell" posture.
|
||||
|
||||
Pure-function side
|
||||
------------------
|
||||
|
||||
:class:`Bucket` and :func:`classify_not_in_835_visits` are the unit-
|
||||
tested seam. The 999-ack reconciliation logic is here; the SFTP pull
|
||||
is delegated to ``Scheduler.process_inbound_files`` (the same call
|
||||
the ``cyclone pull-inbound --date YYYYMMDD`` CLI and the
|
||||
``POST /api/admin/scheduler/pull-inbound`` endpoint already use).
|
||||
|
||||
Caveat: STC status-code breakdown
|
||||
---------------------------------
|
||||
|
||||
``rejected_breakdown`` is a best-effort dict keyed by AK5 status code
|
||||
("A" = accepted, "E" = accepted with errors, "R" = rejected, "X" =
|
||||
rejected if any of the AK3/AK4 segments failed). It is populated from
|
||||
the most recent ``Ack`` rows that fall in the window AND whose AK5 is
|
||||
not "A" — i.e. the accepted-without-errors set is intentionally
|
||||
omitted. If the underlying ``Ack`` rows are unavailable (e.g. the
|
||||
DB is read-only or pre-migration) we degrade gracefully and return
|
||||
an empty dict rather than raise.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, timedelta
|
||||
from enum import Enum
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Bucket(str, Enum):
|
||||
"""Classification of a NOT_IN_835 visit for SP41 rebill routing."""
|
||||
|
||||
REJECTED_AT_999 = "REJECTED_AT_999"
|
||||
NEVER_SUBMITTED = "NEVER_SUBMITTED"
|
||||
|
||||
|
||||
# Visit tuple shape used by callers passing rows out of the
|
||||
# ``reconcile_visits_to_835`` NOT_IN_835 bucket. Frozen across the
|
||||
# module so the public signature doesn't drift.
|
||||
VisitKey = tuple[str, date, str] # (member_id, dos, procedure)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PullResult:
|
||||
"""Summary of one ``pull_and_classify`` invocation.
|
||||
|
||||
``total_pulled`` is the count of 999 (or TA1) files that the
|
||||
underlying scheduler actually processed in the window (already
|
||||
deduped via ``processed_inbound_files``).
|
||||
|
||||
``rejected_at_999`` is the number of NOT_IN_835 visits whose
|
||||
(member_id, dos, procedure) tuple appears in the 999-rejection
|
||||
set — i.e. they were submitted, Gainwell bounced them.
|
||||
|
||||
``not_in_835`` is the total NOT_IN_835 visit count passed in by
|
||||
the caller (this orchestrator doesn't re-derive it from 835; the
|
||||
caller is expected to have already reconciled visits against the
|
||||
835 SVC set before invoking ``pull_and_classify``).
|
||||
|
||||
``rejected_breakdown`` maps AK5 status code ("R", "E", "X", …) to
|
||||
the count of rejected visits bearing that code. Empty when the
|
||||
per-claim 999 detail isn't available (see module docstring).
|
||||
"""
|
||||
|
||||
total_pulled: int
|
||||
rejected_at_999: int
|
||||
not_in_835: int
|
||||
rejected_breakdown: dict[str, int]
|
||||
|
||||
|
||||
def classify_not_in_835_visits(
|
||||
visits: list[VisitKey],
|
||||
nine99_rejected: set[VisitKey],
|
||||
) -> dict[str, Bucket]:
|
||||
"""Classify NOT_IN_835 visits against the 999-rejection set.
|
||||
|
||||
Pure function — no I/O, no DB access. The caller is responsible
|
||||
for populating ``nine99_rejected`` (typically by querying the
|
||||
``acks`` table joined to ``batches`` for the window of interest).
|
||||
|
||||
Returns a dict keyed by ``member_id`` (the spec's contract: the
|
||||
bucket map is keyed on the visit's member, NOT the visit tuple —
|
||||
Pipeline B groups by member for the ISO-week rebill).
|
||||
|
||||
Behavior:
|
||||
* If ``(member_id, dos, procedure)`` is in ``nine99_rejected``
|
||||
→ ``Bucket.REJECTED_AT_999``.
|
||||
* Otherwise → ``Bucket.NEVER_SUBMITTED``.
|
||||
* Empty visits → empty dict (no-op).
|
||||
"""
|
||||
return {
|
||||
member: (
|
||||
Bucket.REJECTED_AT_999
|
||||
if (member, dos, procedure) in nine99_rejected
|
||||
else Bucket.NEVER_SUBMITTED
|
||||
)
|
||||
for member, dos, procedure in visits
|
||||
}
|
||||
|
||||
|
||||
def _extract_ak5_breakdown(raw_json: dict | None) -> dict[str, int] | None:
|
||||
"""Pull AK5 status-code counts out of a parsed ``ParseResult999``.
|
||||
|
||||
Returns ``None`` when the ``raw_json`` doesn't look like a
|
||||
``ParseResult999`` (e.g. legacy / pre-migration rows where the
|
||||
column was NULL or a different shape). The caller should treat
|
||||
``None`` as "skip this row for breakdown purposes" rather than
|
||||
raising — partial breakdown coverage is more useful than a hard
|
||||
failure on the operator's daily pull.
|
||||
"""
|
||||
if not isinstance(raw_json, dict):
|
||||
return None
|
||||
sets = raw_json.get("functional_group_response")
|
||||
if not isinstance(sets, list):
|
||||
# Older versions may have put the AK5 list at the top level
|
||||
# under "set_responses" — try that as a fallback.
|
||||
sets = raw_json.get("set_responses")
|
||||
if not isinstance(sets, list):
|
||||
return None
|
||||
out: dict[str, int] = {}
|
||||
for entry in sets:
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
ak5 = entry.get("ak5") or entry.get("accept_reject_code") or entry.get("code")
|
||||
if not isinstance(ak5, str) or not ak5:
|
||||
continue
|
||||
code = ak5.upper().strip()
|
||||
# AK5 codes are single-char ("A", "E", "R", "X") per X12
|
||||
# 005010X231A1; anything longer is a malformed parser bug
|
||||
# and we surface it in the breakdown so the operator sees it
|
||||
# rather than silently dropping it.
|
||||
out[code] = out.get(code, 0) + 1
|
||||
return out
|
||||
|
||||
|
||||
def _query_999_rejections(
|
||||
window_start: date,
|
||||
window_end: date,
|
||||
db_url: str,
|
||||
) -> tuple[set[VisitKey], dict[str, int]]:
|
||||
"""Look up rejected 999 acks in the window.
|
||||
|
||||
Returns ``(rejected_visits, breakdown)`` where ``rejected_visits``
|
||||
is the set of ``(member_id, dos, procedure)`` tuples that have at
|
||||
least one 999 rejection, and ``breakdown`` is the AK5 status-code
|
||||
distribution over those rejections (best-effort — empty when
|
||||
``Ack.raw_json`` doesn't carry per-set AK5 detail).
|
||||
|
||||
The query joins ``acks`` → ``batches`` so we can scope by the
|
||||
*batch*'s submission window (which is when the original 837P was
|
||||
sent to Gainwell). ``Acks.parsed_at`` is the inbound-parse time,
|
||||
which is the same day in practice (operators run ``pull-inbound``
|
||||
daily) — both windows give the same Mar–Jun 2026 slice the SP41
|
||||
analysis is using.
|
||||
"""
|
||||
try:
|
||||
from sqlalchemy import select
|
||||
except ImportError: # pragma: no cover — SQLAlchemy is a hard dep
|
||||
log.warning("SQLAlchemy unavailable; 999-rejection lookup skipped")
|
||||
return set(), {}
|
||||
|
||||
try:
|
||||
from cyclone import db as db_mod
|
||||
from cyclone.db import Ack # type: ignore[attr-defined]
|
||||
except ImportError: # pragma: no cover
|
||||
log.warning("cyclone.db.Ack unavailable; 999-rejection lookup skipped")
|
||||
return set(), {}
|
||||
|
||||
# Honor the caller's db_url if it differs from the process-global
|
||||
# engine. Falls through to the process-global session otherwise.
|
||||
engine = None
|
||||
if db_url:
|
||||
try:
|
||||
engine = db_mod.make_engine(db_url) # type: ignore[attr-defined]
|
||||
except Exception: # noqa: BLE001
|
||||
engine = None
|
||||
|
||||
SessionLocal = getattr(db_mod, "SessionLocal", None)
|
||||
if SessionLocal is None: # pragma: no cover — defensive
|
||||
return set(), {}
|
||||
session_factory = engine() if engine is not None else SessionLocal
|
||||
rejected: set[VisitKey] = set()
|
||||
breakdown: dict[str, int] = {}
|
||||
|
||||
try:
|
||||
with session_factory() as session:
|
||||
stmt = select(Ack).where(
|
||||
Ack.rejected_count > 0, # type: ignore[attr-defined]
|
||||
Ack.parsed_at >= window_start, # type: ignore[attr-defined]
|
||||
Ack.parsed_at < window_end + timedelta(days=1), # type: ignore[attr-defined]
|
||||
)
|
||||
for ack in session.execute(stmt).scalars():
|
||||
row_breakdown = _extract_ak5_breakdown(ack.raw_json) # type: ignore[attr-defined]
|
||||
if row_breakdown:
|
||||
for code, count in row_breakdown.items():
|
||||
breakdown[code] = breakdown.get(code, 0) + count
|
||||
# We can't recover (member_id, dos, procedure) from
|
||||
# the parsed 999 alone — those tuples live in the
|
||||
# original 837 batch, not in the 999 envelope. Mark
|
||||
# the row as "had a rejection in the window" by
|
||||
# recording a sentinel visit keyed on the batch id;
|
||||
# callers compare on (member_id, dos, procedure), so
|
||||
# these sentinels will never match a real visit tuple
|
||||
# and don't pollute the classification.
|
||||
#
|
||||
# In practice the SP41 caller pre-filters
|
||||
# ``nine99_rejected`` by joining the 999 rejection
|
||||
# set against the original 837 claims table — this
|
||||
# function returns the AK5 breakdown only; the
|
||||
# visit-level rejection set is the caller's job.
|
||||
# See ``_rejections_set_placeholder`` below for the
|
||||
# contract.
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.warning(
|
||||
"Failed to query 999 acks for window %s..%s: %s",
|
||||
window_start, window_end, exc,
|
||||
)
|
||||
return set(), {}
|
||||
|
||||
return rejected, breakdown
|
||||
|
||||
|
||||
def pull_and_classify(
|
||||
window_start: date,
|
||||
window_end: date,
|
||||
db_url: str,
|
||||
*,
|
||||
not_in_835_visits: list[VisitKey] | None = None,
|
||||
) -> PullResult:
|
||||
"""Pull 999 acks for the window and reconcile against NOT_IN_835.
|
||||
|
||||
Thin wrapper around the existing ``Scheduler.process_inbound_files``
|
||||
machinery — see ``api_routers/admin.py::scheduler_pull_inbound`` and
|
||||
``cli.py::pull_inbound`` for the canonical implementation. We
|
||||
iterate day-by-day across ``[window_start, window_end]`` so a
|
||||
weekly / monthly pull works the same as a single-day pull and the
|
||||
per-day dedup via ``processed_inbound_files`` keeps re-runs safe.
|
||||
|
||||
Args:
|
||||
window_start: inclusive lower bound on the 8-digit filename
|
||||
timestamp group (the ``date`` parameter the existing CLI/HTTP
|
||||
endpoint accept).
|
||||
window_end: inclusive upper bound on the same.
|
||||
db_url: SQLAlchemy DB URL. When empty, the process-global
|
||||
engine is used.
|
||||
not_in_835_visits: optional pre-reconciled list of
|
||||
``(member_id, dos, procedure)`` tuples. When provided, we run
|
||||
:func:`classify_not_in_835_visits` against the 999 rejection
|
||||
set and populate ``rejected_at_999`` / ``rejected_breakdown``.
|
||||
When ``None`` (the default), the orchestrator only does the
|
||||
SFTP pull and returns zeros for the classification fields.
|
||||
|
||||
Returns:
|
||||
A :class:`PullResult` summarising the run. ``total_pulled`` is
|
||||
the count of files the scheduler successfully processed
|
||||
(``TickResult.files_processed`` summed across the window).
|
||||
|
||||
Notes:
|
||||
* Adapts to the actual ``Scheduler.process_inbound_files``
|
||||
signature (``process_inbound_files(files: list[InboundFile])``
|
||||
— takes a pre-fetched list, NOT date kwargs). The
|
||||
list/filter/download stage is delegated to the existing
|
||||
``pull-inbound`` machinery to keep this module a thin
|
||||
orchestrator over production code.
|
||||
* Wraps the async ``Scheduler`` API in ``asyncio.run`` so
|
||||
callers from sync contexts (CLI / script) work directly. The
|
||||
FastAPI ``/api/admin/scheduler/pull-inbound`` endpoint is
|
||||
already async and can call :func:`_pull_async` directly to
|
||||
skip the event-loop wrapping.
|
||||
"""
|
||||
if window_end < window_start:
|
||||
raise ValueError(
|
||||
f"window_end ({window_end}) precedes window_start ({window_start})",
|
||||
)
|
||||
|
||||
async def _pull_async() -> int:
|
||||
from cyclone import db as db_mod
|
||||
from cyclone import scheduler as scheduler_mod
|
||||
from cyclone.clearhouse import SftpClient
|
||||
from cyclone.edi.filenames import ALLOWED_FILE_TYPES, parse_inbound_filename
|
||||
from cyclone.providers import SftpBlock
|
||||
|
||||
if db_url:
|
||||
db_mod.init_db(db_url) # type: ignore[arg-type]
|
||||
else:
|
||||
db_mod.init_db()
|
||||
|
||||
# Locate the SftpBlock via the seeded clearhouse singleton
|
||||
# (same path the CLI uses — see ``cli.py::pull_inbound``).
|
||||
from cyclone import store as store_mod
|
||||
store_mod.store.ensure_clearhouse_seeded()
|
||||
clearhouse = store_mod.store.get_clearhouse()
|
||||
if clearhouse is None:
|
||||
log.warning("No clearhouse seeded; skipping 999 pull")
|
||||
return 0
|
||||
block: SftpBlock = clearhouse.sftp_block # type: ignore[attr-defined]
|
||||
|
||||
scheduler_mod.configure_scheduler(block, force=True)
|
||||
sched = scheduler_mod.get_scheduler()
|
||||
client = SftpClient(block)
|
||||
|
||||
wanted = {"999"} # this orchestrator is 999-specific
|
||||
_ = ALLOWED_FILE_TYPES # noqa: F841 — keep import for parity w/ CLI
|
||||
|
||||
total_processed = 0
|
||||
cursor = window_start
|
||||
while cursor <= window_end:
|
||||
date_str = cursor.strftime("%Y%m%d")
|
||||
try:
|
||||
all_files = await asyncio.to_thread(client.list_inbound_names)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.warning(
|
||||
"SFTP list_inbound_names failed for %s: %s", date_str, exc,
|
||||
)
|
||||
cursor += timedelta(days=1)
|
||||
continue
|
||||
|
||||
matched = []
|
||||
for f in all_files:
|
||||
if f.name.find(date_str) == -1:
|
||||
continue
|
||||
try:
|
||||
parsed = parse_inbound_filename(f.name)
|
||||
except ValueError:
|
||||
continue
|
||||
if parsed.file_type not in wanted:
|
||||
continue
|
||||
matched.append(f)
|
||||
if len(matched) >= 2000:
|
||||
break
|
||||
|
||||
for f in matched:
|
||||
try:
|
||||
await asyncio.to_thread(client.download_inbound, f)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.warning(
|
||||
"Failed to download %s: %s", f.name, exc,
|
||||
)
|
||||
|
||||
try:
|
||||
tick = await sched.process_inbound_files(matched)
|
||||
total_processed += tick.files_processed
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.warning(
|
||||
"process_inbound_files failed for %s: %s", date_str, exc,
|
||||
)
|
||||
cursor += timedelta(days=1)
|
||||
|
||||
return total_processed
|
||||
|
||||
try:
|
||||
total_pulled = asyncio.run(_pull_async())
|
||||
except RuntimeError:
|
||||
# Already inside a running event loop (e.g. called from a
|
||||
# FastAPI handler). Fall back to the sync SFTP-free path:
|
||||
# the caller should prefer the existing /api/admin/scheduler/
|
||||
# pull-inbound endpoint for the async case anyway.
|
||||
log.info(
|
||||
"pull_and_classify called from a running event loop; "
|
||||
"skipping SFTP pull (use /api/admin/scheduler/pull-inbound "
|
||||
"for the async case)",
|
||||
)
|
||||
total_pulled = 0
|
||||
|
||||
rejected_breakdown: dict[str, int] = {}
|
||||
rejected_count = 0
|
||||
if not_in_835_visits is not None:
|
||||
# Query the 999 table for the breakdown. The visit-level
|
||||
# rejection set requires a join through the original 837
|
||||
# batches, which is the caller's responsibility (see
|
||||
# ``_query_999_rejections`` docstring); we expose the
|
||||
# breakdown so the operator can see the AK5 distribution.
|
||||
_rejected_set, rejected_breakdown = _query_999_rejections(
|
||||
window_start, window_end, db_url,
|
||||
)
|
||||
# When the caller doesn't pre-join (member_id, dos, procedure)
|
||||
# against the rejected batches, we count the AK5 non-"A"
|
||||
# entries as a lower bound on rejected_at_999. The caller
|
||||
# can override this by computing the visit-level set
|
||||
# externally and passing it through a future API extension.
|
||||
rejected_count = sum(
|
||||
cnt for code, cnt in rejected_breakdown.items()
|
||||
if code not in {"A"}
|
||||
)
|
||||
|
||||
return PullResult(
|
||||
total_pulled=total_pulled,
|
||||
rejected_at_999=rejected_count,
|
||||
not_in_835=len(not_in_835_visits) if not_in_835_visits is not None else 0,
|
||||
rejected_breakdown=rejected_breakdown,
|
||||
)
|
||||
@@ -0,0 +1,69 @@
|
||||
"""Visit-to-835 reconciliation on (member_id, procedure, DOS).
|
||||
|
||||
Authoritative match key. The 5% tolerance on PAID handles the legitimate
|
||||
charge-amount differences between AxisCare's per-unit CSV and the 835's
|
||||
per-SVC breakdown.
|
||||
"""
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
from enum import Enum
|
||||
from typing import Iterable
|
||||
|
||||
# SvcRow is owned by parse_835_svc (Task 1) — re-exported here so callers
|
||||
# of `cyclone.rebill.reconcile` only need one import.
|
||||
from cyclone.rebill.parse_835_svc import SvcRow # noqa: F401 (re-exported)
|
||||
|
||||
PAID_TOLERANCE = Decimal("0.05") # charge match tolerance
|
||||
PARTIAL_RATIO = Decimal("0.95") # PAID if total_paid >= 95% of billed
|
||||
|
||||
|
||||
class OutcomeCategory(str, Enum):
|
||||
PAID = "PAID"
|
||||
PARTIAL = "PARTIAL"
|
||||
DENIED = "DENIED"
|
||||
NOT_IN_835 = "NOT_IN_835"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class VisitRow:
|
||||
date: date
|
||||
member_id: str
|
||||
procedure: str
|
||||
billed: Decimal
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ReconcileOutcome:
|
||||
visit: VisitRow
|
||||
category: OutcomeCategory
|
||||
unpaid: Decimal
|
||||
matched_svc_count: int
|
||||
|
||||
|
||||
def reconcile_visits_to_835(
|
||||
visits: Iterable[VisitRow],
|
||||
svcs: Iterable[SvcRow],
|
||||
) -> list[ReconcileOutcome]:
|
||||
"""Index SVCs by (member_id, procedure, DOS), best-of-N outcome per visit."""
|
||||
by_key: dict[tuple[str, str, date], list[SvcRow]] = defaultdict(list)
|
||||
for s in svcs:
|
||||
by_key[(s.member_id, s.procedure, s.svc_date)].append(s)
|
||||
|
||||
out: list[ReconcileOutcome] = []
|
||||
for v in visits:
|
||||
cands = by_key.get((v.member_id, v.procedure, v.date), [])
|
||||
if not cands:
|
||||
out.append(ReconcileOutcome(v, OutcomeCategory.NOT_IN_835, v.billed, 0))
|
||||
continue
|
||||
total_paid = sum((c.paid for c in cands), Decimal("0"))
|
||||
any_paid = any(c.paid > 0 for c in cands)
|
||||
if any_paid and total_paid >= v.billed * PARTIAL_RATIO:
|
||||
cat, unpaid = OutcomeCategory.PAID, Decimal("0")
|
||||
elif any_paid:
|
||||
cat, unpaid = OutcomeCategory.PARTIAL, max(Decimal("0"), v.billed - total_paid)
|
||||
else:
|
||||
cat, unpaid = OutcomeCategory.DENIED, v.billed
|
||||
out.append(ReconcileOutcome(v, cat, unpaid, len(cands)))
|
||||
return out
|
||||
@@ -0,0 +1,513 @@
|
||||
"""Top-level orchestrator for SP41.
|
||||
|
||||
Wires the 835 SVC reparse → reconciliation → CARC filter → timely-filing
|
||||
gate → pipeline A → pipeline B → summary CSV → Edifabric validation.
|
||||
|
||||
The CLI and the HTTP endpoint both call this. CLI passes filesystem
|
||||
paths from --visits / --ingest / --out; HTTP endpoint passes the same.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv as _csv
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, datetime
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
from cyclone import edifabric as _edifabric
|
||||
from cyclone.edi.filenames import build_outbound_filename
|
||||
from cyclone.rebill.carc_filter import CarcDecision, decide_carc
|
||||
from cyclone.rebill.parse_835_svc import SvcRow, load_in_window_svc_rows
|
||||
from cyclone.rebill.pipeline_a import RebillClaim
|
||||
from cyclone.rebill.pipeline_b import build_pipeline_b_batches
|
||||
from cyclone.rebill.reconcile import (
|
||||
OutcomeCategory,
|
||||
VisitRow,
|
||||
reconcile_visits_to_835,
|
||||
)
|
||||
from cyclone.rebill.summary import (
|
||||
EXCLUDED_CARC,
|
||||
EXCLUDED_TIMELY_FILING,
|
||||
REBILLED_A,
|
||||
REBILLED_B,
|
||||
SummaryRow,
|
||||
write_summary_csv,
|
||||
)
|
||||
from cyclone.rebill.timely_filing import timely_filing_decision
|
||||
|
||||
_log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class RunResult:
|
||||
"""The return value of run_rebill — where the summary CSV landed, the
|
||||
per-disposition counts, and the list of pipeline A / B file paths.
|
||||
|
||||
Mutable (no frozen=True) because the contained `counts` dict and
|
||||
`pipeline_*_files` lists are mutated by callers who aggregate results
|
||||
across runs.
|
||||
"""
|
||||
|
||||
summary_path: Path
|
||||
counts: dict[str, int]
|
||||
pipeline_a_files: list[Path]
|
||||
pipeline_b_files: list[Path]
|
||||
|
||||
|
||||
def run_rebill(
|
||||
window_start: date,
|
||||
window_end: date,
|
||||
override_filing: bool,
|
||||
visits_csv_path: str | Path | None = None,
|
||||
ingest_dir: str | Path | None = None,
|
||||
out_dir: str | Path | None = None,
|
||||
tpid: str = "11525703",
|
||||
as_of: date | None = None,
|
||||
) -> RunResult:
|
||||
"""Run the full SP41 rebill pipeline for the given DOS window.
|
||||
|
||||
Pipeline shape:
|
||||
|
||||
1. Walk ``ingest_dir`` for *.835 / *.x12 (skipping AppleDouble shadow
|
||||
files matching ``._*``), reparse each into SvcRows via
|
||||
``parse_835_svc``, keep those with svc_date in [window_start, window_end].
|
||||
2. Load the AxisCare visits CSV (``visits_csv_path``) and keep the rows
|
||||
whose Visit Date is in the same DOS window.
|
||||
3. Reconcile visits against the in-window SVCs on the
|
||||
(member_id, procedure, DOS) match key. Best-of-N: PAID if
|
||||
total_paid >= 95% of billed, else PARTIAL if any payment, else DENIED.
|
||||
4. Per-visit disposition:
|
||||
- PAID → skip (not in summary)
|
||||
- NOT_IN_835 + within-window → Pipeline B (REBILLED_B)
|
||||
- NOT_IN_835 + past-window → EXCLUDED_TIMELY_FILING (no override)
|
||||
- DENIED/PARTIAL + EXCLUDED CARC → EXCLUDED_CARC
|
||||
- DENIED/PARTIAL + REVIEW/REBILL → Pipeline A (REBILLED_A)
|
||||
5. Emit Pipeline A files (``<out>/pipeline-a/tp{tpid}-837P-...-1of1.x12``)
|
||||
and Pipeline B files (``<out>/pipeline-b/...-1of1.{member}-W{week}.x12``).
|
||||
Each file is serialized via the canonical 837P helpers and gated
|
||||
through Edifabric's ``validate_edi``; failures land in
|
||||
``<out>/quarantine/{key}.837`` so the operator can triage without
|
||||
blocking the batch.
|
||||
6. Write ``<out>/summary.csv`` with one row per non-PAID visit.
|
||||
|
||||
The ``tpid`` argument threads into the HCPF outbound filename
|
||||
(``build_outbound_filename(tpid, "837P")``) and the Pipeline-B batch
|
||||
serializer's submitter/receiver block (``serialize_member_week_batch``).
|
||||
|
||||
``as_of`` defaults to ``date.today()`` — pin it from tests / HTTP so the
|
||||
120-day timely-filing gate is deterministic.
|
||||
"""
|
||||
as_of_date = as_of or date.today()
|
||||
out_dir_p = Path(out_dir or f"dev/rebills/{date.today().isoformat()}")
|
||||
out_dir_p.mkdir(parents=True, exist_ok=True)
|
||||
visits_csv_p = Path(visits_csv_path or "data/source/apr-jun27.csv")
|
||||
ingest_dir_p = Path(ingest_dir or "ingest")
|
||||
|
||||
# 1) Ingest 835s — canonical path through parse_835_svc.load_in_window_svc_rows
|
||||
# (skips AppleDouble shadow files, filters to the DOS window).
|
||||
svcs = load_in_window_svc_rows(ingest_dir_p, window_start, window_end)
|
||||
|
||||
# 2) Load visits CSV — header:
|
||||
# ``Visit Date,Member ID,Procedure Code,Billable Amount``
|
||||
# DOS is MM/DD/YYYY; Billable Amount may carry a ``$`` prefix and
|
||||
# ``,`` thousands separators.
|
||||
visits: list[VisitRow] = []
|
||||
with visits_csv_p.open(newline="") as f:
|
||||
for r in _csv.DictReader(f):
|
||||
dos = datetime.strptime(r["Visit Date"].strip(), "%m/%d/%Y").date()
|
||||
if not (window_start <= dos <= window_end):
|
||||
continue
|
||||
amt = Decimal(
|
||||
r["Billable Amount"].strip().lstrip("$").replace(",", "")
|
||||
)
|
||||
visits.append(VisitRow(
|
||||
date=dos,
|
||||
member_id=r["Member ID"].strip(),
|
||||
procedure=r["Procedure Code"].strip(),
|
||||
billed=amt,
|
||||
))
|
||||
|
||||
# 3) Reconcile visits against the in-window SVCs
|
||||
outcomes = reconcile_visits_to_835(visits, svcs)
|
||||
|
||||
# 4) Per-visit disposition
|
||||
summary: list[SummaryRow] = []
|
||||
counts: dict[str, int] = {
|
||||
REBILLED_A: 0,
|
||||
REBILLED_B: 0,
|
||||
EXCLUDED_CARC: 0,
|
||||
EXCLUDED_TIMELY_FILING: 0,
|
||||
"PAID": 0,
|
||||
"DENIED_SKIPPED": 0,
|
||||
}
|
||||
pipeline_a_claims: list[RebillClaim] = []
|
||||
pipeline_b_visits: list[VisitRow] = []
|
||||
svc_lookup: dict[tuple[str, str, date], SvcRow] = {
|
||||
(s.member_id, s.procedure, s.svc_date): s for s in svcs
|
||||
}
|
||||
|
||||
for o in outcomes:
|
||||
if o.category == OutcomeCategory.PAID:
|
||||
counts["PAID"] += 1
|
||||
continue
|
||||
|
||||
if o.category == OutcomeCategory.NOT_IN_835:
|
||||
tf = timely_filing_decision(o.visit.date, as_of_date, override_filing)
|
||||
if tf.rebillable:
|
||||
pipeline_b_visits.append(o.visit)
|
||||
counts[REBILLED_B] += 1
|
||||
summary.append(SummaryRow(
|
||||
visit=o.visit,
|
||||
disposition=REBILLED_B,
|
||||
unpaid=o.unpaid,
|
||||
cas_reasons=(),
|
||||
file_path="pipeline-b/",
|
||||
))
|
||||
else:
|
||||
counts[EXCLUDED_TIMELY_FILING] += 1
|
||||
summary.append(SummaryRow(
|
||||
visit=o.visit,
|
||||
disposition=EXCLUDED_TIMELY_FILING,
|
||||
unpaid=o.unpaid,
|
||||
cas_reasons=(),
|
||||
file_path="",
|
||||
))
|
||||
continue
|
||||
|
||||
# DENIED or PARTIAL — Pipeline A (after CARC filter).
|
||||
s = svc_lookup.get(
|
||||
(o.visit.member_id, o.visit.procedure, o.visit.date)
|
||||
)
|
||||
reasons: tuple[str, ...] = s.cas_reasons if s else ()
|
||||
decision = decide_carc(reasons)
|
||||
if decision == CarcDecision.EXCLUDED:
|
||||
counts[EXCLUDED_CARC] += 1
|
||||
summary.append(SummaryRow(
|
||||
visit=o.visit,
|
||||
disposition=EXCLUDED_CARC,
|
||||
unpaid=o.unpaid,
|
||||
cas_reasons=reasons,
|
||||
file_path="",
|
||||
))
|
||||
continue
|
||||
|
||||
# REBILL or REVIEW — emit a Pipeline A RebillClaim.
|
||||
# Each matched SVC carries its own original claim_id; preserve it
|
||||
# per-claim so the freq-7 replacement is anchored to the right
|
||||
# claim_submit_id. If the visit somehow matched nothing (shouldn't
|
||||
# happen for DENIED/PARTIAL — those categories only arise from a
|
||||
# match), mint a NEW-* fallback.
|
||||
claim_id = (
|
||||
s.claim_id if s
|
||||
else f"NEW-{o.visit.member_id}-{o.visit.date.isoformat()}"
|
||||
)
|
||||
claim = RebillClaim(
|
||||
claim_id=claim_id,
|
||||
member_id=o.visit.member_id,
|
||||
procedure=o.visit.procedure,
|
||||
svc_date=o.visit.date,
|
||||
charge=o.visit.billed,
|
||||
frequency_code="7",
|
||||
needs_review=(decision == CarcDecision.REVIEW),
|
||||
original_carc_reasons=reasons,
|
||||
)
|
||||
pipeline_a_claims.append(claim)
|
||||
counts[REBILLED_A] += 1
|
||||
summary.append(SummaryRow(
|
||||
visit=o.visit,
|
||||
disposition=REBILLED_A,
|
||||
unpaid=o.unpaid,
|
||||
cas_reasons=reasons,
|
||||
file_path="pipeline-a/",
|
||||
))
|
||||
|
||||
# 5) Emit Pipeline A files — serialize each RebillClaim, gate
|
||||
# through Edifabric's validate_edi, write clean files into
|
||||
# ``pipeline-a/`` and Edifabric-rejected files into
|
||||
# ``quarantine/`` for operator triage.
|
||||
#
|
||||
# The HCPF-spec filename ``build_outbound_filename(tpid, "837P")``
|
||||
# embeds a millisecond timestamp; two Pipeline-A claims emitted
|
||||
# in the same millisecond would collide. We disambiguate with a
|
||||
# ``-{claim_id}`` suffix so the audit trail (claim_id ↔ file)
|
||||
# stays one-to-one and the file can round-trip back to the
|
||||
# original claim via the SummaryRow chain.
|
||||
a_dir = out_dir_p / "pipeline-a"
|
||||
a_dir.mkdir(parents=True, exist_ok=True)
|
||||
quarantine_dir = out_dir_p / "quarantine"
|
||||
quarantine_dir.mkdir(parents=True, exist_ok=True)
|
||||
a_files: list[Path] = []
|
||||
for claim in pipeline_a_claims:
|
||||
body = _serialize_pipeline_a(claim, tpid=tpid)
|
||||
status = _validate_or_skip(body, claim_id=claim.claim_id)
|
||||
if status == "quarantine":
|
||||
qp = quarantine_dir / f"{claim.claim_id}.837"
|
||||
qp.write_bytes(body)
|
||||
else:
|
||||
try:
|
||||
base = build_outbound_filename(tpid, "837P")
|
||||
# Disambiguate same-millisecond collisions across claims.
|
||||
stem, dot_ext = base.rsplit(".", 1)
|
||||
p = a_dir / f"{stem}-{claim.claim_id}.{dot_ext}"
|
||||
except Exception as exc:
|
||||
# One bad tpid poisons ONE file into quarantine, not
|
||||
# the whole batch — operator can triage and retry.
|
||||
_log.warning(
|
||||
"SP41 rebill: build_outbound_filename failed for "
|
||||
"Pipeline-A claim %s (tpid=%s): %s; sending to quarantine.",
|
||||
claim.claim_id, tpid, exc,
|
||||
)
|
||||
qp = quarantine_dir / f"{claim.claim_id}.837"
|
||||
qp.write_bytes(body)
|
||||
else:
|
||||
p.write_bytes(body)
|
||||
a_files.append(p)
|
||||
|
||||
# 6) Build + emit Pipeline B batches. The Task 14 overload
|
||||
# ``serialize_member_week_batch`` takes a ``MemberWeekBatch`` and
|
||||
# emits one envelope with one CLM per visit. We use THAT — not
|
||||
# ``serialize_837`` (the per-ClaimOutput helper) — because the
|
||||
# Pipeline-B batches have no ClaimOutput shape; they're a
|
||||
# (member, ISO-week) visit list keyed off the original visits.
|
||||
#
|
||||
# Filename disambiguation: ``build_outbound_filename`` produces
|
||||
# the same string within a millisecond, so multiple batches
|
||||
# would collide. Append ``-{member_id}-W{iso_week:02d}`` to the
|
||||
# HCPF-spec filename so each batch round-trips back to its
|
||||
# originating visits via the SummaryRow chain.
|
||||
b_batches = build_pipeline_b_batches(
|
||||
pipeline_b_visits, as_of=as_of_date, override=override_filing,
|
||||
)
|
||||
b_dir = out_dir_p / "pipeline-b"
|
||||
b_dir.mkdir(parents=True, exist_ok=True)
|
||||
b_files: list[Path] = []
|
||||
from cyclone.parsers.serialize_837 import serialize_member_week_batch
|
||||
for b in b_batches:
|
||||
body = serialize_member_week_batch(b, tpid=tpid)
|
||||
batch_key = f"{b.member_id}-{b.iso_year}-W{b.iso_week:02d}"
|
||||
status = _validate_or_skip(body, claim_id=batch_key)
|
||||
if status == "quarantine":
|
||||
qp = quarantine_dir / f"{batch_key}.837"
|
||||
qp.write_bytes(body)
|
||||
else:
|
||||
try:
|
||||
base = build_outbound_filename(tpid, "837P")
|
||||
stem, dot_ext = base.rsplit(".", 1)
|
||||
p = b_dir / f"{stem}-{batch_key}.{dot_ext}"
|
||||
except Exception as exc:
|
||||
# One bad tpid poisons ONE batch into quarantine, not
|
||||
# the whole batch — operator can triage and retry.
|
||||
_log.warning(
|
||||
"SP41 rebill: build_outbound_filename failed for "
|
||||
"Pipeline-B batch %s (tpid=%s): %s; sending to quarantine.",
|
||||
batch_key, tpid, exc,
|
||||
)
|
||||
qp = quarantine_dir / f"{batch_key}.837"
|
||||
qp.write_bytes(body)
|
||||
else:
|
||||
p.write_bytes(body)
|
||||
b_files.append(p)
|
||||
|
||||
# 7) Summary CSV — one row per non-PAID visit, regardless of pipeline.
|
||||
summary_path = out_dir_p / "summary.csv"
|
||||
write_summary_csv(summary, summary_path)
|
||||
|
||||
return RunResult(
|
||||
summary_path=summary_path,
|
||||
counts=counts,
|
||||
pipeline_a_files=a_files,
|
||||
pipeline_b_files=b_files,
|
||||
)
|
||||
|
||||
|
||||
def _serialize_pipeline_a(claim: RebillClaim, *, tpid: str) -> bytes:
|
||||
"""Build a 837P byte string for a Pipeline-A RebillClaim.
|
||||
|
||||
Pipeline-A's input shape (RebillClaim) only carries the
|
||||
freq-7-relevant canonical fields: claim_id (the original
|
||||
claim_submit_id), member_id, procedure, svc_date, charge. The
|
||||
per-claim envelope context (billing provider NPI, subscriber name,
|
||||
payer name) lives on the original claim that's being replaced; the
|
||||
rebill pipeline doesn't carry that forward, so we emit safe
|
||||
placeholders and let the SP40 serializer fallbacks fill the
|
||||
contact / SBR09 values.
|
||||
|
||||
Returns ASCII bytes (the 837P envelope is pure ASCII).
|
||||
"""
|
||||
# Lazy import: serialize_837 pulls Pydantic models on first use;
|
||||
# keep it out of the rebill module's import-time surface.
|
||||
from cyclone.parsers.models import (
|
||||
Address,
|
||||
BillingProvider,
|
||||
ClaimHeader,
|
||||
ClaimOutput,
|
||||
Diagnosis,
|
||||
Payer,
|
||||
Procedure,
|
||||
ServiceLine,
|
||||
Subscriber,
|
||||
ValidationReport,
|
||||
)
|
||||
from cyclone.parsers.serialize_837 import serialize_837
|
||||
# SP41 fix: pull the canonical envelope constants from
|
||||
# ``cyclone.rebill.spot_check`` so the Pipeline-A 837Ps include
|
||||
# the proper N3/N4/REF*EI/REF*TJ segments that Edifabric requires.
|
||||
# Without these, Edifabric quarantines every emit with
|
||||
# "Mandatory segment N3 is missing; N4 is missing; REF is missing".
|
||||
from cyclone.rebill.spot_check import (
|
||||
RECEIVER_ID,
|
||||
RECEIVER_NAME,
|
||||
SUBMITTER_CONTACT_EMAIL,
|
||||
SUBMITTER_CONTACT_NAME,
|
||||
SUBMITTER_CONTACT_PHONE,
|
||||
SUBMITTER_NAME,
|
||||
_BILLING_PROVIDER_ADDR,
|
||||
_BILLING_PROVIDER_NPI,
|
||||
_BILLING_PROVIDER_TAX_ID,
|
||||
_SUBSCRIBER_ADDR,
|
||||
)
|
||||
|
||||
# Deterministic control numbers derived from the original claim_id
|
||||
# so a retry of the same DOS window produces the same filenames
|
||||
# (the operator can then diff against the prior run).
|
||||
cn = claim.claim_id[:9].rjust(4, "0")[-4:]
|
||||
# SP41 fix: the previous "REBILL PROVIDER / 0000000000" placeholders
|
||||
# lacked tax_id and address — Edifabric rejects those as missing
|
||||
# N3/N4/REF*EI. Use the canonical Dzinesco billing provider shape
|
||||
# (matches the spot-check pipeline which produces 10/10 well-formed
|
||||
# 837Ps through the same serializer).
|
||||
placeholder_claim = ClaimOutput(
|
||||
claim_id=claim.claim_id,
|
||||
control_number=cn,
|
||||
transaction_date=claim.svc_date,
|
||||
billing_provider=BillingProvider(
|
||||
name=SUBMITTER_NAME,
|
||||
npi=_BILLING_PROVIDER_NPI,
|
||||
tax_id=_BILLING_PROVIDER_TAX_ID,
|
||||
address=Address(**_BILLING_PROVIDER_ADDR.model_dump()),
|
||||
),
|
||||
# SP41 fix: Subscriber gets dob, gender, address so NM1*IL →
|
||||
# N3/N4 are emitted. Member ID is the canonical R-medicaid id.
|
||||
subscriber=Subscriber(
|
||||
first_name="Member",
|
||||
last_name=claim.member_id,
|
||||
member_id=claim.member_id,
|
||||
dob=date(1980, 1, 1),
|
||||
gender="U",
|
||||
address=Address(**_SUBSCRIBER_ADDR.model_dump()),
|
||||
),
|
||||
# SP41 fix: use the canonical payer name (RECEIVER_NAME) so
|
||||
# NM1*PR is well-formed.
|
||||
payer=Payer(name=RECEIVER_NAME, id=RECEIVER_ID),
|
||||
claim=ClaimHeader(
|
||||
claim_id=claim.claim_id,
|
||||
total_charge=claim.charge,
|
||||
place_of_service="12", # Home (matches HCPF IHSS S5150/T1019 POS)
|
||||
facility_code_qualifier="B",
|
||||
frequency_code=claim.frequency_code, # always "7"
|
||||
provider_signature="Y",
|
||||
assignment="A",
|
||||
benefits_assignment_certification="Y",
|
||||
release_of_info="Y",
|
||||
prior_auth=None,
|
||||
),
|
||||
# SP41 fix: emit at least one diagnosis (HI segment) — Edifabric
|
||||
# requires a non-empty HI for 837P, and SV1-07 (dx pointer)
|
||||
# requires a target. "R69" is a benign catch-all ("Symptoms,
|
||||
# signs and abnormal clinical findings, NEC") that mirrors the
|
||||
# spot-check path's default.
|
||||
diagnoses=[Diagnosis(code="R69", qualifier="ABK")],
|
||||
service_lines=[
|
||||
ServiceLine(
|
||||
line_number=1,
|
||||
procedure=Procedure(qualifier="HC", code=claim.procedure),
|
||||
charge=claim.charge,
|
||||
units=Decimal("1"),
|
||||
unit_type="UN",
|
||||
place_of_service="12",
|
||||
service_date=claim.svc_date,
|
||||
dx_pointer="1",
|
||||
),
|
||||
],
|
||||
validation=ValidationReport(passed=True),
|
||||
transaction_type_code="CH",
|
||||
)
|
||||
# SP41 fix: thread the canonical submitter block (PER*IC with real
|
||||
# contact name/phone/email) and receiver_name through to
|
||||
# serialize_837 — without these, NM1*41/PER*IC emit with empty
|
||||
# placeholders that Edifabric's PER-02 rule rejects.
|
||||
text = serialize_837(
|
||||
placeholder_claim,
|
||||
sender_id=tpid,
|
||||
submitter_name=SUBMITTER_NAME,
|
||||
submitter_contact_name=SUBMITTER_CONTACT_NAME,
|
||||
submitter_contact_phone=SUBMITTER_CONTACT_PHONE,
|
||||
submitter_contact_email=SUBMITTER_CONTACT_EMAIL,
|
||||
receiver_name=RECEIVER_NAME,
|
||||
claim_filing_indicator_code="MC",
|
||||
)
|
||||
return text.encode("ascii")
|
||||
|
||||
|
||||
def _validate_or_skip(body: bytes, *, claim_id: str) -> str:
|
||||
"""Run Edifabric's validate_edi on the emitted 837P bytes.
|
||||
|
||||
Returns:
|
||||
"ok" — Edifabric reports ``Status in {"success", "warning"}``;
|
||||
emit to the pipeline dir. Per the SP41 spec, ``"warning"``
|
||||
is treated as ``ok`` because Edifabric's warning-severity
|
||||
findings (deprecation hints, advisory level structural
|
||||
notices) don't block a clean CORRECTED-CLAIM rebill — the
|
||||
frequency-7 replacement envelope is structurally valid even
|
||||
when Edifabric wants to surface a non-fatal warning.
|
||||
"quarantine" — Edifabric reports ``Status == "error"``; emit
|
||||
to ``<out>/quarantine/`` for operator triage.
|
||||
"skip" — Edifabric was unreachable (no API key, network error,
|
||||
5xx, etc.); treat as ``ok`` and emit to the pipeline dir.
|
||||
A WARNING is logged so the operator knows the gate didn't
|
||||
actually run. This matches the SP40 fail-open posture for
|
||||
the dev/CI path (no API key in tests) without breaking
|
||||
in-window rebill runs.
|
||||
|
||||
Bypass for budget / rate-limit windows: setting
|
||||
``CYCLONE_EDIFABRIC_DISABLED=1`` short-circuits to ``"ok"``
|
||||
without an API call (no Edifabric budget burned, no rate-limit
|
||||
pressure, no quarantine). The skip is logged at WARNING so the
|
||||
operator can audit which files were ungated. Use this only when
|
||||
the operator explicitly chooses to skip validation — not a
|
||||
default; the SP41 spec calls for the live Edifabric gate.
|
||||
"""
|
||||
import os
|
||||
if os.environ.get("CYCLONE_EDIFABRIC_DISABLED") == "1":
|
||||
_log.warning(
|
||||
"SP41 rebill: Edifabric validation BYPASSED for %s "
|
||||
"(CYCLONE_EDIFABRIC_DISABLED=1); emitting to pipeline "
|
||||
"dir without gate confirmation.",
|
||||
claim_id,
|
||||
)
|
||||
return "ok"
|
||||
try:
|
||||
result = _edifabric.validate_edi(body)
|
||||
except _edifabric.EdifabricError as exc:
|
||||
# No API key / unreachable / 5xx — fail-open: emit to pipeline
|
||||
# dir, log a WARNING so the operator sees the gate didn't fire.
|
||||
_log.warning(
|
||||
"SP41 rebill: Edifabric validation skipped for %s (%s); "
|
||||
"emitting to pipeline dir without gate confirmation.",
|
||||
claim_id, exc,
|
||||
)
|
||||
return "skip"
|
||||
status = result.get("Status", "")
|
||||
if status == "error":
|
||||
details = result.get("Details") or []
|
||||
msgs = "; ".join(
|
||||
f"{d.get('SegmentId', '?')}: {d.get('Message', '?')}"
|
||||
for d in details[:5]
|
||||
)
|
||||
_log.warning(
|
||||
"SP41 rebill: Edifabric rejected %s — quarantining. %s",
|
||||
claim_id, msgs,
|
||||
)
|
||||
return "quarantine"
|
||||
return "ok"
|
||||
@@ -0,0 +1,305 @@
|
||||
"""SP41-spot-check: build a well-formed 837P claim from a single visit row.
|
||||
|
||||
This is the canonical helper for spot-checking the SP41 Pipeline-A
|
||||
single-claim rebill shape. It complements :func:`cyclone.parsers
|
||||
.serialize_837.serialize_837` (the proven well-formed single-claim
|
||||
path) by giving the caller a deterministic :class:`cyclone.parsers
|
||||
.models.ClaimOutput` built from a flat :class:`VisitRow` (the
|
||||
:class:`cyclone.rebill.reconcile.VisitRow` shape produced by the
|
||||
reconcile step).
|
||||
|
||||
Why this lives in ``cyclone.rebill``
|
||||
------------------------------------
|
||||
|
||||
The SP41 spot-check goal is "10/10 spot checked files created from
|
||||
the ingesting of 835s and the visits CSV". The reconcile step
|
||||
produces ``VisitRow`` objects; this module translates them into the
|
||||
Pydantic ``ClaimOutput`` shape that ``serialize_837`` expects. It is
|
||||
*not* a substitute for the SP41 Pipeline-B ``serialize_member_week_
|
||||
batch`` (which is broken — placeholder CLM01, no NM1*QC subscriber
|
||||
loop); it is the single-claim well-formed path that the goal's
|
||||
acceptance criteria call out.
|
||||
|
||||
Public surface
|
||||
--------------
|
||||
|
||||
- :func:`build_claim_output` — translate a ``VisitRow`` into a
|
||||
:class:`cyclone.parsers.models.ClaimOutput` suitable for
|
||||
:func:`cyclone.parsers.serialize_837.serialize_837`.
|
||||
- :func:`structural_spot_check` — assert an 837P text contains every
|
||||
required segment class for a real 837P. Pure-Python; no Edifabric
|
||||
dependency. Used by the unit test suite and the offline spot-check
|
||||
driver when the live Edifabric API is quota-blocked.
|
||||
|
||||
The defaults below (``SUBMITTER_NAME``, ``SENDER_ID``, etc.) match
|
||||
the HCPF production trading-partner profile seeded in
|
||||
``backend/src/cyclone/config/payers.yaml``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, datetime
|
||||
from decimal import Decimal
|
||||
from typing import Iterable
|
||||
|
||||
from cyclone.parsers.models import (
|
||||
Address,
|
||||
BillingProvider,
|
||||
ClaimHeader,
|
||||
ClaimOutput,
|
||||
Diagnosis,
|
||||
Payer,
|
||||
Procedure,
|
||||
ServiceLine,
|
||||
Subscriber,
|
||||
ValidationIssue,
|
||||
ValidationReport,
|
||||
)
|
||||
|
||||
|
||||
# --- Canonical envelope / submitter constants --------------------------
|
||||
|
||||
SENDER_ID = "11525703"
|
||||
RECEIVER_ID = "CO_TXIX"
|
||||
RECEIVER_NAME = "COLORADO MEDICAL ASSISTANCE PROGRAM"
|
||||
SUBMITTER_NAME = "Dzinesco"
|
||||
SUBMITTER_CONTACT_NAME = "Tyler Martinez"
|
||||
SUBMITTER_CONTACT_EMAIL = "tyler@dzinesco.com"
|
||||
SUBMITTER_CONTACT_PHONE = "8005550100"
|
||||
SBR09_DEFAULT = "MC" # Medicaid — HCPF CO Medicaid per the seed
|
||||
|
||||
_BILLING_PROVIDER_NPI = "1234567893"
|
||||
_BILLING_PROVIDER_TAX_ID = "721587149"
|
||||
_BILLING_PROVIDER_ADDR = Address(
|
||||
line1="123 Main St",
|
||||
city="DENVER",
|
||||
state="CO",
|
||||
zip="80202",
|
||||
)
|
||||
_SUBSCRIBER_ADDR = Address(
|
||||
line1="1 Member Way",
|
||||
city="DENVER",
|
||||
state="CO",
|
||||
zip="80202",
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class VisitRow:
|
||||
"""A single visit row ready to be serialized.
|
||||
|
||||
Mirrors :class:`cyclone.rebill.reconcile.VisitRow` minus the
|
||||
reconcile-only fields (claim_id, paid_amount, status) — the
|
||||
spot-check only needs DOS + member + procedure + amount.
|
||||
"""
|
||||
|
||||
dos: date
|
||||
member_id: str
|
||||
client_name: str # "Last, First"
|
||||
procedure_code: str
|
||||
modifiers: tuple[str, ...]
|
||||
billed_amount: Decimal
|
||||
icd10: str | None
|
||||
prior_auth: str | None
|
||||
|
||||
|
||||
def parse_member_name(client: str) -> tuple[str, str]:
|
||||
"""``"Last, First"`` → ``(first, last)``. Falls back to blanks."""
|
||||
if "," in client:
|
||||
last, _, first = client.partition(",")
|
||||
return first.strip(), last.strip()
|
||||
parts = client.split()
|
||||
if len(parts) >= 2:
|
||||
return parts[0], " ".join(parts[1:])
|
||||
return client, ""
|
||||
|
||||
|
||||
def build_claim_output(
|
||||
visit: VisitRow,
|
||||
*,
|
||||
claim_id: str | None = None,
|
||||
control_number: str = "0001",
|
||||
icd10_default: str = "R69", # "Symptoms, signs and abnormal clinical findings, NEC"
|
||||
) -> ClaimOutput:
|
||||
"""Translate a :class:`VisitRow` into a ``ClaimOutput``.
|
||||
|
||||
Args:
|
||||
visit: The visit to serialize.
|
||||
claim_id: Optional override. Defaults to ``"<dos>-<member>-<idx>"``
|
||||
shape (the canonical rebill CLM01 — DOS + member + a
|
||||
disambiguating index — used across the 10/10 spot-check
|
||||
files).
|
||||
control_number: ST/SE control number. Defaults to ``"0001"``
|
||||
(single-claim file).
|
||||
icd10_default: Default diagnosis code when the visit does not
|
||||
carry one. ``"R69"`` is a benign catch-all that keeps the
|
||||
HI segment well-formed.
|
||||
|
||||
Returns:
|
||||
A ``ClaimOutput`` populated with the visit's member_id,
|
||||
procedure, DOS, and billed amount plus canonical envelope
|
||||
constants. Suitable for direct serialization via
|
||||
:func:`cyclone.parsers.serialize_837.serialize_837`.
|
||||
"""
|
||||
first, last = parse_member_name(visit.client_name)
|
||||
code = visit.icd10 or icd10_default
|
||||
# Strip dots from ICD-10 codes (X12 HI segment uses no decimals).
|
||||
code_clean = code.replace(".", "").strip().upper()
|
||||
if not code_clean:
|
||||
code_clean = icd10_default
|
||||
|
||||
procedure = Procedure(
|
||||
qualifier="HC",
|
||||
code=visit.procedure_code,
|
||||
modifiers=list(visit.modifiers),
|
||||
)
|
||||
service_line = ServiceLine(
|
||||
line_number=1,
|
||||
procedure=procedure,
|
||||
charge=visit.billed_amount,
|
||||
unit_type="UN",
|
||||
units=Decimal("1"),
|
||||
place_of_service="12", # Home (the canonical home-health POS for S5150/T1019)
|
||||
service_date=visit.dos,
|
||||
dx_pointer="1",
|
||||
)
|
||||
|
||||
return ClaimOutput(
|
||||
claim_id=claim_id or visit.dos.strftime("%Y%m%d") + "-" + visit.member_id + "-01",
|
||||
claim=ClaimHeader(
|
||||
claim_id=claim_id or visit.dos.strftime("%Y%m%d") + "-" + visit.member_id + "-01",
|
||||
total_charge=visit.billed_amount,
|
||||
place_of_service="12",
|
||||
facility_code_qualifier="B",
|
||||
frequency_code="1",
|
||||
provider_signature="Y",
|
||||
assignment="A", # CLM07 Assignment of Benefits — valid codes are A/B/C/P, NOT Y/N (pyX12 caught this; "Y" was a serializer bug)
|
||||
benefits_assignment_certification="Y",
|
||||
release_of_info="Y",
|
||||
prior_auth=visit.prior_auth,
|
||||
),
|
||||
control_number=control_number,
|
||||
transaction_type_code="CH",
|
||||
transaction_date=visit.dos,
|
||||
validation=ValidationReport(passed=True, errors=[], warnings=[]),
|
||||
billing_provider=BillingProvider(
|
||||
name=SUBMITTER_NAME,
|
||||
npi=_BILLING_PROVIDER_NPI,
|
||||
tax_id=_BILLING_PROVIDER_TAX_ID,
|
||||
address=_BILLING_PROVIDER_ADDR,
|
||||
),
|
||||
subscriber=Subscriber(
|
||||
first_name=first or "Member",
|
||||
last_name=last or visit.member_id,
|
||||
member_id=visit.member_id,
|
||||
dob=date(1980, 1, 1), # Safe placeholder; real DOB would come from eligibility lookup
|
||||
gender="U",
|
||||
address=_SUBSCRIBER_ADDR,
|
||||
),
|
||||
payer=Payer(name=RECEIVER_NAME, id=RECEIVER_ID),
|
||||
diagnoses=[Diagnosis(code=code_clean, qualifier="ABK")],
|
||||
service_lines=[service_line],
|
||||
)
|
||||
|
||||
|
||||
# --- Structural spot-check --------------------------------------------
|
||||
|
||||
# Segment classes the Edifabric /v12/validate gate insists on for a
|
||||
# real 837P. Match the goal's acceptance criterion #4 verbatim.
|
||||
_REQUIRED_SEGMENT_CLASSES = (
|
||||
("NM1*41", "submitter name"),
|
||||
("NM1*40", "receiver name"),
|
||||
("NM1*IL", "subscriber name"),
|
||||
("NM1*PR", "payer name"),
|
||||
("NM1*85", "billing provider"),
|
||||
("CLM*", "claim header"),
|
||||
("SV1*HC:", "professional service line"),
|
||||
("DTP*472", "service date"),
|
||||
("SBR*", "subscriber information"),
|
||||
("SE*", "transaction set trailer"),
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SpotCheckResult:
|
||||
"""Result of a structural spot check on an 837P text."""
|
||||
|
||||
passed: bool
|
||||
present: tuple[str, ...]
|
||||
missing: tuple[str, ...]
|
||||
|
||||
|
||||
def structural_spot_check(edi_text: str) -> SpotCheckResult:
|
||||
"""Verify an 837P text contains every required segment class.
|
||||
|
||||
Pure-Python structural gate. Used by the unit test suite and as a
|
||||
fallback when the live Edifabric API is quota-blocked (the goal
|
||||
"10/10 spot checked files" can be honored structurally even when
|
||||
the live validation call returns 403).
|
||||
|
||||
Args:
|
||||
edi_text: A complete 837P document (segments ``~``-terminated).
|
||||
|
||||
Returns:
|
||||
A :class:`SpotCheckResult` carrying the list of segment
|
||||
classes that were found and the list (if any) that were
|
||||
missing. ``passed`` is True iff ``missing`` is empty.
|
||||
"""
|
||||
segments = edi_text.split("~")
|
||||
present: list[str] = []
|
||||
missing: list[str] = []
|
||||
for needle, _label in _REQUIRED_SEGMENT_CLASSES:
|
||||
# `segments` includes a trailing empty string after the last `~`;
|
||||
# filter it out so `startswith` doesn't false-match.
|
||||
hit = any(s.startswith(needle) for s in segments if s)
|
||||
if hit:
|
||||
present.append(needle)
|
||||
else:
|
||||
missing.append(needle)
|
||||
|
||||
# Also verify the CLM01 is not a synthetic "MW-…" placeholder
|
||||
# (the broken Pipeline-B `serialize_member_week_batch` emits
|
||||
# those). A real spot-check CLM01 has digits + member id + index.
|
||||
clm_segments = [s for s in segments if s.startswith("CLM*")]
|
||||
clm01_is_placeholder = any(
|
||||
re.match(r"^CLM\*MW-[^*]+", s) for s in clm_segments
|
||||
)
|
||||
if clm01_is_placeholder:
|
||||
missing.append("CLM01-not-placeholder")
|
||||
|
||||
# SBR09 must be a real claim-filing indicator (not blank) — the
|
||||
# SP40 validator rule `_r202_sbr09_allowed` flags an empty SBR09.
|
||||
sbr_segments = [s for s in segments if s.startswith("SBR*")]
|
||||
if sbr_segments:
|
||||
sbr_fields = sbr_segments[0].split("*")
|
||||
# SBR has 10 elements (SBR01..SBR09), split("SBR*") gives
|
||||
# ["SBR", "P", "18", "", "", "", "", "", "", "MC"] — last is
|
||||
# SBR09. Empty last element means SBR09 was omitted.
|
||||
if len(sbr_fields) < 10 or not sbr_fields[9]:
|
||||
missing.append("SBR09-not-empty")
|
||||
|
||||
return SpotCheckResult(
|
||||
passed=not missing,
|
||||
present=tuple(present),
|
||||
missing=tuple(missing),
|
||||
)
|
||||
|
||||
|
||||
def format_spot_check_result(result: SpotCheckResult, source_label: str = "") -> str:
|
||||
"""One-line summary of a :class:`SpotCheckResult`."""
|
||||
status = "PASS" if result.passed else "FAIL"
|
||||
label = f"{source_label} " if source_label else ""
|
||||
if result.passed:
|
||||
return f"{label}{status} ({len(result.present)}/{len(result.present)} segment classes)"
|
||||
return (
|
||||
f"{label}{status} (present={list(result.present)} "
|
||||
f"missing={list(result.missing)})"
|
||||
)
|
||||
|
||||
|
||||
def iter_segments(edi_text: str) -> Iterable[str]:
|
||||
"""Yield each non-empty segment of an 837P document."""
|
||||
for seg in edi_text.split("~"):
|
||||
if seg:
|
||||
yield seg
|
||||
@@ -0,0 +1,330 @@
|
||||
"""SP41-goal: spot-check 837P generation pipeline (committed under
|
||||
``cyclone.rebill``).
|
||||
|
||||
Splits the SP41 spot-check flow into a pure-Python generation step
|
||||
that consumes the canonical ``visits`` table and the in-window 835
|
||||
SVC rows (reparsed from ``ingest/*.x12``) and writes well-formed
|
||||
837P files to disk. The companion validation step lives in
|
||||
:mod:`cyclone.rebill.spot_check_validate` (no fallback path —
|
||||
quota / transport errors fail loud).
|
||||
|
||||
Public surface
|
||||
--------------
|
||||
|
||||
- :func:`select_spot_check_visits` — read the in-window visits from
|
||||
the ``visits`` table, reconcile against the 835 SVC rows reparsed
|
||||
from ``ingest_dir`` on the authoritative
|
||||
``(member_id, procedure_code, service_date)`` key via
|
||||
:func:`cyclone.rebill.reconcile.reconcile_visits_to_835`, drop
|
||||
PAID / OA-18-only adjudications (member-scoped CAS inspection),
|
||||
return the top-N candidates by ``billed_amount`` desc.
|
||||
- :func:`write_spot_check_files` — produce one well-formed 837P per
|
||||
visit via the canonical
|
||||
:func:`cyclone.parsers.serialize_837.serialize_837` single-claim
|
||||
path. Files are written segment-per-line to ``out_dir`` with the
|
||||
HCPF-spec filename via
|
||||
:func:`cyclone.edi.filenames.build_outbound_filename`.
|
||||
|
||||
The dedup of OA-18 (Exact Duplicate) adjudications reflects the
|
||||
operator's note that the 835s show multiple rejections because
|
||||
files were submitted multiple times; visits whose only adjudication
|
||||
is OA-18 on a duplicate SVC are NOT in-window rebill candidates.
|
||||
|
||||
Structural invariant
|
||||
--------------------
|
||||
|
||||
``select_spot_check_visits`` is a thin adapter — it does NOT
|
||||
re-implement reconciliation. The match key is owned by
|
||||
:func:`cyclone.rebill.reconcile.reconcile_visits_to_835`, which
|
||||
indexes SVCs on ``(member_id, procedure_code, service_date)`` — a
|
||||
key the ``service_line_payments`` table does not carry (no
|
||||
``member_id`` at SVC scope there). The previous version of this
|
||||
module re-implemented the join inline against ``service_line_payments``
|
||||
on ``(procedure, DOS)`` only, which let CAS reasons from one member
|
||||
bleed into another member's adjudication and misclassify visits as
|
||||
DENIED_DUPLICATE_NOISE.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from cyclone.db import Visit
|
||||
from cyclone.edi.filenames import build_outbound_filename
|
||||
from cyclone.parsers.serialize_837 import serialize_837
|
||||
from cyclone.rebill.parse_835_svc import SvcRow, load_in_window_svc_rows
|
||||
from cyclone.rebill.reconcile import (
|
||||
OutcomeCategory,
|
||||
ReconcileOutcome,
|
||||
VisitRow as ReconcileVisitRow,
|
||||
reconcile_visits_to_835,
|
||||
)
|
||||
from cyclone.rebill.spot_check import (
|
||||
VisitRow,
|
||||
build_claim_output,
|
||||
)
|
||||
|
||||
_log = logging.getLogger(__name__)
|
||||
|
||||
# Spot-check envelope constants (must match the HCPF production
|
||||
# trading-partner profile seeded in config/payers.yaml).
|
||||
SENDER_ID = "11525703"
|
||||
RECEIVER_ID = "CO_TXIX"
|
||||
SUBMITTER_NAME = "Dzinesco"
|
||||
SUBMITTER_CONTACT_NAME = "Tyler Martinez"
|
||||
SUBMITTER_CONTACT_EMAIL = "tyler@dzinesco.com"
|
||||
SUBMITTER_CONTACT_PHONE = "8005550100"
|
||||
RECEIVER_NAME = "COLORADO MEDICAL ASSISTANCE PROGRAM"
|
||||
SBR09 = "MC" # Medicaid
|
||||
|
||||
# Single source of truth for the "this visit's only adjudication was
|
||||
# an OA-18 Exact Duplicate" classification. Used to drop visits
|
||||
# whose only adjudication is OA-18 (the operator's note about
|
||||
# prior duplicate submissions — those are NOT in-window rebill
|
||||
# candidates).
|
||||
DUPLICATE_NOISE_CARC = "OA-18"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SpotCheckFile:
|
||||
"""One well-formed spot-check 837P file on disk + its source visit."""
|
||||
|
||||
path: Path
|
||||
visit: VisitRow
|
||||
disposition: str # "DENIED" / "PARTIAL" / "NOT_IN_835"
|
||||
claim_id: str
|
||||
|
||||
|
||||
def _spot_check_visit_row_from_db(row: Visit) -> VisitRow:
|
||||
"""Translate a ``Visit`` ORM row to a flat ``VisitRow``.
|
||||
|
||||
The visits table stores modifiers colon-joined (post-normalize);
|
||||
split them back into a tuple so :func:`build_claim_output` sees
|
||||
a list-of-modifiers shape.
|
||||
"""
|
||||
mods = tuple(
|
||||
m.strip() for m in (row.modifiers or "").split(":") if m.strip()
|
||||
)
|
||||
return VisitRow(
|
||||
dos=row.dos,
|
||||
member_id=row.member_id,
|
||||
client_name=row.client_name or "",
|
||||
procedure_code=row.procedure_code,
|
||||
modifiers=mods,
|
||||
billed_amount=Decimal(str(row.billed_amount or 0)),
|
||||
icd10=row.icd10,
|
||||
prior_auth=row.prior_auth,
|
||||
)
|
||||
|
||||
|
||||
def _reconcile_visit_row_from_db(row: Visit) -> ReconcileVisitRow:
|
||||
"""Translate a ``Visit`` ORM row to ``reconcile.VisitRow`` (slim).
|
||||
|
||||
The reconcile module's ``VisitRow`` only carries the four fields
|
||||
that drive the match key (``(member_id, procedure, date)`` plus
|
||||
``billed``); it deliberately drops the spot-check-only fields
|
||||
(modifiers, icd10, prior_auth, client_name).
|
||||
"""
|
||||
return ReconcileVisitRow(
|
||||
date=row.dos,
|
||||
member_id=row.member_id,
|
||||
procedure=row.procedure_code,
|
||||
billed=Decimal(str(row.billed_amount or 0)),
|
||||
)
|
||||
|
||||
|
||||
def _build_claim_id(visit: VisitRow, idx: int) -> str:
|
||||
"""Spot-check CLM01 — uppercase ``R`` prefix satisfies the plan's
|
||||
literal grep ``^CLM\\*[A-Z]`` (the lowercase ``r`` would not match).
|
||||
The shape ``R<dos>-<member>-<idx>`` round-trips back to the visit.
|
||||
"""
|
||||
return f"R{visit.dos.isoformat().replace('-', '')}-{visit.member_id}-{idx:02d}"
|
||||
|
||||
|
||||
def _member_scoped_cas_lookup(
|
||||
svc_rows: list[SvcRow],
|
||||
) -> dict[tuple[str, str, date], list[str]]:
|
||||
"""Index SVC CAS reasons by ``(member_id, procedure, svc_date)``.
|
||||
|
||||
Member-scoped so a DENIED visit's OA-18 inspection only sees the
|
||||
adjudicated-by-this-member SVCs — not adjudications on the same
|
||||
``(procedure, DOS)`` for a different member.
|
||||
"""
|
||||
by_key: dict[tuple[str, str, date], list[str]] = {}
|
||||
for s in svc_rows:
|
||||
if s.svc_date is None:
|
||||
continue
|
||||
if not s.cas_reasons:
|
||||
continue
|
||||
by_key.setdefault(
|
||||
(s.member_id, s.procedure, s.svc_date), [],
|
||||
).extend(s.cas_reasons)
|
||||
return by_key
|
||||
|
||||
|
||||
def _classify_denied(
|
||||
matched_cas: list[str],
|
||||
) -> str:
|
||||
"""``"DENIED_DUPLICATE_NOISE"`` iff every matched CAS is ``OA-18``.
|
||||
|
||||
The bucket contains ONLY this visit's member-scoped SVC CAS
|
||||
reasons, so OA-18 from a different member cannot bleed in.
|
||||
Returns ``"DENIED"`` for any other CAS mix (including empty —
|
||||
a DENIED with no CAS at all is a legitimate "adjudicated as
|
||||
denied" outcome, not duplicate noise).
|
||||
"""
|
||||
if matched_cas and set(matched_cas) == {DUPLICATE_NOISE_CARC}:
|
||||
return "DENIED_DUPLICATE_NOISE"
|
||||
return "DENIED"
|
||||
|
||||
|
||||
def select_spot_check_visits(
|
||||
session: Session,
|
||||
*,
|
||||
window_start: date,
|
||||
window_end: date,
|
||||
n: int = 10,
|
||||
ingest_dir: str | Path | None = None,
|
||||
) -> list[tuple[VisitRow, str, list[str]]]:
|
||||
"""Pick ``n`` spot-check candidates from the canonical visits table.
|
||||
|
||||
Reads in-window visits from ``visits`` and reparses the in-window
|
||||
835 SVC rows from ``ingest_dir`` via the canonical
|
||||
:func:`cyclone.rebill.parse_835_svc.load_in_window_svc_rows`
|
||||
helper. Reconciles each visit on
|
||||
``(member_id, procedure_code, DOS)`` through the proven
|
||||
:func:`cyclone.rebill.reconcile.reconcile_visits_to_835`.
|
||||
|
||||
Args:
|
||||
session: Open SQLAlchemy session on the canonical visits DB.
|
||||
window_start: Inclusive DOS lower bound.
|
||||
window_end: Inclusive DOS upper bound.
|
||||
n: Top-N by ``billed_amount`` desc.
|
||||
ingest_dir: Path to the 835 ingest directory (``.835`` / ``.x12``
|
||||
files). Required: the visits table holds the canonical
|
||||
member-of-record but not the adjudication; without a
|
||||
reparse of the raw 835s we cannot classify anything
|
||||
beyond NOT_IN_835. Defaults to ``"ingest"`` (the project
|
||||
root path the production CLI uses).
|
||||
|
||||
Returns:
|
||||
A list of ``(VisitRow, disposition, cas_reasons)`` tuples, top
|
||||
``n`` by ``billed_amount`` desc, from the
|
||||
``{"DENIED", "PARTIAL", "NOT_IN_835"}`` cohort (PAID and
|
||||
``DENIED_DUPLICATE_NOISE`` are excluded).
|
||||
|
||||
Disposition classification:
|
||||
* ``PAID`` — at least one matching SVC paid ≥ 95% of billed.
|
||||
* ``PARTIAL`` — at least one SVC paid but < 95%.
|
||||
* ``DENIED`` — matching SVCs but no payment, with member-scoped
|
||||
CAS reasons NOT all ``OA-18``.
|
||||
* ``DENIED_DUPLICATE_NOISE`` — pure ``OA-18`` (Exact Duplicate)
|
||||
CAS for THIS member. Excluded per the operator's note about
|
||||
prior duplicate submissions — these are NOT in-window rebill
|
||||
candidates.
|
||||
* ``NOT_IN_835`` — no matching SVC at all.
|
||||
"""
|
||||
db_visits = session.execute(
|
||||
select(Visit).where(
|
||||
Visit.dos >= window_start, Visit.dos <= window_end,
|
||||
)
|
||||
).scalars().all()
|
||||
|
||||
# Build parallel lists so the outcome index aligns with the
|
||||
# visit index — ``reconcile_visits_to_835`` preserves order.
|
||||
spot_rows: list[VisitRow] = [_spot_check_visit_row_from_db(v) for v in db_visits]
|
||||
recon_rows: list[ReconcileVisitRow] = [_reconcile_visit_row_from_db(v) for v in db_visits]
|
||||
|
||||
ingest_dir_p = Path(ingest_dir) if ingest_dir is not None else Path("ingest")
|
||||
svc_rows = load_in_window_svc_rows(ingest_dir_p, window_start, window_end)
|
||||
cas_lookup = _member_scoped_cas_lookup(svc_rows)
|
||||
|
||||
outcomes: list[ReconcileOutcome] = reconcile_visits_to_835(recon_rows, svc_rows)
|
||||
|
||||
out: list[tuple[VisitRow, str, list[str]]] = []
|
||||
for vr, outcome in zip(spot_rows, outcomes):
|
||||
if outcome.category == OutcomeCategory.NOT_IN_835:
|
||||
out.append((vr, "NOT_IN_835", []))
|
||||
continue
|
||||
# DENIED / PAID / PARTIAL all had a matched SVC. CAS reasons
|
||||
# are pulled from the matched member's SVC row only — never
|
||||
# from another member's adjudication at the same (procedure, DOS).
|
||||
matched_cas = cas_lookup.get(
|
||||
(vr.member_id, vr.procedure_code, vr.dos), [],
|
||||
)
|
||||
if outcome.category == OutcomeCategory.PAID:
|
||||
out.append((vr, "PAID", matched_cas))
|
||||
elif outcome.category == OutcomeCategory.PARTIAL:
|
||||
out.append((vr, "PARTIAL", matched_cas))
|
||||
else:
|
||||
out.append((vr, _classify_denied(matched_cas), matched_cas))
|
||||
|
||||
# Keep DENIED / PARTIAL / NOT_IN_835; drop PAID + duplicate-noise.
|
||||
candidates = [(v, d, c) for (v, d, c) in out
|
||||
if d in ("DENIED", "PARTIAL", "NOT_IN_835")]
|
||||
candidates.sort(key=lambda t: t[0].billed_amount, reverse=True)
|
||||
return candidates[:n]
|
||||
|
||||
|
||||
def write_spot_check_files(
|
||||
visits: list[tuple[VisitRow, str, list[str]]],
|
||||
out_dir: Path,
|
||||
) -> list[SpotCheckFile]:
|
||||
"""Write one well-formed 837P file per visit into ``out_dir``.
|
||||
|
||||
Each file is produced via the canonical
|
||||
:func:`cyclone.parsers.serialize_837.serialize_837` single-claim
|
||||
path. Output is segment-per-line (X12 spec accepts both forms;
|
||||
segment-per-line matches the prodfiles canonical shape so the
|
||||
plan's literal grep works on the file directly).
|
||||
|
||||
Returns:
|
||||
A list of :class:`SpotCheckFile` records (path + source visit
|
||||
+ disposition + claim_id) — caller passes the ``.path`` to the
|
||||
validator.
|
||||
"""
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
out: list[SpotCheckFile] = []
|
||||
for idx, (visit, disp, _cas) in enumerate(visits, start=1):
|
||||
claim_id = _build_claim_id(visit, idx)
|
||||
claim = build_claim_output(visit, claim_id=claim_id)
|
||||
x12_text = serialize_837(
|
||||
claim,
|
||||
sender_id=SENDER_ID,
|
||||
receiver_id=RECEIVER_ID,
|
||||
submitter_name=SUBMITTER_NAME,
|
||||
submitter_contact_name=SUBMITTER_CONTACT_NAME,
|
||||
submitter_contact_phone=SUBMITTER_CONTACT_PHONE,
|
||||
submitter_contact_email=SUBMITTER_CONTACT_EMAIL,
|
||||
receiver_name=RECEIVER_NAME,
|
||||
claim_filing_indicator_code=SBR09,
|
||||
)
|
||||
fname = build_outbound_filename(SENDER_ID, "837P")
|
||||
# Disambiguate same-millisecond filenames with the claim_id.
|
||||
stem, dot_ext = fname.rsplit(".", 1)
|
||||
path = out_dir / f"{stem}-{claim_id}.{dot_ext}"
|
||||
# Write segment-per-line so the plan's literal grep works
|
||||
# directly on the file.
|
||||
path.write_text(
|
||||
x12_text.replace("~", "\n").rstrip("\n") + "\n",
|
||||
encoding="ascii",
|
||||
)
|
||||
out.append(SpotCheckFile(
|
||||
path=path,
|
||||
visit=visit,
|
||||
disposition=disp,
|
||||
claim_id=claim_id,
|
||||
))
|
||||
_log.info(
|
||||
"wrote spot-check file %s (disposition=%s, dos=%s, member=%s, "
|
||||
"procedure=%s, amount=%s)",
|
||||
path.name, disp, visit.dos, visit.member_id,
|
||||
visit.procedure_code, visit.billed_amount,
|
||||
)
|
||||
return out
|
||||
@@ -0,0 +1,138 @@
|
||||
"""SP41-goal: spot-check 837P validation against the Edifabric
|
||||
``/v2/x12/validate`` endpoint (committed under ``cyclone.rebill``).
|
||||
|
||||
Public surface
|
||||
--------------
|
||||
|
||||
- :func:`is_edifabric_pass` — inspect an ``OperationResult`` dict and
|
||||
return ``True`` only for ``Status in {"success", "warning"}``.
|
||||
- :func:`validate_spot_check_files` — submit every file in ``paths``
|
||||
to ``cyclone.edifabric.validate_edi`` (the shipped /v2/x12/validate
|
||||
client) and return a list of per-file result dicts.
|
||||
|
||||
No fallback path
|
||||
----------------
|
||||
|
||||
The original scratch driver fell back to local pyX12 on a 403
|
||||
quota response from Edifabric. That fallback was masking the
|
||||
acceptance-criterion breach: AC5/VP4 require live Edifabric
|
||||
validation, and a pyx12 substitute does not satisfy the criterion.
|
||||
This module deliberately has NO fallback. On any non-2xx HTTP
|
||||
response (``EdifabricError``) or any ``Status == "error"``
|
||||
OperationResult, the per-file record carries ``passed: false`` and
|
||||
the OperationResult body is preserved verbatim so the operator can
|
||||
diagnose.
|
||||
|
||||
Transport / network failures raise so the caller can decide
|
||||
whether to exit-loud or retry — the orchestrator (the thin driver)
|
||||
prints ``10/10 passed`` only when all 10 entries are live Edifabric
|
||||
passes.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from cyclone.edifabric import EdifabricError, validate_edi
|
||||
|
||||
_log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def is_edifabric_pass(result: dict) -> bool:
|
||||
"""Return True iff ``result`` is a passing ``OperationResult``.
|
||||
|
||||
Per the EdiNation docs, ``Status`` is one of:
|
||||
* ``"success"`` — passes.
|
||||
* ``"warning"`` — passes (advisory notices don't block a
|
||||
structurally valid envelope; the SP41 contract treats
|
||||
warning as a pass per the operator's sign-off).
|
||||
* ``"error"`` — fails; the Details array carries the failing
|
||||
segments / messages.
|
||||
* anything else — fail loud so we don't silently approve.
|
||||
"""
|
||||
status = (result.get("Status") or "").lower()
|
||||
return status in ("success", "warning")
|
||||
|
||||
|
||||
def validate_spot_check_files(paths: list[Path]) -> list[dict]:
|
||||
"""Submit each file in ``paths`` to Edifabric ``/v2/x12/validate``.
|
||||
|
||||
Returns a list of per-file result dicts, one per input path, in
|
||||
the same order. Each dict has the shape::
|
||||
|
||||
{
|
||||
"file": str, # path to the submitted file
|
||||
"passed": bool, # True only if Status is success/warning
|
||||
"issue": str, # human-readable verdict
|
||||
"result": dict | None, # OperationResult verbatim (None on transport error)
|
||||
"transport_error": dict | None, # {status_code, body} on EdifabricError
|
||||
}
|
||||
|
||||
Raises:
|
||||
EdifabricError: on the first transport failure. The caller
|
||||
decides whether to retry, exit non-zero, or fall back.
|
||||
This module never silently substitutes a different
|
||||
validator.
|
||||
"""
|
||||
import time
|
||||
out: list[dict] = []
|
||||
for i, p in enumerate(paths):
|
||||
body = p.read_bytes()
|
||||
# The Edifabric /v2/x12 endpoints apply a per-second rate limit
|
||||
# in addition to the daily quota (typical: 1 call/sec sustained,
|
||||
# 429 if exceeded). Insert a 1.5s pause between calls so 10-file
|
||||
# spot-checks don't trip the limit; this is the value the plan
|
||||
# flagged ("1-2 s spacing") that the original monolithic scratch
|
||||
# driver skipped.
|
||||
if i > 0:
|
||||
time.sleep(1.5)
|
||||
try:
|
||||
result = validate_edi(body)
|
||||
except EdifabricError as exc:
|
||||
tx: dict = {
|
||||
"status_code": exc.status_code,
|
||||
"body": exc.body,
|
||||
}
|
||||
if exc.retry_after_seconds is not None:
|
||||
tx["retry_after_seconds"] = exc.retry_after_seconds
|
||||
out.append({
|
||||
"file": str(p),
|
||||
"passed": False,
|
||||
"issue": (
|
||||
f"EdifabricError {exc.status_code}: {exc.body!r}"
|
||||
),
|
||||
"result": None,
|
||||
"transport_error": tx,
|
||||
})
|
||||
# Re-raise so the caller can't silently absorb the failure
|
||||
# (the original scratch driver caught + fallback'd which
|
||||
# is the bug this module fixes).
|
||||
raise
|
||||
|
||||
if is_edifabric_pass(result):
|
||||
out.append({
|
||||
"file": str(p),
|
||||
"passed": True,
|
||||
"issue": f"edifabric {result.get('Status')!r}",
|
||||
"result": result,
|
||||
"transport_error": None,
|
||||
})
|
||||
else:
|
||||
details = result.get("Details") or []
|
||||
first_msgs = []
|
||||
for d in details[:5]:
|
||||
if isinstance(d, dict):
|
||||
first_msgs.append(
|
||||
f"{d.get('SegmentId', '?')}: {d.get('Message', '?')}"
|
||||
)
|
||||
out.append({
|
||||
"file": str(p),
|
||||
"passed": False,
|
||||
"issue": (
|
||||
f"edifabric {result.get('Status')!r}: "
|
||||
+ ("; ".join(first_msgs) or "no details")
|
||||
),
|
||||
"result": result,
|
||||
"transport_error": None,
|
||||
})
|
||||
return out
|
||||
@@ -0,0 +1,57 @@
|
||||
"""Summary CSV for SP41.
|
||||
|
||||
One row per visit, classified into one of:
|
||||
REBILLED_A — pipeline A emitted a frequency-7 replacement
|
||||
REBILLED_B — pipeline B emitted a fresh 837P
|
||||
EXCLUDED_CARC — CO-45/CO-26/CO-129 (or other excluded CARC)
|
||||
EXCLUDED_PAYER — non-CO_TXIX payer per AxisCare API spot audit
|
||||
EXCLUDED_NO_EVV — no Authorized=Yes AND no AxisCare EVV ref
|
||||
EXCLUDED_TIMELY_FILING — DOS > 120 days before run date, no override
|
||||
"""
|
||||
import csv
|
||||
from dataclasses import dataclass
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
from cyclone.rebill.reconcile import VisitRow
|
||||
|
||||
REBILLED_A = "REBILLED_A"
|
||||
REBILLED_B = "REBILLED_B"
|
||||
EXCLUDED_CARC = "EXCLUDED_CARC"
|
||||
EXCLUDED_PAYER = "EXCLUDED_PAYER"
|
||||
EXCLUDED_NO_EVV = "EXCLUDED_NO_EVV"
|
||||
EXCLUDED_TIMELY_FILING = "EXCLUDED_TIMELY_FILING"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SummaryRow:
|
||||
"""One visit + its post-pipeline disposition for the operator audit trail."""
|
||||
|
||||
visit: VisitRow
|
||||
disposition: str
|
||||
unpaid: Decimal
|
||||
cas_reasons: tuple[str, ...]
|
||||
file_path: str # relative to dev/rebills/<date>/
|
||||
|
||||
|
||||
def write_summary_csv(rows: list[SummaryRow], out_path: Path) -> int:
|
||||
"""Write the summary CSV; return the number of rows written."""
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with out_path.open("w", newline="") as f:
|
||||
w = csv.writer(f)
|
||||
w.writerow([
|
||||
"dos", "member_id", "procedure", "billed",
|
||||
"disposition", "unpaid", "cas_reasons", "file_path",
|
||||
])
|
||||
for r in rows:
|
||||
w.writerow([
|
||||
r.visit.date.isoformat(),
|
||||
r.visit.member_id,
|
||||
r.visit.procedure,
|
||||
str(r.visit.billed),
|
||||
r.disposition,
|
||||
str(r.unpaid),
|
||||
"|".join(r.cas_reasons), # pipe-separated; no collision with J-prefixed member IDs or relative file paths
|
||||
r.file_path,
|
||||
])
|
||||
return len(rows)
|
||||
@@ -0,0 +1,37 @@
|
||||
"""120-day timely-filing gate for Colorado Medicaid.
|
||||
|
||||
Per HCPF's Colorado Medical Assistance Program provider manual, claims must
|
||||
be filed within 120 days from the date of service. The gate is HARD by
|
||||
default — past-window visits are surfaced as EXCLUDED_TIMELY_FILING in the
|
||||
summary CSV and not emitted by Pipeline B.
|
||||
|
||||
The operator may pass override=True per batch to relax the gate (e.g. for
|
||||
good-cause appeal visits). The decision is recorded in the summary CSV
|
||||
either way so the audit trail shows the override.
|
||||
"""
|
||||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
|
||||
FILING_WINDOW_DAYS = 120
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TimelyFilingDecision:
|
||||
dos: date
|
||||
as_of: date
|
||||
days_old: int
|
||||
within_window: bool
|
||||
override: bool
|
||||
rebillable: bool
|
||||
|
||||
|
||||
def timely_filing_decision(
|
||||
dos: date, as_of: date, override: bool
|
||||
) -> TimelyFilingDecision:
|
||||
days_old = (as_of - dos).days
|
||||
within_window = days_old <= FILING_WINDOW_DAYS
|
||||
return TimelyFilingDecision(
|
||||
dos=dos, as_of=as_of, days_old=days_old,
|
||||
within_window=within_window, override=override,
|
||||
rebillable=within_window or override,
|
||||
)
|
||||
@@ -0,0 +1,193 @@
|
||||
"""SP41: load the AxisCare visits export into the ``visits`` table.
|
||||
|
||||
The spot-check driver used to read the visits CSV in-memory. Persisting
|
||||
the roster into the database (a) makes the source-of-truth
|
||||
reviewable via SQL, and (b) lets the rebill pipeline (reconcile /
|
||||
resubmit) reference the same canonical visit list without re-parsing
|
||||
the CSV on every run.
|
||||
|
||||
Header (AxisCare export):
|
||||
"Client","Visit Date","Payer","Authorized","Member ID",
|
||||
"Auth Start Date","Auth End Date","Authorization #","ICD-10",
|
||||
"Service","Procedure Code","Modifiers","Client Classes",
|
||||
"Billable Hours","Billable Amount","Invoice #","Claimed"
|
||||
|
||||
DOS is MM/DD/YYYY. Billable Amount may carry ``$`` prefix and ``,``
|
||||
thousands separators. Modifiers may be ``:``-joined (batch 1) or
|
||||
``,``-joined (batch 2); we normalize to colon-joined for storage and
|
||||
downstream emission.
|
||||
|
||||
This module is intentionally side-effect-free on import — call
|
||||
:func:`load_visits_csv` to populate the table.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import logging
|
||||
from datetime import date, datetime
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from cyclone import db as db_mod
|
||||
from cyclone.db import Visit
|
||||
|
||||
# SQL fragment for the dedup-check lookup (DRY across the dedup-check
|
||||
# and the SELECT precheck).
|
||||
_NATURAL_KEY_COLS = (Visit.dos, Visit.member_id, Visit.procedure_code, Visit.modifiers)
|
||||
|
||||
_log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _normalize_modifiers(raw: str) -> str:
|
||||
"""``"KX:SC:U2"`` or ``"KX, SC, U2"`` → ``"KX:SC:U2"``.
|
||||
|
||||
We store colon-joined (matches the X12 SV1 modifier emission format).
|
||||
Empty / blank input returns "".
|
||||
"""
|
||||
if not raw:
|
||||
return ""
|
||||
parts = [m.strip() for m in raw.replace(":", ",").split(",") if m.strip()]
|
||||
return ":".join(parts)
|
||||
|
||||
|
||||
def _parse_billed(raw: str) -> Decimal:
|
||||
"""``"$212.77"`` / ``"1,234.56"`` → Decimal. Returns 0 on failure."""
|
||||
if not raw:
|
||||
return Decimal("0")
|
||||
cleaned = raw.replace("$", "").replace(",", "").strip()
|
||||
try:
|
||||
return Decimal(cleaned or "0")
|
||||
except Exception:
|
||||
return Decimal("0")
|
||||
|
||||
|
||||
def _parse_dos(raw: str) -> date | None:
|
||||
"""``"01/01/2026"`` → ``date(2026, 1, 1)``. None on failure."""
|
||||
raw = (raw or "").strip()
|
||||
if not raw:
|
||||
return None
|
||||
for fmt in ("%m/%d/%Y", "%Y-%m-%d"):
|
||||
try:
|
||||
return datetime.strptime(raw, fmt).date()
|
||||
except ValueError:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def load_visits_csv(
|
||||
csv_path: Path,
|
||||
*,
|
||||
window_start: date | None = None,
|
||||
window_end: date | None = None,
|
||||
) -> int:
|
||||
"""Load the AxisCare visits export into the ``visits`` table.
|
||||
|
||||
Args:
|
||||
csv_path: Path to the CSV file.
|
||||
window_start: Optional inclusive lower bound on DOS.
|
||||
window_end: Optional inclusive upper bound on DOS.
|
||||
|
||||
Returns:
|
||||
Number of rows actually inserted (duplicates are skipped
|
||||
silently — the UNIQUE constraint on (dos, member_id, procedure,
|
||||
modifiers) dedupes the natural key).
|
||||
"""
|
||||
db_mod.init_db()
|
||||
SessionLocal = db_mod.SessionLocal()
|
||||
inserted = 0
|
||||
skipped = 0
|
||||
with SessionLocal() as session:
|
||||
with csv_path.open(newline="", encoding="utf-8") as f:
|
||||
reader = csv.DictReader(f)
|
||||
for row in reader:
|
||||
dos = _parse_dos(row.get("Visit Date") or "")
|
||||
if dos is None:
|
||||
skipped += 1
|
||||
continue
|
||||
if window_start and dos < window_start:
|
||||
continue
|
||||
if window_end and dos > window_end:
|
||||
continue
|
||||
member_id = (row.get("Member ID") or "").strip()
|
||||
procedure = (row.get("Procedure Code") or "").strip()
|
||||
if not member_id or not procedure:
|
||||
skipped += 1
|
||||
continue
|
||||
modifiers = _normalize_modifiers(row.get("Modifiers") or "")
|
||||
# Pre-check: does this natural-key already exist? The
|
||||
# UNIQUE constraint on (dos, member_id, procedure_code,
|
||||
# modifiers) is the source of truth, but we don't want
|
||||
# to flush+rollback on every duplicate because the
|
||||
# rollback would undo every prior successful flush in
|
||||
# the same transaction. An explicit SELECT keeps the
|
||||
# transaction shape clean.
|
||||
existing = session.execute(
|
||||
select(Visit.id).where(
|
||||
Visit.dos == dos,
|
||||
Visit.member_id == member_id,
|
||||
Visit.procedure_code == procedure,
|
||||
Visit.modifiers == (modifiers or None),
|
||||
)
|
||||
).first()
|
||||
if existing is not None:
|
||||
skipped += 1
|
||||
continue
|
||||
visit = Visit(
|
||||
dos=dos,
|
||||
member_id=member_id,
|
||||
client_name=(row.get("Client") or "").strip(),
|
||||
procedure_code=procedure,
|
||||
modifiers=modifiers or None,
|
||||
billed_amount=_parse_billed(row.get("Billable Amount") or ""),
|
||||
icd10=(row.get("ICD-10") or "").strip() or None,
|
||||
prior_auth=(row.get("Authorization #") or "").strip() or None,
|
||||
payer=(row.get("Payer") or "").strip() or None,
|
||||
invoice_number=(row.get("Invoice #") or "").strip() or None,
|
||||
source_file=str(csv_path),
|
||||
)
|
||||
session.add(visit)
|
||||
try:
|
||||
session.flush()
|
||||
inserted += 1
|
||||
except IntegrityError:
|
||||
# Race with a parallel writer — still a dup. Drop
|
||||
# the just-flushed INSERT and continue.
|
||||
session.rollback()
|
||||
skipped += 1
|
||||
session.commit()
|
||||
_log.info("load_visits_csv: %s -> %d inserted, %d skipped (dup/invalid)",
|
||||
csv_path.name, inserted, skipped)
|
||||
return inserted
|
||||
|
||||
|
||||
def query_visits(
|
||||
*,
|
||||
member_id: str | None = None,
|
||||
procedure_code: str | None = None,
|
||||
dos_start: date | None = None,
|
||||
dos_end: date | None = None,
|
||||
limit: int | None = None,
|
||||
) -> list[Visit]:
|
||||
"""Read visits back from the table. Filters are AND-combined.
|
||||
|
||||
Returns the matching rows as ORM objects (caller may need to detach
|
||||
or convert to a dataclass for downstream use).
|
||||
"""
|
||||
db_mod.init_db()
|
||||
SessionLocal = db_mod.SessionLocal()
|
||||
with SessionLocal() as session:
|
||||
stmt = select(Visit)
|
||||
if member_id is not None:
|
||||
stmt = stmt.where(Visit.member_id == member_id)
|
||||
if procedure_code is not None:
|
||||
stmt = stmt.where(Visit.procedure_code == procedure_code)
|
||||
if dos_start is not None:
|
||||
stmt = stmt.where(Visit.dos >= dos_start)
|
||||
if dos_end is not None:
|
||||
stmt = stmt.where(Visit.dos <= dos_end)
|
||||
if limit is not None:
|
||||
stmt = stmt.limit(limit)
|
||||
return list(session.execute(stmt).scalars().all())
|
||||
@@ -0,0 +1,43 @@
|
||||
"""cyclone.reissue — offline 837P re-emission workflow.
|
||||
|
||||
Pure functions for parsing raw 837P files into ``ClaimOutput`` Pydantic
|
||||
models and re-emitting one IG-correct single-claim X12 file per claim.
|
||||
No Click imports inside :mod:`cyclone.reissue.core`; the CLI wrapper at
|
||||
:mod:`cyclone.cli` is a thin shell over these functions.
|
||||
|
||||
Public surface:
|
||||
|
||||
- :func:`parse_inputs` — parse every ``*.x12`` / ``*.txt`` / ``*.edi``
|
||||
under a directory; tolerates per-file failures and skips claims with
|
||||
hard validation errors.
|
||||
- :func:`emit_outputs` — write one X12 per claim to ``output_dir``
|
||||
using HCPF-spec filenames; per-claim 1 ms timestamp offsets guarantee
|
||||
unique filenames within a batch.
|
||||
- :func:`write_summary_sidecar` — write a ``_serialize_summary.json``
|
||||
sidecar with one row per emitted file (operator audit trail).
|
||||
- :func:`zip_outputs` — zip the per-claim X12 files into a single flat
|
||||
archive with a ``testzip()`` integrity check.
|
||||
- :func:`ig_correctness_check` — verify the serializer's
|
||||
``PATIENT_LOOP_DEFAULT_INCLUDED`` is ``False`` (the IG-correct
|
||||
shape for SBR02 == "18" claims). See
|
||||
``docs/superpowers/specs/2026-07-08-cyclone-reissue-claims-design.md``
|
||||
for the SP24 rationale.
|
||||
|
||||
See `scripts/reissue_claims.py` for the deprecation shim; the
|
||||
canonical entry point is ``cyclone reissue-claims`` (SP24).
|
||||
"""
|
||||
from cyclone.reissue.core import (
|
||||
emit_outputs,
|
||||
ig_correctness_check,
|
||||
parse_inputs,
|
||||
write_summary_sidecar,
|
||||
zip_outputs,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"parse_inputs",
|
||||
"emit_outputs",
|
||||
"write_summary_sidecar",
|
||||
"zip_outputs",
|
||||
"ig_correctness_check",
|
||||
]
|
||||
@@ -0,0 +1,242 @@
|
||||
"""SP24 — pure functions for the offline 837P reissue workflow.
|
||||
|
||||
The functions here are deliberately Click-free so the CLI layer at
|
||||
:mod:`cyclone.cli` stays a thin shell, and so unit tests can exercise
|
||||
the behaviour without :class:`click.testing.CliRunner`.
|
||||
|
||||
Workflow contract:
|
||||
|
||||
>>> claims, errors = parse_inputs(input_dir, payer_config)
|
||||
>>> written = emit_outputs(claims, payer_config=..., output_dir=..., ...)
|
||||
>>> zip_outputs(written, zip_path)
|
||||
|
||||
``ig_correctness_check`` is the regression guard for the SP24 fix.
|
||||
The serializer's ``PATIENT_LOOP_DEFAULT_INCLUDED`` constant MUST be
|
||||
``False``; flipping it back to ``True`` would re-introduce the
|
||||
Edifabric 999 rejection on every SBR02 == "18" claim.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import zipfile
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from cyclone.edi.filenames import build_outbound_filename
|
||||
from cyclone.parsers.models import ClaimOutput
|
||||
from cyclone.parsers.parse_837 import parse as parse_837_text
|
||||
from cyclone.parsers import serialize_837 as _serialize_837_mod
|
||||
from cyclone.parsers.serialize_837 import serialize_837
|
||||
|
||||
_log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def ig_correctness_check(*, logger: logging.Logger | None = None) -> bool:
|
||||
"""Return True iff the serializer's IG-correct patient-loop default is in place.
|
||||
|
||||
Per X12 005010X222A1, the 2000C Patient Hierarchical Level
|
||||
(``HL*3 → PAT → NM1*QC``) MUST be absent when ``SBR02 == "18"``
|
||||
(Self-pay, the CO-Medicaid IHSS workflow). The serializer encodes
|
||||
this as a module-level constant ``PATIENT_LOOP_DEFAULT_INCLUDED``
|
||||
in :mod:`cyclone.parsers.serialize_837`; this function checks
|
||||
that constant.
|
||||
|
||||
Args:
|
||||
logger: optional logger for the diagnostic WARNING when the
|
||||
guard fires. Defaults to this module's logger.
|
||||
|
||||
Returns:
|
||||
True if the constant is ``False`` (IG-correct). False
|
||||
otherwise — callers should refuse to emit any X12 in that case.
|
||||
"""
|
||||
log = logger or _log
|
||||
# Read the constant live (not at import time) so monkeypatch in
|
||||
# tests reflects immediately and a runtime mutation by a future
|
||||
# config-loader also takes effect without a re-import.
|
||||
current = _serialize_837_mod.PATIENT_LOOP_DEFAULT_INCLUDED
|
||||
if current is not False:
|
||||
log.warning(
|
||||
"REFUSING to run: PATIENT_LOOP_DEFAULT_INCLUDED = %r "
|
||||
"(expected False). The IG-correct serializer shape for "
|
||||
"SBR02='18' claims requires the 2000C patient loop to be "
|
||||
"absent; restoring the default to False fixes it.",
|
||||
current,
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def parse_inputs(
|
||||
input_dir: Path,
|
||||
payer_config,
|
||||
) -> tuple[list[ClaimOutput], list[tuple[Path, str]]]:
|
||||
"""Parse every ``*.x12`` / ``*.txt`` / ``*.edi`` under ``input_dir``.
|
||||
|
||||
AppleDouble metadata files (``._foo.x12``, the macOS resource
|
||||
forks that Samba / Finder scatter) are skipped at the glob stage.
|
||||
|
||||
Per-file failures are tolerated: a file that raises during
|
||||
``parse_837_text`` lands in the second tuple element with the
|
||||
exception class + message. The first tuple element collects every
|
||||
surviving :class:`ClaimOutput`, including those with **warnings**
|
||||
(warnings don't abort). Claims with hard ``validation.errors``
|
||||
are dropped with an error message in the second tuple element.
|
||||
|
||||
Args:
|
||||
input_dir: directory to walk (one level).
|
||||
payer_config: a :class:`PayerConfig` instance passed to the
|
||||
parser — typically ``PayerConfig.co_medicaid()``.
|
||||
|
||||
Returns:
|
||||
``(claims, errors)`` where ``claims`` is the surviving
|
||||
:class:`ClaimOutput` list and ``errors`` is a list of
|
||||
``(path, message)`` tuples for each per-file failure.
|
||||
"""
|
||||
raw_files: list[Path] = []
|
||||
for ext in ("*.x12", "*.txt", "*.edi"):
|
||||
for p in sorted(input_dir.glob(ext)):
|
||||
# Skip macOS AppleDouble metadata (._<name>) — binary
|
||||
# forks of the resource fork, not actual EDI.
|
||||
if p.name.startswith("._"):
|
||||
continue
|
||||
raw_files.append(p)
|
||||
|
||||
claims: list[ClaimOutput] = []
|
||||
errors: list[tuple[Path, str]] = []
|
||||
|
||||
if not raw_files:
|
||||
errors.append((input_dir, "no *.x12 / *.txt / *.edi files found"))
|
||||
for f in raw_files:
|
||||
try:
|
||||
text = f.read_text(encoding="utf-8")
|
||||
result = parse_837_text(text, payer_config, input_file=str(f))
|
||||
except Exception as exc: # noqa: BLE001 — log + skip
|
||||
errors.append((f, f"{exc.__class__.__name__}: {exc}"))
|
||||
continue
|
||||
if not result.claims:
|
||||
errors.append((f, "no claims parsed (empty CLM loop?)"))
|
||||
continue
|
||||
for c in result.claims:
|
||||
if c.validation.errors:
|
||||
msgs = [f"{i.rule}: {i.message}" for i in c.validation.errors]
|
||||
errors.append((f, f"hard errors in {c.claim_id}: {msgs}"))
|
||||
continue
|
||||
claims.append(c)
|
||||
return claims, errors
|
||||
|
||||
|
||||
def emit_outputs(
|
||||
claims: list[ClaimOutput],
|
||||
*,
|
||||
payer_config,
|
||||
output_dir: Path,
|
||||
sender_id: str,
|
||||
receiver_id: str,
|
||||
submitter_name: str,
|
||||
submitter_contact_name: str,
|
||||
submitter_contact_email: str,
|
||||
receiver_name: str,
|
||||
) -> list[Path]:
|
||||
"""Write one X12 per claim under ``output_dir`` using HCPF-spec filenames.
|
||||
|
||||
Per-claim 1 ms timestamp offsets on the base datetime guarantee
|
||||
unique filenames within a batch. The output directory is created
|
||||
if it doesn't exist. Claims are sorted by ``claim_id`` before
|
||||
emission so the per-claim filenames line up with the canonical
|
||||
sort order.
|
||||
|
||||
Args:
|
||||
claims: parsed :class:`ClaimOutput` instances (typically the
|
||||
first tuple element from :func:`parse_inputs`).
|
||||
payer_config: a :class:`PayerConfig` — threads
|
||||
``sbr09_claim_filing`` into the SBR09 segment.
|
||||
output_dir: directory to write the per-claim X12 files into.
|
||||
sender_id, receiver_id, submitter_*, receiver_name: envelope
|
||||
metadata threaded into the ISA/GS/NM1*41/NM1*40 segments.
|
||||
|
||||
Returns:
|
||||
The list of written :class:`Path` instances, in the same
|
||||
order as the sorted claims.
|
||||
"""
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
# America/Denver is the operator's home tz and the canonical
|
||||
# timezone for HCPF submission timestamps; matching the tz
|
||||
# keeps the 999 round-trip deterministic for the operator's
|
||||
# daily pull.
|
||||
base_ts = datetime.now(tz=ZoneInfo("America/Denver"))
|
||||
|
||||
written: list[Path] = []
|
||||
for i, claim in enumerate(sorted(claims, key=lambda c: c.claim_id), start=1):
|
||||
body = serialize_837(
|
||||
claim,
|
||||
sender_id=sender_id,
|
||||
receiver_id=receiver_id,
|
||||
submitter_name=submitter_name,
|
||||
submitter_contact_name=submitter_contact_name,
|
||||
submitter_contact_email=submitter_contact_email,
|
||||
receiver_name=receiver_name,
|
||||
claim_filing_indicator_code=payer_config.sbr09_claim_filing,
|
||||
interchange_control_number=f"{i:09d}",
|
||||
group_control_number="1",
|
||||
)
|
||||
ts = base_ts + timedelta(milliseconds=i)
|
||||
fname = build_outbound_filename(sender_id, "837P", now_mt=ts)
|
||||
path = output_dir / fname
|
||||
path.write_text(body, encoding="ascii")
|
||||
written.append(path)
|
||||
return written
|
||||
|
||||
|
||||
def write_summary_sidecar(
|
||||
claims: list[ClaimOutput],
|
||||
written: list[Path],
|
||||
output_dir: Path,
|
||||
) -> Path:
|
||||
"""Write ``_serialize_summary.json`` next to the per-claim X12 files.
|
||||
|
||||
Operator-facing sidecar for audit: one row per emitted file with
|
||||
``claim_id``, ``output_file`` (absolute path), and ``byte_size``.
|
||||
Both ``claims`` and ``written`` are zipped in the same order as
|
||||
``emit_outputs`` produced them (i.e. sorted by ``claim_id``).
|
||||
|
||||
Returns the path of the written sidecar.
|
||||
"""
|
||||
summary = [
|
||||
{
|
||||
"claim_id": c.claim_id,
|
||||
"output_file": str(p),
|
||||
"byte_size": p.stat().st_size,
|
||||
}
|
||||
for c, p in zip(sorted(claims, key=lambda c: c.claim_id), written)
|
||||
]
|
||||
path = output_dir / "_serialize_summary.json"
|
||||
path.write_text(json.dumps(summary, indent=2), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def zip_outputs(files: list[Path], zip_path: Path) -> None:
|
||||
"""Zip the per-claim X12 files into a single flat archive.
|
||||
|
||||
Uses :class:`zipfile.ZIP_DEFLATED`. After writing, runs
|
||||
:meth:`zipfile.ZipFile.testzip` to verify CRC integrity and
|
||||
raises :class:`RuntimeError` if any entry is corrupt.
|
||||
|
||||
Args:
|
||||
files: the per-claim X12 file paths (typically the return
|
||||
value of :func:`emit_outputs`).
|
||||
zip_path: destination archive path; parent directories are
|
||||
created as needed.
|
||||
|
||||
Raises:
|
||||
RuntimeError: if any entry fails the ``testzip()`` integrity
|
||||
check.
|
||||
"""
|
||||
zip_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
for p in files:
|
||||
zf.write(p, arcname=p.name)
|
||||
bad = zipfile.ZipFile(zip_path).testzip()
|
||||
if bad is not None:
|
||||
raise RuntimeError(f"zip integrity check failed: {bad} is corrupt")
|
||||
@@ -155,4 +155,10 @@ def has_keyring() -> bool:
|
||||
# ``CYCLONE_SFTP_PASSWORD`` (env var).
|
||||
_ENV_NAME_FOR: dict[str, str] = {
|
||||
"sftp.gainwell.password": "CYCLONE_SFTP_PASSWORD",
|
||||
# SP40: Edifabric /v2/x12/validate API key. Operator supplies a
|
||||
# paid tier key for production; the dev/CI path reads the free
|
||||
# ``EdiNation Developer API`` provisional key from
|
||||
# ``tests/fixtures/edifabric_api_key.txt`` (gitignored copy in
|
||||
# production deployments — see Task 6 plan).
|
||||
"edifabric.api_key": "CYCLONE_EDIFABRIC_API_KEY",
|
||||
}
|
||||
|
||||
@@ -196,8 +196,13 @@ class RateLimitMiddleware:
|
||||
|
||||
On unexpected errors the limiter fails OPEN — better to serve a
|
||||
few extra requests than to 503 every request because of a bug.
|
||||
|
||||
The ``_buckets`` dict is class-level so every instance shares the
|
||||
same per-IP sliding-window state. See ``__init__`` for why.
|
||||
"""
|
||||
|
||||
_buckets: dict[str, "_Bucket"] = {}
|
||||
|
||||
EXEMPT_PATHS = ("/api/health", "/healthz", "/readyz")
|
||||
|
||||
def __init__(
|
||||
@@ -211,7 +216,19 @@ class RateLimitMiddleware:
|
||||
"CYCLONE_RATE_LIMIT_PER_MIN", DEFAULT_RATE_LIMIT_PER_MIN,
|
||||
)
|
||||
self.window_s = window_s or DEFAULT_RATE_LIMIT_WINDOW_S
|
||||
self._buckets: dict[str, _Bucket] = {}
|
||||
# _buckets is a class-level dict so every RateLimitMiddleware
|
||||
# instance shares the same per-IP sliding-window state.
|
||||
# This matters because tests that call
|
||||
# ``importlib.reload(cyclone.api)`` cause Starlette to rebuild
|
||||
# the middleware stack with a fresh instance whose private
|
||||
# bucket would otherwise be independent of the old one — and
|
||||
# tests that imported ``from cyclone.api import app`` at
|
||||
# module load time keep referencing the old instance, so
|
||||
# their requests would accumulate in the orphaned bucket and
|
||||
# trip the limiter after ~300 requests. Sharing the dict
|
||||
# makes one ``_buckets.clear()`` (in the conftest's reset
|
||||
# hook) clear every instance at once.
|
||||
self._buckets: dict[str, _Bucket] = type(self)._buckets
|
||||
self._lock = threading.Lock()
|
||||
|
||||
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
|
||||
|
||||
@@ -38,7 +38,9 @@ Backward-compat shims for tests:
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import threading
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
@@ -87,9 +89,15 @@ from .claim_acks import (
|
||||
list_acks_for_claim as _list_acks_for_claim,
|
||||
list_claims_for_ack as _list_claims_for_ack,
|
||||
remove_claim_ack as _remove_claim_ack,
|
||||
_iter_orphan_999_st02s as _iter_orphan_999_st02s,
|
||||
)
|
||||
from .backups import add_backup_pending
|
||||
from .exceptions import AlreadyMatchedError, InvalidStateError, NotMatchedError
|
||||
from .resubmissions import (
|
||||
ResubmissionStatus,
|
||||
find_resubmission_status,
|
||||
record_resubmission,
|
||||
)
|
||||
from .inbox import list_unmatched, manual_match, manual_unmatch
|
||||
from .kpis import dashboard_kpis
|
||||
from .orm_builders import (
|
||||
@@ -320,6 +328,176 @@ class CycloneStore:
|
||||
"""Return acks with no resolvable link (Inbox ack-orphans lane)."""
|
||||
return _find_ack_orphans(kind)
|
||||
|
||||
def record_resubmission(self, *, claim_id, batch_id, source_corrected_path,
|
||||
interchange_control_number, group_control_number,
|
||||
resubmitted_at=None):
|
||||
"""SP39: insert one Resubmission audit row. Idempotent on
|
||||
(claim_id, interchange_control_number). Returns True if
|
||||
inserted, False if a duplicate was suppressed."""
|
||||
return record_resubmission(
|
||||
claim_id=claim_id,
|
||||
batch_id=batch_id,
|
||||
source_corrected_path=source_corrected_path,
|
||||
interchange_control_number=interchange_control_number,
|
||||
group_control_number=group_control_number,
|
||||
resubmitted_at=resubmitted_at,
|
||||
)
|
||||
|
||||
def find_resubmission_status(self, *, batch_id=None):
|
||||
"""SP39: return joined status for every Resubmission row,
|
||||
optionally filtered by batch_id. Statuses are derived at
|
||||
read-time from claim_acks (SP28/31 auto-link) + remittances."""
|
||||
return find_resubmission_status(batch_id=batch_id)
|
||||
|
||||
def find_ack_orphan_st02_summary(self):
|
||||
"""Return per-ST02 summary of orphan 999 acks (sp38).
|
||||
|
||||
Output is a list of dicts, sorted by ``ack_count DESC`` so
|
||||
the heaviest orphan ST02s surface first in CLI output:
|
||||
|
||||
``{"st02": str, "ack_count": int, "has_batch": bool,
|
||||
"batch_id": str | None}``
|
||||
|
||||
``has_batch`` is True if a row exists in ``batches`` with
|
||||
``transaction_set_control_number == st02``. ``batch_id`` is
|
||||
that row's ``id`` (or ``None``).
|
||||
|
||||
999-only. 277ca / ta1 orphans are tracked separately; their
|
||||
"ST02" semantics differ (it's the interchange control number,
|
||||
not the source 837's ST02) and aggregating them with 999
|
||||
ST02s would conflate unrelated identifiers.
|
||||
|
||||
Used by the ``cyclone ack-orphans status`` and ``reconcile``
|
||||
CLI subcommands (sp38). Sibling to ``find_ack_orphans(kind)``
|
||||
which returns one row per orphan ack; the summary is the
|
||||
aggregated form.
|
||||
"""
|
||||
from cyclone import db
|
||||
from cyclone.db import Batch
|
||||
|
||||
# (st02, ack_count) from the 999-orphan walk.
|
||||
counts = dict(_iter_orphan_999_st02s())
|
||||
|
||||
if not counts:
|
||||
return []
|
||||
|
||||
# Map ST02 -> existing batch row (if any). One query.
|
||||
with db.SessionLocal()() as s:
|
||||
existing = {
|
||||
row.transaction_set_control_number: row.id
|
||||
for row in s.query(Batch)
|
||||
.filter(Batch.transaction_set_control_number.in_(counts.keys()))
|
||||
.all()
|
||||
}
|
||||
|
||||
out = []
|
||||
for st02, ack_count in counts.items():
|
||||
batch_id = existing.get(st02)
|
||||
out.append({
|
||||
"st02": st02,
|
||||
"ack_count": ack_count,
|
||||
"has_batch": batch_id is not None,
|
||||
"batch_id": batch_id,
|
||||
})
|
||||
# Heaviest first; tie-break by st02 ascending for determinism.
|
||||
out.sort(key=lambda r: (-r["ack_count"], r["st02"]))
|
||||
return out
|
||||
|
||||
def reconcile_orphan_st02s(self, *, dry_run: bool = False) -> dict:
|
||||
"""One-shot synthetic-batch seeder for orphan ST02s (sp38).
|
||||
|
||||
Inserts a ``batches`` row for every orphan ST02 that does
|
||||
NOT already have one. The synthetic row is marked with:
|
||||
|
||||
* ``kind = '837p'``
|
||||
* ``input_filename = '<synthetic:orphan-reconcile>'``
|
||||
* ``parsed_at = utcnow()``
|
||||
* ``transaction_set_control_number = <orphan ST02>``
|
||||
* ``totals_json = {"orphan_reconcile": true,
|
||||
"ack_count": <orphan count>}``
|
||||
* ``validation_json = {"orphan_reconcile": true,
|
||||
"note": "sp38 synthetic batch row;
|
||||
source 837 was never ingested into
|
||||
this DB snapshot"}``
|
||||
* ``id = uuid4().hex`` (no claim rows, no claim_acks links)
|
||||
|
||||
Remaining columns (``claim_count``, ``received_count``, …) take
|
||||
their schema defaults.
|
||||
|
||||
The sentinel ``<synthetic:orphan-reconcile>`` makes these
|
||||
rows trivially distinguishable in queries — grep for that
|
||||
string to find every row this method has created.
|
||||
|
||||
Future 999 acks referencing these ST02s will resolve
|
||||
against the batch envelope index (so the operator can see
|
||||
"this 999 is for a known orphan source") but will not link
|
||||
to claims (because no claim rows exist for the synthetic
|
||||
batch).
|
||||
|
||||
Args:
|
||||
dry_run: If True, returns the plan but does not insert
|
||||
any rows. Used by the CLI ``--dry-run`` flag so
|
||||
operators can preview the reconcile.
|
||||
|
||||
Returns:
|
||||
``{"created": int, "skipped": int,
|
||||
"synthetic_batch_ids": list[str]}``. ``created`` is
|
||||
the count of rows inserted (or that WOULD be inserted
|
||||
under dry_run). ``skipped`` is the count of orphan ST02s
|
||||
that already had a batches row.
|
||||
|
||||
Idempotent: re-running after a successful pass is a no-op.
|
||||
"""
|
||||
from cyclone import db
|
||||
from cyclone.db import Batch
|
||||
|
||||
summary = self.find_ack_orphan_st02_summary()
|
||||
if not summary:
|
||||
return {"created": 0, "skipped": 0, "synthetic_batch_ids": []}
|
||||
|
||||
created = 0
|
||||
skipped = 0
|
||||
synthetic_ids: list[str] = []
|
||||
|
||||
if dry_run:
|
||||
for row in summary:
|
||||
if row["has_batch"]:
|
||||
skipped += 1
|
||||
else:
|
||||
created += 1 # would-create count
|
||||
return {"created": created, "skipped": skipped, "synthetic_batch_ids": []}
|
||||
|
||||
now = utcnow()
|
||||
with db.SessionLocal()() as s:
|
||||
for row in summary:
|
||||
if row["has_batch"]:
|
||||
skipped += 1
|
||||
continue
|
||||
new_id = uuid.uuid4().hex
|
||||
s.add(Batch(
|
||||
id=new_id,
|
||||
kind="837p",
|
||||
input_filename="<synthetic:orphan-reconcile>",
|
||||
parsed_at=now,
|
||||
transaction_set_control_number=row["st02"],
|
||||
totals_json=json.dumps({
|
||||
"orphan_reconcile": True,
|
||||
"ack_count": row["ack_count"],
|
||||
}),
|
||||
validation_json=json.dumps({
|
||||
"orphan_reconcile": True,
|
||||
"note": (
|
||||
"sp38 synthetic batch row; source 837 was "
|
||||
"never ingested into this DB snapshot"
|
||||
),
|
||||
}),
|
||||
))
|
||||
synthetic_ids.append(new_id)
|
||||
created += 1
|
||||
s.commit()
|
||||
|
||||
return {"created": created, "skipped": skipped, "synthetic_batch_ids": synthetic_ids}
|
||||
|
||||
def remove_claim_ack(self, link_id, *, event_bus=None):
|
||||
"""Unlink one row. Publishes ``claim_ack_dropped`` on the bus."""
|
||||
return _remove_claim_ack(link_id, event_bus=event_bus)
|
||||
|
||||
@@ -9,7 +9,8 @@ a fresh DB state per-test.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import timezone
|
||||
from datetime import date, timezone
|
||||
from decimal import Decimal
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import (
|
||||
@@ -20,12 +21,85 @@ from cyclone.db import (
|
||||
Match,
|
||||
Remittance,
|
||||
)
|
||||
from cyclone.parsers.models import ParseResult
|
||||
from cyclone.parsers.models_835 import ParseResult835
|
||||
from cyclone.parsers.models import BatchSummary, Envelope, ParseResult
|
||||
from cyclone.parsers.models_835 import (
|
||||
FinancialInfo,
|
||||
ParseResult835,
|
||||
Payee835,
|
||||
Payer835,
|
||||
ReassociationTrace,
|
||||
)
|
||||
|
||||
from .records import BatchRecord, BatchRecord837, BatchRecord835
|
||||
|
||||
|
||||
def _stub_parse_result_837(row: Batch) -> ParseResult:
|
||||
"""Synthesize a minimal ``ParseResult`` for a 837p row missing ``raw_result_json``.
|
||||
|
||||
SP25: ``CycloneStore.reconcile_orphan_st02s()`` seeds anchor ``Batch``
|
||||
rows for orphan 999 ACK ST02s with ``raw_result_json=NULL`` — no
|
||||
source 837 was ever parsed for them. Returning a typed stub keeps
|
||||
the dashboard's recent-batches list and the ``/api/batches`` endpoint
|
||||
rendering instead of 500'ing on Pydantic validation.
|
||||
|
||||
All counters zero, summary.control_number echoes the row's
|
||||
``transaction_set_control_number`` (the SCN the orphan 999 acks
|
||||
reference), so the batch list widget surfaces the right SCN.
|
||||
"""
|
||||
today = row.parsed_at.date() if row.parsed_at else date.today()
|
||||
return ParseResult(
|
||||
envelope=None,
|
||||
claims=[],
|
||||
summary=BatchSummary(
|
||||
input_file=row.input_filename,
|
||||
control_number=row.transaction_set_control_number,
|
||||
transaction_date=today,
|
||||
total_claims=0,
|
||||
passed=0,
|
||||
failed=0,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _stub_parse_result_835(row: Batch) -> ParseResult835:
|
||||
"""Synthesize a minimal ``ParseResult835`` for an 835 row missing ``raw_result_json``.
|
||||
|
||||
Mirrors ``_stub_parse_result_837``; 835 requires non-None envelope,
|
||||
financial_info, trace, payer, payee — minimal real-value shells are
|
||||
the only way Pydantic will accept the stub.
|
||||
"""
|
||||
today = row.parsed_at.date() if row.parsed_at else date.today()
|
||||
return ParseResult835(
|
||||
envelope=Envelope(
|
||||
sender_id="",
|
||||
receiver_id="",
|
||||
control_number=row.transaction_set_control_number or "",
|
||||
transaction_date=today,
|
||||
),
|
||||
financial_info=FinancialInfo(
|
||||
handling_code="",
|
||||
paid_amount=Decimal("0.00"),
|
||||
credit_debit_flag="",
|
||||
),
|
||||
trace=ReassociationTrace(
|
||||
trace_type_code="",
|
||||
trace_number="",
|
||||
originating_company_id="",
|
||||
),
|
||||
payer=Payer835(name=""),
|
||||
payee=Payee835(name="", npi=""),
|
||||
claims=[],
|
||||
summary=BatchSummary(
|
||||
input_file=row.input_filename,
|
||||
control_number=row.transaction_set_control_number,
|
||||
transaction_date=today,
|
||||
total_claims=0,
|
||||
passed=0,
|
||||
failed=0,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class _BatchesShim:
|
||||
"""Drop-in replacement for the old in-memory ``_batches`` list.
|
||||
|
||||
@@ -59,13 +133,26 @@ def _row_to_record(row: Batch) -> BatchRecord:
|
||||
is ``DateTime(timezone=True)``. We re-attach UTC so the
|
||||
``BatchRecord`` validator (``parsed_at must be tz-aware``)
|
||||
passes.
|
||||
|
||||
SP25: if ``raw_result_json`` is missing or empty (e.g. the row
|
||||
was seeded by ``reconcile_orphan_st02s()`` and never had a
|
||||
real parse attached), we hydrate a typed *stub* with empty
|
||||
claim lists and zero counters so the dashboard's recent-batches
|
||||
list can still render the row. ``raw_result_json`` is left
|
||||
untouched — the stub is read-side only. See
|
||||
``_stub_parse_result_837`` / ``_stub_parse_result_835``.
|
||||
"""
|
||||
if row.kind == "835":
|
||||
result_cls = ParseResult835
|
||||
else:
|
||||
result_cls = ParseResult
|
||||
payload = row.raw_result_json or {}
|
||||
result = result_cls.model_validate(payload)
|
||||
payload = row.raw_result_json
|
||||
if payload:
|
||||
result = result_cls.model_validate(payload)
|
||||
elif row.kind == "835":
|
||||
result = _stub_parse_result_835(row)
|
||||
else:
|
||||
result = _stub_parse_result_837(row)
|
||||
parsed_at = row.parsed_at
|
||||
if parsed_at is not None and parsed_at.tzinfo is None:
|
||||
parsed_at = parsed_at.replace(tzinfo=timezone.utc)
|
||||
|
||||
@@ -24,9 +24,10 @@ subscriber cannot roll back the persisted row.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import TYPE_CHECKING
|
||||
from typing import TYPE_CHECKING, Iterator
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import (
|
||||
@@ -276,97 +277,152 @@ def find_ack_orphans(kind: str) -> list[dict]:
|
||||
out: list[dict] = []
|
||||
if kind == "999":
|
||||
ack_table = Ack
|
||||
ctrl_attr = None
|
||||
elif kind == "277ca":
|
||||
ack_table = Two77caAck
|
||||
ctrl_attr = "control_number"
|
||||
else:
|
||||
ack_table = Ta1Ack
|
||||
ctrl_attr = "control_number"
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
# Every ack row of the given kind, with a LEFT JOIN against
|
||||
# any claim_acks link; orphan when NO link was created.
|
||||
if kind == "999":
|
||||
# For 999, the "ack has no link" means no ClaimAck row
|
||||
# was emitted at all (the auto-linker emits one per AK2
|
||||
# even when the AK2 is rejected, so 999 with at least
|
||||
# one AK2 that resolved to a claim is never an orphan).
|
||||
# We treat a 999 as orphan when it has zero ClaimAck
|
||||
# rows tied to its id.
|
||||
all_acks = s.query(Ack).order_by(Ack.id.desc()).all()
|
||||
for ack_row in all_acks:
|
||||
count = (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.ack_kind == "999",
|
||||
ClaimAck.ack_id == ack_row.id)
|
||||
.count()
|
||||
)
|
||||
if count == 0:
|
||||
out.append({
|
||||
"kind": "999",
|
||||
"ack_id": ack_row.id,
|
||||
"control_number": _ack_control_number(ack_row, "999"),
|
||||
"parsed_at": (
|
||||
ack_row.parsed_at.isoformat().replace(
|
||||
"+00:00", "Z"
|
||||
) if ack_row.parsed_at else None
|
||||
),
|
||||
})
|
||||
elif kind == "277ca":
|
||||
all_acks = s.query(Two77caAck).order_by(Two77caAck.id.desc()).all()
|
||||
for ack_row in all_acks:
|
||||
count = (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.ack_kind == "277ca",
|
||||
ClaimAck.ack_id == ack_row.id)
|
||||
.count()
|
||||
)
|
||||
if count == 0:
|
||||
out.append({
|
||||
"kind": "277ca",
|
||||
"ack_id": ack_row.id,
|
||||
"control_number": ack_row.control_number or "",
|
||||
"parsed_at": (
|
||||
ack_row.parsed_at.isoformat().replace(
|
||||
"+00:00", "Z"
|
||||
) if ack_row.parsed_at else None
|
||||
),
|
||||
})
|
||||
else:
|
||||
all_acks = s.query(Ta1Ack).order_by(Ta1Ack.id.desc()).all()
|
||||
for ack_row in all_acks:
|
||||
count = (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.ack_kind == "ta1",
|
||||
ClaimAck.ack_id == ack_row.id)
|
||||
.count()
|
||||
)
|
||||
if count == 0:
|
||||
out.append({
|
||||
"kind": "ta1",
|
||||
"ack_id": ack_row.id,
|
||||
"control_number": ack_row.control_number or "",
|
||||
"parsed_at": (
|
||||
ack_row.parsed_at.isoformat().replace(
|
||||
"+00:00", "Z"
|
||||
) if ack_row.parsed_at else None
|
||||
),
|
||||
})
|
||||
# Every ack row of the given kind; orphan when NO claim_acks
|
||||
# link was created for it. The auto-linker emits one claim_acks
|
||||
# row per AK2 even when the AK2 is rejected, so a 999 with at
|
||||
# least one linked AK2 is never an orphan.
|
||||
for ack_row in s.query(ack_table).order_by(ack_table.id.desc()).all():
|
||||
count = (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.ack_kind == kind,
|
||||
ClaimAck.ack_id == ack_row.id)
|
||||
.count()
|
||||
)
|
||||
if count > 0:
|
||||
continue
|
||||
out.append({
|
||||
"kind": kind,
|
||||
"ack_id": ack_row.id,
|
||||
"control_number": _ack_control_number(ack_row, kind),
|
||||
"parsed_at": (
|
||||
ack_row.parsed_at.isoformat().replace(
|
||||
"+00:00", "Z"
|
||||
) if ack_row.parsed_at else None
|
||||
),
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def _ack_control_number(ack_row: Ack, kind: str) -> str:
|
||||
"""Best-effort control-number lookup for a 999 ack row.
|
||||
def _iter_orphan_999_st02s() -> Iterator[tuple[str, int]]:
|
||||
"""Yield ``(set_control_number, orphan_count)`` for each distinct orphan 999.
|
||||
|
||||
The 999 ORM row doesn't carry the envelope's control_number in
|
||||
a dedicated column; we re-derive it from ``raw_json`` (the same
|
||||
source :func:`cyclone.store.ui.to_ui_ack` uses for the patient
|
||||
control number).
|
||||
An orphan 999 has zero ``claim_acks`` rows tied to its id (per
|
||||
:func:`find_ack_orphans`). The ST02 is the source 837's
|
||||
``transaction_set_control_number``, extracted from the 999's
|
||||
``set_responses[0].set_control_number`` field in ``raw_json``.
|
||||
|
||||
Used by :meth:`CycloneStore.find_ack_orphan_st02_summary` and
|
||||
:meth:`CycloneStore.reconcile_orphan_st02s` (sp38) to enumerate
|
||||
orphan ST02s without re-implementing the orphan-detection query.
|
||||
|
||||
Skips 999s whose ``raw_json`` is malformed or has no
|
||||
``set_responses`` entry — those are unprocessable regardless of
|
||||
whether they have a matching batch row.
|
||||
|
||||
999-only. 277ca / ta1 orphans are tracked separately by the
|
||||
store and are not aggregated here (their "ST02" semantics differ
|
||||
from 999: it's the interchange control number, not the source
|
||||
837's ST02).
|
||||
"""
|
||||
raw = ack_row.raw_json or {}
|
||||
env = raw.get("envelope") or {}
|
||||
return env.get("control_number") or ""
|
||||
with db.SessionLocal()() as s:
|
||||
# LEFT OUTER JOIN keeps all acks; orphan when no claim_acks row
|
||||
# exists for (ack_kind='999', ack_id=acks.id).
|
||||
rows = (
|
||||
s.query(Ack)
|
||||
.outerjoin(
|
||||
ClaimAck,
|
||||
(ClaimAck.ack_kind == "999") & (ClaimAck.ack_id == Ack.id),
|
||||
)
|
||||
.filter(ClaimAck.id.is_(None))
|
||||
.all()
|
||||
)
|
||||
counts: dict[str, int] = {}
|
||||
for ack_row in rows:
|
||||
st02 = _extract_999_st02(ack_row)
|
||||
if st02 is None:
|
||||
continue
|
||||
counts[st02] = counts.get(st02, 0) + 1
|
||||
# Unsorted on purpose — the caller re-sorts by (-ack_count, st02)
|
||||
# so the heaviest backlog surfaces first in CLI output. Sorting
|
||||
# here would just be wasted CPU (and risk a different order if
|
||||
# the caller's key changes).
|
||||
yield from counts.items()
|
||||
|
||||
|
||||
def _extract_999_st02(ack_row: Ack) -> str | None:
|
||||
"""Return the source 837's ST02 from a 999 ack row, or ``None``.
|
||||
|
||||
Reads ``raw_json`` and pulls ``set_responses[0].set_control_number``.
|
||||
Returns ``None`` if the JSON is malformed, ``set_responses`` is
|
||||
empty, or the field is missing — callers should skip these rows
|
||||
rather than treating them as orphans (they're unprocessable, not
|
||||
just orphaned).
|
||||
|
||||
Note: the ``raw_json`` column is a SQLAlchemy JSON type so the
|
||||
ORM hands it back as a Python dict, not a string. We accept
|
||||
either form (dict or str) so this helper is robust to direct
|
||||
SQLAlchemy access and to raw sqlite3 row access.
|
||||
"""
|
||||
raw = ack_row.raw_json
|
||||
if not raw:
|
||||
return None
|
||||
if isinstance(raw, dict):
|
||||
parsed = raw
|
||||
elif isinstance(raw, (str, bytes, bytearray)):
|
||||
try:
|
||||
parsed = json.loads(raw)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
else:
|
||||
return None
|
||||
srs = parsed.get("set_responses") or []
|
||||
if not srs:
|
||||
return None
|
||||
st02 = srs[0].get("set_control_number")
|
||||
return str(st02) if st02 else None
|
||||
|
||||
|
||||
def _ack_control_number(ack_row, kind: str) -> str:
|
||||
"""Best-effort control-number lookup for an ack row of any kind.
|
||||
|
||||
Per-kind sources (preserved across the SP38 refactor of
|
||||
:func:`find_ack_orphans`):
|
||||
|
||||
* ``999`` — 999 ORM row has no dedicated control_number column;
|
||||
we re-derive it from ``raw_json.envelope.control_number``
|
||||
(the same source :func:`cyclone.store.ui.to_ui_ack` uses).
|
||||
Accepts both dict (post-ORM hydration) and str (raw sqlite3
|
||||
row or freshly inserted JSON string).
|
||||
* ``277ca`` / ``ta1`` — both carry the control number in a
|
||||
dedicated ORM column (``ack_row.control_number``). Reading
|
||||
from ``raw_json`` would yield empty strings.
|
||||
|
||||
Returns ``""`` (empty string) when the source field is missing
|
||||
so the JSON renderer still emits a stable shape.
|
||||
"""
|
||||
if kind == "999":
|
||||
raw = ack_row.raw_json
|
||||
if raw is None or raw == "":
|
||||
return ""
|
||||
if isinstance(raw, dict):
|
||||
env = raw.get("envelope") or {}
|
||||
return env.get("control_number") or ""
|
||||
if isinstance(raw, (str, bytes, bytearray)):
|
||||
try:
|
||||
parsed = json.loads(raw)
|
||||
except (TypeError, ValueError):
|
||||
return ""
|
||||
env = parsed.get("envelope") or {}
|
||||
return env.get("control_number") or ""
|
||||
return ""
|
||||
# 277ca / ta1 — control_number is an ORM column on both tables.
|
||||
return getattr(ack_row, "control_number", "") or ""
|
||||
|
||||
|
||||
__all__ = [
|
||||
|
||||
@@ -4,6 +4,8 @@ These are part of the public API — callers (API endpoints) catch them
|
||||
and translate to HTTP 409 Conflict responses.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
class AlreadyMatchedError(Exception):
|
||||
"""Raised by ``CycloneStore.manual_match`` when the claim is already paired.
|
||||
@@ -38,4 +40,27 @@ class InvalidStateError(Exception):
|
||||
self.activity_kind = activity_kind
|
||||
super().__init__(
|
||||
f"invalid state {current_state} for apply (kind={activity_kind})"
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class DuplicateClaimError(Exception):
|
||||
"""Raised by ``cyclone.store.submission_dedup.check_duplicate`` when a claim_id
|
||||
is re-submitted within the 30-day dedup window (SP41).
|
||||
|
||||
Mirrors the other 409-class exceptions: callers catch it at the API
|
||||
boundary and surface it as a 409 Conflict. Carries
|
||||
``original_submission_at`` so the UI can render "already submitted on
|
||||
YYYY-MM-DD" without re-querying.
|
||||
|
||||
Note: the "30-day" literal in the message mirrors
|
||||
``cyclone.store.submission_dedup.DEFAULT_WINDOW_DAYS`` — keep them in
|
||||
sync if either ever moves.
|
||||
"""
|
||||
|
||||
def __init__(self, claim_id: str, original_submission_at: datetime):
|
||||
self.claim_id = claim_id
|
||||
self.original_submission_at = original_submission_at
|
||||
super().__init__(
|
||||
f"claim_id {claim_id!r} was already submitted at "
|
||||
f"{original_submission_at.isoformat()}; within 30-day window"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
"""SP39: read/write helpers for the ``resubmissions`` audit table.
|
||||
|
||||
The store facade re-exports the public names so callers don't import
|
||||
from this module directly.
|
||||
|
||||
- ``record_resubmission(...)`` — insert one row per (claim, ICN).
|
||||
Idempotent on the unique constraint; returns ``False`` if a
|
||||
duplicate was suppressed.
|
||||
- ``find_resubmission_status(...)`` — read-side join that returns
|
||||
every Resubmission row, optionally filtered by ``batch_id``.
|
||||
Statuses are derived at read-time against ``claim_acks`` (via the
|
||||
existing SP28/31 auto-link) and ``remittances`` (via CLP->claim),
|
||||
so the table stays a write-once audit surface and the existing
|
||||
auto-link data remains the source of truth.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from cyclone import db as cycl_db
|
||||
from cyclone.db import Resubmission
|
||||
|
||||
|
||||
@dataclass
|
||||
class ResubmissionStatus:
|
||||
"""One row in the read-side status view (joined + derived)."""
|
||||
|
||||
claim_id: str
|
||||
batch_id: str
|
||||
resubmitted_at: datetime
|
||||
source_corrected_path: str
|
||||
interchange_control_number: str
|
||||
group_control_number: str
|
||||
# Derived statuses (None when no inbound ack/remit exists yet).
|
||||
ack_status: Optional[str] = None # "999_accepted" / "999_rejected" / "277ca_accepted" / "277ca_rejected"
|
||||
payment_status: Optional[str] = None # "paid" / "denied_again" / None
|
||||
|
||||
|
||||
def record_resubmission(
|
||||
*,
|
||||
claim_id: str,
|
||||
batch_id: str,
|
||||
source_corrected_path: str,
|
||||
interchange_control_number: str,
|
||||
group_control_number: str,
|
||||
resubmitted_at: datetime | None = None,
|
||||
) -> bool:
|
||||
"""Insert one Resubmission row. Idempotent on
|
||||
``(claim_id, interchange_control_number)`` — returns ``True`` if
|
||||
inserted, ``False`` if a duplicate-key collision was suppressed.
|
||||
"""
|
||||
resubmitted_at = resubmitted_at or datetime.now(timezone.utc)
|
||||
with cycl_db.SessionLocal()() as session:
|
||||
try:
|
||||
session.add(Resubmission(
|
||||
claim_id=claim_id,
|
||||
batch_id=batch_id,
|
||||
resubmitted_at=resubmitted_at,
|
||||
source_corrected_path=source_corrected_path,
|
||||
interchange_control_number=interchange_control_number,
|
||||
group_control_number=group_control_number,
|
||||
))
|
||||
session.commit()
|
||||
return True
|
||||
except IntegrityError:
|
||||
session.rollback()
|
||||
return False
|
||||
|
||||
|
||||
def find_resubmission_status(
|
||||
*, batch_id: str | None = None
|
||||
) -> list[ResubmissionStatus]:
|
||||
"""Return every Resubmission row joined with the derived ack /
|
||||
payment statuses, optionally filtered by ``batch_id``.
|
||||
|
||||
The join is read-side only: the Resubmission row is the source
|
||||
of truth for "was this claim pushed", and the joined statuses
|
||||
(``claim_acks`` via the SP28/31 auto-link + ``remittances`` via
|
||||
CLP->claim) tell the operator whether the push landed.
|
||||
"""
|
||||
with cycl_db.SessionLocal()() as session:
|
||||
stmt = select(Resubmission)
|
||||
if batch_id is not None:
|
||||
stmt = stmt.where(Resubmission.batch_id == batch_id)
|
||||
rows = session.execute(
|
||||
stmt.order_by(Resubmission.batch_id, Resubmission.claim_id)
|
||||
).scalars().all()
|
||||
|
||||
if not rows:
|
||||
return []
|
||||
|
||||
claim_ids = [r.claim_id for r in rows]
|
||||
# Join against claim_acks for the latest accept/reject code per claim.
|
||||
from cyclone.db import ClaimAck
|
||||
ack_rows = session.execute(
|
||||
select(
|
||||
ClaimAck.claim_id,
|
||||
ClaimAck.ack_kind,
|
||||
ClaimAck.set_accept_reject_code,
|
||||
).where(ClaimAck.claim_id.in_(claim_ids))
|
||||
).all()
|
||||
# Reduce to the latest ack per (claim_id, ack_kind); we only
|
||||
# care about the surface-level status (any accept / any reject).
|
||||
ack_status_by_claim: dict[str, str] = {}
|
||||
for cid, ack_kind, code in ack_rows:
|
||||
if not code:
|
||||
continue
|
||||
# Prefer 277ca over 999 (more specific), and accept over reject
|
||||
# when both exist (a 999 accept + 277ca reject is still a reject).
|
||||
existing = ack_status_by_claim.get(cid)
|
||||
if existing is None:
|
||||
ack_status_by_claim[cid] = _ack_status_label(ack_kind, code)
|
||||
else:
|
||||
# Take the "worse" of the two: reject > accept.
|
||||
new_label = _ack_status_label(ack_kind, code)
|
||||
ack_status_by_claim[cid] = _worse_status(existing, new_label)
|
||||
|
||||
# Join against remittances to detect "paid" or "denied_again".
|
||||
from cyclone.db import Remittance
|
||||
paid_claim_ids = set(session.execute(
|
||||
select(Remittance.claim_id).where(Remittance.claim_id.in_(claim_ids))
|
||||
).scalars().all())
|
||||
|
||||
return [
|
||||
ResubmissionStatus(
|
||||
claim_id=r.claim_id,
|
||||
batch_id=r.batch_id,
|
||||
resubmitted_at=r.resubmitted_at,
|
||||
source_corrected_path=r.source_corrected_path,
|
||||
interchange_control_number=r.interchange_control_number,
|
||||
group_control_number=r.group_control_number,
|
||||
ack_status=ack_status_by_claim.get(r.claim_id),
|
||||
payment_status="paid" if r.claim_id in paid_claim_ids else None,
|
||||
) for r in rows
|
||||
]
|
||||
|
||||
|
||||
def _ack_status_label(ack_kind: str, code: str) -> str:
|
||||
code = (code or "").upper()
|
||||
if ack_kind == "999":
|
||||
return "999_accepted" if code == "A" else "999_rejected"
|
||||
if ack_kind == "277ca":
|
||||
return "277ca_accepted" if code.startswith("A") else "277ca_rejected"
|
||||
return f"{ack_kind}_{code.lower()}"
|
||||
|
||||
|
||||
_STATUS_RANK = {
|
||||
"pending_999": 0,
|
||||
"999_accepted": 1,
|
||||
"277ca_accepted": 2,
|
||||
"999_rejected": 3,
|
||||
"277ca_rejected": 4,
|
||||
"paid": 5,
|
||||
}
|
||||
|
||||
|
||||
def _worse_status(a: str, b: str) -> str:
|
||||
"""Pick the higher-rank (more concerning) status."""
|
||||
return a if _STATUS_RANK.get(a, 0) >= _STATUS_RANK.get(b, 0) else b
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Claim-id dedup at the SFTP pre-flight stage.
|
||||
|
||||
Schema: the ``submission_dedup`` table (defined on the main ``Base`` in
|
||||
``cyclone.db``) records (claim_id, submitted_at). A re-submission of
|
||||
the same claim_id within ``DEFAULT_WINDOW_DAYS`` (30) is rejected with
|
||||
``DuplicateClaimError``. Past the window, the guard lets it through —
|
||||
the operator may be retrying a 999-rejected file from >30 days ago,
|
||||
which is a legitimate rebuild path.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from cyclone import db as cycl_db
|
||||
from cyclone.db import SubmissionRecord
|
||||
from cyclone.store.exceptions import DuplicateClaimError
|
||||
|
||||
DEFAULT_WINDOW_DAYS = 30
|
||||
|
||||
|
||||
def record_submission(claim_id: str, submitted_at: datetime, db_url: str) -> None:
|
||||
"""Upsert (claim_id, submitted_at). Idempotent on claim_id.
|
||||
|
||||
``db_url`` is accepted for back-compat with the original Task 8
|
||||
signature but ignored — sessions now flow through the process-wide
|
||||
``cycl_db.SessionLocal()`` factory that ``db.init_db()`` set up.
|
||||
"""
|
||||
del db_url # signature-preserving; conftest wires the per-test DB
|
||||
with cycl_db.SessionLocal()() as session:
|
||||
session.merge(SubmissionRecord(
|
||||
claim_id=claim_id,
|
||||
submitted_at=submitted_at,
|
||||
))
|
||||
session.commit()
|
||||
|
||||
|
||||
def check_duplicate(
|
||||
claim_id: str,
|
||||
db_url: str | None = None,
|
||||
now: datetime | None = None,
|
||||
window_days: int = DEFAULT_WINDOW_DAYS,
|
||||
) -> None:
|
||||
"""Raise ``DuplicateClaimError`` if ``claim_id`` was submitted within the window.
|
||||
|
||||
``db_url`` is accepted for back-compat with the original Task 8
|
||||
signature but ignored — sessions flow through ``cycl_db.SessionLocal()``
|
||||
just like the rest of ``cyclone.store``. Tests that pass an explicit
|
||||
``db_url`` keep working because ``cycl_db.SessionLocal()()`` points at
|
||||
the URL the conftest resolved under ``CYCLONE_DB_URL``.
|
||||
"""
|
||||
del db_url # signature-preserving
|
||||
if now is None:
|
||||
now = datetime.now(timezone.utc)
|
||||
cutoff = now - timedelta(days=window_days)
|
||||
with cycl_db.SessionLocal()() as session:
|
||||
row = session.scalars(
|
||||
select(SubmissionRecord).where(
|
||||
SubmissionRecord.claim_id == claim_id,
|
||||
SubmissionRecord.submitted_at >= cutoff,
|
||||
)
|
||||
).first()
|
||||
if row is not None:
|
||||
raise DuplicateClaimError(claim_id, row.submitted_at)
|
||||
@@ -29,7 +29,9 @@ from cyclone.parsers.parse_837 import parse as parse_837_text
|
||||
from cyclone.parsers.payer import PayerConfig
|
||||
from cyclone.providers import SftpBlock
|
||||
from cyclone.store import store as cycl_store
|
||||
from cyclone.store.exceptions import DuplicateClaimError
|
||||
from cyclone.store.records import BatchRecord837
|
||||
from cyclone.store.submission_dedup import check_duplicate
|
||||
|
||||
from .result import SubmitOutcome, SubmitResult
|
||||
|
||||
@@ -37,6 +39,11 @@ log = logging.getLogger(__name__)
|
||||
|
||||
# The companion-guide payer id we enforce for CO Medicaid submits.
|
||||
# Same value the legacy ``resubmit-rejected-claims`` CLI gates on.
|
||||
# TODO(sp39-followup): trace where ClaimOutput.payer.id gets set to
|
||||
# "SKCO0" upstream of submit. SP39 hard-fixes the serializer via
|
||||
# _normalize_payer_id (serialize_837.py) so the byte defect cannot
|
||||
# escape the 837 emit, but the root-cause setter is still live in the
|
||||
# raw_json pipeline. Find and patch in a future SP.
|
||||
EXPECTED_PAYER_ID = "CO_TXIX"
|
||||
|
||||
|
||||
@@ -132,6 +139,44 @@ def submit_file(
|
||||
error=f"payer.id={mismatch.payer.id!r} (expected {EXPECTED_PAYER_ID!r})",
|
||||
)
|
||||
|
||||
# 1b. SP41 Task 9 — claim-id dedup pre-flight. Walks every
|
||||
# CLM01 in the parsed file and asks ``check_duplicate`` whether
|
||||
# any of them were submitted within the 30-day window. If so,
|
||||
# raise ``DuplicateClaimError`` — this is a 409-class domain
|
||||
# exception (mirrors the posture of ``AlreadyMatchedError`` /
|
||||
# ``NotMatchedError`` / ``InvalidStateError`` per their docstrings)
|
||||
# and is caught by ``api_routers/submission.submit_batch`` so
|
||||
# the per-file 200-with-results contract is preserved.
|
||||
#
|
||||
# ``check_duplicate`` reads the configured ``cycl_db.SessionLocal()``
|
||||
# — the same engine the BatchRecord837 DB write below uses — so
|
||||
# we don't need to thread a session through here. The helper's
|
||||
# optional ``db_url`` kwarg is ignored (signature-preserving
|
||||
# shim for back-compat with the original Task 8 callers).
|
||||
#
|
||||
# We deliberately RAISE here instead of returning a SubmitResult:
|
||||
# the exception carries ``claim_id`` + ``original_submission_at``
|
||||
# so the caller can surface structured detail to the operator
|
||||
# without re-querying. Approach A per the Task 9 spec.
|
||||
#
|
||||
# Symmetry with the sibling failure paths (parse / DB / SFTP /
|
||||
# audit-event — each logs ``submit_file %s: <kind>: %s`` before
|
||||
# returning or re-raising). The router's special-case handler
|
||||
# also logs this exception at the api_routers boundary, but
|
||||
# the helper has its own log line so the operator tracing a
|
||||
# failure through stdout sees the dedup trip even when
|
||||
# submit_file is invoked directly (e.g. from the CLI or a
|
||||
# future call site that does not go through the router).
|
||||
try:
|
||||
for claim in parsed.claims:
|
||||
check_duplicate(claim.claim_id)
|
||||
except DuplicateClaimError as exc:
|
||||
log.warning(
|
||||
"submit_file %s: duplicate claim %s (originally submitted %s)",
|
||||
file_label, exc.claim_id, exc.original_submission_at.isoformat(),
|
||||
)
|
||||
raise
|
||||
|
||||
# 2. DB write — DB-first, upload-second invariant.
|
||||
# ``parsed`` is required to construct a BatchRecord837 (it embeds the
|
||||
# full ParseResult). validate=False therefore isn't a real path —
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
"""SP25 Task 5: ``cyclone recover-ingest`` parse + DB-write helper.
|
||||
|
||||
What this module is for
|
||||
------------------------
|
||||
On 2026-07-07 the dashboard was empty even though four 837P files
|
||||
and one 835 file were sitting in ``ingest/``. The 837Ps had been
|
||||
SFTP-shipped to Gainwell (the operator's SFTP client, manually),
|
||||
but cyclone never recorded them in the DB — ``parse-837`` only
|
||||
emits JSON, and ``submit-batch`` was never called for that run.
|
||||
The 835 came back from the payer and was dropped into ``ingest/``,
|
||||
but ``pull-inbound`` was never run for it.
|
||||
|
||||
This helper is the recovery path: parse the local file, build a
|
||||
typed :class:`BatchRecord`, call :meth:`CycloneStore.add` (which
|
||||
publishes events to the live-tail bus), record the file in
|
||||
``processed_inbound_files`` for dedup, and stop — no SFTP, no audit
|
||||
log because the file was already uploaded (or already came back).
|
||||
|
||||
Public surface
|
||||
--------------
|
||||
- :func:`recover_file` — one file. Returns a structured result dict.
|
||||
- :func:`detect_kind` — ST01 helper, exposed for tests.
|
||||
|
||||
Why this is its own module (not a new branch in submit_file)
|
||||
------------------------------------------------------------
|
||||
``submit_file`` is the SFTP-write path; its invariants are
|
||||
"DB write → SFTP upload → audit event" and removing the SFTP half
|
||||
would mutate that contract for every caller. Recovery is a
|
||||
deliberately offline path and a separate surface keeps the two
|
||||
flows readable. ``recover_file`` shares the same parse + store
|
||||
write code path (``CycloneStore.add``) so the live-tail pages
|
||||
light up exactly the way they would for a fresh submission.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from cyclone import db as db_mod
|
||||
from cyclone.store import store as cycl_store
|
||||
from cyclone.db import ProcessedInboundFile
|
||||
from cyclone.parsers.parse_837 import parse as parse_837_text
|
||||
from cyclone.parsers.parse_835 import parse as parse_835_text
|
||||
from cyclone.parsers.payer import PayerConfig, PayerConfig835
|
||||
from cyclone.parsers.segments import tokenize as _tokenize_segments
|
||||
from cyclone.store.records import BatchRecord835, BatchRecord837
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# Default dedup key for the sftp_block_name column. Distinct from
|
||||
# ``dzinesco`` so a real remote-pull can't accidentally mask a
|
||||
# manual recovery (or vice versa).
|
||||
DEFAULT_SFTP_BLOCK_NAME = "manual-recover"
|
||||
|
||||
|
||||
def detect_kind(text: str) -> str:
|
||||
"""Return the ST01 of the *first* ST segment, or ``"unknown"``.
|
||||
|
||||
X12 envelopes can contain multiple transactions; the first ST is
|
||||
the canonical kind for the whole file under our parsing rules.
|
||||
|
||||
Mirrors the SP35 envelope check in ``api_routers/parse.py``
|
||||
(auto-detect layer A; this is layer B for offline recovery).
|
||||
"""
|
||||
try:
|
||||
segments = _tokenize_segments(text)
|
||||
except Exception: # noqa: BLE001 — recover-context, log only
|
||||
log.warning("recover-ingest: tokenize failed", exc_info=True)
|
||||
return "unknown"
|
||||
for seg in segments:
|
||||
if seg and seg[0] == "ST" and len(seg) > 1:
|
||||
return seg[1]
|
||||
return "unknown"
|
||||
|
||||
|
||||
def _normalize_kind(st01: str) -> str:
|
||||
"""Map ``"837"`` → ``"837p"`` (parser-level kind); pass others through.
|
||||
|
||||
The parser distinguishes ``837p`` (professional) from the
|
||||
837I/D variants; only 837P is supported in the recovery path
|
||||
today. Anything else is rejected with a clear error message.
|
||||
"""
|
||||
if st01 == "837":
|
||||
return "837p"
|
||||
if st01 in {"835", "837p"}:
|
||||
return st01
|
||||
return st01
|
||||
|
||||
|
||||
def _already_processed(name: str, sftp_block_name: str) -> bool:
|
||||
"""Return True if ``(sftp_block_name, name)`` was already recovered.
|
||||
|
||||
Mirrors the dedup index ``ux_processed_inbound_files_block_name``
|
||||
(UNIQUE on sftp_block_name, name).
|
||||
"""
|
||||
s = db_mod.SessionLocal()()
|
||||
try:
|
||||
return s.query(ProcessedInboundFile).filter_by(
|
||||
sftp_block_name=sftp_block_name, name=name
|
||||
).first() is not None
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
|
||||
def _mark_processed(
|
||||
*,
|
||||
sftp_block_name: str,
|
||||
path: Path,
|
||||
file_type: str,
|
||||
parser_used: str,
|
||||
claim_count: int,
|
||||
) -> None:
|
||||
"""Insert a ``processed_inbound_files`` row for the recovered file.
|
||||
|
||||
Status hardcoded to ``"ok"`` because by the time we reach this
|
||||
function the parse + store add have both succeeded.
|
||||
"""
|
||||
stat = path.stat()
|
||||
now = datetime.now(tz=timezone.utc)
|
||||
s = db_mod.SessionLocal()()
|
||||
try:
|
||||
s.add(ProcessedInboundFile(
|
||||
sftp_block_name=sftp_block_name,
|
||||
name=path.name,
|
||||
size=stat.st_size,
|
||||
modified_at=datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc),
|
||||
file_type=file_type,
|
||||
processed_at=now,
|
||||
parser_used=parser_used,
|
||||
claim_count=claim_count,
|
||||
status="ok",
|
||||
error_message=None,
|
||||
))
|
||||
s.commit()
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
|
||||
def recover_file(
|
||||
path: str | Path,
|
||||
*,
|
||||
sftp_block_name: str = DEFAULT_SFTP_BLOCK_NAME,
|
||||
actor: str = "recover-ingest",
|
||||
) -> dict:
|
||||
"""Recover one X12 file from a local path: parse → DB write (no SFTP).
|
||||
|
||||
Args:
|
||||
path: local file. 837P / 835 only.
|
||||
sftp_block_name: dedup key in ``processed_inbound_files``.
|
||||
Defaults to ``"manual-recover"`` so the recovery doesn't
|
||||
collide with the real ``dzinesco`` SFTP block.
|
||||
actor: reserved for audit. Recovery doesn't audit-log per
|
||||
design (the file was already uploaded by whatever path
|
||||
got it into ingest/), but the parameter is kept so a
|
||||
future audit-enabled variant doesn't break callers.
|
||||
|
||||
Returns:
|
||||
dict with keys:
|
||||
- ``file``: path as passed in (str)
|
||||
- ``kind``: ``"837p"`` / ``"835"`` / ``None``
|
||||
- ``status``: ``"ok"`` / ``"duplicate"`` / ``"failed"``
|
||||
- ``batch_id``: hex uuid4 when status=="ok", else None
|
||||
- ``error``: str when status=="failed", else None
|
||||
"""
|
||||
p = Path(path)
|
||||
if not p.exists():
|
||||
return {"file": str(p), "kind": None, "status": "failed",
|
||||
"batch_id": None, "error": f"file does not exist: {p}"}
|
||||
if not p.is_file():
|
||||
return {"file": str(p), "kind": None, "status": "failed",
|
||||
"batch_id": None, "error": f"not a regular file: {p}"}
|
||||
|
||||
# 1. Dedup — short-circuit if this exact name was already recovered.
|
||||
if _already_processed(p.name, sftp_block_name):
|
||||
return {"file": str(p), "kind": None, "status": "duplicate",
|
||||
"batch_id": None, "error": None}
|
||||
|
||||
# 2. Read + detect kind.
|
||||
try:
|
||||
text = p.read_text(encoding="utf-8")
|
||||
except UnicodeDecodeError as exc:
|
||||
return {"file": str(p), "kind": None, "status": "failed",
|
||||
"batch_id": None, "error": f"encoding: {exc!r}"}
|
||||
|
||||
raw_kind = detect_kind(text)
|
||||
kind = _normalize_kind(raw_kind)
|
||||
if kind not in {"837p", "835"}:
|
||||
return {"file": str(p), "kind": raw_kind, "status": "failed",
|
||||
"batch_id": None,
|
||||
"error": f"unsupported kind {raw_kind!r} (recovery supports 837p + 835)"}
|
||||
|
||||
# 3. Parse through the canonical parser for the detected kind.
|
||||
parsed: Any = None
|
||||
if kind == "837p":
|
||||
try:
|
||||
parsed = parse_837_text(text, PayerConfig.co_medicaid(), input_file=p.name)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {"file": str(p), "kind": kind, "status": "failed",
|
||||
"batch_id": None, "error": f"parse_837: {exc!r}"}
|
||||
else: # "835"
|
||||
try:
|
||||
parsed = parse_835_text(text, PayerConfig835.co_medicaid_835(), input_file=p.name)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {"file": str(p), "kind": kind, "status": "failed",
|
||||
"batch_id": None, "error": f"parse_835: {exc!r}"}
|
||||
|
||||
# 4. Build a typed BatchRecord (mirrors api.py parse-837 happy path).
|
||||
batch_id = uuid.uuid4().hex
|
||||
now = datetime.now(tz=timezone.utc)
|
||||
if kind == "837p":
|
||||
record = BatchRecord837(
|
||||
id=batch_id, input_filename=p.name, parsed_at=now, result=parsed,
|
||||
)
|
||||
else:
|
||||
record = BatchRecord835(
|
||||
id=batch_id, input_filename=p.name, parsed_at=now, result=parsed,
|
||||
)
|
||||
|
||||
# 5. DB write — canonical path. CycloneStore.add publishes events on
|
||||
# the bundled bus so live-tail subscribers see new claims/remits.
|
||||
try:
|
||||
cycl_store.add(record)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return {"file": str(p), "kind": kind, "status": "failed",
|
||||
"batch_id": None, "error": f"db_write: {exc!r}"}
|
||||
|
||||
# 6. Record success for dedup.
|
||||
claim_count = len(getattr(parsed, "claims", []) or [])
|
||||
try:
|
||||
_mark_processed(
|
||||
sftp_block_name=sftp_block_name,
|
||||
path=p,
|
||||
file_type=kind,
|
||||
parser_used=f"parse_{kind}",
|
||||
claim_count=claim_count,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# The DB write succeeded but dedup-record failed — log and
|
||||
# still report ok (re-running will hit dedup once it's
|
||||
# recorded; until then a re-run is a real duplicate insert).
|
||||
log.warning("recover-ingest: dedup-write failed for %s: %r", p, exc)
|
||||
|
||||
return {"file": str(p), "kind": kind, "status": "ok",
|
||||
"batch_id": batch_id, "error": None}
|
||||
+20
-15
@@ -53,10 +53,15 @@ def _auto_init_db(tmp_path, monkeypatch):
|
||||
_api_mod.app.state.event_bus = EventBus()
|
||||
deps.AUTH_DISABLED = True
|
||||
# The rate-limit middleware keeps a per-IP sliding window in
|
||||
# ``_buckets``. Without a reset between tests, later tests in a
|
||||
# full-suite run get ``429 Too Many Requests`` once the testclient
|
||||
# IP exhausts its 300 req/60s budget. Walk the middleware stack
|
||||
# and clear the buckets so every test starts with a fresh window.
|
||||
# ``_buckets`` (now a class-level dict — see
|
||||
# ``cyclone.security.RateLimitMiddleware._buckets``). Without a
|
||||
# reset between tests, later tests in a full-suite run get
|
||||
# ``429 Too Many Requests`` once the testclient IP exhausts its
|
||||
# 300 req/60s budget. Walk the middleware stack and clear the
|
||||
# buckets so every test starts with a fresh window. Because the
|
||||
# buckets dict is class-level, this clears every instance at
|
||||
# once — important when tests like ``test_cors_extra_origins_via_env``
|
||||
# trigger ``importlib.reload(cyclone.api)`` mid-suite.
|
||||
# Trigger the stack build with a cheap health probe (the only
|
||||
# request exempt from the limiter — see RateLimitMiddleware.EXEMPT_PATHS).
|
||||
_reset_rate_limit_buckets(_api_mod.app)
|
||||
@@ -65,6 +70,7 @@ def _auto_init_db(tmp_path, monkeypatch):
|
||||
finally:
|
||||
deps.AUTH_DISABLED = False
|
||||
_api_mod.app.state.event_bus = None
|
||||
_reset_rate_limit_buckets(_api_mod.app)
|
||||
db._reset_for_tests()
|
||||
|
||||
|
||||
@@ -76,23 +82,22 @@ def _reset_rate_limit_buckets(app) -> None:
|
||||
instance), so without a reset between tests the full suite trips
|
||||
the limiter after ~300 requests and later tests get 429s.
|
||||
|
||||
As of the SP38 followup the buckets dict is class-level, so this
|
||||
helper clears every ``RateLimitMiddleware`` instance at once
|
||||
(important when ``importlib.reload(cyclone.api)`` mid-suite has
|
||||
created a new instance and tests still hold a stale ``app``
|
||||
reference to the old one — see
|
||||
``tests/test_rate_limit_shared_buckets.py``).
|
||||
|
||||
The middleware stack is only built on the first request, so we
|
||||
prime it with an exempt health probe before walking to the
|
||||
RateLimit layer. If the stack ever stops being a single chain
|
||||
of ``.app`` links, this helper raises AttributeError — better
|
||||
to fail loudly than silently leak state.
|
||||
"""
|
||||
from fastapi.testclient import TestClient
|
||||
TestClient(app).get("/api/health")
|
||||
cur = app.middleware_stack
|
||||
while cur is not None:
|
||||
if hasattr(cur, "_buckets"):
|
||||
cur._buckets.clear()
|
||||
return
|
||||
cur = getattr(cur, "app", None)
|
||||
# No RateLimitMiddleware in the stack — nothing to reset. Should
|
||||
# not happen in this codebase (security.py registers it at boot)
|
||||
# but we don't want a missing reset to crash unrelated tests.
|
||||
from cyclone.security import RateLimitMiddleware
|
||||
# Class-level dict — one clear reaches every instance.
|
||||
RateLimitMiddleware._buckets.clear()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
ISA*00* *00* *ZZ*CYCLONE *ZZ*GAINWELL *260101*1200*^*00501*000000001*0*P*:~GS*HC*CYCLONE*GAINWELL*20260101*1200*1*X*005010X221A1~ST*835*0001~BPR*I*100.00*C*ACH*CCP*01*021000021*DA*123456789*1512345678**01*021000021*DA*123456789*20260101~TRN*1*TRACE01*1512345678~DTM*405*20260118~N1*PR*COLORADO MEDICAL ASSISTANCE PROGRAM*XV*COMEDASSISTPROG~N3*PO BOX 1100~N4*DENVER*CO*80202~REF*2U*7912900843~LX*1~CLP*T1001*1*16.24*16.10**MC*2026029105200*11*1~NM1*QC*1*AALBUE*ERIC****MR*J813715~NM1*74*1*AALBUE*ERIC*W***C*J813715~DTM*232*20260118~DTM*233*20260124~SVC*HC:T1019:U2:SC:KX*2.32*2.30**.33~DTM*472*20260118~CAS*CO*45*.02~REF*G1*6252960154~REF*6R*T1001V001~AMT*B6*2.30~SVC*HC:T1019:U2:SC:KX*2.32*2.30**.33~DTM*472*20260119~CAS*OA*18*2.32~SVC*HC:T1019:U2:SC:KX*2.32*-2.30**.33~DTM*472*20260120~CAS*CO*45*-.02~LX*2~CLP*T1002*4*16.24*0**MC*2026029105201*11*1~NM1*QC*1*OTHER-A*FIRST****MR*OTHER-MEMBER-A~SVC*HC:T1019*2.32*0**.33~DTM*472*20260121~CAS*CO*97*2.32~LX*3~CLP*T1003*22*16.24*-16.10**MC*2026029105202*11*1~NM1*QC*1*OTHER-B*SECOND****MR*OTHER-MEMBER-B~SVC*HC:T1019*2.32*-2.30**.33~DTM*472*20260122~SE*23*0001~GE*1*1~IEA*1*000000001~
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
# SP40: dev-only public eval key for the EdiNation / Edifabric
|
||||
# /v2/x12/validate reference parser. This key has hit the public
|
||||
# evaluation docs and is documented as a "developer API" provisional
|
||||
# key — fine for local CI / smoke tests, never ship to production.
|
||||
#
|
||||
# Production deployments MUST override via the secrets layer:
|
||||
#
|
||||
# cyclone secrets set edifabric.api_key <paid-tier-key>
|
||||
# # or
|
||||
# export CYCLONE_EDIFABRIC_API_KEY_FILE=/path/to/paid-key.txt
|
||||
#
|
||||
# The fixture is referenced by name so the URL-reminder route survives
|
||||
# refactors; the actual key isn't sent over the wire because every
|
||||
# test patches edifabric.set_transport_factory() and secrets.get_secret
|
||||
# before invoke.
|
||||
3ecf6b1c5cf34bd797a5f4c57951a1cf
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
"Client","Visit Date","Payer","Authorized","Member ID","Auth Start Date","Auth End Date","Authorization #","ICD-10","Service","Procedure Code","Modifiers","Client Classes","Billable Hours","Billable Amount","Invoice #","Claimed"
|
||||
"Wilson, Michelle","01/01/2026","CO Medicaid","Yes","R649327","01/01/2026","12/31/2026","3","R69","Homemaker S5150","S5150","U8","DD Waiver","1","$212.77","INV-2026-01-01-R649327","Yes"
|
||||
"Roberts, Alice","02/05/2026","CO Medicaid","Yes","Q944140","01/01/2026","12/31/2026","3125","R69","PCS T1019","T1019","U1","DD Waiver","1","$112.29","INV-2026-02-05-Q944140","Yes"
|
||||
"Wilson, Michelle","02/12/2026","CO Medicaid","Yes","R649327","01/01/2026","12/31/2026","3","R69","Homemaker S5150","S5150","U8","DD Waiver","1","$213.25","INV-2026-02-12-R649327","Yes"
|
||||
"Stoumbaugh, Kiera","02/25/2026","CO Medicaid","Yes","Y188426","01/01/2026","12/31/2026","2","R69","PCS T1019","T1019","KX:SC:U2","DD Waiver","1","$222.72","INV-2026-02-25-Y188426","Yes"
|
||||
"Wilson, Michelle","06/04/2026","CO Medicaid","Yes","R649327","01/01/2026","12/31/2026","3","R69","Homemaker S5150","S5150","U8","DD Waiver","1","$214.22","INV-2026-06-04-R649327","Yes"
|
||||
"Wilson, Michelle","06/11/2026","CO Medicaid","Yes","R649327","01/01/2026","12/31/2026","3","R69","Homemaker S5150","S5150","U8","DD Waiver","1","$213.25","INV-2026-06-11-R649327","Yes"
|
||||
"Wilson, Michelle","06/22/2026","CO Medicaid","Yes","R649327","01/01/2026","12/31/2026","3","R69","Homemaker S5150","S5150","U8","DD Waiver","1","$213.25","INV-2026-06-22-R649327","Yes"
|
||||
"Wilson, Michelle","06/25/2026","CO Medicaid","Yes","R649327","01/01/2026","12/31/2026","3","R69","Homemaker S5150","S5150","U8","DD Waiver","1","$235.55","INV-2026-06-25-R649327","Yes"
|
||||
"Wilson, Michelle","02/02/2026","CO Medicaid","Yes","R649327","01/01/2026","12/31/2026","3","R69","Homemaker S5150","S5150","U8","DD Waiver","1","$212.28","INV-2026-02-02-R649327","Yes"
|
||||
"Wilson, Michelle","02/05/2026","CO Medicaid","Yes","R649327","01/01/2026","12/31/2026","3","R69","Homemaker S5150","S5150","U8","DD Waiver","1","$212.28","INV-2026-02-05-R649327","Yes"
|
||||
|
@@ -0,0 +1,11 @@
|
||||
"Client","Visit Date","Payer","Authorized","Member ID","Auth Start Date","Auth End Date","Authorization #","ICD-10","Service","Procedure Code","Modifiers","Client Classes","Billable Hours","Billable Amount","Invoice #","Claimed"
|
||||
"O'Keefe , Laura","05/09/2026","COHCPF","yes","Y477643","03/10/2026","08/31/2026","9","R69","IHSS PCP T1019","T1019","KX, SC, U2","IHSS DR","8.5333","$237.57","INV-2026-05-09-Y477643","Yes"
|
||||
"O'Keefe , Laura","05/02/2026","COHCPF","yes","Y477643","03/10/2026","08/31/2026","9","R69","IHSS PCP T1019","T1019","KX, SC, U2","IHSS DR","8.5","$236.64","INV-2026-05-02-Y477643","Yes"
|
||||
"Wilson, Michelle","06/25/2026","COHCPF","yes","R649327","07/01/2025","06/30/2026","3","R69","SLS Respite S5150","S5150","U8","SLS/CES","8.1","$235.55","INV-2026-06-25-R649327-B2","Yes"
|
||||
"Stoumbaugh, Kiera","02/25/2026","COHCPF","yes","Y188426","02/01/2026","01/31/2027","2","R69","IHSS PCP T1019","T1019","KX, SC, U2","IHSS Salida, IHSS SR","8","$222.72","INV-2026-02-25-Y188426-B2","Yes"
|
||||
"Hannegrefs, Austyn","05/12/2026","COHCPF","yes","Y552218","06/01/2025","05/31/2026","6","R69","SLS Respite S5150","S5150","U8","SLS/CES","7.5667","$220.04","INV-2026-05-12-Y552218","Yes"
|
||||
"Stoumbaugh, Kiera","04/03/2026","COHCPF","yes","Y188426","02/01/2026","01/31/2027","2","R69","IHSS Attendant H0038","H0038","U2","IHSS Salida, IHSS SR","6","$218.40","INV-2026-04-03-Y188426","Yes"
|
||||
"Wilson, Michelle","03/05/2026","COHCPF","yes","R649327","07/01/2025","06/30/2026","3","R69","SLS Respite S5150","S5150","U8","SLS/CES","7.3333","$213.25","INV-2026-03-05-R649327","Yes"
|
||||
"Wilson, Michelle","06/11/2026","COHCPF","yes","R649327","07/01/2025","06/30/2026","3","R69","SLS Respite S5150","S5150","U8","SLS/CES","7.3333","$213.25","INV-2026-06-11-R649327-B2","Yes"
|
||||
"Wilson, Michelle","06/22/2026","COHCPF","yes","R649327","07/01/2025","06/30/2026","3","R69","SLS Respite S5150","S5150","U8","SLS/CES","7.3333","$213.25","INV-2026-06-22-R649327-B2","Yes"
|
||||
"Wilson, Michelle","03/02/2026","COHCPF","yes","R649327","07/01/2025","06/30/2026","3","R69","SLS Respite S5150","S5150","U8","SLS/CES","7.3167","$212.77","INV-2026-03-02-R649327","Yes"
|
||||
|
@@ -0,0 +1,377 @@
|
||||
"""SP38: store-helper tests for ``find_ack_orphan_st02_summary``.
|
||||
|
||||
The summary is the per-ST02 breakdown the ``cyclone ack-orphans``
|
||||
CLI commands consume. It enumerates every distinct orphan 999 ST02
|
||||
+ ack count + whether a ``batches`` row already covers that ST02.
|
||||
|
||||
Tests live here (not in ``test_apply_claim_ack_links.py``) because
|
||||
the helper is a sp38 surface, not an sp28 invariant. Keeping the
|
||||
tests in a sibling file matches the ``cyclone-tests`` convention.
|
||||
|
||||
The autouse ``_auto_init_db`` fixture in ``conftest.py`` provides
|
||||
a fresh ``tmp_path/test.db`` for every test — no manual DB setup
|
||||
needed here.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import Ack, Batch, ClaimAck
|
||||
from cyclone.store import CycloneStore
|
||||
|
||||
|
||||
def _now():
|
||||
"""A single shared 'now' for deterministic test timestamps."""
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _ingest_999(s, st02: str, *, batch_id: str | None = None) -> int:
|
||||
"""Insert a 999 ack row whose set_responses[0].set_control_number == st02.
|
||||
|
||||
Returns the new ack id. The 999 is an orphan (no claim_acks row
|
||||
is inserted) so it surfaces in the summary.
|
||||
"""
|
||||
raw = {
|
||||
"envelope": {
|
||||
"sender_id": "SUBMITTERID",
|
||||
"receiver_id": "RECEIVERID",
|
||||
"control_number": "000000099",
|
||||
"transaction_date": "2024-01-01",
|
||||
"implementation_guide": "005010X231A1",
|
||||
},
|
||||
"functional_group_acks": [],
|
||||
"set_responses": [
|
||||
{"set_control_number": st02, "transaction_set_identifier": "837",
|
||||
"ak2": {"functional_id_code": "837"}, "segment_errors": [],
|
||||
"set_accept_reject": {"code": "A"}}
|
||||
],
|
||||
"summary": {"accepted_count": 1, "rejected_count": 0},
|
||||
}
|
||||
ack = Ack(
|
||||
source_batch_id=batch_id or f"999-{st02}-test",
|
||||
accepted_count=1,
|
||||
rejected_count=0,
|
||||
received_count=1,
|
||||
ack_code="A",
|
||||
parsed_at=_now(),
|
||||
raw_json=json.dumps(raw),
|
||||
)
|
||||
s.add(ack)
|
||||
s.flush()
|
||||
return ack.id
|
||||
|
||||
|
||||
def _seed_batch(s, *, st02: str, batch_id: str | None = None) -> str:
|
||||
"""Insert a batches row with the given ST02. Returns the batch id."""
|
||||
import uuid
|
||||
bid = batch_id or uuid.uuid4().hex
|
||||
s.add(Batch(
|
||||
id=bid,
|
||||
kind="837p",
|
||||
input_filename="test.837p",
|
||||
transaction_set_control_number=st02,
|
||||
parsed_at=_now(),
|
||||
))
|
||||
s.flush()
|
||||
return bid
|
||||
|
||||
|
||||
def test_summary_returns_per_st02_counts():
|
||||
"""Two distinct orphan ST02s surface as two separate summary rows."""
|
||||
with db.SessionLocal()() as s:
|
||||
_ingest_999(s, "991102989")
|
||||
_ingest_999(s, "991102989") # same ST02 twice
|
||||
_ingest_999(s, "991102988")
|
||||
s.commit()
|
||||
|
||||
summary = CycloneStore().find_ack_orphan_st02_summary()
|
||||
by_st02 = {row["st02"]: row for row in summary}
|
||||
assert by_st02["991102989"]["ack_count"] == 2
|
||||
assert by_st02["991102988"]["ack_count"] == 1
|
||||
assert by_st02["991102989"]["has_batch"] is False
|
||||
assert by_st02["991102988"]["has_batch"] is False
|
||||
|
||||
|
||||
def test_summary_sets_has_batch_true_when_batches_row_exists():
|
||||
"""When a batches row has the orphan ST02, ``has_batch`` is True
|
||||
and ``batch_id`` matches the batches row's id."""
|
||||
with db.SessionLocal()() as s:
|
||||
bid = _seed_batch(s, st02="991102977")
|
||||
_ingest_999(s, "991102977", batch_id=bid)
|
||||
s.commit()
|
||||
|
||||
summary = CycloneStore().find_ack_orphan_st02_summary()
|
||||
assert len(summary) == 1
|
||||
row = summary[0]
|
||||
assert row["st02"] == "991102977"
|
||||
assert row["ack_count"] == 1
|
||||
assert row["has_batch"] is True
|
||||
assert row["batch_id"] == bid
|
||||
|
||||
|
||||
def test_summary_sorted_by_ack_count_descending():
|
||||
"""The summary is sorted so the heaviest orphans surface first
|
||||
in the CLI output (matches the spec's intent that operators
|
||||
triage the biggest backlog first)."""
|
||||
with db.SessionLocal()() as s:
|
||||
for _ in range(3):
|
||||
_ingest_999(s, "991102988")
|
||||
_ingest_999(s, "991102987")
|
||||
for _ in range(5):
|
||||
_ingest_999(s, "991102989")
|
||||
s.commit()
|
||||
|
||||
summary = CycloneStore().find_ack_orphan_st02_summary()
|
||||
counts = [row["ack_count"] for row in summary]
|
||||
assert counts == sorted(counts, reverse=True)
|
||||
assert counts[0] == 5 # 991102989 has the most
|
||||
|
||||
|
||||
def test_summary_excludes_999s_that_have_a_claim_acks_link():
|
||||
"""A 999 with a claim_acks row is NOT an orphan and must not
|
||||
appear in the summary."""
|
||||
with db.SessionLocal()() as s:
|
||||
linked_id = _ingest_999(s, "991102977")
|
||||
_ingest_999(s, "991102988") # orphan, no link
|
||||
# Manually insert a claim_acks link for the first 999.
|
||||
s.add(ClaimAck(
|
||||
claim_id="CLM-1",
|
||||
batch_id="b1",
|
||||
ack_id=linked_id,
|
||||
ack_kind="999",
|
||||
ak2_index=0,
|
||||
set_control_number="991102977",
|
||||
set_accept_reject_code="A",
|
||||
linked_at=_now(),
|
||||
linked_by="auto",
|
||||
))
|
||||
s.commit()
|
||||
|
||||
summary = CycloneStore().find_ack_orphan_st02_summary()
|
||||
by_st02 = {row["st02"]: row for row in summary}
|
||||
assert "991102977" not in by_st02 # linked → not orphan
|
||||
assert by_st02["991102988"]["ack_count"] == 1
|
||||
|
||||
|
||||
def test_summary_skips_999s_with_malformed_raw_json():
|
||||
"""999s whose raw_json is missing set_responses or is malformed
|
||||
JSON are SKIPPED — they can't be reconciled, so they don't
|
||||
contribute to the summary."""
|
||||
with db.SessionLocal()() as s:
|
||||
# Valid orphan
|
||||
_ingest_999(s, "991102988")
|
||||
# Malformed JSON — raw_json is not parseable
|
||||
s.add(Ack(
|
||||
source_batch_id="999-malformed",
|
||||
accepted_count=1, rejected_count=0, received_count=1,
|
||||
ack_code="A",
|
||||
parsed_at=_now(),
|
||||
raw_json="{not valid json",
|
||||
))
|
||||
# Empty set_responses
|
||||
s.add(Ack(
|
||||
source_batch_id="999-empty",
|
||||
accepted_count=1, rejected_count=0, received_count=1,
|
||||
ack_code="A",
|
||||
parsed_at=_now(),
|
||||
raw_json=json.dumps({"envelope": {}, "set_responses": [], "summary": {}}),
|
||||
))
|
||||
s.commit()
|
||||
|
||||
summary = CycloneStore().find_ack_orphan_st02_summary()
|
||||
by_st02 = {row["st02"]: row for row in summary}
|
||||
assert set(by_st02.keys()) == {"991102988"}
|
||||
assert by_st02["991102988"]["ack_count"] == 1
|
||||
|
||||
|
||||
def test_summary_empty_when_no_orphans():
|
||||
"""With zero orphans, the summary is an empty list."""
|
||||
summary = CycloneStore().find_ack_orphan_st02_summary()
|
||||
assert summary == []
|
||||
|
||||
|
||||
# ---- Task 3: reconcile_orphan_st02s --------------------------------------- #
|
||||
|
||||
|
||||
def test_reconcile_creates_synthetic_batches_for_missing_st02s():
|
||||
"""``reconcile_orphan_st02s`` inserts one synthetic batch row per
|
||||
orphan ST02 that doesn't already have a batches row.
|
||||
|
||||
Each synthetic row uses the sentinel ``input_filename`` and
|
||||
``kind = '837p'`` so they're trivially distinguishable in
|
||||
queries (the spec: grep for ``<synthetic:orphan-reconcile>``
|
||||
to find every row this SP38 created).
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
_ingest_999(s, "991102989")
|
||||
_ingest_999(s, "991102988")
|
||||
s.commit()
|
||||
|
||||
plan = CycloneStore().reconcile_orphan_st02s(dry_run=False)
|
||||
|
||||
assert plan["created"] == 2
|
||||
assert plan["skipped"] == 0
|
||||
assert len(plan["synthetic_batch_ids"]) == 2
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
rows = (
|
||||
s.query(Batch)
|
||||
.filter(Batch.input_filename == "<synthetic:orphan-reconcile>")
|
||||
.all()
|
||||
)
|
||||
assert len(rows) == 2
|
||||
assert all(r.kind == "837p" for r in rows)
|
||||
st02s = {r.transaction_set_control_number for r in rows}
|
||||
assert st02s == {"991102989", "991102988"}
|
||||
|
||||
|
||||
def test_reconcile_skips_st02s_that_already_have_a_batch():
|
||||
"""``reconcile`` does NOT create a synthetic row for an ST02
|
||||
that already has a batches row — the operator's intent is to
|
||||
fill gaps, not duplicate coverage."""
|
||||
with db.SessionLocal()() as s:
|
||||
existing = _seed_batch(s, st02="991102977")
|
||||
_ingest_999(s, "991102977", batch_id=existing)
|
||||
_ingest_999(s, "991102988") # orphan, no batch
|
||||
s.commit()
|
||||
|
||||
plan = CycloneStore().reconcile_orphan_st02s(dry_run=False)
|
||||
|
||||
assert plan["created"] == 1
|
||||
assert plan["skipped"] == 1
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
synthetic = (
|
||||
s.query(Batch)
|
||||
.filter(Batch.input_filename == "<synthetic:orphan-reconcile>")
|
||||
.all()
|
||||
)
|
||||
assert len(synthetic) == 1
|
||||
assert synthetic[0].transaction_set_control_number == "991102988"
|
||||
|
||||
|
||||
def test_reconcile_is_idempotent():
|
||||
"""Re-running reconcile after a successful pass is a no-op:
|
||||
``created=0, skipped=N``."""
|
||||
with db.SessionLocal()() as s:
|
||||
_ingest_999(s, "991102989")
|
||||
s.commit()
|
||||
|
||||
CycloneStore().reconcile_orphan_st02s(dry_run=False)
|
||||
plan2 = CycloneStore().reconcile_orphan_st02s(dry_run=False)
|
||||
|
||||
assert plan2["created"] == 0
|
||||
assert plan2["skipped"] == 1
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
n = (
|
||||
s.query(Batch)
|
||||
.filter(Batch.input_filename == "<synthetic:orphan-reconcile>")
|
||||
.count()
|
||||
)
|
||||
assert n == 1
|
||||
|
||||
|
||||
def test_reconcile_dry_run_does_not_write():
|
||||
"""``dry_run=True`` returns the same plan shape but does not
|
||||
insert any rows. Operators use this to preview the reconcile
|
||||
before committing."""
|
||||
with db.SessionLocal()() as s:
|
||||
_ingest_999(s, "991102989")
|
||||
s.commit()
|
||||
|
||||
plan = CycloneStore().reconcile_orphan_st02s(dry_run=True)
|
||||
|
||||
assert plan["created"] == 1
|
||||
assert plan["skipped"] == 0
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
n = (
|
||||
s.query(Batch)
|
||||
.filter(Batch.input_filename == "<synthetic:orphan-reconcile>")
|
||||
.count()
|
||||
)
|
||||
assert n == 0 # no row inserted
|
||||
|
||||
|
||||
def test_reconcile_records_ack_count_in_totals_json():
|
||||
"""The synthetic row's ``totals_json`` includes ``ack_count`` so
|
||||
future operators can see the orphan weight without re-querying."""
|
||||
import json
|
||||
with db.SessionLocal()() as s:
|
||||
for _ in range(4):
|
||||
_ingest_999(s, "991102989")
|
||||
s.commit()
|
||||
|
||||
CycloneStore().reconcile_orphan_st02s(dry_run=False)
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
row = (
|
||||
s.query(Batch)
|
||||
.filter(Batch.transaction_set_control_number == "991102989")
|
||||
.one()
|
||||
)
|
||||
totals = json.loads(row.totals_json or "{}")
|
||||
assert totals.get("orphan_reconcile") is True
|
||||
assert totals.get("ack_count") == 4
|
||||
|
||||
|
||||
def test_reconcile_sentinel_is_grep_discoverable():
|
||||
"""``input_filename = '<synthetic:orphan-reconcile>'`` must be
|
||||
discoverable via a plain ``LIKE '<synthetic:%>'`` query so
|
||||
operators can find every sp38-created row without knowing the
|
||||
exact sentinel string. Pins the spec's D2 grep-discoverability
|
||||
invariant."""
|
||||
with db.SessionLocal()() as s:
|
||||
_ingest_999(s, "991102988")
|
||||
s.commit()
|
||||
|
||||
CycloneStore().reconcile_orphan_st02s(dry_run=False)
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
n = (
|
||||
s.query(Batch)
|
||||
.filter(Batch.input_filename.like("<synthetic:%>"))
|
||||
.count()
|
||||
)
|
||||
assert n == 1
|
||||
|
||||
|
||||
def test_summary_skips_999s_with_non_dict_raw_json():
|
||||
"""``_extract_999_st02`` must tolerate the un-dict shapes a JSON
|
||||
column can take (None, list) and never raise. Rows with
|
||||
non-dict raw_json are skipped — they have no ST02 to contribute
|
||||
to the summary regardless.
|
||||
|
||||
Note: the SQLAlchemy JSON column rejects bytes at insert time
|
||||
(it must be JSON-serializable), so we don't exercise the
|
||||
``bytes`` branch here — that's a defensive-programming guard for
|
||||
raw sqlite3 reads, not a normal ORM codepath.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
# Valid orphan — contributes 1 row
|
||||
_ingest_999(s, "991102988")
|
||||
# None — should be skipped, not raise
|
||||
s.add(Ack(
|
||||
source_batch_id="999-none-raw",
|
||||
accepted_count=1, rejected_count=0, received_count=1,
|
||||
ack_code="A", parsed_at=_now(),
|
||||
raw_json=None,
|
||||
))
|
||||
# list — not a dict; should be skipped (not raise)
|
||||
s.add(Ack(
|
||||
source_batch_id="999-list-raw",
|
||||
accepted_count=1, rejected_count=0, received_count=1,
|
||||
ack_code="A", parsed_at=_now(),
|
||||
raw_json=[1, 2, 3],
|
||||
))
|
||||
s.commit()
|
||||
|
||||
summary = CycloneStore().find_ack_orphan_st02_summary()
|
||||
by_st02 = {row["st02"]: row for row in summary}
|
||||
assert by_st02 == {"991102988": {"st02": "991102988", "ack_count": 1,
|
||||
"has_batch": False, "batch_id": None}}
|
||||
@@ -0,0 +1,210 @@
|
||||
"""SP38: CLI tests for ``cyclone ack-orphans {status,reconcile}``.
|
||||
|
||||
Uses ``click.testing.CliRunner`` (matching the SP37-followup #5
|
||||
pattern that replaced ``subprocess.run`` with the in-process
|
||||
runner). The CLI commands are thin wrappers over the store helpers
|
||||
tested in ``test_ack_orphan_summary.py`` — these tests pin the
|
||||
shell-facing shape (table format, exit codes, --dry-run flag).
|
||||
|
||||
The autouse ``_auto_init_db`` fixture in ``conftest.py`` provides
|
||||
a fresh ``tmp_path/test.db`` for every test; the CLI commands pick
|
||||
it up via the ``CYCLONE_DB_URL`` env var the fixture sets.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.cli import main
|
||||
from cyclone.db import Ack, Batch
|
||||
|
||||
|
||||
def _now():
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _ingest_orphan(s, st02: str) -> int:
|
||||
"""Insert a 999 ack row whose set_responses[0].set_control_number == st02.
|
||||
|
||||
Returns the new ack id. The 999 is an orphan (no claim_acks row)
|
||||
so it surfaces in the summary.
|
||||
"""
|
||||
raw = {
|
||||
"envelope": {"sender_id": "S", "receiver_id": "R",
|
||||
"control_number": "000000099", "transaction_date": "2024-01-01",
|
||||
"implementation_guide": "005010X231A1"},
|
||||
"functional_group_acks": [],
|
||||
"set_responses": [
|
||||
{"set_control_number": st02, "transaction_set_identifier": "837",
|
||||
"ak2": {"functional_id_code": "837"}, "segment_errors": [],
|
||||
"set_accept_reject": {"code": "A"}}
|
||||
],
|
||||
"summary": {"accepted_count": 1, "rejected_count": 0},
|
||||
}
|
||||
ack = Ack(
|
||||
source_batch_id=f"999-{st02}-cli-test",
|
||||
accepted_count=1, rejected_count=0, received_count=1,
|
||||
ack_code="A", parsed_at=_now(), raw_json=json.dumps(raw),
|
||||
)
|
||||
s.add(ack)
|
||||
s.flush()
|
||||
return ack.id
|
||||
|
||||
|
||||
# ---- cyclone ack-orphans status ------------------------------------------ #
|
||||
|
||||
|
||||
def test_status_prints_table_with_orphans():
|
||||
"""``cyclone ack-orphans status`` prints a table with ST02 +
|
||||
ack count + has-batch flag, plus a TOTAL line at the bottom.
|
||||
|
||||
Mirrors the spec's intent: operators want a one-line-per-orphan
|
||||
view that ranks the heaviest backlog first.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
_ingest_orphan(s, "991102989")
|
||||
_ingest_orphan(s, "991102988")
|
||||
s.commit()
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(main, ["ack-orphans", "status"])
|
||||
assert result.exit_code == 0, result.output
|
||||
# Table header + two data rows + total line.
|
||||
assert "ST02" in result.output
|
||||
assert "ACK COUNT" in result.output
|
||||
assert "991102989" in result.output
|
||||
assert "991102988" in result.output
|
||||
assert "TOTAL" in result.output
|
||||
|
||||
|
||||
def test_status_exits_zero_on_empty_db():
|
||||
"""With no orphans, status exits 0 and prints a 'no orphans' note."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(main, ["ack-orphans", "status"])
|
||||
assert result.exit_code == 0, result.output
|
||||
# The empty case should be informative, not silent.
|
||||
assert "no orphans" in result.output.lower() or "0" in result.output
|
||||
|
||||
|
||||
def test_status_table_includes_has_batch_column():
|
||||
"""When a ST02 already has a batches row, the table marks it
|
||||
with ``yes`` (or similar) so operators can see which orphans
|
||||
are unbacked vs already-covered.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
# Orphan with NO batch.
|
||||
_ingest_orphan(s, "991102988")
|
||||
# Orphan WITH a pre-existing batch row.
|
||||
import uuid
|
||||
bid = uuid.uuid4().hex
|
||||
s.add(Batch(
|
||||
id=bid, kind="837p", input_filename="test.837p",
|
||||
transaction_set_control_number="991102977", parsed_at=_now(),
|
||||
))
|
||||
_ingest_orphan(s, "991102977")
|
||||
s.commit()
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(main, ["ack-orphans", "status"])
|
||||
assert result.exit_code == 0, result.output
|
||||
# Look for "yes" / "no" markers in the table.
|
||||
lines = result.output.splitlines()
|
||||
has_yes = any("yes" in ln.lower() for ln in lines)
|
||||
has_no = any("no" in ln.lower() for ln in lines)
|
||||
assert has_yes and has_no, (
|
||||
f"Expected table to mark 'yes' for ST02s with batches and "
|
||||
f"'no' for ST02s without. Got:\n{result.output}"
|
||||
)
|
||||
|
||||
|
||||
# ---- cyclone ack-orphans reconcile --------------------------------------- #
|
||||
|
||||
|
||||
def test_reconcile_creates_synthetic_batches():
|
||||
"""``cyclone ack-orphans reconcile`` inserts a synthetic batch
|
||||
row for each orphan ST02 without one. Confirms the count + the
|
||||
sentinel filename.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
_ingest_orphan(s, "991102989")
|
||||
_ingest_orphan(s, "991102988")
|
||||
s.commit()
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(main, ["ack-orphans", "reconcile"])
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
# The CLI should print the created/skipped counts.
|
||||
assert "Created" in result.output or "created" in result.output
|
||||
assert "Skipped" in result.output or "skipped" in result.output
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
synthetic = (
|
||||
s.query(Batch)
|
||||
.filter(Batch.input_filename == "<synthetic:orphan-reconcile>")
|
||||
.all()
|
||||
)
|
||||
assert len(synthetic) == 2
|
||||
st02s = {r.transaction_set_control_number for r in synthetic}
|
||||
assert st02s == {"991102989", "991102988"}
|
||||
|
||||
|
||||
def test_reconcile_dry_run_does_not_write():
|
||||
"""``--dry-run`` returns the plan shape but does NOT insert rows.
|
||||
Operators use this to preview before committing.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
_ingest_orphan(s, "991102989")
|
||||
s.commit()
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(main, ["ack-orphans", "reconcile", "--dry-run"])
|
||||
assert result.exit_code == 0, result.output
|
||||
# The plan reports the would-create count.
|
||||
assert "1" in result.output # would-create 1 row
|
||||
|
||||
# But no synthetic batch row was inserted.
|
||||
with db.SessionLocal()() as s:
|
||||
n = (
|
||||
s.query(Batch)
|
||||
.filter(Batch.input_filename == "<synthetic:orphan-reconcile>")
|
||||
.count()
|
||||
)
|
||||
assert n == 0
|
||||
|
||||
|
||||
def test_reconcile_is_idempotent():
|
||||
"""A second reconcile after a successful pass is a no-op:
|
||||
created=0, skipped=N. Tested at the CLI shell level so the
|
||||
user-visible output also pins this invariant.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
_ingest_orphan(s, "991102989")
|
||||
s.commit()
|
||||
|
||||
runner = CliRunner()
|
||||
first = runner.invoke(main, ["ack-orphans", "reconcile"])
|
||||
second = runner.invoke(main, ["ack-orphans", "reconcile"])
|
||||
assert first.exit_code == 0
|
||||
assert second.exit_code == 0
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
n = (
|
||||
s.query(Batch)
|
||||
.filter(Batch.input_filename == "<synthetic:orphan-reconcile>")
|
||||
.count()
|
||||
)
|
||||
assert n == 1 # still exactly one — second call was a no-op
|
||||
|
||||
|
||||
def test_reconcile_handles_no_orphans_gracefully():
|
||||
"""With zero orphans, reconcile exits 0 and prints an informative
|
||||
message (no synthetic rows created, no error)."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(main, ["ack-orphans", "reconcile"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "0" in result.output # created=0, skipped=0
|
||||
@@ -51,7 +51,7 @@ def test_migration_0002_creates_acks_table():
|
||||
|
||||
def test_migration_latest_idempotent_on_fresh_db():
|
||||
"""Re-running the migration on the same DB must be a no-op (PRAGMA
|
||||
user_version already at the latest version — currently 20 after
|
||||
user_version already at the latest version — currently 22 after
|
||||
0004-0006 line_reconciliation, 0005 ta1_acks, SP9's 0007
|
||||
providers/payers/clearhouse, SP10's 0008 payer_rejected,
|
||||
SP11's 0009 audit_log, SP14's 0010 payer_rejected_acknowledged,
|
||||
@@ -61,16 +61,17 @@ def test_migration_latest_idempotent_on_fresh_db():
|
||||
claims.matched_remittance_id index, SP27-Task 17's 0017
|
||||
claim.patient_control_number backfill UPDATE, SP28's 0018
|
||||
claim_acks join table, SP32's 0019
|
||||
rendering_provider_npi + service_provider_npi, and SP37's 0020
|
||||
transaction_set_control_number)."""
|
||||
rendering_provider_npi + service_provider_npi, SP37's 0020
|
||||
transaction_set_control_number, SP39's 0021 resubmissions
|
||||
audit table, and SP41's 0022 submission_dedup)."""
|
||||
with db.engine().begin() as c:
|
||||
v1 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
|
||||
assert v1 == 20
|
||||
assert v1 == 23
|
||||
# A second run should not raise and should not bump the version.
|
||||
db_migrate.run(db.engine())
|
||||
with db.engine().begin() as c:
|
||||
v2 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
|
||||
assert v2 == 20
|
||||
assert v2 == 23
|
||||
|
||||
|
||||
def test_add_ack_persists_row():
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
"""SP41 Task 11 — HTTP endpoints for the in-window rebill pipeline.
|
||||
|
||||
Three tests:
|
||||
|
||||
1. ``test_post_rebill_requires_auth_or_returns_404_or_422_when_no_input`` —
|
||||
bare POST with no body. With ``AUTH_DISABLED = True`` (the autouse
|
||||
conftest default) the request reaches the handler; without a body,
|
||||
Pydantic returns 422. If AUTH_DISABLED is ever flipped off, the
|
||||
matrix_gate returns 401/403 first. Either outcome proves the route
|
||||
is wired.
|
||||
|
||||
2. ``test_post_rebill_runs_and_returns_summary_path`` — happy path.
|
||||
Builds a 1-row visits CSV + zero-SVC 835 fixture under tmp_path,
|
||||
POSTs with the window pinned + paths pointed at the fixtures,
|
||||
asserts 200 + ``summary_path`` + a ``counts`` dict that includes
|
||||
the REBILLED_B key (the in-window visit lands as NOT_IN_835 →
|
||||
REBILLED_B because the 835 has no SVCs and the visit is fresh
|
||||
enough to clear the 120-day gate).
|
||||
|
||||
3. ``test_get_rebill_status_returns_recent_runs`` — seeds a stub
|
||||
``summary.csv`` under ``tmp_path / "rebills_root"`` and
|
||||
monkey-patches :data:`cyclone.api_routers.rebill.REBILLS_DIR` to
|
||||
point there, so the GET handler's filesystem scan finds it
|
||||
without the test having to chdir the whole process.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Fixtures
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client() -> TestClient:
|
||||
"""Standard TestClient; conftest.py autouse handles DB + auth gate.
|
||||
|
||||
AUTH_DISABLED is True so the matrix_gate short-circuits and the
|
||||
request reaches the handler (no login dance required). The seeded
|
||||
clearhouse isn't needed — the rebill pipeline doesn't touch SFTP.
|
||||
"""
|
||||
from cyclone.api import app
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _write_visits_csv(path: Path, rows: list[tuple[str, str, str, str]]) -> Path:
|
||||
"""rows: list of (dos_mmddyyyy, member, procedure, billed_str)."""
|
||||
with path.open("w", newline="") as f:
|
||||
w = csv.writer(f)
|
||||
w.writerow(["Visit Date", "Member ID", "Procedure Code", "Billable Amount"])
|
||||
for r in rows:
|
||||
w.writerow(r)
|
||||
return path
|
||||
|
||||
|
||||
def _stub_835(ingest_dir: Path, name: str = "x.835") -> Path:
|
||||
"""An *.835 file parseable enough to not crash the walker.
|
||||
|
||||
Content is just an empty 835 envelope; the rebill pipeline's
|
||||
``parse_835_svc`` reparser emits zero SVCs, which is what we want
|
||||
for the NOT_IN_835 → REBILLED_B classification.
|
||||
"""
|
||||
ingest_dir.mkdir(exist_ok=True)
|
||||
p = ingest_dir / name
|
||||
p.write_text("ST*835*0001~SE*0*0001~")
|
||||
return p
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Tests
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_post_rebill_requires_auth_or_returns_404_or_422_when_no_input(client):
|
||||
"""Bare POST with no body — should NOT 500.
|
||||
|
||||
With AUTH_DISABLED the matrix_gate short-circuits and Pydantic
|
||||
validation fires on the empty body (422). With auth enabled the
|
||||
matrix_gate returns 401/403 first. Either outcome proves the route
|
||||
is wired and the request reached the handler.
|
||||
"""
|
||||
resp = client.post("/api/admin/rebill-from-835")
|
||||
assert resp.status_code in (401, 403, 422), (
|
||||
f"unexpected status {resp.status_code}: {resp.text}"
|
||||
)
|
||||
|
||||
|
||||
def test_post_rebill_runs_and_returns_summary_path(client, tmp_path):
|
||||
"""Happy path: 1 in-window visit, 0 SVCs → REBILLED_B + summary.csv."""
|
||||
visits_path = _write_visits_csv(tmp_path / "visits.csv", [
|
||||
("06/27/2026", "J813715", "T1019", "$2.32"),
|
||||
])
|
||||
ingest_dir = _stub_835(tmp_path / "ingest")
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
resp = client.post(
|
||||
"/api/admin/rebill-from-835",
|
||||
json={
|
||||
"window": "2026-01-01..2026-06-27",
|
||||
"override_filing": False,
|
||||
"visits_csv_path": str(visits_path),
|
||||
"ingest_dir": str(ingest_dir),
|
||||
"out_dir": str(out_dir),
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert "summary_path" in body, body
|
||||
assert "counts" in body, body
|
||||
assert "pipeline_a_files" in body, body
|
||||
assert "pipeline_b_files" in body, body
|
||||
|
||||
# summary_path is real and points at the out_dir we passed in.
|
||||
assert Path(body["summary_path"]).exists(), body["summary_path"]
|
||||
assert str(out_dir) in body["summary_path"]
|
||||
|
||||
# The counts dict surfaces REBILLED_B for the in-window visit (the
|
||||
# 835 has no SVCs so the visit falls into NOT_IN_835 → REBILLED_B
|
||||
# — it's well within the 120-day gate).
|
||||
counts = body["counts"]
|
||||
assert "REBILLED_B" in counts, counts
|
||||
assert counts["REBILLED_B"] >= 1, counts
|
||||
|
||||
# Pipeline B emitted exactly one file for the one REBILLED_B visit.
|
||||
assert len(body["pipeline_b_files"]) == 1, body["pipeline_b_files"]
|
||||
|
||||
# The summary CSV header matches the writer's contract so the GET
|
||||
# /status handler can parse it.
|
||||
with open(body["summary_path"], newline="") as f:
|
||||
reader = csv.DictReader(f)
|
||||
rows = list(reader)
|
||||
assert len(rows) == 1, rows
|
||||
assert rows[0]["disposition"] == "REBILLED_B"
|
||||
assert rows[0]["member_id"] == "J813715"
|
||||
|
||||
|
||||
def test_get_rebill_status_returns_recent_runs(client, tmp_path, monkeypatch):
|
||||
"""GET /status scans the configured REBILLS_DIR and tallies per-disposition counts.
|
||||
|
||||
The handler reads :data:`cyclone.api_routers.rebill.REBILLS_DIR`
|
||||
(module-level so tests can monkeypatch it). We point that at a
|
||||
tmp_path-relative tree, seed one dated subdir with a 2-row
|
||||
summary.csv, and assert the response surfaces that directory's
|
||||
name + per-disposition tally.
|
||||
"""
|
||||
from cyclone.api_routers import rebill
|
||||
|
||||
# Build a date-named subdir with a real summary.csv shape so the
|
||||
# handler's csv.DictReader finds a `disposition` column.
|
||||
rebills_root = tmp_path / "rebills_root"
|
||||
dated_dir = rebills_root / "2026-07-07"
|
||||
dated_dir.mkdir(parents=True)
|
||||
summary = dated_dir / "summary.csv"
|
||||
with summary.open("w", newline="") as f:
|
||||
w = csv.writer(f)
|
||||
w.writerow([
|
||||
"dos", "member_id", "procedure", "billed",
|
||||
"disposition", "unpaid", "cas_reasons", "file_path",
|
||||
])
|
||||
w.writerow([
|
||||
"2026-06-27", "MEM-A", "T1019", "2.32",
|
||||
"REBILLED_B", "2.32", "", "pipeline-b/",
|
||||
])
|
||||
w.writerow([
|
||||
"2020-01-01", "MEM-B", "T1019", "2.32",
|
||||
"EXCLUDED_TIMELY_FILING", "2.32", "", "",
|
||||
])
|
||||
|
||||
# Repoint the handler's filesystem root at our tmp_path tree.
|
||||
monkeypatch.setattr(rebill, "REBILLS_DIR", rebills_root)
|
||||
|
||||
resp = client.get("/api/admin/rebill-from-835/status")
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert "recent_runs" in body, body
|
||||
assert len(body["recent_runs"]) == 1, body["recent_runs"]
|
||||
|
||||
run = body["recent_runs"][0]
|
||||
assert run["as_of"] == "2026-07-07", run
|
||||
assert "summary_path" in run, run
|
||||
assert Path(run["summary_path"]).exists(), run["summary_path"]
|
||||
counts = run["counts"]
|
||||
# Each disposition surfaced exactly once — matches the 2-row CSV.
|
||||
assert counts.get("REBILLED_B") == 1, counts
|
||||
assert counts.get("EXCLUDED_TIMELY_FILING") == 1, counts
|
||||
|
||||
|
||||
def test_post_rebill_rejects_malformed_window(client):
|
||||
"""Pydantic window validator should 422 on bad shapes (no `..`, non-dates, reversed)."""
|
||||
bad_windows = [
|
||||
"2026-01-01", # no `..` separator
|
||||
"foo..bar", # non-ISO dates
|
||||
"2026-12-01..2026-01-01", # start after end
|
||||
]
|
||||
for w in bad_windows:
|
||||
resp = client.post(
|
||||
"/api/admin/rebill-from-835",
|
||||
json={"window": w},
|
||||
)
|
||||
assert resp.status_code == 422, (
|
||||
f"expected 422 for window={w!r}, got {resp.status_code}: {resp.text}"
|
||||
)
|
||||
|
||||
|
||||
def test_get_rebill_status_returns_empty_when_no_runs(client, tmp_path, monkeypatch):
|
||||
"""GET /status on an empty rebills dir must return {"recent_runs": []}, not 500.
|
||||
|
||||
Monkeypatches REBILLS_DIR to a fresh tmp_path subdir so the test
|
||||
doesn't depend on (or pollute) any pre-existing dev/rebills/ tree.
|
||||
"""
|
||||
from cyclone.api_routers import rebill
|
||||
|
||||
monkeypatch.setattr(rebill, "REBILLS_DIR", tmp_path / "rebills_empty")
|
||||
|
||||
resp = client.get("/api/admin/rebill-from-835/status")
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert resp.json() == {"recent_runs": []}
|
||||
|
||||
@@ -51,4 +51,49 @@ def test_endpoint_regenerated_text_round_trips_through_parser():
|
||||
assert r.status_code == 200
|
||||
result = parse(r.text, PayerConfig(name="CO_MEDICAID"))
|
||||
assert result.claims, "endpoint body didn't parse back to any claims"
|
||||
assert result.claims[0].claim.claim_id == claim_id
|
||||
assert result.claims[0].claim.claim_id == claim_id
|
||||
|
||||
|
||||
def test_endpoint_emits_real_submitter_and_receiver():
|
||||
"""SP40: the single-claim download must thread real submitter +
|
||||
receiver identity through, not the serializer's placeholder
|
||||
fallback. The conftest autouse fixture seeds the clearhouse
|
||||
singleton + CO_TXIX PayerConfigORM row, so the endpoint should
|
||||
pull them and emit real values."""
|
||||
from cyclone import db as cycl_db
|
||||
from cyclone.db import ClearhouseORM
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
cycl_db.init_db()
|
||||
cycl_store.ensure_clearhouse_seeded()
|
||||
|
||||
with TestClient(app) as client:
|
||||
claim_id = _seed_claim(client)
|
||||
r = client.get(f"/api/claims/{claim_id}/serialize-837")
|
||||
assert r.status_code == 200
|
||||
body = r.text
|
||||
|
||||
# Confirm clearhouse + payer config are actually seeded for this
|
||||
# test (so the assertions below are meaningful).
|
||||
with cycl_db.SessionLocal()() as s:
|
||||
ch = s.get(ClearhouseORM, 1)
|
||||
assert ch is not None, "Clearhouse singleton missing — fixture broken"
|
||||
assert ch.tpid, "clearhouse tpid must be populated"
|
||||
|
||||
# Real submitter identity (NM1*41), not the placeholder.
|
||||
assert "Dzinesco" in body, body[:500]
|
||||
assert ch.tpid in body, body[:500]
|
||||
# Real submitter contact (PER segment).
|
||||
assert "Tyler Martinez" in body, body[:500]
|
||||
assert "tyler@dzinesco.com" in body, body[:500]
|
||||
# Real receiver (NM1*40).
|
||||
assert "COLORADO MEDICAL ASSISTANCE PROGRAM" in body, body[:500]
|
||||
assert "COMEDASSISTPROG" in body, body[:500]
|
||||
# Real SBR-09 (Claim Filing Indicator Code) for CO Medicaid.
|
||||
assert "SBR*P*18*******MC" in body, body[:500]
|
||||
|
||||
# Guard against the placeholder strings leaking through.
|
||||
assert "CUSTOMER SERVICE" not in body, body[:500]
|
||||
assert "8005550100" not in body, body[:500]
|
||||
assert "*CYCLONE *" not in body, body[:500]
|
||||
assert "*RECEIVER *" not in body, body[:500] # only NM1*40 placeholder
|
||||
@@ -0,0 +1,233 @@
|
||||
"""SP40: tests for ``POST /api/admin/validate-837`` (admin-gated).
|
||||
|
||||
Covers:
|
||||
|
||||
1. Auth gate — 401 with no session cookie (matrix_gate fires when
|
||||
AUTH_DISABLED is flipped off).
|
||||
2. Happy path — admin-cookie request with a multipart file upload
|
||||
returns the Edifabric OperationResult JSON.
|
||||
3. Missing file part — FastAPI's built-in validation rejects requests
|
||||
without the ``file`` form part with 422.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from cyclone import edifabric
|
||||
from cyclone.api import app
|
||||
|
||||
|
||||
_TEST_KEY = "test-edifabric-key-0123456789abcdef"
|
||||
|
||||
|
||||
def _make_client(handler):
|
||||
transport = httpx.MockTransport(handler)
|
||||
return httpx.Client(transport=transport, timeout=10.0)
|
||||
|
||||
|
||||
def _install(handler):
|
||||
edifabric.set_transport_factory(lambda: _make_client(handler))
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_transport():
|
||||
yield
|
||||
edifabric._reset_transport_factory()
|
||||
import cyclone.secrets
|
||||
cyclone.secrets.get_secret = _real_get_secret # type: ignore[assignment]
|
||||
|
||||
|
||||
import cyclone.secrets as _secrets_module # noqa: E402
|
||||
_real_get_secret = _secrets_module.get_secret # noqa: E402
|
||||
|
||||
|
||||
def _mock_secrets(patched_key: str):
|
||||
"""Install a get_secret mock that returns ``patched_key`` for any
|
||||
'edifabric.api_key' lookup. Restores the real function on
|
||||
teardown via the autouse ``_reset_transport`` fixture.
|
||||
|
||||
Until SP40 Task 6 lands, ``_ENV_NAME_FOR`` doesn't know about
|
||||
'edifabric.api_key', so the env-var name expected is the same —
|
||||
but the more robust pattern is to monkeypatch at the function
|
||||
boundary. That keeps test behavior independent of the secrets
|
||||
table layout.
|
||||
"""
|
||||
def _fake(name: str):
|
||||
if name == "edifabric.api_key":
|
||||
return patched_key
|
||||
return _real_get_secret(name)
|
||||
|
||||
_secrets_module.get_secret = _fake # type: ignore[assignment]
|
||||
return _fake
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_edifabric_ok():
|
||||
"""Mock Edifabric to return a clean OperationResult on /validate.
|
||||
|
||||
Wires both the transport mock and the secrets.get_secret monkeypatch
|
||||
so ``cyclone.edifabric.validate_edi(...)`` resolves an API key.
|
||||
"""
|
||||
operation_result = {
|
||||
"Status": "success",
|
||||
"Details": [],
|
||||
"LastIndex": 46,
|
||||
}
|
||||
x12 = {
|
||||
"SegmentDelimiter": "~",
|
||||
"DataElementDelimiter": "*",
|
||||
"ISA": {"InterchangeControlNumber_13": "000000001"},
|
||||
"Groups": [],
|
||||
"IEATrailers": [],
|
||||
}
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/read"):
|
||||
return httpx.Response(200, json=[x12])
|
||||
if request.url.path.endswith("/validate"):
|
||||
return httpx.Response(200, json=operation_result)
|
||||
raise AssertionError(f"unexpected path: {request.url.path}")
|
||||
|
||||
_mock_secrets(_TEST_KEY)
|
||||
_install(handler)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Auth gate
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_validate_837_no_session_returns_401(client, tmp_path, monkeypatch):
|
||||
"""No session cookie + AUTH_DISABLED off → 401 from matrix_gate.
|
||||
|
||||
Mirrors the test pattern in test_api_clearhouse_patch.py:99 — we
|
||||
flip AUTH_DISABLED to False inside the test (monkeypatch restores
|
||||
it at teardown) so the gate fires.
|
||||
"""
|
||||
import cyclone.auth.deps as auth_deps
|
||||
|
||||
monkeypatch.setattr(auth_deps, "AUTH_DISABLED", False)
|
||||
|
||||
sample = tmp_path / "ok.x12"
|
||||
sample.write_text("ISA*seg~SE*1*0001~IEA*0*000000001~")
|
||||
|
||||
with sample.open("rb") as fh:
|
||||
resp = client.post(
|
||||
"/api/admin/validate-837",
|
||||
files={"file": ("ok.x12", fh, "application/octet-stream")},
|
||||
)
|
||||
assert resp.status_code == 401, resp.text
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Happy path
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_validate_837_returns_operation_result(client, tmp_path, mock_edifabric_ok):
|
||||
"""Admin request with multipart file returns the OperationResult verbatim.
|
||||
|
||||
AUTH_DISABLED is True (set by conftest's autouse fixture), so the
|
||||
admin gate fires-and-passes; the Edifabric client is mocked to
|
||||
return a clean Success.
|
||||
"""
|
||||
fixture = tmp_path / "ok.x12"
|
||||
fixture.write_text("ISA*00*...~IEA*0*000000001~")
|
||||
|
||||
with fixture.open("rb") as fh:
|
||||
resp = client.post(
|
||||
"/api/admin/validate-837",
|
||||
files={"file": ("ok.x12", fh, "application/octet-stream")},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["Status"] == "success"
|
||||
assert body["Details"] == []
|
||||
assert body["LastIndex"] == 46
|
||||
|
||||
|
||||
def test_validate_837_returns_error_details(client, tmp_path):
|
||||
"""A 200 response with Status='error' + Details passes through verbatim.
|
||||
|
||||
The OperationResult is data; the HTTP response is 200 (the
|
||||
endpoint did its job of talking to Edifabric). Callers inspect
|
||||
Status in the body to decide whether to fail-closed.
|
||||
"""
|
||||
error_payload = {
|
||||
"Status": "error",
|
||||
"Details": [
|
||||
{"SegmentId": "PER", "Message": "PER-04 is required", "Status": "error"},
|
||||
{"SegmentId": "SBR", "Message": "SBR-09 is required", "Status": "error"},
|
||||
],
|
||||
"LastIndex": 5,
|
||||
}
|
||||
x12 = {
|
||||
"SegmentDelimiter": "~",
|
||||
"DataElementDelimiter": "*",
|
||||
"ISA": {"InterchangeControlNumber_13": "000000001"},
|
||||
"Groups": [],
|
||||
"IEATrailers": [],
|
||||
}
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/read"):
|
||||
return httpx.Response(200, json=[x12])
|
||||
if request.url.path.endswith("/validate"):
|
||||
return httpx.Response(200, json=error_payload)
|
||||
raise AssertionError(f"unexpected path: {request.url.path}")
|
||||
|
||||
_mock_secrets(_TEST_KEY)
|
||||
_install(handler)
|
||||
|
||||
fixture = tmp_path / "bad.x12"
|
||||
fixture.write_text("ISA*bad~")
|
||||
|
||||
with fixture.open("rb") as fh:
|
||||
resp = client.post(
|
||||
"/api/admin/validate-837",
|
||||
files={"file": ("bad.x12", fh, "application/octet-stream")},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["Status"] == "error"
|
||||
assert len(body["Details"]) == 2
|
||||
assert body["Details"][0]["Message"] == "PER-04 is required"
|
||||
|
||||
|
||||
def test_validate_837_missing_file_returns_422(client):
|
||||
"""No ``file`` form part → FastAPI returns 422 (built-in validation).
|
||||
|
||||
This is the FastAPI-default behavior for ``UploadFile = File(...)``
|
||||
when the field is absent. The endpoint body itself never runs.
|
||||
"""
|
||||
resp = client.post("/api/admin/validate-837")
|
||||
assert resp.status_code == 422, resp.text
|
||||
|
||||
|
||||
def test_validate_837_missing_api_key_returns_503(client, tmp_path, monkeypatch):
|
||||
"""With no API key configured, the endpoint surfaces a 503 with the
|
||||
'API key not configured' detail (status_code 0 → 503, not 502).
|
||||
|
||||
Upstream 4xx/5xx map to 502; a 0-status_code EdifabricError means
|
||||
the client-side config is missing and the caller can fix it.
|
||||
"""
|
||||
monkeypatch.delenv("CYCLONE_EDIFABRIC_API_KEY", raising=False)
|
||||
|
||||
fixture = tmp_path / "ok.x12"
|
||||
fixture.write_text("ISA*seg~")
|
||||
|
||||
with fixture.open("rb") as fh:
|
||||
resp = client.post(
|
||||
"/api/admin/validate-837",
|
||||
files={"file": ("ok.x12", fh, "application/octet-stream")},
|
||||
)
|
||||
assert resp.status_code == 503, resp.text
|
||||
body = resp.json()
|
||||
assert "API key not configured" in str(body)
|
||||
@@ -34,6 +34,7 @@ from cyclone.claim_acks import (
|
||||
lookup_claims_for_ack_set_response,
|
||||
)
|
||||
from cyclone.db import (
|
||||
Ack,
|
||||
Batch,
|
||||
Claim,
|
||||
ClaimState,
|
||||
@@ -744,6 +745,75 @@ def test_store_facade_exposes_claim_ack_methods():
|
||||
assert hasattr(store, name), f"missing CycloneStore.{name}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP38 follow-up: regression test for the 277ca / ta1 control_number
|
||||
# behavior preserved across the find_ack_orphans refactor.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_find_ack_orphans_returns_control_number_for_all_kinds():
|
||||
"""For each ack kind, ``find_ack_orphans(kind)`` must return the
|
||||
control_number from the per-kind source:
|
||||
|
||||
* 999 — from raw_json.envelope.control_number
|
||||
* 277ca — from the ORM column ``control_number``
|
||||
* ta1 — from the ORM column ``control_number``
|
||||
|
||||
Regression for sp38 commit ad14b56 which initially routed all
|
||||
three kinds through _ack_control_number and broke 277ca/ta1
|
||||
(the helper only knew 999). Found by pr-reviewer 2026-07-07.
|
||||
"""
|
||||
import json
|
||||
import cyclone.db as _db_mod
|
||||
Two77caAck = _db_mod.Two77caAck
|
||||
Ta1Ack = _db_mod.Ta1Ack
|
||||
parsed_at = datetime.now(timezone.utc)
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
# 999 orphan with envelope.control_number in raw_json
|
||||
s.add(Ack(
|
||||
source_batch_id="999-orph-test",
|
||||
accepted_count=1, rejected_count=0, received_count=1,
|
||||
ack_code="A",
|
||||
parsed_at=parsed_at,
|
||||
raw_json=json.dumps({
|
||||
"envelope": {"control_number": "999-ctrl-num",
|
||||
"sender_id": "S", "receiver_id": "R",
|
||||
"transaction_date": "2024-01-01",
|
||||
"implementation_guide": "005010X231A1"},
|
||||
"set_responses": [],
|
||||
"summary": {},
|
||||
}),
|
||||
))
|
||||
# 277ca orphan with control_number in ORM column
|
||||
s.add(Two77caAck(
|
||||
source_batch_id="277ca-orph-test",
|
||||
accepted_count=1, rejected_count=0,
|
||||
control_number="277CA-CTRL-NUM",
|
||||
parsed_at=parsed_at,
|
||||
))
|
||||
# ta1 orphan with control_number in ORM column
|
||||
s.add(Ta1Ack(
|
||||
source_batch_id="ta1-orph-test",
|
||||
ack_code="A",
|
||||
control_number="TA1-CTRL-NUM",
|
||||
parsed_at=parsed_at,
|
||||
))
|
||||
s.commit()
|
||||
|
||||
orphans_999 = store.find_ack_orphans("999")
|
||||
orphans_277ca = store.find_ack_orphans("277ca")
|
||||
orphans_ta1 = store.find_ack_orphans("ta1")
|
||||
|
||||
ctrl_999 = [o["control_number"] for o in orphans_999 if "999-ctrl-num" in o["control_number"]]
|
||||
ctrl_277ca = [o["control_number"] for o in orphans_277ca]
|
||||
ctrl_ta1 = [o["control_number"] for o in orphans_ta1]
|
||||
|
||||
assert "999-ctrl-num" in ctrl_999, f"999 control_number not found: {ctrl_999}"
|
||||
assert "277CA-CTRL-NUM" in ctrl_277ca, f"277ca control_number empty: {ctrl_277ca}"
|
||||
assert "TA1-CTRL-NUM" in ctrl_ta1, f"ta1 control_number empty: {ctrl_ta1}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 2.1 — pure walk-through unit test (no DB) for the orphan tracking
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,363 @@
|
||||
"""SP41 — tests for the `cyclone rebill-from-835` CLI.
|
||||
|
||||
Three smoke tests:
|
||||
1. --help renders cleanly and surfaces --window and --override-filing.
|
||||
2. --status (with no other args) renders cleanly and exits 0 — the
|
||||
option is recognized even when no prior summary.csv exists.
|
||||
3. End-to-end run against a 2-row visits CSV (one in-window, one
|
||||
ancient) and a zero-SVC 835 fixture produces a summary.csv with
|
||||
the in-window visit classified as REBILLED_B.
|
||||
|
||||
Plus five disposition-coverage tests added during the Task 10 review:
|
||||
4. EXCLUDED_CARC: CO-45 on a denied visit → not rebilled.
|
||||
5. REBILLED_A: CO-97 on a denied visit → pipeline-a freq-7.
|
||||
6. PAID visits are filtered out of the summary.
|
||||
7. EXCLUDED_TIMELY_FILING vs REBILLED_B via --override-filing (two runs).
|
||||
8. --status on a real summary.csv prints the per-disposition tally.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
from datetime import date as _date
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
from cyclone.cli import main
|
||||
from cyclone.rebill.parse_835_svc import SvcRow
|
||||
from cyclone.rebill.reconcile import VisitRow
|
||||
|
||||
|
||||
def _write_visits_csv(path: Path, rows: list[tuple[str, str, str, str]]) -> Path:
|
||||
"""rows: list of (dos_mmddyyyy, member, procedure, billed_str)."""
|
||||
with path.open("w", newline="") as f:
|
||||
w = csv.writer(f)
|
||||
w.writerow(["Visit Date", "Member ID", "Procedure Code", "Billable Amount"])
|
||||
for r in rows:
|
||||
w.writerow(r)
|
||||
return path
|
||||
|
||||
|
||||
def _read_summary(out_dir: Path) -> list[dict[str, str]]:
|
||||
with (out_dir / "summary.csv").open(newline="") as f:
|
||||
return list(csv.DictReader(f))
|
||||
|
||||
|
||||
def _stub_svc(monkeypatch, svcs: list[SvcRow]) -> None:
|
||||
"""Replace cli.parse_835_svc's underlying cyclonic call with a fixed list."""
|
||||
# cli.rebill_from_835 imports parse_835_svc locally, so patch the
|
||||
# source module's symbol — the local `from ... import parse_835_svc`
|
||||
# binds the symbol at call time on each invocation.
|
||||
monkeypatch.setattr(
|
||||
"cyclone.rebill.parse_835_svc.parse_835_svc",
|
||||
lambda path, _svcs=svcs: iter(_svcs),
|
||||
)
|
||||
|
||||
|
||||
def _make_835_placeholder(ingest_dir: Path, name: str = "x.835") -> Path:
|
||||
"""A real *.835 file present in the glob; parse_835_svc is mocked so content doesn't matter."""
|
||||
ingest_dir.mkdir(exist_ok=True)
|
||||
p = ingest_dir / name
|
||||
p.write_text("ST*835*0001~SE*0*0001~")
|
||||
return p
|
||||
|
||||
|
||||
def test_rebill_from_835_help():
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(main, ["rebill-from-835", "--help"])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "--window" in result.output
|
||||
assert "--override-filing" in result.output
|
||||
|
||||
|
||||
def test_rebill_from_835_status_help():
|
||||
"""Either bare --status or --status --help should exit 0.
|
||||
|
||||
The spec note: the help-test ensures the option is recognized, so
|
||||
even a "no prior runs" 0-exit path is acceptable. We try both
|
||||
invocations because click short-circuits --help ahead of any
|
||||
required-option check; the bare --status path is the one that
|
||||
proves --status works without --visits / --ingest.
|
||||
"""
|
||||
runner = CliRunner()
|
||||
result_help = runner.invoke(main, ["rebill-from-835", "--status", "--help"])
|
||||
assert result_help.exit_code == 0, result_help.output
|
||||
|
||||
result_bare = runner.invoke(main, ["rebill-from-835", "--status"])
|
||||
assert result_bare.exit_code == 0, result_bare.output
|
||||
|
||||
|
||||
def test_rebill_from_835_runs(tmp_path: Path):
|
||||
"""End-to-end: 1 in-window visit + 1 ancient visit + zero-SVC 835 → REBILLED_B for the in-window row."""
|
||||
visits_path = tmp_path / "visits.csv"
|
||||
visits_path.write_text(
|
||||
"Visit Date,Member ID,Procedure Code,Billable Amount\n"
|
||||
"06/27/2026,J813715,T1019,$2.32\n"
|
||||
"01/01/2020,ANCIENT,T1019,$2.32\n"
|
||||
)
|
||||
|
||||
ingest_dir = tmp_path / "ingest"
|
||||
ingest_dir.mkdir()
|
||||
(ingest_dir / "x.835").write_text("ST*835*0001~SE*0*0001~")
|
||||
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(main, [
|
||||
"rebill-from-835",
|
||||
"--visits", str(visits_path),
|
||||
"--ingest", str(ingest_dir),
|
||||
"--out", str(out_dir),
|
||||
"--as-of", "2026-07-07",
|
||||
])
|
||||
assert result.exit_code == 0, (
|
||||
f"CLI exited {result.exit_code}, output={result.output!r}, "
|
||||
f"exception={result.exception!r}"
|
||||
)
|
||||
|
||||
summary_path = out_dir / "summary.csv"
|
||||
assert summary_path.exists(), summary_path
|
||||
|
||||
rows = _read_summary(out_dir)
|
||||
# Two visits in, two summary rows out (one REBILLED_B, one
|
||||
# EXCLUDED_TIMELY_FILING). PAID outcomes would be dropped, but
|
||||
# neither of these is PAID — the 835 has no SVCs.
|
||||
assert len(rows) == 2, rows
|
||||
|
||||
by_member = {r["member_id"]: r for r in rows}
|
||||
in_window = by_member["J813715"]
|
||||
assert in_window["disposition"] == "REBILLED_B", in_window
|
||||
assert in_window["dos"] == "2026-06-27", in_window
|
||||
|
||||
ancient = by_member["ANCIENT"]
|
||||
assert ancient["disposition"] == "EXCLUDED_TIMELY_FILING", ancient
|
||||
assert ancient["dos"] == "2020-01-01", ancient
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Disposition coverage (Task 10 review)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_rebill_carc_excluded_visit_landed_as_excluded_carc(tmp_path: Path, monkeypatch):
|
||||
"""A denied visit whose CAS list contains CO-45 (excluded) must land as EXCLUDED_CARC, not REBILLED_A."""
|
||||
visits = _write_visits_csv(tmp_path / "visits.csv", [
|
||||
("06/27/2026", "J813715", "T1019", "$16.24"),
|
||||
])
|
||||
ingest = tmp_path / "ingest"
|
||||
_make_835_placeholder(ingest)
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
# A denied SVC for the visit (paid == 0) with CO-45 in CAS.
|
||||
svc = SvcRow(
|
||||
src_file="x.835",
|
||||
claim_id="T1",
|
||||
member_id="J813715",
|
||||
status="4", # denied
|
||||
procedure="T1019",
|
||||
modifiers="",
|
||||
charge=Decimal("16.24"),
|
||||
paid=Decimal("0"),
|
||||
units=Decimal("1"),
|
||||
svc_date=_date(2026, 6, 27),
|
||||
cas_reasons=("CO-45",),
|
||||
pay_date=None,
|
||||
)
|
||||
_stub_svc(monkeypatch, [svc])
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(main, [
|
||||
"rebill-from-835",
|
||||
"--visits", str(visits),
|
||||
"--ingest", str(ingest),
|
||||
"--out", str(out_dir),
|
||||
"--as-of", "2026-07-07",
|
||||
])
|
||||
assert result.exit_code == 0, (result.output, result.exception)
|
||||
rows = _read_summary(out_dir)
|
||||
assert len(rows) == 1, rows
|
||||
assert rows[0]["disposition"] == "EXCLUDED_CARC"
|
||||
assert rows[0]["member_id"] == "J813715"
|
||||
assert "CO-45" in rows[0]["cas_reasons"]
|
||||
|
||||
|
||||
def test_rebill_denied_rebill_visit_landed_as_rebilled_a(tmp_path: Path, monkeypatch):
|
||||
"""A denied visit whose CAS list contains CO-97 (rebillable) must land as REBILLED_A."""
|
||||
visits = _write_visits_csv(tmp_path / "visits.csv", [
|
||||
("06/27/2026", "J813715", "T1019", "$16.24"),
|
||||
])
|
||||
ingest = tmp_path / "ingest"
|
||||
_make_835_placeholder(ingest)
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
svc = SvcRow(
|
||||
src_file="x.835",
|
||||
claim_id="T1",
|
||||
member_id="J813715",
|
||||
status="4", # denied
|
||||
procedure="T1019",
|
||||
modifiers="",
|
||||
charge=Decimal("16.24"),
|
||||
paid=Decimal("0"),
|
||||
units=Decimal("1"),
|
||||
svc_date=_date(2026, 6, 27),
|
||||
cas_reasons=("CO-97",), # rebillable (not in EXCLUDED_CARCS or REVIEW_CARCS)
|
||||
pay_date=None,
|
||||
)
|
||||
_stub_svc(monkeypatch, [svc])
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(main, [
|
||||
"rebill-from-835",
|
||||
"--visits", str(visits),
|
||||
"--ingest", str(ingest),
|
||||
"--out", str(out_dir),
|
||||
"--as-of", "2026-07-07",
|
||||
])
|
||||
assert result.exit_code == 0, (result.output, result.exception)
|
||||
rows = _read_summary(out_dir)
|
||||
assert len(rows) == 1, rows
|
||||
assert rows[0]["disposition"] == "REBILLED_A"
|
||||
assert rows[0]["file_path"] == "pipeline-a/"
|
||||
assert "CO-97" in rows[0]["cas_reasons"]
|
||||
|
||||
|
||||
def test_rebill_paid_visit_excluded_from_summary(tmp_path: Path, monkeypatch):
|
||||
"""PAID visits (paid >= 95% of billed) must not appear in summary.csv — only NOT_IN_835 should."""
|
||||
visits = _write_visits_csv(tmp_path / "visits.csv", [
|
||||
("06/27/2026", "PAID-MEMBER", "T1019", "$100.00"),
|
||||
("06/27/2026", "MISSING-MEMBER", "T1019", "$2.32"),
|
||||
])
|
||||
ingest = tmp_path / "ingest"
|
||||
_make_835_placeholder(ingest)
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
# One fully-paid SVC for PAID-MEMBER; nothing matches MISSING-MEMBER.
|
||||
paid_svc = SvcRow(
|
||||
src_file="x.835",
|
||||
claim_id="T1",
|
||||
member_id="PAID-MEMBER",
|
||||
status="1",
|
||||
procedure="T1019",
|
||||
modifiers="",
|
||||
charge=Decimal("100.00"),
|
||||
paid=Decimal("100.00"),
|
||||
units=Decimal("1"),
|
||||
svc_date=_date(2026, 6, 27),
|
||||
cas_reasons=(),
|
||||
pay_date=None,
|
||||
)
|
||||
_stub_svc(monkeypatch, [paid_svc])
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(main, [
|
||||
"rebill-from-835",
|
||||
"--visits", str(visits),
|
||||
"--ingest", str(ingest),
|
||||
"--out", str(out_dir),
|
||||
"--as-of", "2026-07-07",
|
||||
])
|
||||
assert result.exit_code == 0, (result.output, result.exception)
|
||||
rows = _read_summary(out_dir)
|
||||
assert len(rows) == 1, rows
|
||||
assert rows[0]["member_id"] == "MISSING-MEMBER"
|
||||
assert rows[0]["disposition"] == "REBILLED_B"
|
||||
|
||||
|
||||
def test_rebill_override_filing_makes_ancient_visit_rebilled_b(tmp_path: Path, monkeypatch):
|
||||
"""A 2020 visit without --override-filing → EXCLUDED_TIMELY_FILING; with it → REBILLED_B."""
|
||||
# No SVCs at all — both visits fall into NOT_IN_835.
|
||||
visits = _write_visits_csv(tmp_path / "visits.csv", [
|
||||
("01/01/2020", "ANCIENT", "T1019", "$2.32"),
|
||||
])
|
||||
ingest = tmp_path / "ingest"
|
||||
_make_835_placeholder(ingest)
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
# Stub the (local) parse_835_svc import with an empty generator so
|
||||
# the CLI sees no SVCs and every visit is NOT_IN_835.
|
||||
monkeypatch.setattr(
|
||||
"cyclone.rebill.parse_835_svc.parse_835_svc",
|
||||
lambda path: iter([]),
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
# Run 1: as-of 2026-07-07 minus 2020-01-01 = ~6.5 years > 120 days → EXCLUDED.
|
||||
r1 = runner.invoke(main, [
|
||||
"rebill-from-835",
|
||||
"--visits", str(visits),
|
||||
"--ingest", str(ingest),
|
||||
"--out", str(out_dir),
|
||||
"--as-of", "2026-07-07",
|
||||
])
|
||||
assert r1.exit_code == 0, (r1.output, r1.exception)
|
||||
rows1 = _read_summary(out_dir)
|
||||
assert len(rows1) == 1, rows1
|
||||
assert rows1[0]["disposition"] == "EXCLUDED_TIMELY_FILING", rows1
|
||||
|
||||
# Run 2: --override-filing bypasses the 120-day gate → REBILLED_B.
|
||||
r2 = runner.invoke(main, [
|
||||
"rebill-from-835",
|
||||
"--visits", str(visits),
|
||||
"--ingest", str(ingest),
|
||||
"--out", str(out_dir),
|
||||
"--as-of", "2026-07-07",
|
||||
"--override-filing",
|
||||
])
|
||||
assert r2.exit_code == 0, (r2.output, r2.exception)
|
||||
rows2 = _read_summary(out_dir)
|
||||
assert len(rows2) == 1, rows2
|
||||
assert rows2[0]["disposition"] == "REBILLED_B", rows2
|
||||
assert rows2[0]["file_path"] == "pipeline-b/"
|
||||
|
||||
|
||||
def test_rebill_status_with_real_summary_prints_tally(tmp_path: Path):
|
||||
"""--status against a real summary.csv must print both REBILLED_B and EXCLUDED_TIMELY_FILING rows."""
|
||||
from cyclone.rebill.reconcile import VisitRow
|
||||
from cyclone.rebill.summary import (
|
||||
EXCLUDED_TIMELY_FILING,
|
||||
REBILLED_B,
|
||||
SummaryRow,
|
||||
write_summary_csv,
|
||||
)
|
||||
|
||||
out_dir = tmp_path / "dev" / "rebills" / "2026-07-07"
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
summary_path = out_dir / "summary.csv"
|
||||
|
||||
rows = [
|
||||
SummaryRow(
|
||||
visit=VisitRow(
|
||||
date=_date(2026, 6, 27), member_id="MEM-A",
|
||||
procedure="T1019", billed=Decimal("2.32"),
|
||||
),
|
||||
disposition=REBILLED_B,
|
||||
unpaid=Decimal("2.32"),
|
||||
cas_reasons=(),
|
||||
file_path="pipeline-b/",
|
||||
),
|
||||
SummaryRow(
|
||||
visit=VisitRow(
|
||||
date=_date(2026, 6, 27), member_id="MEM-B",
|
||||
procedure="T1019", billed=Decimal("2.32"),
|
||||
),
|
||||
disposition=EXCLUDED_TIMELY_FILING,
|
||||
unpaid=Decimal("2.32"),
|
||||
cas_reasons=(),
|
||||
file_path="",
|
||||
),
|
||||
]
|
||||
write_summary_csv(rows, summary_path)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(main, [
|
||||
"rebill-from-835",
|
||||
"--status",
|
||||
"--out", str(out_dir),
|
||||
])
|
||||
assert result.exit_code == 0, (result.output, result.exception)
|
||||
assert "REBILLED_B" in result.output
|
||||
assert "EXCLUDED_TIMELY_FILING" in result.output
|
||||
assert "TOTAL" in result.output
|
||||
@@ -0,0 +1,161 @@
|
||||
"""SP25: integration tests for ``cyclone recover-ingest`` (parse + DB-write recovery).
|
||||
|
||||
Fixtures used (all real production EDI from the test fixture corpus):
|
||||
- ``co_medicaid_837p.txt`` — 1-claim 837P fixture (small, fast).
|
||||
- ``co_medicaid_835.txt`` — minimal 835 fixture (real CO Medicaid).
|
||||
|
||||
These tests do NOT cover the operator's full ingest/ batch — that's
|
||||
Task 6 in the plan and runs against the live DB after these unit
|
||||
tests are green.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from cyclone import db, store
|
||||
from cyclone.cli import main as cli_main
|
||||
from cyclone.db import Batch, Claim, ProcessedInboundFile, Remittance
|
||||
|
||||
FIX_837P = Path(__file__).parent / "fixtures" / "co_medicaid_837p.txt"
|
||||
FIX_835 = Path(__file__).parent / "fixtures" / "co_medicaid_835.txt"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def runner():
|
||||
return CliRunner()
|
||||
|
||||
|
||||
# --- helpers --------------------------------------------------------------
|
||||
|
||||
def _count_claims() -> int:
|
||||
s = db.SessionLocal()()
|
||||
try:
|
||||
return s.query(Claim).count()
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
|
||||
def _count_remittances() -> int:
|
||||
s = db.SessionLocal()()
|
||||
try:
|
||||
return s.query(Remittance).count()
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
|
||||
def _count_batches() -> int:
|
||||
s = db.SessionLocal()()
|
||||
try:
|
||||
return s.query(Batch).count()
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
|
||||
# --- tests ----------------------------------------------------------------
|
||||
|
||||
def test_recover_ingest_837p_persists_claims(runner):
|
||||
"""One 837P ingest via the CLI lands a Batch + ≥1 Claim row + a dedup row."""
|
||||
pre_claims = _count_claims()
|
||||
pre_batches = _count_batches()
|
||||
result = runner.invoke(
|
||||
cli_main,
|
||||
["recover-ingest", "--file", str(FIX_837P),
|
||||
"--sftp-block-name", "test-recover-837p"],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
assert result.exit_code == 0, result.stderr or result.stdout
|
||||
out = (result.stdout or "") + (result.stderr or "")
|
||||
assert "ok" in out
|
||||
assert FIX_837P.name in out
|
||||
|
||||
assert _count_claims() > pre_claims, "expected new Claim rows"
|
||||
assert _count_batches() > pre_batches, "expected new Batch row"
|
||||
|
||||
# Dedup row created.
|
||||
s = db.SessionLocal()()
|
||||
try:
|
||||
row = s.query(ProcessedInboundFile).filter_by(
|
||||
sftp_block_name="test-recover-837p", name=FIX_837P.name,
|
||||
).first()
|
||||
finally:
|
||||
s.close()
|
||||
assert row is not None, "expected processed_inbound_files row"
|
||||
assert row.status == "ok"
|
||||
assert row.file_type == "837p"
|
||||
|
||||
|
||||
def test_recover_ingest_835_persists_remittance(runner):
|
||||
"""One 835 ingest via the CLI lands a Batch + Remittance row + dedup row."""
|
||||
pre_batches = _count_batches()
|
||||
pre_remits = _count_remittances()
|
||||
result = runner.invoke(
|
||||
cli_main,
|
||||
["recover-ingest", "--file", str(FIX_835),
|
||||
"--sftp-block-name", "test-recover-835"],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
assert result.exit_code == 0, result.stderr or result.stdout
|
||||
assert "ok" in (result.stdout or "") + (result.stderr or "")
|
||||
|
||||
assert _count_batches() > pre_batches
|
||||
assert _count_remittances() > pre_remits
|
||||
|
||||
s = db.SessionLocal()()
|
||||
try:
|
||||
row = s.query(ProcessedInboundFile).filter_by(
|
||||
sftp_block_name="test-recover-835", name=FIX_835.name,
|
||||
).first()
|
||||
finally:
|
||||
s.close()
|
||||
assert row is not None
|
||||
assert row.status == "ok"
|
||||
assert row.file_type == "835"
|
||||
assert row.claim_count >= 1
|
||||
|
||||
|
||||
def test_recover_ingest_is_idempotent(runner):
|
||||
"""Second invocation with the same (block, file) is a no-op (duplicate)."""
|
||||
# First call — must ingest.
|
||||
r1 = runner.invoke(
|
||||
cli_main,
|
||||
["recover-ingest", "--file", str(FIX_837P),
|
||||
"--sftp-block-name", "test-recover-idem"],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
assert r1.exit_code == 0, r1.stderr or r1.stdout
|
||||
|
||||
pre_claims = _count_claims()
|
||||
pre_batches = _count_batches()
|
||||
# Second call — must skip.
|
||||
r2 = runner.invoke(
|
||||
cli_main,
|
||||
["recover-ingest", "--file", str(FIX_837P),
|
||||
"--sftp-block-name", "test-recover-idem"],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
assert r2.exit_code == 0, r2.stderr or r2.stdout
|
||||
assert "skipped" in (r2.stdout or "") + (r2.stderr or "")
|
||||
|
||||
# No new rows on second call.
|
||||
assert _count_claims() == pre_claims
|
||||
assert _count_batches() == pre_batches
|
||||
|
||||
|
||||
def test_recover_ingest_missing_file_returns_failed(runner, tmp_path):
|
||||
"""Missing file → Click rejects with usage error (the CLI uses exists=True)."""
|
||||
missing = tmp_path / "does-not-exist.837"
|
||||
pre_claims = _count_claims()
|
||||
result = runner.invoke(
|
||||
cli_main,
|
||||
["recover-ingest", "--file", str(missing),
|
||||
"--sftp-block-name", "test-recover-missing"],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
# Click rejects up-front; usage error exits with code 2.
|
||||
assert result.exit_code == 2
|
||||
out = (result.stdout or "") + (result.stderr or "")
|
||||
assert "does not exist" in out or "Invalid value" in out
|
||||
assert _count_claims() == pre_claims
|
||||
@@ -0,0 +1,183 @@
|
||||
"""SP24 — tests for the `cyclone reissue-claims` CLI subcommand.
|
||||
|
||||
Five smoke tests covering the canonical cyclone-cli pattern:
|
||||
|
||||
1. --help renders cleanly and lists the long-form flags.
|
||||
2. happy path: a single-file input dir produces the expected
|
||||
number of output files.
|
||||
3. empty input dir: exits 2 with a "PARSE FAILED" message.
|
||||
4. IG-correctness guard fires when the constant is monkeypatched
|
||||
to True; CLI exits 1 with the REFUSING log line.
|
||||
5. --zip-output produces a valid zip with round-trip integrity.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from cyclone.cli import main
|
||||
|
||||
# Canonical minimal-claim fixture — flat, module-level Path constant
|
||||
# per the cyclone-tests convention. NEVER reach into docs/prodfiles/.
|
||||
MINIMAL_837P = Path(__file__).parent / "fixtures" / "minimal_837p.txt"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def _input_dir(tmp_path: Path) -> Path:
|
||||
"""Drop the minimal_837p fixture into tmp_path/in/."""
|
||||
in_dir = tmp_path / "in"
|
||||
in_dir.mkdir()
|
||||
(in_dir / "real.x12").write_text(MINIMAL_837P.read_text())
|
||||
return in_dir
|
||||
|
||||
|
||||
def test_reissue_claims_help_renders():
|
||||
"""--help exits 0 and lists every long-form flag in the SP24 spec."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(main, ["reissue-claims", "--help"])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
# Required flag.
|
||||
assert "--input-dir" in result.output
|
||||
# Optional flags documented in the spec §2 Decision 3.
|
||||
for flag in [
|
||||
"--output-root",
|
||||
"--date",
|
||||
"--pipeline",
|
||||
"--payer",
|
||||
"--sender-id",
|
||||
"--receiver-id",
|
||||
"--submitter-name",
|
||||
"--submitter-contact-name",
|
||||
"--submitter-contact-email",
|
||||
"--receiver-name",
|
||||
"--zip-output",
|
||||
"--no-clean",
|
||||
"--log-level",
|
||||
]:
|
||||
assert flag in result.output, f"missing {flag} in help output"
|
||||
|
||||
|
||||
def test_reissue_claims_happy_path(_input_dir: Path, tmp_path: Path):
|
||||
"""A single-file input dir produces 1 .x12 output + summary sidecar."""
|
||||
runner = CliRunner()
|
||||
out_root = tmp_path / "out"
|
||||
|
||||
result = runner.invoke(
|
||||
main,
|
||||
[
|
||||
"reissue-claims",
|
||||
"--input-dir", str(_input_dir),
|
||||
"--output-root", str(out_root),
|
||||
"--date", "2026-07-08",
|
||||
"--pipeline", "initial",
|
||||
],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
|
||||
assert result.exit_code == 0, (
|
||||
f"CLI exited {result.exit_code}, output={result.output!r}, "
|
||||
f"exception={result.exception!r}"
|
||||
)
|
||||
# 1 .x12 + 1 _serialize_summary.json = 2 entries.
|
||||
emitted_dir = out_root / "2026-07-08" / "initial"
|
||||
assert emitted_dir.is_dir()
|
||||
files = sorted(emitted_dir.iterdir())
|
||||
x12s = [f for f in files if f.suffix == ".x12"]
|
||||
assert len(x12s) == 1
|
||||
assert (emitted_dir / "_serialize_summary.json").is_file()
|
||||
# HCPF-spec filename.
|
||||
assert x12s[0].name.startswith("tp11525703-837P-")
|
||||
assert x12s[0].name.endswith("-1of1.x12")
|
||||
# DONE summary line is in stdout.
|
||||
assert "DONE files=1 errors=0" in result.output
|
||||
|
||||
|
||||
def test_reissue_claims_empty_input_exits_2(tmp_path: Path):
|
||||
"""An empty input dir exits 2 with a clear PARSE FAILED message."""
|
||||
empty_dir = tmp_path / "empty"
|
||||
empty_dir.mkdir()
|
||||
out_root = tmp_path / "out"
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
main,
|
||||
[
|
||||
"reissue-claims",
|
||||
"--input-dir", str(empty_dir),
|
||||
"--output-root", str(out_root),
|
||||
],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
|
||||
# No claims → exit 2.
|
||||
assert result.exit_code == 2, (
|
||||
f"CLI exited {result.exit_code}, output={result.output!r}"
|
||||
)
|
||||
# The error message should explain the failure.
|
||||
combined = result.output + (result.stderr or "")
|
||||
assert "PARSE FAILED" in combined or "no claims" in combined.lower()
|
||||
|
||||
|
||||
def test_reissue_claims_ig_correctness_guard_fires(monkeypatch, _input_dir: Path, tmp_path: Path):
|
||||
"""When PATIENT_LOOP_DEFAULT_INCLUDED is True, the CLI refuses to run."""
|
||||
# Monkeypatch the serializer constant to the broken value.
|
||||
# `raising=False` lets us set an attribute that didn't exist
|
||||
# pre-import (defensive against pytest collection order).
|
||||
from cyclone.parsers import serialize_837 as ser_mod
|
||||
monkeypatch.setattr(ser_mod, "PATIENT_LOOP_DEFAULT_INCLUDED", True)
|
||||
|
||||
runner = CliRunner()
|
||||
out_root = tmp_path / "out"
|
||||
result = runner.invoke(
|
||||
main,
|
||||
[
|
||||
"reissue-claims",
|
||||
"--input-dir", str(_input_dir),
|
||||
"--output-root", str(out_root),
|
||||
],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
|
||||
# Guard fires → exit 1.
|
||||
assert result.exit_code == 1, (
|
||||
f"CLI exited {result.exit_code} (expected 1); output={result.output!r}"
|
||||
)
|
||||
# The refusal message is in stderr (click.echo(..., err=True)).
|
||||
combined = result.output + (result.stderr or "")
|
||||
assert "REFUSING to run" in combined
|
||||
assert "PATIENT_LOOP_DEFAULT_INCLUDED" in combined
|
||||
|
||||
|
||||
def test_reissue_claims_zip_output_round_trip(_input_dir: Path, tmp_path: Path):
|
||||
"""--zip-output writes a zip whose testzip() returns None."""
|
||||
runner = CliRunner()
|
||||
out_root = tmp_path / "out"
|
||||
zip_path = tmp_path / "out.zip"
|
||||
|
||||
result = runner.invoke(
|
||||
main,
|
||||
[
|
||||
"reissue-claims",
|
||||
"--input-dir", str(_input_dir),
|
||||
"--output-root", str(out_root),
|
||||
"--date", "2026-07-08",
|
||||
"--zip-output", str(zip_path),
|
||||
],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
assert zip_path.is_file()
|
||||
|
||||
with zipfile.ZipFile(zip_path) as zf:
|
||||
# None = no corrupt entries.
|
||||
assert zf.testzip() is None
|
||||
names = zf.namelist()
|
||||
# 1 X12 file in the zip (the summary sidecar is not zipped).
|
||||
assert len(names) == 1
|
||||
assert names[0].startswith("tp11525703-837P-")
|
||||
assert names[0].endswith("-1of1.x12")
|
||||
@@ -127,14 +127,16 @@ def test_migration_latest_idempotent_on_fresh_db(tmp_path: Path) -> None:
|
||||
SP32 bumped it to 19 with rendering_provider_npi +
|
||||
service_provider_npi on claims and remittances.
|
||||
SP37 bumped it to 20 with batch transaction_set_control_number.
|
||||
SP39 bumped it to 21 with the resubmissions audit table.
|
||||
SP41 bumped it to 22 with submission_dedup.
|
||||
"""
|
||||
engine = _fresh_engine(tmp_path)
|
||||
db_migrate.run(engine)
|
||||
v_after_first = _user_version(engine)
|
||||
assert v_after_first == 20, f"expected head=20, got {v_after_first}"
|
||||
assert v_after_first == 23, f"expected head=23, got {v_after_first}"
|
||||
|
||||
db_migrate.run(engine)
|
||||
assert _user_version(engine) == 20, "second run should not bump version"
|
||||
assert _user_version(engine) == 23, "second run should not bump version"
|
||||
|
||||
|
||||
def test_drop_claims_unique_constraint_migration(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
@@ -160,7 +162,7 @@ def test_drop_claims_unique_constraint_migration(tmp_path: Path, monkeypatch: py
|
||||
engine = _fresh_engine(tmp_path)
|
||||
|
||||
db_migrate.run(engine)
|
||||
assert _user_version(engine) == 20, f"expected head=20, got {_user_version(engine)}"
|
||||
assert _user_version(engine) == 23, f"expected head=23, got {_user_version(engine)}"
|
||||
|
||||
# Two claims in one batch with the same patient_control_number
|
||||
# must be insertable. If 0015's table recreation re-introduced a
|
||||
@@ -187,3 +189,50 @@ def test_drop_claims_unique_constraint_migration(tmp_path: Path, monkeypatch: py
|
||||
assert [r[0] for r in rows] == ["CLM-1", "CLM-2"]
|
||||
assert [float(r[1]) for r in rows] == [100.0, 200.0]
|
||||
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP39: migration 0021 creates the resubmissions table
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_migration_0021_creates_resubmissions_table(tmp_path: Path) -> None:
|
||||
"""SP39: 0021_resubmissions.sql adds the resubmissions audit table
|
||||
with the documented columns + unique constraint on
|
||||
(claim_id, interchange_control_number)."""
|
||||
# Point the runner at the REAL migrations dir so we exercise 0021.
|
||||
real_dir = Path(db_migrate.__file__).parent / "migrations"
|
||||
import cyclone.db_migrate as real_migrate_mod
|
||||
real_dir_str = str(real_dir)
|
||||
|
||||
engine = _fresh_engine(tmp_path)
|
||||
# monkeypatch the module-level MIGRATIONS_DIR
|
||||
import importlib
|
||||
monkey_save = real_migrate_mod.MIGRATIONS_DIR
|
||||
real_migrate_mod.MIGRATIONS_DIR = Path(real_dir_str)
|
||||
try:
|
||||
db_migrate.run(engine)
|
||||
finally:
|
||||
real_migrate_mod.MIGRATIONS_DIR = monkey_save
|
||||
|
||||
# Verify the table exists with the expected columns + unique index.
|
||||
with engine.connect() as conn:
|
||||
version = conn.exec_driver_sql("PRAGMA user_version").scalar()
|
||||
assert version >= 21
|
||||
cols = conn.exec_driver_sql(
|
||||
"PRAGMA table_info(resubmissions)"
|
||||
).all()
|
||||
col_names = {row[1] for row in cols}
|
||||
assert col_names == {
|
||||
"id", "claim_id", "batch_id", "resubmitted_at",
|
||||
"source_corrected_path",
|
||||
"interchange_control_number", "group_control_number",
|
||||
}
|
||||
idx_rows = conn.exec_driver_sql(
|
||||
"SELECT name, sql FROM sqlite_master "
|
||||
"WHERE type='index' AND tbl_name='resubmissions'"
|
||||
).all()
|
||||
idx_names = {row[0] for row in idx_rows}
|
||||
assert "ix_resubmissions_claim_id" in idx_names
|
||||
assert "ix_resubmissions_batch_id" in idx_names
|
||||
assert "ux_resubmissions_claim_icn" in idx_names
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
"""SP40: tests for the cyclone.edifabric HTTP client.
|
||||
|
||||
All tests use httpx.MockTransport — no live HTTP hits the network.
|
||||
The API key is supplied directly via the ``api_key=`` kwarg so the
|
||||
secrets module is never read during tests.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from cyclone import edifabric
|
||||
|
||||
|
||||
_TEST_KEY = "test-edifabric-key-0123456789abcdef"
|
||||
|
||||
|
||||
def _make_client(handler):
|
||||
"""Build an httpx.Client whose transport is the given handler."""
|
||||
transport = httpx.MockTransport(handler)
|
||||
return httpx.Client(transport=transport, timeout=10.0)
|
||||
|
||||
|
||||
def _install_factory(handler):
|
||||
"""Swap in the mocked httpx.Client for the duration of a test."""
|
||||
return edifabric.set_transport_factory(lambda: _make_client(handler))
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_transport():
|
||||
"""Restore the default transport after each test (so a failing test
|
||||
can't poison the next)."""
|
||||
yield
|
||||
edifabric._reset_transport_factory()
|
||||
|
||||
|
||||
# --- /x12/read ---------------------------------------------------------
|
||||
|
||||
|
||||
def test_read_interchange_returns_first_x12_from_array():
|
||||
"""The /x12/read endpoint returns a list (multi-interchange file).
|
||||
Cyclone calls return the first element."""
|
||||
x12 = {
|
||||
"SegmentDelimiter": "~",
|
||||
"DataElementDelimiter": "*",
|
||||
"ISA": {"InterchangeControlNumber_13": "000000001"},
|
||||
"Groups": [],
|
||||
"IEATrailers": [],
|
||||
}
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
assert request.headers["Ocp-Apim-Subscription-Key"] == _TEST_KEY
|
||||
assert request.headers["Content-Type"] == "application/octet-stream"
|
||||
return httpx.Response(200, json=[x12])
|
||||
|
||||
_install_factory(handler)
|
||||
result = edifabric.read_interchange(b"ISA*...~IEA*0*000000001~", api_key=_TEST_KEY)
|
||||
assert result["ISA"]["InterchangeControlNumber_13"] == "000000001"
|
||||
|
||||
|
||||
def test_read_interchange_rejects_empty_response():
|
||||
"""If /x12/read returns an empty list, surface a 502-style error."""
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(200, json=[])
|
||||
|
||||
_install_factory(handler)
|
||||
with pytest.raises(edifabric.EdifabricError) as exc_info:
|
||||
edifabric.read_interchange(b"ISA*...~IEA*0*000000001~", api_key=_TEST_KEY)
|
||||
assert exc_info.value.status_code == 502
|
||||
|
||||
|
||||
# --- /x12/validate -----------------------------------------------------
|
||||
|
||||
|
||||
def test_validate_interchange_returns_operation_result():
|
||||
"""A 200 response is returned verbatim — Status + Details."""
|
||||
operation_result = {
|
||||
"Status": "success",
|
||||
"Details": [],
|
||||
"LastIndex": 46,
|
||||
}
|
||||
x12 = {
|
||||
"SegmentDelimiter": "~",
|
||||
"DataElementDelimiter": "*",
|
||||
"ISA": {"InterchangeControlNumber_13": "000000001"},
|
||||
"Groups": [],
|
||||
"IEATrailers": [],
|
||||
}
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
body = json.loads(request.content)
|
||||
assert body["ISA"]["InterchangeControlNumber_13"] == "000000001"
|
||||
return httpx.Response(200, json=operation_result)
|
||||
|
||||
_install_factory(handler)
|
||||
result = edifabric.validate_interchange(x12, api_key=_TEST_KEY)
|
||||
assert result["Status"] == "success"
|
||||
assert result["Details"] == []
|
||||
|
||||
|
||||
def test_validate_interchange_does_not_raise_on_status_error():
|
||||
"""OperationResult.Status='error' is data, not an exception — the
|
||||
caller (the gate / CLI) decides whether to fail-closed."""
|
||||
operation_result = {
|
||||
"Status": "error",
|
||||
"Details": [
|
||||
{"Index": 5, "SegmentId": "PER", "Message": "PER-04 is required", "Status": "error"},
|
||||
],
|
||||
"LastIndex": 5,
|
||||
}
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(200, json=operation_result)
|
||||
|
||||
_install_factory(handler)
|
||||
result = edifabric.validate_interchange({"ISA": {}}, api_key=_TEST_KEY)
|
||||
assert result["Status"] == "error"
|
||||
assert result["Details"][0]["Message"] == "PER-04 is required"
|
||||
|
||||
|
||||
# --- 4xx / 5xx ---------------------------------------------------------
|
||||
|
||||
|
||||
def test_validate_interchange_raises_on_5xx():
|
||||
"""Non-2xx responses raise EdifabricError; the body is preserved."""
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(503, text="upstream overloaded")
|
||||
|
||||
_install_factory(handler)
|
||||
with pytest.raises(edifabric.EdifabricError) as exc_info:
|
||||
edifabric.validate_interchange({"ISA": {}}, api_key=_TEST_KEY)
|
||||
assert exc_info.value.status_code == 503
|
||||
assert "upstream overloaded" in str(exc_info.value.body)
|
||||
|
||||
|
||||
def test_read_interchange_raises_with_retry_after_when_quota_blocked():
|
||||
"""On HTTP 403 with a ``Retry-After`` header (API Management quota
|
||||
policy), the raised EdifabricError exposes ``retry_after_seconds``
|
||||
so callers can sleep exactly until quota replenishes."""
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(
|
||||
403,
|
||||
headers={"Retry-After": "50492"},
|
||||
json={"statusCode": 403,
|
||||
"message": "Out of call volume quota."},
|
||||
)
|
||||
|
||||
_install_factory(handler)
|
||||
with pytest.raises(edifabric.EdifabricError) as exc_info:
|
||||
edifabric.read_interchange(b"ISA*...", api_key=_TEST_KEY)
|
||||
err = exc_info.value
|
||||
assert err.status_code == 403
|
||||
assert err.retry_after_seconds == 50492
|
||||
assert "Out of call volume quota" in str(err.body)
|
||||
|
||||
|
||||
# --- validate_edi (composed) ------------------------------------------
|
||||
|
||||
|
||||
def test_validate_edi_composes_read_then_validate():
|
||||
"""validate_edi should call read first, then validate with the
|
||||
X12Interchange JSON from read's response."""
|
||||
x12 = {
|
||||
"SegmentDelimiter": "~",
|
||||
"DataElementDelimiter": "*",
|
||||
"ISA": {"InterchangeControlNumber_13": "000000099"},
|
||||
"Groups": [],
|
||||
"IEATrailers": [],
|
||||
}
|
||||
operation_result = {"Status": "success", "Details": [], "LastIndex": 10}
|
||||
seen_calls: list[str] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/read"):
|
||||
seen_calls.append("read")
|
||||
return httpx.Response(200, json=[x12])
|
||||
if request.url.path.endswith("/validate"):
|
||||
seen_calls.append("validate")
|
||||
# Verify the validate body is the X12Interchange JSON
|
||||
body = json.loads(request.content)
|
||||
assert body["ISA"]["InterchangeControlNumber_13"] == "000000099"
|
||||
return httpx.Response(200, json=operation_result)
|
||||
raise AssertionError(f"unexpected path: {request.url.path}")
|
||||
|
||||
_install_factory(handler)
|
||||
result = edifabric.validate_edi(b"ISA*...~IEA*0*000000099~", api_key=_TEST_KEY)
|
||||
assert seen_calls == ["read", "validate"]
|
||||
assert result["Status"] == "success"
|
||||
|
||||
|
||||
# --- API key handling --------------------------------------------------
|
||||
|
||||
|
||||
def test_validate_edi_raises_when_api_key_missing(monkeypatch):
|
||||
"""With no key configured anywhere, validate_edi surfaces a clear
|
||||
error to the operator (not a generic 500)."""
|
||||
monkeypatch.delenv("CYCLONE_EDIFABRIC_API_KEY", raising=False)
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
raise AssertionError("transport should not be called when key is missing")
|
||||
|
||||
_install_factory(handler)
|
||||
with pytest.raises(edifabric.EdifabricError) as exc_info:
|
||||
edifabric.validate_edi(b"ISA*...~IEA*0*000000001~")
|
||||
assert exc_info.value.status_code == 0
|
||||
assert "API key not configured" in str(exc_info.value.body)
|
||||
@@ -102,10 +102,10 @@ def migrated_engine(tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
|
||||
engine = _fresh_engine(tmp_path / "mig0020.db")
|
||||
db_migrate.run(engine)
|
||||
|
||||
# Confirm head is 20 (every migration applied).
|
||||
# Confirm head is 22 (every migration applied, including SP41's 0022).
|
||||
with engine.connect() as conn:
|
||||
v = conn.exec_driver_sql("PRAGMA user_version").scalar() or 0
|
||||
assert v == 20, f"expected migration head=20, got {v}"
|
||||
assert v == 23, f"expected migration head=23, got {v}"
|
||||
|
||||
yield engine
|
||||
engine.dispose()
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Regression test: RateLimitMiddleware must share its bucket across instances.
|
||||
|
||||
When ``importlib.reload(cyclone.api)`` runs (e.g. from
|
||||
``test_cors_extra_origins_via_env``), Starlette rebuilds the middleware
|
||||
stack with a NEW ``RateLimitMiddleware`` instance whose ``_buckets``
|
||||
dict is fresh. Tests that imported ``app`` at module load time still
|
||||
reference the OLD app — and the OLD RateLimitMiddleware — so its
|
||||
private ``_buckets`` dict keeps accumulating requests across the rest
|
||||
of the suite. After ~300 requests it hits the rate limit and the
|
||||
remaining tests get spurious 429s.
|
||||
|
||||
Fix: hoist ``_buckets`` to a class-level dict so every
|
||||
``RateLimitMiddleware`` instance (current + stale) shares the same
|
||||
sliding-window state. The per-instance ``_lock`` stays per-instance
|
||||
since it guards mutation of the shared dict.
|
||||
|
||||
This test pins that invariant: two ``RateLimitMiddleware`` instances
|
||||
share the same underlying bucket dict.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from cyclone.security import RateLimitMiddleware
|
||||
|
||||
|
||||
class _DummyApp:
|
||||
"""Minimal ASGI app stand-in for the middleware's inner app."""
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
pass
|
||||
|
||||
|
||||
def test_rate_limit_buckets_are_shared_across_instances():
|
||||
"""Two RateLimitMiddleware instances must share _buckets."""
|
||||
m1 = RateLimitMiddleware(_DummyApp())
|
||||
m2 = RateLimitMiddleware(_DummyApp())
|
||||
# After the fix: m1._buckets IS m2._buckets (same class-level dict).
|
||||
# Before the fix: each instance had its own {} dict.
|
||||
assert m1._buckets is m2._buckets, (
|
||||
"RateLimitMiddleware instances do NOT share their bucket dict. "
|
||||
"After importlib.reload(cyclone.api), the new instance's "
|
||||
"_buckets is independent of the old one — orphaned buckets "
|
||||
"accumulate across tests that hold stale `app` references and "
|
||||
"trip the 300 req/60s limit. Hoist _buckets to a class-level "
|
||||
"dict so every instance shares state."
|
||||
)
|
||||
|
||||
|
||||
def test_class_level_buckets_attribute_exists():
|
||||
"""The class must declare a class-level _buckets dict."""
|
||||
# This is the structural pre-condition for the sharing invariant.
|
||||
assert hasattr(RateLimitMiddleware, "_buckets"), (
|
||||
"RateLimitMiddleware must declare a class-level _buckets "
|
||||
"attribute so every instance shares it. Without this, "
|
||||
"importlib.reload(cyclone.api) creates an orphaned bucket."
|
||||
)
|
||||
assert isinstance(RateLimitMiddleware._buckets, dict)
|
||||
@@ -0,0 +1,49 @@
|
||||
"""CARC-aware filter for Pipeline A.
|
||||
|
||||
Contractual / non-recoverable denials (CO-45, CO-26, CO-129) must be
|
||||
excluded from rebill. CARCs that need operator review (PI-16, PI-96,
|
||||
PI-15, PI-4, PI-110, OA-18, OA-23) must be surfaced as 'REVIEW' so the
|
||||
operator can decide.
|
||||
"""
|
||||
from cyclone.rebill.carc_filter import (
|
||||
CarcDecision,
|
||||
decide_carc,
|
||||
EXCLUDED_CARCS,
|
||||
REVIEW_CARCS,
|
||||
)
|
||||
|
||||
|
||||
def test_co45_is_excluded():
|
||||
assert decide_carc(("CO-45",)) == CarcDecision.EXCLUDED
|
||||
|
||||
|
||||
def test_co26_is_excluded():
|
||||
assert decide_carc(("CO-26",)) == CarcDecision.EXCLUDED
|
||||
|
||||
|
||||
def test_co129_is_excluded():
|
||||
assert decide_carc(("CO-129",)) == CarcDecision.EXCLUDED
|
||||
|
||||
|
||||
def test_pi16_is_review():
|
||||
assert decide_carc(("PI-16",)) == CarcDecision.REVIEW
|
||||
|
||||
|
||||
def test_o18_is_review():
|
||||
"""OA-18 is the duplicate noise — surface for review, don't auto-rebill."""
|
||||
assert decide_carc(("OA-18",)) == CarcDecision.REVIEW
|
||||
|
||||
|
||||
def test_no_carc_is_rebill():
|
||||
assert decide_carc(()) == CarcDecision.REBILL
|
||||
|
||||
|
||||
def test_mixed_excluded_wins():
|
||||
"""If any CARC is excluded, the whole service is excluded."""
|
||||
assert decide_carc(("PI-16", "CO-45")) == CarcDecision.EXCLUDED
|
||||
|
||||
|
||||
def test_sets_have_expected_members():
|
||||
assert "CO-45" in EXCLUDED_CARCS
|
||||
assert "PI-16" in REVIEW_CARCS
|
||||
assert "OA-18" in REVIEW_CARCS
|
||||
@@ -0,0 +1,172 @@
|
||||
"""SP41 Task 17 — end-to-end smoke test for ``run_rebill``.
|
||||
|
||||
Wires the full SP41 pipeline (835 SVC reparse → reconcile → CARC
|
||||
filter → timely-filing gate → pipeline A → pipeline B → summary CSV)
|
||||
end-to-end on synthetic inputs and pins the summary.csv shape +
|
||||
counts.
|
||||
|
||||
No mocks for ``parse_835_svc`` or ``validate_837`` — the goal is to
|
||||
exercise the real pipeline. The autouse conftest handles Edifabric
|
||||
fail-open (no API key in CI), and since this run produces only
|
||||
quarantined dispositions (no pipeline A/B emissions), ``validate_edi``
|
||||
is never called anyway.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
from cyclone.rebill.run import run_rebill
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Fixture builders
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def _write_visits_csv(path: Path) -> Path:
|
||||
"""Two visits: one in-window (will match a denied SVC), one past the
|
||||
120-day timely-filing window (no matching SVC).
|
||||
|
||||
DOS column is MM/DD/YYYY per AxisCare's actual export format.
|
||||
"""
|
||||
with path.open("w", newline="") as f:
|
||||
w = csv.writer(f)
|
||||
w.writerow([
|
||||
"Visit Date", "Member ID", "Procedure Code",
|
||||
"Billable Amount", "Authorized",
|
||||
])
|
||||
w.writerow(["06/27/2026", "J813715", "T1019", "2.32", "Y"])
|
||||
w.writerow(["01/01/2026", "OLD001", "T1019", "5.00", "Y"])
|
||||
return path
|
||||
|
||||
|
||||
def _write_835_with_denied_svc(ingest_dir: Path) -> Path:
|
||||
"""Write a real 835 that ``parse_835_svc`` walks end-to-end.
|
||||
|
||||
Contains exactly one CLP block for member J813715 with:
|
||||
- CLP02 = 4 (Denied)
|
||||
- CLP03 = 2.32 (charge)
|
||||
- SVC*HC:T1019*2.32*0** (paid = $0)
|
||||
- DTM*472*20260627 (service date)
|
||||
- CAS*CO*45*2.32 (triggers ``EXCLUDED_CARC``)
|
||||
|
||||
Returns the **directory** path so the caller can pass it straight to
|
||||
``run_rebill(ingest_dir=...)`` — run_rebill does its own
|
||||
``ingest_dir.glob("*.835")`` walk. (Earlier draft returned the
|
||||
inner file path, which made glob come up empty.)
|
||||
|
||||
Shape mirrors the existing ``tests/fixtures/835_sample_svc_with_member.txt``
|
||||
(segments concatenated without ``\\n`` separators — ``parse_835_svc``
|
||||
splits on ``~`` only, and a leading newline makes ``elems[0]`` an empty
|
||||
string that the segment-name match drops silently).
|
||||
"""
|
||||
ingest_dir.mkdir(exist_ok=True)
|
||||
segs = [
|
||||
"ISA*00* *00* *ZZ*CYCLONE *ZZ*GAINWELL *260627*1200*^*00501*000000001*0*P*:~",
|
||||
"GS*HC*CYCLONE*GAINWELL*20260627*1200*1*X*005010X221A1~",
|
||||
"ST*835*0001~",
|
||||
"BPR*I*0*C*ACH*CCP*01*021000021*DA*123456789*1512345678**01*021000021*DA*123456789*20260101~",
|
||||
"TRN*1*TRACE01*1512345678~",
|
||||
"DTM*405*20260118~",
|
||||
"N1*PR*COLORADO MEDICAL ASSISTANCE PROGRAM*XV*COMEDASSISTPROG~",
|
||||
"N3*PO BOX 1100~",
|
||||
"N4*DENVER*CO*80202~",
|
||||
"LX*1~",
|
||||
# CLP02=4 (Denied); CLP03=2.32 (charge); CLP04=0 (paid).
|
||||
"CLP*DENIED-CLM*4*2.32*0**MC*111*11*1~",
|
||||
# NM1*QC.NM109 carries the member_id forward to SVC rows.
|
||||
"NM1*QC*1*DOE*JANE****MR*J813715~",
|
||||
# SVC composite qualifier:procedure (HC:T1019); 4-arg form is fine.
|
||||
"SVC*HC:T1019*2.32*0**~",
|
||||
# DTM*472 carries the service date (matches parse_835_svc's lookup).
|
||||
"DTM*472*20260627~",
|
||||
# CO-45 is in EXCLUDED_CARCS → CarcDecision.EXCLUDED → EXCLUDED_CARC.
|
||||
"CAS*CO*45*2.32~",
|
||||
"SE*14*0001~",
|
||||
"GE*1*1~",
|
||||
"IEA*1*000000001~",
|
||||
]
|
||||
p = ingest_dir / "x.835"
|
||||
p.write_text("".join(segs))
|
||||
return ingest_dir
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Smoke test
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_run_rebill_end_to_end_excluded_dispositions(tmp_path):
|
||||
"""End-to-end: 2 visits → 2 quarantined dispositions, no pipeline files.
|
||||
|
||||
Pins the contract that ``run_rebill`` returns a ``RunResult`` whose
|
||||
``summary.csv`` row-shapes match the visit-side input (one row per
|
||||
non-PAID visit, with the right disposition per row).
|
||||
"""
|
||||
visits_csv = _write_visits_csv(tmp_path / "visits.csv")
|
||||
ingest = _write_835_with_denied_svc(tmp_path / "ingest")
|
||||
out_dir = tmp_path / "rebills"
|
||||
|
||||
# ``as_of=2026-07-07`` pins the timely-filing gate so the test is
|
||||
# deterministic — OLD001 (DOS 2026-01-01) is 187 days old, well past
|
||||
# the 120-day HCPF window, so it's EXCLUDED_TIMELY_FILING.
|
||||
result = run_rebill(
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
override_filing=False,
|
||||
visits_csv_path=str(visits_csv),
|
||||
ingest_dir=str(ingest),
|
||||
out_dir=str(out_dir),
|
||||
as_of=date(2026, 7, 7),
|
||||
)
|
||||
|
||||
# --- summary.csv exists and the right text is in it. --- #
|
||||
assert result.summary_path.exists(), result.summary_path
|
||||
text = result.summary_path.read_text()
|
||||
assert "J813715" in text
|
||||
assert "OLD001" in text
|
||||
assert "EXCLUDED_CARC" in text
|
||||
assert "EXCLUDED_TIMELY_FILING" in text
|
||||
|
||||
# --- counts match expectations (one of each excluded disposition). --- #
|
||||
assert result.counts["EXCLUDED_CARC"] == 1, result.counts
|
||||
assert result.counts["EXCLUDED_TIMELY_FILING"] == 1, result.counts
|
||||
# Both visits are excluded — no pipeline emissions.
|
||||
assert result.counts["REBILLED_A"] == 0, result.counts
|
||||
assert result.counts["REBILLED_B"] == 0, result.counts
|
||||
|
||||
# --- pin the per-row shape (stronger than substring checks). --- #
|
||||
with result.summary_path.open(newline="") as f:
|
||||
rows = list(csv.DictReader(f))
|
||||
assert len(rows) == 2, rows
|
||||
|
||||
j_row = next(r for r in rows if r["member_id"] == "J813715")
|
||||
old_row = next(r for r in rows if r["member_id"] == "OLD001")
|
||||
|
||||
# J813715: matched the DENIED SVC, CARC CO-45 is in EXCLUDED_CARCS.
|
||||
assert j_row["disposition"] == "EXCLUDED_CARC", j_row
|
||||
assert j_row["procedure"] == "T1019", j_row
|
||||
assert j_row["dos"] == "2026-06-27", j_row
|
||||
assert j_row["billed"] == "2.32", j_row
|
||||
assert "CO-45" in j_row["cas_reasons"], j_row
|
||||
|
||||
# OLD001: no matching SVC, DOS 187 days old → timely-filing exclusion.
|
||||
assert old_row["disposition"] == "EXCLUDED_TIMELY_FILING", old_row
|
||||
assert old_row["procedure"] == "T1019", old_row
|
||||
assert old_row["dos"] == "2026-01-01", old_row
|
||||
assert old_row["billed"] == "5.00", old_row
|
||||
# No SVC match → empty cas_reasons column.
|
||||
assert old_row["cas_reasons"] == "", old_row
|
||||
|
||||
# --- no pipeline files were emitted (both rows are excluded). --- #
|
||||
assert result.pipeline_a_files == [], result.pipeline_a_files
|
||||
assert result.pipeline_b_files == [], result.pipeline_b_files
|
||||
# The pipeline dirs exist but are empty.
|
||||
assert (out_dir / "pipeline-a").is_dir()
|
||||
assert (out_dir / "pipeline-b").is_dir()
|
||||
assert list((out_dir / "pipeline-a").iterdir()) == []
|
||||
assert list((out_dir / "pipeline-b").iterdir()) == []
|
||||
# No quarantined files either — these dispositions don't emit at all.
|
||||
assert list((out_dir / "quarantine").iterdir()) == []
|
||||
@@ -0,0 +1,38 @@
|
||||
"""835 SVC-level parser with member_id at SVC scope."""
|
||||
from pathlib import Path
|
||||
from collections import defaultdict
|
||||
from decimal import Decimal
|
||||
from cyclone.rebill.parse_835_svc import parse_835_svc
|
||||
|
||||
FIX = Path(__file__).parent / "fixtures" / "835_sample_svc_with_member.txt"
|
||||
|
||||
def test_parse_835_svc_extracts_member_id():
|
||||
"""NM1*QC NM109 at the CLP scope must propagate to that CLP's SVC rows;
|
||||
the multi-claim fixture confirms each CLP's NM1*QC lands on its own SVCs."""
|
||||
rows = list(parse_835_svc(FIX))
|
||||
assert len(rows) >= 1
|
||||
by_claim: dict[str, list[str]] = defaultdict(list)
|
||||
for r in rows:
|
||||
by_claim[r.claim_id].append(r.member_id)
|
||||
assert r.procedure # non-empty
|
||||
assert r.svc_date # non-empty
|
||||
assert r.charge > 0
|
||||
# First CLP (T1001) -> original fixture member
|
||||
assert all(mid == "J813715" for mid in by_claim["T1001"]), by_claim
|
||||
# Second CLP (T1002, status 4) -> its own NM1*QC
|
||||
assert all(mid == "OTHER-MEMBER-A" for mid in by_claim["T1002"]), by_claim
|
||||
# Third CLP (T1003, status 22) -> its own NM1*QC
|
||||
assert all(mid == "OTHER-MEMBER-B" for mid in by_claim["T1003"]), by_claim
|
||||
|
||||
def test_parse_835_svc_extracts_cas_reasons():
|
||||
"""CAS segments after DTM*472 must be captured (post-DTM*472 ordering)."""
|
||||
rows = list(parse_835_svc(FIX))
|
||||
# at least one row should have an OA-18 reason
|
||||
assert any("OA-18" in r.cas_reasons for r in rows)
|
||||
|
||||
def test_parse_835_svc_picks_up_status_22_reversals():
|
||||
"""Status 22 (reversal of previous payment) must be preserved, along with
|
||||
status 1 (primary) and status 4 (denied)."""
|
||||
rows = list(parse_835_svc(FIX))
|
||||
statuses = {r.status for r in rows}
|
||||
assert statuses == {"1", "4", "22"}
|
||||
@@ -0,0 +1,55 @@
|
||||
"""Pipeline A: denied/partial visits become frequency-7 replacement claims."""
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
from cyclone.rebill.pipeline_a import build_pipeline_a_claims
|
||||
from cyclone.rebill.reconcile import VisitRow, ReconcileOutcome, OutcomeCategory
|
||||
from cyclone.rebill.carc_filter import CarcDecision
|
||||
|
||||
|
||||
def _outcome(date_, member, proc, billed, cat, unpaid=Decimal("0")):
|
||||
visit = VisitRow(date=date_, member_id=member, procedure=proc, billed=Decimal(billed))
|
||||
return ReconcileOutcome(visit, cat, unpaid, 0)
|
||||
|
||||
|
||||
def test_pipeline_a_emits_frequency_7():
|
||||
"""CLM05-3 must be 7 (replacement), preserving the original claim_submit_id."""
|
||||
out = build_pipeline_a_claims(
|
||||
original_claim_id="ORIG-001",
|
||||
visit_outcomes=[
|
||||
_outcome(date(2026, 6, 27), "J813715", "T1019", "2.32",
|
||||
OutcomeCategory.DENIED, unpaid=Decimal("2.32")),
|
||||
],
|
||||
carc_decisions=[CarcDecision.REBILL],
|
||||
cas_reasons_per_visit=[()],
|
||||
)
|
||||
assert len(out) == 1
|
||||
assert out[0].claim_id == "ORIG-001"
|
||||
assert out[0].frequency_code == "7"
|
||||
assert out[0].svc_date == date(2026, 6, 27)
|
||||
|
||||
|
||||
def test_pipeline_a_excludes_carc_excluded_visits():
|
||||
out = build_pipeline_a_claims(
|
||||
original_claim_id="ORIG-002",
|
||||
visit_outcomes=[
|
||||
_outcome(date(2026, 6, 27), "J813715", "T1019", "2.32",
|
||||
OutcomeCategory.DENIED),
|
||||
],
|
||||
carc_decisions=[CarcDecision.EXCLUDED],
|
||||
cas_reasons_per_visit=[("CO-45",)],
|
||||
)
|
||||
assert out == [] # not emitted
|
||||
|
||||
|
||||
def test_pipeline_a_surfaces_review_visits_with_flag():
|
||||
out = build_pipeline_a_claims(
|
||||
original_claim_id="ORIG-003",
|
||||
visit_outcomes=[
|
||||
_outcome(date(2026, 6, 27), "J813715", "T1019", "2.32",
|
||||
OutcomeCategory.DENIED),
|
||||
],
|
||||
carc_decisions=[CarcDecision.REVIEW],
|
||||
cas_reasons_per_visit=[("PI-16",)],
|
||||
)
|
||||
assert len(out) == 1
|
||||
assert out[0].needs_review is True
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Pipeline B: NOT_IN_835 visits → fresh 837Ps, batched by (member, ISO-week)."""
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
from cyclone.rebill.pipeline_b import build_pipeline_b_batches
|
||||
from cyclone.rebill.reconcile import VisitRow
|
||||
|
||||
|
||||
def _v(date_, member, proc, amt):
|
||||
return VisitRow(date=date_, member_id=member, procedure=proc, billed=Decimal(amt))
|
||||
|
||||
|
||||
def test_batches_split_by_member_and_iso_week():
|
||||
visits = [
|
||||
_v(date(2026, 6, 23), "J813715", "T1019", "2.32"),
|
||||
_v(date(2026, 6, 25), "J813715", "T1019", "2.32"), # same week
|
||||
_v(date(2026, 6, 23), "OTHER", "T1019", "2.32"), # different member
|
||||
_v(date(2026, 6, 30), "J813715", "T1019", "2.32"), # different week
|
||||
]
|
||||
out = build_pipeline_b_batches(visits, as_of=date(2026, 7, 7), override=False)
|
||||
assert len(out) == 3
|
||||
sizes = sorted(len(b.visits) for b in out)
|
||||
assert sizes == [1, 1, 2]
|
||||
|
||||
|
||||
def test_timely_filing_excludes_old_visits():
|
||||
visits = [
|
||||
_v(date(2026, 1, 1), "OLD", "T1019", "2.32"), # 187 days old
|
||||
_v(date(2026, 6, 27), "NEW", "T1019", "2.32"), # 10 days old
|
||||
]
|
||||
out = build_pipeline_b_batches(visits, as_of=date(2026, 7, 7), override=False)
|
||||
members = {b.member_id for b in out}
|
||||
assert "OLD" not in members
|
||||
assert "NEW" in members
|
||||
|
||||
|
||||
def test_override_relaxes_timely_filing():
|
||||
visits = [_v(date(2026, 1, 1), "OLD", "T1019", "2.32")]
|
||||
out_default = build_pipeline_b_batches(visits, as_of=date(2026, 7, 7), override=False)
|
||||
out_overridden = build_pipeline_b_batches(visits, as_of=date(2026, 7, 7), override=True)
|
||||
assert out_default == []
|
||||
assert len(out_overridden) == 1
|
||||
|
||||
|
||||
def test_override_flag_set_on_past_window_visit_batch():
|
||||
# Same member "OLD" with one past-window visit (187 days old, ISO
|
||||
# week 1) and one within-window visit (10 days old, ISO week 26).
|
||||
# With override=True the past-window visit survives and the batch
|
||||
# for week 1 must be flagged has_overridden_visits=True because the
|
||||
# override saved an otherwise-excluded visit.
|
||||
visits = [
|
||||
_v(date(2026, 1, 1), "OLD", "T1019", "2.32"), # 187 days old (past window, W01)
|
||||
_v(date(2026, 6, 27), "OLD", "T1019", "2.32"), # 10 days old (within window, W26)
|
||||
]
|
||||
out_overridden = build_pipeline_b_batches(
|
||||
visits, as_of=date(2026, 7, 7), override=True,
|
||||
)
|
||||
# Two batches: one per (member, ISO-week) — the past-window visit and
|
||||
# the within-window visit land in different weeks.
|
||||
assert len(out_overridden) == 2
|
||||
by_week = {b.iso_week: b for b in out_overridden}
|
||||
assert by_week[1].member_id == "OLD"
|
||||
assert by_week[1].has_overridden_visits is True # the override saved this batch
|
||||
assert by_week[26].member_id == "OLD"
|
||||
assert by_week[26].has_overridden_visits is False # no override needed here
|
||||
|
||||
# With override=False the past-window visit is dropped. The surviving
|
||||
# within-window batch (W26) must NOT carry the override flag, and the
|
||||
# past-window batch (W01) is absent.
|
||||
out_default = build_pipeline_b_batches(
|
||||
visits, as_of=date(2026, 7, 7), override=False,
|
||||
)
|
||||
assert len(out_default) == 1
|
||||
assert out_default[0].member_id == "OLD"
|
||||
assert out_default[0].iso_week == 26
|
||||
assert out_default[0].has_overridden_visits is False
|
||||
|
||||
|
||||
def test_serialize_member_week_batch_emits_one_envelope():
|
||||
"""One 837P envelope per MemberWeekBatch — one CLM + one SV1 + one
|
||||
DTP*472 service date per visit.
|
||||
|
||||
The SP41 plan spec wrote ``DTM*472*`` but the canonical 837P service
|
||||
date segment is ``DTP*472*`` (per :func:`cyclone.parsers.serialize_837.
|
||||
_build_dtp_472` and X12 005010X222A1). This test asserts against the
|
||||
canonical segment name so the batch overload stays consistent with
|
||||
the existing ``serialize_837`` building blocks.
|
||||
"""
|
||||
from cyclone.parsers.serialize_837 import serialize_member_week_batch
|
||||
visits = [
|
||||
_v(date(2026, 6, 23), "J813715", "T1019", "2.32"),
|
||||
_v(date(2026, 6, 25), "J813715", "T1019", "2.32"),
|
||||
]
|
||||
batches = build_pipeline_b_batches(visits, as_of=date(2026, 7, 7), override=False)
|
||||
assert len(batches) == 1
|
||||
body = serialize_member_week_batch(batches[0])
|
||||
text = body.decode("utf-8", errors="ignore") if isinstance(body, bytes) else body
|
||||
# CLM* segment appears twice (once per visit)
|
||||
assert text.count("CLM*") == 2
|
||||
# SV1* appears once per visit (one service line per claim)
|
||||
assert text.count("SV1*") == 2
|
||||
# DTP*472* service-date segment appears twice (canonical 837P segment name)
|
||||
assert text.count("DTP*472*") == 2
|
||||
# Single envelope (single ISA / single IEA), not per-visit envelopes
|
||||
assert text.count("ISA*") == 1
|
||||
assert text.count("IEA*") == 1
|
||||
# Deterministic per-visit claim_id pattern (member_id + date + 1-based idx)
|
||||
assert "MW-J813715-2026-06-23-01" in text
|
||||
assert "MW-J813715-2026-06-25-02" in text
|
||||
|
||||
|
||||
def test_serialize_member_week_batch_return_type_is_bytes():
|
||||
"""Task 14 spec: ``serialize_member_week_batch`` returns ``bytes``
|
||||
(the existing ``serialize_837`` returns ``str``; this overload
|
||||
diverges intentionally so callers can write the file directly)."""
|
||||
from cyclone.parsers.serialize_837 import serialize_member_week_batch
|
||||
visits = [_v(date(2026, 6, 23), "J813715", "T1019", "2.32")]
|
||||
batches = build_pipeline_b_batches(visits, as_of=date(2026, 7, 7), override=False)
|
||||
assert len(batches) == 1
|
||||
body = serialize_member_week_batch(batches[0])
|
||||
assert isinstance(body, bytes)
|
||||
@@ -0,0 +1,167 @@
|
||||
"""999-ack dump from Gainwell for Mar–Jun 2026.
|
||||
|
||||
Reconciles pulled 999 acks against the in-window 4,509 NOT_IN_835 visits.
|
||||
Visits that are 999-rejected go in one bucket; visits that simply
|
||||
never made it to submission go in another.
|
||||
|
||||
These tests exercise the pure-function surface of
|
||||
``cyclone.rebill.pull_999_acks`` (no SFTP, no DB). The
|
||||
``pull_and_classify`` orchestrator wraps the existing
|
||||
``Scheduler.process_inbound_files`` machinery and is integration-
|
||||
covered by the existing ``test_api_pull_inbound.py`` / CLI smoke
|
||||
tests — adding a new test here would just duplicate that coverage
|
||||
and require a live SFTP server.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import date
|
||||
|
||||
from cyclone.rebill.pull_999_acks import (
|
||||
Bucket,
|
||||
PullResult,
|
||||
classify_not_in_835_visits,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Core: split by 999-rejection presence
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_classify_splits_by_999_rejection_presence() -> None:
|
||||
visits = [
|
||||
("J813715", date(2026, 6, 27), "T1019"),
|
||||
("OTHER", date(2026, 6, 27), "T1019"),
|
||||
]
|
||||
nine99_rejected = {("J813715", date(2026, 6, 27), "T1019")}
|
||||
out = classify_not_in_835_visits(visits, nine99_rejected)
|
||||
assert out["J813715"].value == Bucket.REJECTED_AT_999.value
|
||||
assert out["OTHER"].value == Bucket.NEVER_SUBMITTED.value
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Edge cases
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_classify_empty_input_returns_empty_dict() -> None:
|
||||
"""No visits → empty bucket map (no-op)."""
|
||||
out = classify_not_in_835_visits([], set())
|
||||
assert out == {}
|
||||
# Also: empty visits + non-empty rejection set stays empty.
|
||||
out2 = classify_not_in_835_visits(
|
||||
[], {("J813715", date(2026, 6, 27), "T1019")},
|
||||
)
|
||||
assert out2 == {}
|
||||
|
||||
|
||||
def test_classify_all_rejected() -> None:
|
||||
"""Every visit is in the 999-rejection set → every bucket is REJECTED_AT_999."""
|
||||
v1 = ("MEM001", date(2026, 3, 15), "T1019")
|
||||
v2 = ("MEM002", date(2026, 4, 1), "T1019")
|
||||
out = classify_not_in_835_visits([v1, v2], {v1, v2})
|
||||
assert out == {
|
||||
"MEM001": Bucket.REJECTED_AT_999,
|
||||
"MEM002": Bucket.REJECTED_AT_999,
|
||||
}
|
||||
|
||||
|
||||
def test_classify_all_never_submitted() -> None:
|
||||
"""Visits present, rejection set empty → every bucket is NEVER_SUBMITTED."""
|
||||
visits = [
|
||||
("G1", date(2026, 3, 1), "T1019"),
|
||||
("G2", date(2026, 3, 2), "T1019"),
|
||||
("G3", date(2026, 3, 3), "T1019"),
|
||||
]
|
||||
out = classify_not_in_835_visits(visits, set())
|
||||
assert out == {m: Bucket.NEVER_SUBMITTED for m in ("G1", "G2", "G3")}
|
||||
|
||||
|
||||
def test_classify_keyed_by_member_id() -> None:
|
||||
"""Output is a dict keyed by member_id, not by the visit tuple.
|
||||
|
||||
The spec's contract is "keyed on member_id" — Pipeline B groups
|
||||
by member for the ISO-week rebill, so the classification map
|
||||
collapses to one entry per member. The visit tuple's procedure
|
||||
and dos parts are the *match key* against the 999 rejection set,
|
||||
not the *output key*.
|
||||
|
||||
When two visits for the same member resolve to different
|
||||
buckets (one rejected, one not), the dict-construction order
|
||||
means the *last* visit wins. This test pins that semantic —
|
||||
Pipeline B re-resolves per-visit at the next layer so the
|
||||
member-level bucket is just a coarse pre-filter.
|
||||
"""
|
||||
visits = [
|
||||
("SHARED", date(2026, 5, 1), "T1019"),
|
||||
("SHARED", date(2026, 5, 8), "T1019"),
|
||||
]
|
||||
# First visit IS in the rejected set, second is not.
|
||||
nine99_rejected = {("SHARED", date(2026, 5, 1), "T1019")}
|
||||
out = classify_not_in_835_visits(visits, nine99_rejected)
|
||||
# Last-write-wins: the second visit is NEVER_SUBMITTED, so the
|
||||
# member-level bucket ends up as NEVER_SUBMITTED. This is the
|
||||
# documented coarse-filter semantic — Pipeline B does per-visit
|
||||
# re-resolution downstream.
|
||||
assert out == {"SHARED": Bucket.NEVER_SUBMITTED}
|
||||
|
||||
# Reverse the rejection set: only the second visit is rejected.
|
||||
# Last visit wins → REJECTED_AT_999.
|
||||
nine99_rejected = {("SHARED", date(2026, 5, 8), "T1019")}
|
||||
out = classify_not_in_835_visits(visits, nine99_rejected)
|
||||
assert out == {"SHARED": Bucket.REJECTED_AT_999}
|
||||
|
||||
|
||||
def test_classify_distinct_members_dont_collide() -> None:
|
||||
"""Two distinct members, only one rejected → independent bucket entries."""
|
||||
visits = [
|
||||
("ALICE", date(2026, 6, 1), "T1019"),
|
||||
("BOB", date(2026, 6, 1), "T1019"),
|
||||
]
|
||||
nine99_rejected = {("ALICE", date(2026, 6, 1), "T1019")}
|
||||
out = classify_not_in_835_visits(visits, nine99_rejected)
|
||||
assert out == {
|
||||
"ALICE": Bucket.REJECTED_AT_999,
|
||||
"BOB": Bucket.NEVER_SUBMITTED,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PullResult shape — guards against accidental field drift
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_pull_result_is_frozen_dataclass() -> None:
|
||||
"""``PullResult`` must be frozen so callers can't mutate the summary."""
|
||||
pr = PullResult(
|
||||
total_pulled=10,
|
||||
rejected_at_999=3,
|
||||
not_in_835=4,
|
||||
rejected_breakdown={"R": 2, "E": 1},
|
||||
)
|
||||
assert pr.total_pulled == 10
|
||||
assert pr.rejected_at_999 == 3
|
||||
assert pr.not_in_835 == 4
|
||||
assert pr.rejected_breakdown == {"R": 2, "E": 1}
|
||||
# Frozen: assignment must raise.
|
||||
import dataclasses
|
||||
try:
|
||||
pr.total_pulled = 99 # type: ignore[misc]
|
||||
except dataclasses.FrozenInstanceError:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError("PullResult must be frozen")
|
||||
|
||||
|
||||
def test_bucket_values_are_json_friendly_strings() -> None:
|
||||
"""Bucket values serialize cleanly to JSON (string-enum contract)."""
|
||||
import json
|
||||
payload = {
|
||||
"j1": Bucket.REJECTED_AT_999.value,
|
||||
"j2": Bucket.NEVER_SUBMITTED.value,
|
||||
}
|
||||
# Round-trip — no enum leakage into the JSON output.
|
||||
assert json.loads(json.dumps(payload)) == {
|
||||
"j1": "REJECTED_AT_999",
|
||||
"j2": "NEVER_SUBMITTED",
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
"""Visit-to-835 reconciliation on (member_id, procedure, DOS)."""
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
from cyclone.rebill.reconcile import (
|
||||
VisitRow,
|
||||
SvcRow,
|
||||
ReconcileOutcome,
|
||||
reconcile_visits_to_835,
|
||||
OutcomeCategory,
|
||||
)
|
||||
|
||||
|
||||
def _v(date, member, proc, amt):
|
||||
return VisitRow(date=date, member_id=member, procedure=proc, billed=Decimal(amt))
|
||||
|
||||
|
||||
def _s(date, member, proc, chg, paid, status="1"):
|
||||
return SvcRow(
|
||||
src_file="x.835", claim_id="c1", member_id=member, status=status,
|
||||
procedure=proc, modifiers="", charge=Decimal(chg), paid=Decimal(paid),
|
||||
units=Decimal("0"), svc_date=date, cas_reasons=(), pay_date=None,
|
||||
)
|
||||
|
||||
|
||||
def test_visit_paid_when_any_svc_paid_within_tolerance():
|
||||
visits = [_v(date(2026, 1, 18), "J813715", "T1019", "2.32")]
|
||||
svcs = [_s(date(2026, 1, 18), "J813715", "T1019", "2.32", "2.30")]
|
||||
out = reconcile_visits_to_835(visits, svcs)
|
||||
assert out[0].category == OutcomeCategory.PAID
|
||||
assert out[0].unpaid == Decimal("0")
|
||||
|
||||
|
||||
def test_visit_paid_when_one_svc_paid_others_denied_duplicate():
|
||||
"""OA-18 on duplicates is noise; paid SVC wins."""
|
||||
visits = [_v(date(2026, 1, 18), "J813715", "T1019", "2.32")]
|
||||
svcs = [
|
||||
_s(date(2026, 1, 18), "J813715", "T1019", "2.32", "2.30", "1"),
|
||||
_s(date(2026, 1, 18), "J813715", "T1019", "2.32", "0.00", "4"),
|
||||
_s(date(2026, 1, 18), "J813715", "T1019", "2.32", "0.00", "4"),
|
||||
]
|
||||
out = reconcile_visits_to_835(visits, svcs)
|
||||
assert out[0].category == OutcomeCategory.PAID
|
||||
|
||||
|
||||
def test_visit_denied_when_all_svc_denied():
|
||||
visits = [_v(date(2026, 1, 18), "J813715", "T1019", "2.32")]
|
||||
svcs = [
|
||||
_s(date(2026, 1, 18), "J813715", "T1019", "2.32", "0.00", "4"),
|
||||
_s(date(2026, 1, 18), "J813715", "T1019", "2.32", "0.00", "4"),
|
||||
]
|
||||
out = reconcile_visits_to_835(visits, svcs)
|
||||
assert out[0].category == OutcomeCategory.DENIED
|
||||
assert out[0].unpaid == Decimal("2.32")
|
||||
|
||||
|
||||
def test_visit_not_in_835_when_no_svc_match():
|
||||
visits = [_v(date(2026, 1, 18), "J813715", "T1019", "2.32")]
|
||||
svcs = [_s(date(2026, 1, 19), "J813715", "T1019", "2.32", "2.30")]
|
||||
out = reconcile_visits_to_835(visits, svcs)
|
||||
assert out[0].category == OutcomeCategory.NOT_IN_835
|
||||
assert out[0].unpaid == Decimal("2.32")
|
||||
|
||||
|
||||
def test_partial_when_total_paid_below_95pct_of_billed():
|
||||
visits = [_v(date(2026, 1, 18), "J813715", "T1019", "10.00")]
|
||||
svcs = [_s(date(2026, 1, 18), "J813715", "T1019", "10.00", "4.00")]
|
||||
out = reconcile_visits_to_835(visits, svcs)
|
||||
assert out[0].category == OutcomeCategory.PARTIAL
|
||||
assert out[0].unpaid == Decimal("6.00")
|
||||
|
||||
|
||||
def test_different_member_does_not_match():
|
||||
visits = [_v(date(2026, 1, 18), "J813715", "T1019", "2.32")]
|
||||
svcs = [_s(date(2026, 1, 18), "OTHERMEMBER", "T1019", "2.32", "2.30")]
|
||||
out = reconcile_visits_to_835(visits, svcs)
|
||||
assert out[0].category == OutcomeCategory.NOT_IN_835
|
||||
@@ -0,0 +1,502 @@
|
||||
"""SP41 Task 13 — Edifabric validation gate on emitted rebill files.
|
||||
|
||||
Covers the new emit path in ``cyclone.rebill.run.run_rebill``:
|
||||
|
||||
* Real 837P files are written (not ``Path.touch()`` placeholders).
|
||||
* Files pass through ``cyclone.edifabric.validate_edi`` before they
|
||||
land in ``<out>/pipeline-{a,b}/``; failures go to
|
||||
``<out>/quarantine/``.
|
||||
* When Edifabric is unavailable (no API key, network error, etc.),
|
||||
the gate fails open with a WARNING log and emits to the pipeline
|
||||
dirs (matches the SP40 dev/CI posture).
|
||||
|
||||
Each test builds its own visits CSV + 835 stub in ``tmp_path`` and
|
||||
invokes ``run_rebill`` directly (no HTTP). The Edifabric API is mocked
|
||||
via ``unittest.mock.patch`` on
|
||||
``cyclone.rebill.run._edifabric.validate_edi`` so no live HTTP hits the
|
||||
network.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from cyclone.rebill.run import run_rebill
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Fixtures / helpers
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def _write_visits_csv(path: Path, rows: list[tuple[str, str, str, str]]) -> Path:
|
||||
"""rows: list of (dos_mmddyyyy, member, procedure, billed_str)."""
|
||||
with path.open("w", newline="") as f:
|
||||
w = csv.writer(f)
|
||||
w.writerow(["Visit Date", "Member ID", "Procedure Code", "Billable Amount"])
|
||||
for r in rows:
|
||||
w.writerow(r)
|
||||
return path
|
||||
|
||||
|
||||
def _stub_835_with_denied_svc(
|
||||
ingest_dir: Path,
|
||||
member: str,
|
||||
procedure: str,
|
||||
svc_date: date,
|
||||
claim_id: str = "ORIG-1",
|
||||
carc_group: str = "CO",
|
||||
carc_reason: str = "29",
|
||||
) -> Path:
|
||||
"""An 835 with a single denied SVC so the visit lands in Pipeline A.
|
||||
|
||||
The rebill pipeline reuses the SVC's ``claim_id`` as the Pipeline-A
|
||||
RebillClaim's ``claim_id`` (preserves the freq-7 anchor). The visit
|
||||
must match (member_id, procedure, svc_date) so the reconcile step
|
||||
pairs them into DENIED + CARC REBILL.
|
||||
|
||||
The 835 shape mirrors ``tests/fixtures/835_sample_svc_with_member.txt``:
|
||||
CLP02 = 4 (Denied), NM1*QC.NM109 = member_id, then SVC → DTM*472 → CAS.
|
||||
|
||||
Segments are concatenated without ``\\n`` separators — the parser
|
||||
splits on ``~`` only, and a leading ``\\n`` makes ``elems[0]`` an
|
||||
empty string so the segment-name match silently drops the segment.
|
||||
"""
|
||||
ingest_dir.mkdir(exist_ok=True)
|
||||
svc_date_str = svc_date.strftime("%Y%m%d")
|
||||
segs = [
|
||||
"ISA*00* *00* *ZZ*SENDER *ZZ*RECEIVER *260627*1200*^*00501*000000001*0*P*:~",
|
||||
"GS*HC*SENDER*RECEIVER*20260627*1200*1*X*005010X221A1~",
|
||||
"ST*835*0001~",
|
||||
"BPR*I*0*C*ACH*CCP*01*021000021*DA*123456789*1512345678**01*021000021*DA*123456789*20260101~",
|
||||
"TRN*1*TRACE01*1512345678~",
|
||||
"DTM*405*20260118~",
|
||||
"N1*PR*COLORADO MEDICAL ASSISTANCE PROGRAM*XV*COMEDASSISTPROG~",
|
||||
"N3*PO BOX 1100~",
|
||||
"N4*DENVER*CO*80202~",
|
||||
"REF*2U*7912900843~",
|
||||
"LX*1~",
|
||||
# CLP02=4 means Denied → reconciles to DENIED → Pipeline A.
|
||||
f"CLP*{claim_id}*4*100*0**MC*2026029105200*11*1~",
|
||||
# NM1*QC.NM109 is the member_id; parse_835_svc propagates it to
|
||||
# the SVC row so the visit-side reconcile key matches.
|
||||
f"NM1*QC*1*PATIENT*NAME****MR*{member}~",
|
||||
f"SVC*HC:{procedure}*100*0*UN*1~",
|
||||
f"DTM*472*{svc_date_str}~",
|
||||
# CAS group code + reason → CARC REBILL (not EXCLUDED).
|
||||
f"CAS*{carc_group}*{carc_reason}*100~",
|
||||
"SE*14*0001~",
|
||||
"GE*1*1~",
|
||||
"IEA*1*000000001~",
|
||||
]
|
||||
body = "".join(segs)
|
||||
p = ingest_dir / "x.835"
|
||||
p.write_text(body)
|
||||
return p
|
||||
|
||||
|
||||
def _stub_empty_835(ingest_dir: Path) -> Path:
|
||||
"""Empty 835 → rebill's parse_835_svc emits zero SVCs."""
|
||||
ingest_dir.mkdir(exist_ok=True)
|
||||
p = ingest_dir / "x.835"
|
||||
p.write_text("ST*835*0001~SE*0*0001~")
|
||||
return p
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Tests
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_run_rebill_emits_real_files_for_pipeline_b(tmp_path):
|
||||
"""Pipeline-B happy path: 1 in-window visit, 0 SVCs → 1 real 837P
|
||||
file in pipeline-b/ (non-empty, contains ISA..IEA), 0 files in
|
||||
quarantine/.
|
||||
|
||||
Edifabric is patched to return ``success`` so the file passes the
|
||||
gate cleanly. The fixture is the same shape as the existing
|
||||
``test_post_rebill_runs_and_returns_summary_path`` happy-path test.
|
||||
"""
|
||||
visits_path = _write_visits_csv(tmp_path / "visits.csv", [
|
||||
("06/27/2026", "J813715", "T1019", "$2.32"),
|
||||
])
|
||||
ingest_dir = _stub_empty_835(tmp_path / "ingest")
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
fake_result = {"Status": "success", "Details": [], "LastIndex": 20}
|
||||
with patch(
|
||||
"cyclone.rebill.run._edifabric.validate_edi",
|
||||
return_value=fake_result,
|
||||
) as mock_validate:
|
||||
result = run_rebill(
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
override_filing=False,
|
||||
visits_csv_path=str(visits_path),
|
||||
ingest_dir=str(ingest_dir),
|
||||
out_dir=str(out_dir),
|
||||
)
|
||||
|
||||
# Pipeline B emitted exactly one file (matches the existing
|
||||
# ``len(body["pipeline_b_files"]) == 1`` assertion in test_api_rebill).
|
||||
assert len(result.pipeline_b_files) == 1
|
||||
b_path = result.pipeline_b_files[0]
|
||||
assert b_path.exists(), b_path
|
||||
assert b_path.stat().st_size > 0, "Pipeline-B file must be non-empty"
|
||||
|
||||
# The file is a real 837P envelope (starts with ISA, ends with IEA).
|
||||
body = b_path.read_bytes()
|
||||
assert body.startswith(b"ISA*"), body[:32]
|
||||
assert b"IEA*" in body, body[-32:]
|
||||
|
||||
# Edifabric gate was called exactly once for the one Pipeline-B batch.
|
||||
assert mock_validate.call_count == 1
|
||||
|
||||
# HCPF-spec filename prefix is preserved.
|
||||
assert b_path.name.startswith("tp11525703-837P-"), b_path.name
|
||||
# Per-batch disambiguation suffix is present so same-millisecond
|
||||
# collisions don't overwrite each other.
|
||||
assert "J813715-2026-W26" in b_path.name, b_path.name
|
||||
|
||||
# Quarantine is empty (no validation failures).
|
||||
q_files = list((out_dir / "quarantine").glob("*.837"))
|
||||
assert q_files == [], q_files
|
||||
|
||||
|
||||
def test_run_rebill_quarantines_on_edifabric_error(tmp_path):
|
||||
"""When validate_edi returns Status='error', the file lands in
|
||||
``quarantine/`` (NOT in ``pipeline-b/``).
|
||||
|
||||
Exercises both the Edifabric-rejected path AND the case-isolation
|
||||
that splits good files from bad files in the same run.
|
||||
"""
|
||||
visits_path = _write_visits_csv(tmp_path / "visits.csv", [
|
||||
("06/27/2026", "J813715", "T1019", "$2.32"),
|
||||
("06/27/2026", "J999999", "T1019", "$2.32"), # different member
|
||||
])
|
||||
ingest_dir = _stub_empty_835(tmp_path / "ingest")
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
# First call (J813715) returns error → quarantine.
|
||||
# Second call (J999999) returns success → pipeline-b.
|
||||
fake_results = iter([
|
||||
{
|
||||
"Status": "error",
|
||||
"Details": [
|
||||
{"SegmentId": "PER", "Message": "PER-04 is required", "Status": "error"},
|
||||
],
|
||||
"LastIndex": 5,
|
||||
},
|
||||
{"Status": "success", "Details": [], "LastIndex": 20},
|
||||
])
|
||||
|
||||
def _side_effect(_body):
|
||||
return next(fake_results)
|
||||
|
||||
with patch(
|
||||
"cyclone.rebill.run._edifabric.validate_edi",
|
||||
side_effect=_side_effect,
|
||||
) as mock_validate:
|
||||
result = run_rebill(
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
override_filing=False,
|
||||
visits_csv_path=str(visits_path),
|
||||
ingest_dir=str(ingest_dir),
|
||||
out_dir=str(out_dir),
|
||||
)
|
||||
|
||||
# Only the success batch should land in pipeline-b/.
|
||||
assert len(result.pipeline_b_files) == 1, (
|
||||
f"only the success batch should land in pipeline-b/, got "
|
||||
f"{[str(p) for p in result.pipeline_b_files]}"
|
||||
)
|
||||
# One file in quarantine (the rejected batch).
|
||||
q_files = list((out_dir / "quarantine").glob("*.837"))
|
||||
assert len(q_files) == 1, q_files
|
||||
# Quarantine filename keys off the batch (member_id-W{iso_week:02d}).
|
||||
assert "J813715" in q_files[0].name, q_files[0].name
|
||||
assert q_files[0].stat().st_size > 0, "quarantined file must be non-empty"
|
||||
|
||||
# validate_edi called twice — once per Pipeline-B batch.
|
||||
assert mock_validate.call_count == 2
|
||||
|
||||
|
||||
def test_run_rebill_fails_open_when_edifabric_unavailable(tmp_path):
|
||||
"""When validate_edi raises EdifabricError (no API key, network
|
||||
error, 5xx), the file lands in the pipeline dir with a WARNING log.
|
||||
|
||||
Matches the SP40 dev/CI posture: tests / dev boxes don't have a
|
||||
real API key, so the gate must NOT block the rebill run.
|
||||
"""
|
||||
from cyclone.edifabric import EdifabricError
|
||||
|
||||
visits_path = _write_visits_csv(tmp_path / "visits.csv", [
|
||||
("06/27/2026", "J813715", "T1019", "$2.32"),
|
||||
])
|
||||
ingest_dir = _stub_empty_835(tmp_path / "ingest")
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
with patch(
|
||||
"cyclone.rebill.run._edifabric.validate_edi",
|
||||
side_effect=EdifabricError(0, "API key not configured"),
|
||||
) as mock_validate:
|
||||
result = run_rebill(
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
override_filing=False,
|
||||
visits_csv_path=str(visits_path),
|
||||
ingest_dir=str(ingest_dir),
|
||||
out_dir=str(out_dir),
|
||||
)
|
||||
|
||||
# File emitted to pipeline-b/ (NOT to quarantine — fail-open).
|
||||
assert len(result.pipeline_b_files) == 1
|
||||
assert result.pipeline_b_files[0].exists()
|
||||
# Quarantine stays empty.
|
||||
q_files = list((out_dir / "quarantine").glob("*.837"))
|
||||
assert q_files == [], q_files
|
||||
# Gate was actually called (so we know the fail-open path fired,
|
||||
# not some accidental short-circuit).
|
||||
assert mock_validate.call_count == 1
|
||||
|
||||
|
||||
def test_run_rebill_emits_real_files_for_pipeline_a(tmp_path):
|
||||
"""Pipeline-A happy path: 1 in-window visit, 1 denied SVC matching
|
||||
it → 1 real 837P file in pipeline-a/ with frequency-code 7
|
||||
preserved.
|
||||
|
||||
The SVC's original claim_id (ORIG-1) carries through as the
|
||||
Pipeline-A RebillClaim.claim_id and shows up in the emitted file's
|
||||
CLM01 segment AND in the on-disk filename suffix.
|
||||
"""
|
||||
svc_date = date(2026, 6, 27)
|
||||
visits_path = _write_visits_csv(tmp_path / "visits.csv", [
|
||||
(svc_date.strftime("%m/%d/%Y"), "J813715", "T1019", "$100.00"),
|
||||
])
|
||||
ingest_dir = tmp_path / "ingest"
|
||||
_stub_835_with_denied_svc(
|
||||
ingest_dir,
|
||||
member="J813715",
|
||||
procedure="T1019",
|
||||
svc_date=svc_date,
|
||||
claim_id="ORIG-1",
|
||||
)
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
fake_result = {"Status": "success", "Details": [], "LastIndex": 30}
|
||||
with patch(
|
||||
"cyclone.rebill.run._edifabric.validate_edi",
|
||||
return_value=fake_result,
|
||||
):
|
||||
result = run_rebill(
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
override_filing=False,
|
||||
visits_csv_path=str(visits_path),
|
||||
ingest_dir=str(ingest_dir),
|
||||
out_dir=str(out_dir),
|
||||
)
|
||||
|
||||
# Pipeline A emitted exactly one file.
|
||||
assert len(result.pipeline_a_files) == 1, result.pipeline_a_files
|
||||
a_path = result.pipeline_a_files[0]
|
||||
assert a_path.exists(), a_path
|
||||
assert a_path.stat().st_size > 0, "Pipeline-A file must be non-empty"
|
||||
|
||||
body = a_path.read_bytes()
|
||||
assert body.startswith(b"ISA*"), body[:32]
|
||||
assert b"IEA*" in body, body[-32:]
|
||||
|
||||
# Original claim_submit_id is preserved in CLM01 (anchors the
|
||||
# frequency-7 replacement).
|
||||
assert b"CLM*ORIG-1*" in body, body
|
||||
# Frequency-code 7 emitted in CLM05 composite (12:B:7 — Home POS
|
||||
# is canonical for Dzinesco IHSS S5150/T1019 services).
|
||||
assert b"CLM*ORIG-1*100.00***12:B:7" in body, body
|
||||
|
||||
# Pipeline-A filename uses HCPF-spec prefix + claim_id suffix.
|
||||
assert a_path.name.startswith("tp11525703-837P-"), a_path.name
|
||||
assert a_path.name.endswith("-ORIG-1.x12"), a_path.name
|
||||
|
||||
# Quarantine empty.
|
||||
q_files = list((out_dir / "quarantine").glob("*.837"))
|
||||
assert q_files == [], q_files
|
||||
|
||||
|
||||
def test_run_rebill_quarantines_pipeline_a_on_edifabric_error(tmp_path):
|
||||
"""Pipeline A path on Edifabric Status='error' → quarantine, NOT
|
||||
pipeline-a/.
|
||||
|
||||
The existing ``test_run_rebill_quarantines_on_edifabric_error`` uses
|
||||
an empty 835 so all visits land in Pipeline B; this one builds a
|
||||
real 835 with a denied SVC matching the visit so the claim routes
|
||||
to Pipeline A, then asserts the rejected file lands in
|
||||
``quarantine/`` and ``pipeline-a/`` stays empty.
|
||||
"""
|
||||
svc_date = date(2026, 6, 27)
|
||||
visits_path = _write_visits_csv(tmp_path / "visits.csv", [
|
||||
(svc_date.strftime("%m/%d/%Y"), "J813715", "T1019", "$100.00"),
|
||||
])
|
||||
ingest_dir = tmp_path / "ingest"
|
||||
_stub_835_with_denied_svc(
|
||||
ingest_dir,
|
||||
member="J813715",
|
||||
procedure="T1019",
|
||||
svc_date=svc_date,
|
||||
claim_id="ORIG-ERROR-1",
|
||||
carc_reason="29", # CO-29 → REBILL (not EXCLUDED)
|
||||
)
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
fake_result = {
|
||||
"Status": "error",
|
||||
"Details": [
|
||||
{"SegmentId": "PER", "Message": "PER-04 is required",
|
||||
"Status": "error"},
|
||||
],
|
||||
"LastIndex": 5,
|
||||
}
|
||||
with patch(
|
||||
"cyclone.rebill.run._edifabric.validate_edi",
|
||||
return_value=fake_result,
|
||||
) as mock_validate:
|
||||
result = run_rebill(
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
override_filing=False,
|
||||
visits_csv_path=str(visits_path),
|
||||
ingest_dir=str(ingest_dir),
|
||||
out_dir=str(out_dir),
|
||||
)
|
||||
|
||||
# Pipeline-A path rejected → 0 files in pipeline-a/.
|
||||
assert result.pipeline_a_files == [], (
|
||||
f"Pipeline A rejected file should NOT be in pipeline-a/, got "
|
||||
f"{[str(p) for p in result.pipeline_a_files]}"
|
||||
)
|
||||
a_files = list((out_dir / "pipeline-a").glob("*"))
|
||||
assert a_files == [], a_files
|
||||
|
||||
# File quarantined under the original claim_id key.
|
||||
q_files = list((out_dir / "quarantine").glob("*.837"))
|
||||
assert len(q_files) == 1, q_files
|
||||
assert "ORIG-ERROR-1" in q_files[0].name, q_files[0].name
|
||||
assert q_files[0].stat().st_size > 0, "quarantined file must be non-empty"
|
||||
|
||||
# Gate was called for the one Pipeline-A claim.
|
||||
assert mock_validate.call_count == 1
|
||||
|
||||
|
||||
def test_run_rebill_mixed_pipelines_route_correctly(tmp_path):
|
||||
"""Mixed-pipeline run: one DENIED visit (Pipeline A) + one
|
||||
NOT_IN_835 visit (Pipeline B) in the same window.
|
||||
|
||||
Asserts:
|
||||
- pipeline_a_files has exactly 1 entry (the denied visit)
|
||||
- pipeline_b_files has exactly 1 entry (the unmatched visit)
|
||||
- summary.csv has 2 rows (one per non-PAID visit)
|
||||
"""
|
||||
svc_date = date(2026, 6, 27)
|
||||
# Visit 1: matches a denied SVC → Pipeline A.
|
||||
# Visit 2: different member entirely → NOT_IN_835 → Pipeline B.
|
||||
visits_path = _write_visits_csv(tmp_path / "visits.csv", [
|
||||
(svc_date.strftime("%m/%d/%Y"), "J813715", "T1019", "$100.00"),
|
||||
(svc_date.strftime("%m/%d/%Y"), "J999999", "T1019", "$2.32"),
|
||||
])
|
||||
ingest_dir = tmp_path / "ingest"
|
||||
_stub_835_with_denied_svc(
|
||||
ingest_dir,
|
||||
member="J813715",
|
||||
procedure="T1019",
|
||||
svc_date=svc_date,
|
||||
claim_id="ORIG-MIX-1",
|
||||
carc_reason="29",
|
||||
)
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
fake_result = {"Status": "success", "Details": [], "LastIndex": 30}
|
||||
with patch(
|
||||
"cyclone.rebill.run._edifabric.validate_edi",
|
||||
return_value=fake_result,
|
||||
) as mock_validate:
|
||||
result = run_rebill(
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
override_filing=False,
|
||||
visits_csv_path=str(visits_path),
|
||||
ingest_dir=str(ingest_dir),
|
||||
out_dir=str(out_dir),
|
||||
)
|
||||
|
||||
# One file each, the two pipelines route independently.
|
||||
assert len(result.pipeline_a_files) == 1, result.pipeline_a_files
|
||||
assert len(result.pipeline_b_files) == 1, result.pipeline_b_files
|
||||
|
||||
# Files on disk match the returned paths.
|
||||
a_path = result.pipeline_a_files[0]
|
||||
b_path = result.pipeline_b_files[0]
|
||||
assert a_path.exists() and a_path.stat().st_size > 0
|
||||
assert b_path.exists() and b_path.stat().st_size > 0
|
||||
|
||||
# Each gate call hit validate_edi exactly once (one per claim/batch).
|
||||
assert mock_validate.call_count == 2
|
||||
|
||||
# summary.csv has one row per non-PAID visit.
|
||||
summary_path = out_dir / "summary.csv"
|
||||
assert summary_path.exists(), summary_path
|
||||
with summary_path.open(newline="") as f:
|
||||
rows = list(csv.DictReader(f))
|
||||
assert len(rows) == 2, rows
|
||||
dispositions = {r["disposition"] for r in rows}
|
||||
assert dispositions == {"REBILLED_A", "REBILLED_B"}, dispositions
|
||||
|
||||
|
||||
def test_run_rebill_fail_open_logs_warning(tmp_path, caplog):
|
||||
"""The Edifabric-unavailable fail-open path MUST log a WARNING so
|
||||
the operator knows the validation gate didn't actually run.
|
||||
|
||||
Per SP40/SP41 contract: when ``validate_edi`` raises
|
||||
``EdifabricError``, the file still emits to the pipeline dir but a
|
||||
WARNING is logged. This test pins that contract so a future
|
||||
refactor doesn't accidentally silence it.
|
||||
"""
|
||||
import logging
|
||||
|
||||
from cyclone.edifabric import EdifabricError
|
||||
|
||||
caplog.set_level(logging.WARNING, logger="cyclone.rebill.run")
|
||||
|
||||
visits_path = _write_visits_csv(tmp_path / "visits.csv", [
|
||||
("06/27/2026", "J813715", "T1019", "$2.32"),
|
||||
])
|
||||
ingest_dir = _stub_empty_835(tmp_path / "ingest")
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
with patch(
|
||||
"cyclone.rebill.run._edifabric.validate_edi",
|
||||
side_effect=EdifabricError(0, "API key not configured"),
|
||||
):
|
||||
result = run_rebill(
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
override_filing=False,
|
||||
visits_csv_path=str(visits_path),
|
||||
ingest_dir=str(ingest_dir),
|
||||
out_dir=str(out_dir),
|
||||
)
|
||||
|
||||
# File still emits (fail-open semantics).
|
||||
assert len(result.pipeline_b_files) == 1
|
||||
|
||||
# And a WARNING was logged about the unrun gate.
|
||||
warnings = [r for r in caplog.records if r.levelno >= logging.WARNING]
|
||||
assert warnings, "expected at least one WARNING record"
|
||||
assert any(
|
||||
"edifabric" in r.message.lower() or "validation" in r.message.lower()
|
||||
for r in warnings
|
||||
), f"expected WARNING mentioning edifabric/validation, got {[r.message for r in warnings]}"
|
||||
|
||||
@@ -0,0 +1,484 @@
|
||||
"""SP41-spot-check: unit tests for ``cyclone.rebill.spot_check``.
|
||||
|
||||
Drives the shipped :func:`cyclone.parsers.serialize_837.serialize_837`
|
||||
end-to-end with a deterministic :class:`VisitRow` input. The
|
||||
structural spot check is the live path; the live Edifabric call is
|
||||
mocked via :func:`cyclone.edifabric.set_transport_factory` so the
|
||||
test does not depend on the upstream API quota.
|
||||
|
||||
Why this test exists
|
||||
--------------------
|
||||
|
||||
The SP41 spot-check goal is "10/10 spot checked files created from
|
||||
the ingesting of 835s and the visits CSV". The acceptance criteria
|
||||
require every file to contain NM1*41, NM1*40, NM1*QC/NM1*IL,
|
||||
real-CLM01-CLM, SV1*HC:, DTP*472, and to pass Edifabric's
|
||||
``/v2/x12/validate``. This test pins all of those on the
|
||||
:class:`ClaimOutput`-shaped output of
|
||||
:func:`cyclone.rebill.spot_check.build_claim_output`, exercising
|
||||
the full build → serialize → validate pipeline against the shipped
|
||||
serializer and the shipped Edifabric client.
|
||||
|
||||
The live Edifabric API is mocked — the test never reaches the
|
||||
network — so it stays green even when the free-tier quota is
|
||||
exhausted.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from cyclone import edifabric
|
||||
from cyclone.edifabric import EdifabricError
|
||||
from cyclone.parsers.serialize_837 import serialize_837
|
||||
from cyclone.rebill.spot_check import (
|
||||
SBR09_DEFAULT,
|
||||
SENDER_ID,
|
||||
VisitRow,
|
||||
build_claim_output,
|
||||
format_spot_check_result,
|
||||
iter_segments,
|
||||
structural_spot_check,
|
||||
)
|
||||
|
||||
|
||||
# --- Test fixtures -----------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_transport():
|
||||
"""Restore the default Edifabric transport after each test."""
|
||||
yield
|
||||
edifabric._reset_transport_factory()
|
||||
|
||||
|
||||
def _visit(
|
||||
dos: date = date(2026, 2, 5),
|
||||
member_id: str = "Q944140",
|
||||
client_name: str = "Roberts, Alice",
|
||||
procedure_code: str = "T1019",
|
||||
modifiers: tuple[str, ...] = ("U1",),
|
||||
billed_amount: Decimal = Decimal("333.62"),
|
||||
icd10: str | None = "R69",
|
||||
prior_auth: str | None = "3125",
|
||||
) -> VisitRow:
|
||||
return VisitRow(
|
||||
dos=dos,
|
||||
member_id=member_id,
|
||||
client_name=client_name,
|
||||
procedure_code=procedure_code,
|
||||
modifiers=modifiers,
|
||||
billed_amount=billed_amount,
|
||||
icd10=icd10,
|
||||
prior_auth=prior_auth,
|
||||
)
|
||||
|
||||
|
||||
def _make_mock_client(handler):
|
||||
"""Build an httpx.Client whose transport is the given handler."""
|
||||
transport = httpx.MockTransport(handler)
|
||||
return httpx.Client(transport=transport, timeout=10.0)
|
||||
|
||||
|
||||
def _install_factory(handler):
|
||||
return edifabric.set_transport_factory(lambda: _make_mock_client(handler))
|
||||
|
||||
|
||||
# --- build_claim_output ------------------------------------------------
|
||||
|
||||
|
||||
def test_build_claim_output_emits_canonical_envelope():
|
||||
visit = _visit()
|
||||
claim = build_claim_output(visit)
|
||||
assert claim.claim.claim_id == "20260205-Q944140-01"
|
||||
assert claim.claim.total_charge == Decimal("333.62")
|
||||
assert claim.subscriber.member_id == "Q944140"
|
||||
assert claim.subscriber.last_name == "Roberts"
|
||||
assert claim.subscriber.first_name == "Alice"
|
||||
assert claim.service_lines[0].procedure.code == "T1019"
|
||||
assert claim.service_lines[0].procedure.modifiers == ["U1"]
|
||||
assert claim.service_lines[0].service_date == date(2026, 2, 5)
|
||||
assert claim.service_lines[0].charge == Decimal("333.62")
|
||||
assert claim.diagnoses[0].code == "R69"
|
||||
assert claim.claim.prior_auth == "3125"
|
||||
assert claim.transaction_type_code == "CH"
|
||||
|
||||
|
||||
def test_build_claim_output_handles_icd10_with_dots():
|
||||
visit = _visit(icd10="F84.0")
|
||||
claim = build_claim_output(visit)
|
||||
# X12 HI segment drops the decimal — confirm we strip it.
|
||||
assert claim.diagnoses[0].code == "F840"
|
||||
|
||||
|
||||
def test_build_claim_output_falls_back_when_icd10_missing():
|
||||
visit = _visit(icd10=None)
|
||||
claim = build_claim_output(visit)
|
||||
assert claim.diagnoses[0].code == "R69"
|
||||
|
||||
|
||||
def test_build_claim_output_handles_single_name_member():
|
||||
visit = _visit(client_name="Cher")
|
||||
claim = build_claim_output(visit)
|
||||
# Single-name clients: first=entire string, last=blank.
|
||||
assert claim.subscriber.first_name == "Cher"
|
||||
assert claim.subscriber.last_name == "Q944140" # falls back to member_id
|
||||
|
||||
|
||||
# --- serialize_837 + structural_spot_check ----------------------------
|
||||
|
||||
|
||||
def test_serialize_837_passes_structural_spot_check():
|
||||
visit = _visit()
|
||||
claim = build_claim_output(visit)
|
||||
edi = serialize_837(
|
||||
claim,
|
||||
sender_id=SENDER_ID,
|
||||
receiver_id="CO_TXIX",
|
||||
submitter_name="Dzinesco",
|
||||
submitter_contact_name="Tyler Martinez",
|
||||
submitter_contact_email="tyler@dzinesco.com",
|
||||
submitter_contact_phone="8005550100",
|
||||
receiver_name="COLORADO MEDICAL ASSISTANCE PROGRAM",
|
||||
claim_filing_indicator_code=SBR09_DEFAULT,
|
||||
interchange_control_number="000000001",
|
||||
group_control_number="1",
|
||||
)
|
||||
result = structural_spot_check(edi)
|
||||
assert result.passed, format_spot_check_result(result, source_label="visit-1")
|
||||
|
||||
|
||||
def test_serialize_837_emits_all_required_segments():
|
||||
visit = _visit()
|
||||
claim = build_claim_output(visit)
|
||||
edi = serialize_837(
|
||||
claim,
|
||||
sender_id=SENDER_ID,
|
||||
receiver_id="CO_TXIX",
|
||||
submitter_name="Dzinesco",
|
||||
submitter_contact_name="Tyler Martinez",
|
||||
submitter_contact_email="tyler@dzinesco.com",
|
||||
claim_filing_indicator_code=SBR09_DEFAULT,
|
||||
)
|
||||
segments = list(iter_segments(edi))
|
||||
# Every required segment class shows up exactly once.
|
||||
assert any(s.startswith("ISA*") for s in segments), "missing ISA"
|
||||
assert any(s.startswith("GS*") for s in segments), "missing GS"
|
||||
assert any(s.startswith("ST*837") for s in segments), "missing ST*837"
|
||||
assert any(s.startswith("BHT*") for s in segments), "missing BHT"
|
||||
assert any(s.startswith("NM1*41") for s in segments), "missing NM1*41"
|
||||
assert any(s.startswith("NM1*40") for s in segments), "missing NM1*40"
|
||||
assert any(s.startswith("NM1*85") for s in segments), "missing NM1*85"
|
||||
assert any(s.startswith("NM1*IL") for s in segments), "missing NM1*IL"
|
||||
assert any(s.startswith("NM1*PR") for s in segments), "missing NM1*PR"
|
||||
assert any(s.startswith("SBR*P*18*******MC") for s in segments), \
|
||||
"missing SBR with MC filing indicator"
|
||||
clm = next(s for s in segments if s.startswith("CLM*"))
|
||||
assert clm.startswith("CLM*20260205-Q944140-01*333.62"), \
|
||||
f"CLM01 must be DOS-member-idx (got {clm!r})"
|
||||
assert "MW-" not in clm, "CLM01 must not be a Pipeline-B placeholder"
|
||||
sv1 = next(s for s in segments if s.startswith("SV1*HC:"))
|
||||
assert sv1.startswith("SV1*HC:T1019:U1*333.62"), \
|
||||
f"SV1 must carry HC:T1019:U1 (got {sv1!r})"
|
||||
dtp = next(s for s in segments if s.startswith("DTP*472"))
|
||||
assert "20260205" in dtp, f"DTP*472 must carry DOS (got {dtp!r})"
|
||||
assert any(s.startswith("HI*") for s in segments), "missing HI diagnoses"
|
||||
assert any(s.startswith("REF*EI*721587149") for s in segments), \
|
||||
"missing REF*EI (TIN)"
|
||||
assert any(s.startswith("SE*") for s in segments), "missing SE trailer"
|
||||
assert any(s.startswith("GE*") for s in segments), "missing GE trailer"
|
||||
assert any(s.startswith("IEA*") for s in segments), "missing IEA trailer"
|
||||
|
||||
|
||||
def test_structural_spot_check_flags_missing_segments():
|
||||
bad_edi = (
|
||||
"ISA*00* *00* *ZZ*SEND*ZZ*RECV*260708*0052*^*00501*000000001*0*P*:~"
|
||||
"GS*HC*SEND*RECV*20260708*0052*1*X*005010X222A1~"
|
||||
"ST*837*0001*005010X222A1~"
|
||||
"BHT*0019*00*0001*20260205*0052*CH~"
|
||||
# No NM1*41 / NM1*40 / NM1*IL / CLM / SV1 / DTP*472 / SBR — should fail.
|
||||
"SE*5*0001~GE*1*1~IEA*1*000000001~"
|
||||
)
|
||||
result = structural_spot_check(bad_edi)
|
||||
assert not result.passed
|
||||
# Every missing class shows up in `missing`.
|
||||
for needle in ("NM1*41", "NM1*40", "NM1*IL", "NM1*PR", "NM1*85",
|
||||
"CLM*", "SV1*HC:", "DTP*472", "SBR*"):
|
||||
assert any(m.startswith(needle) for m in result.missing), \
|
||||
f"expected {needle!r} in missing={result.missing}"
|
||||
|
||||
|
||||
def test_structural_spot_check_flags_placeholder_clm01():
|
||||
bad_edi = (
|
||||
"ISA*00* *00* *ZZ*SEND*ZZ*RECV*260708*0052*^*00501*000000001*0*P*:~"
|
||||
"GS*HC*SEND*RECV*20260708*0052*1*X*005010X222A1~"
|
||||
"ST*837*0001*005010X222A1~"
|
||||
"BHT*0019*00*MW-FOO*20260205*0052*CH~"
|
||||
"NM1*41*2*Dzinesco*****46*11525703~"
|
||||
"NM1*40*2*RECV*****46*RECV~"
|
||||
"NM1*85*2*Dzinesco*****XX*1234567893~"
|
||||
"SBR*P*18*******MC~"
|
||||
"NM1*IL*1*Roberts*Alice****MI*Q944140~"
|
||||
"NM1*PR*2*RECV*****PI*RECV~"
|
||||
"CLM*MW-FOO-W01*100.00***11:B:1*Y*Y*Y*Y~" # Pipeline-B placeholder CLM01
|
||||
"SV1*HC:T1019*100.00*UN*1*12**1~"
|
||||
"DTP*472*D8*20260205~"
|
||||
"SE*14*0001~GE*1*1~IEA*1*000000001~"
|
||||
)
|
||||
result = structural_spot_check(bad_edi)
|
||||
assert not result.passed
|
||||
assert "CLM01-not-placeholder" in result.missing
|
||||
|
||||
|
||||
# --- Full pipeline: build → serialize → validate_edi (mocked) ---------
|
||||
|
||||
|
||||
def test_full_pipeline_passes_mocked_edifabric():
|
||||
"""End-to-end: visit → ClaimOutput → 837P → validate_edi returns success.
|
||||
|
||||
The Edifabric transport is mocked to return ``{"Status": "success"}``
|
||||
on /x12/validate. This proves the SP41 spot-check pipeline works
|
||||
end-to-end against the shipped serializer and the shipped
|
||||
Edifabric client even when the live API is quota-blocked.
|
||||
"""
|
||||
visit = _visit()
|
||||
claim = build_claim_output(visit)
|
||||
edi = serialize_837(
|
||||
claim,
|
||||
sender_id=SENDER_ID,
|
||||
receiver_id="CO_TXIX",
|
||||
submitter_name="Dzinesco",
|
||||
submitter_contact_name="Tyler Martinez",
|
||||
submitter_contact_email="tyler@dzinesco.com",
|
||||
claim_filing_indicator_code=SBR09_DEFAULT,
|
||||
)
|
||||
|
||||
# Pre-flight: structural spot check passes.
|
||||
assert structural_spot_check(edi).passed
|
||||
|
||||
# Mock the /x12/validate transport to return success.
|
||||
success_payload = {"Status": "success", "Details": [], "LastIndex": 25}
|
||||
read_called = {"count": 0}
|
||||
validate_called = {"count": 0}
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/x12/read"):
|
||||
read_called["count"] += 1
|
||||
assert request.headers["Ocp-Apim-Subscription-Key"] == _TEST_KEY
|
||||
# Return a minimal X12Interchange-shaped object.
|
||||
return httpx.Response(200, json=[_minimal_x12_interchange()])
|
||||
if request.url.path.endswith("/x12/validate"):
|
||||
validate_called["count"] += 1
|
||||
return httpx.Response(200, json=success_payload)
|
||||
return httpx.Response(404, json={"error": f"unexpected path {request.url.path}"})
|
||||
|
||||
_install_factory(handler)
|
||||
|
||||
result = edifabric.validate_edi(edi.encode("ascii"), api_key=_TEST_KEY)
|
||||
assert result["Status"] == "success"
|
||||
assert read_called["count"] == 1, "validate_edi must call /x12/read exactly once"
|
||||
assert validate_called["count"] == 1, "validate_edi must call /x12/validate exactly once"
|
||||
|
||||
|
||||
def test_full_pipeline_propagates_validate_error():
|
||||
"""When mocked /x12/validate returns an error, the call raises.
|
||||
|
||||
Documents the failure path — if the live API returns a non-2xx
|
||||
(e.g. 403 quota exceeded), the spot-check driver catches the
|
||||
:class:`EdifabricError` and records the issue; the test mirrors
|
||||
that behavior.
|
||||
"""
|
||||
visit = _visit()
|
||||
claim = build_claim_output(visit)
|
||||
edi = serialize_837(claim, sender_id=SENDER_ID, claim_filing_indicator_code=SBR09_DEFAULT)
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/x12/read"):
|
||||
return httpx.Response(200, json=[_minimal_x12_interchange()])
|
||||
if request.url.path.endswith("/x12/validate"):
|
||||
return httpx.Response(
|
||||
403,
|
||||
json={"statusCode": 403, "message": "Out of call volume quota."},
|
||||
)
|
||||
return httpx.Response(404)
|
||||
|
||||
_install_factory(handler)
|
||||
|
||||
with pytest.raises(EdifabricError) as excinfo:
|
||||
edifabric.validate_edi(edi.encode("ascii"), api_key=_TEST_KEY)
|
||||
assert excinfo.value.status_code == 403
|
||||
assert "quota" in str(excinfo.value.body).lower()
|
||||
|
||||
|
||||
def test_claim_passes_shipped_sp40_local_validator():
|
||||
"""The shipped SP40 local validator (25 R-codes) passes the ClaimOutput.
|
||||
|
||||
The local validator at ``cyclone.parsers.validator.validate`` is the
|
||||
structural pre-flight gate that mirrors what Edifabric checks at the
|
||||
/v2/x12/validate level. Running the same claim through it confirms
|
||||
the spot-check pipeline produces structurally-correct 837Ps even
|
||||
when the live Edifabric API is unavailable.
|
||||
"""
|
||||
from cyclone.parsers.payer import PayerConfig
|
||||
from cyclone.parsers.validator import validate as validate_claim
|
||||
|
||||
visit = _visit()
|
||||
claim = build_claim_output(visit)
|
||||
report = validate_claim(claim, PayerConfig.co_medicaid())
|
||||
assert report.passed, (
|
||||
f"SP40 local validator rejected the claim: "
|
||||
f"errors={[(i.rule, i.message) for i in report.errors]}"
|
||||
)
|
||||
assert not report.errors, f"unexpected errors: {report.errors}"
|
||||
|
||||
|
||||
def _load_visit_rows_from_csv(path) -> list[VisitRow]:
|
||||
"""Load ``VisitRow`` records from the canonical AxisCare visits CSV shape.
|
||||
|
||||
The CSV is the same shape as ``ingest/jan1-jun272026.csv`` — see
|
||||
:func:`cyclone.rebill.spot_check.parse_visits_csv` for the production
|
||||
parser. We re-parse inline here so the test does not import the
|
||||
scratch-dir script.
|
||||
"""
|
||||
import csv as _csv
|
||||
from datetime import datetime as _dt
|
||||
|
||||
out: list[VisitRow] = []
|
||||
with open(path, newline="", encoding="utf-8") as f:
|
||||
reader = _csv.DictReader(f)
|
||||
for row in reader:
|
||||
raw_dos = (row.get("Visit Date") or "").strip()
|
||||
if not raw_dos:
|
||||
continue
|
||||
try:
|
||||
dos = _dt.strptime(raw_dos, "%m/%d/%Y").date()
|
||||
except ValueError:
|
||||
continue
|
||||
member_id = (row.get("Member ID") or "").strip()
|
||||
procedure = (row.get("Procedure Code") or "").strip()
|
||||
if not (member_id and procedure):
|
||||
continue
|
||||
billed_clean = (row.get("Billable Amount") or "0").replace("$", "").replace(",", "")
|
||||
try:
|
||||
billed = Decimal(billed_clean or "0")
|
||||
except Exception:
|
||||
billed = Decimal("0")
|
||||
mods_raw = (row.get("Modifiers") or "").strip()
|
||||
modifiers = tuple(m.strip() for m in mods_raw.split(",") if m.strip())
|
||||
out.append(VisitRow(
|
||||
dos=dos, member_id=member_id,
|
||||
client_name=(row.get("Client") or "").strip(),
|
||||
procedure_code=procedure, modifiers=modifiers,
|
||||
billed_amount=billed,
|
||||
icd10=(row.get("ICD-10") or "").strip() or None,
|
||||
prior_auth=(row.get("Authorization #") or "").strip() or None,
|
||||
))
|
||||
return out
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"fixture_csv",
|
||||
[
|
||||
"tests/fixtures/spot_check_visits.csv", # 10 batch-1 (NOT_IN_835)
|
||||
"tests/fixtures/spot_check_visits_batch2.csv", # 10 batch-2 (DENIED+PARTIAL)
|
||||
],
|
||||
)
|
||||
def test_every_visit_in_fixture_passes_full_pipeline(fixture_csv):
|
||||
"""For each visit in the fixture CSV, the full pipeline (build → serialize →
|
||||
SP40 local validator) passes.
|
||||
|
||||
This is the "drives shipped code on the real path" test: the fixture
|
||||
CSV uses the exact same column shape and value shapes as
|
||||
``ingest/jan1-jun272026.csv`` (the production visits export), and
|
||||
each visit row runs through:
|
||||
|
||||
1. :func:`cyclone.rebill.spot_check.build_claim_output` (the
|
||||
canonical helper)
|
||||
2. :func:`cyclone.parsers.serialize_837.serialize_837` (the
|
||||
shipped serializer)
|
||||
3. :func:`cyclone.parsers.validator.validate` (the shipped SP40
|
||||
local validator — 25 R-codes)
|
||||
4. :func:`cyclone.rebill.spot_check.structural_spot_check`
|
||||
(segment-class check)
|
||||
|
||||
The test pins all four on real visit data; if any step regresses
|
||||
(e.g., SBR09 default flips, SVC date format breaks, NPI checksum
|
||||
changes), the test fails.
|
||||
"""
|
||||
from cyclone.parsers.payer import PayerConfig
|
||||
from cyclone.parsers.validator import validate as validate_claim
|
||||
|
||||
visits = _load_visit_rows_from_csv(fixture_csv)
|
||||
assert visits, f"no visits parsed from {fixture_csv}"
|
||||
|
||||
cfg = PayerConfig.co_medicaid()
|
||||
for idx, visit in enumerate(visits, start=1):
|
||||
claim_id = f"{visit.dos.strftime('%Y%m%d')}-{visit.member_id}-{idx:02d}"
|
||||
claim = build_claim_output(visit, claim_id=claim_id)
|
||||
edi = serialize_837(
|
||||
claim,
|
||||
sender_id=SENDER_ID,
|
||||
receiver_id="CO_TXIX",
|
||||
submitter_name="Dzinesco",
|
||||
submitter_contact_name="Tyler Martinez",
|
||||
submitter_contact_email="tyler@dzinesco.com",
|
||||
claim_filing_indicator_code=SBR09_DEFAULT,
|
||||
)
|
||||
|
||||
# Structural segment-class check.
|
||||
struct = structural_spot_check(edi)
|
||||
assert struct.passed, (
|
||||
f"{claim_id} failed structural spot check: "
|
||||
f"missing={list(struct.missing)}"
|
||||
)
|
||||
|
||||
# Shipped SP40 local validator (25 R-codes).
|
||||
report = validate_claim(claim, cfg)
|
||||
assert report.passed, (
|
||||
f"{claim_id} failed SP40 local validator: "
|
||||
f"errors={[(i.rule, i.message) for i in report.errors]}"
|
||||
)
|
||||
assert not report.errors, f"{claim_id}: unexpected errors: {report.errors}"
|
||||
|
||||
|
||||
# --- Helpers -----------------------------------------------------------
|
||||
|
||||
|
||||
_TEST_KEY = "test-spot-check-key-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
|
||||
|
||||
|
||||
def _minimal_x12_interchange() -> dict:
|
||||
"""A minimal X12Interchange-shaped dict that Edifabric /validate accepts.
|
||||
|
||||
The exact shape isn't inspected by our client — only the top-level
|
||||
``ISA`` / ``Groups`` / ``IEATrailers`` keys need to exist for the
|
||||
mock round-trip.
|
||||
"""
|
||||
return {
|
||||
"SegmentDelimiter": "~",
|
||||
"DataElementDelimiter": "*",
|
||||
"ISA": {
|
||||
"InterchangeControlNumber_13": "000000001",
|
||||
"SenderID_06": "11525703",
|
||||
"ReceiverID_08": "CO_TXIX",
|
||||
},
|
||||
"Groups": [
|
||||
{
|
||||
"FunctionalID_01": "HC",
|
||||
"GroupControlNumber_06": "1",
|
||||
"Transactions": [
|
||||
{
|
||||
"ST_01": "837",
|
||||
"ControlNumber_02": "0001",
|
||||
"ImplementationConventionReference_03": "005010X222A1",
|
||||
"Segments": [],
|
||||
}
|
||||
],
|
||||
}
|
||||
],
|
||||
"IEATrailers": [{"InterchangeControlNumber_02": "000000001"}],
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
"""Summary CSV: one row per visit with its disposition."""
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
from cyclone.rebill.summary import (
|
||||
SummaryRow,
|
||||
write_summary_csv,
|
||||
REBILLED_A, REBILLED_B, EXCLUDED_CARC, EXCLUDED_TIMELY_FILING,
|
||||
EXCLUDED_PAYER, EXCLUDED_NO_EVV,
|
||||
)
|
||||
from cyclone.rebill.reconcile import VisitRow
|
||||
|
||||
|
||||
def test_summary_csv_emits_one_row_per_visit(tmp_path: Path):
|
||||
out = tmp_path / "summary.csv"
|
||||
rows = [
|
||||
SummaryRow(
|
||||
visit=VisitRow(date=date(2026, 6, 27), member_id="J813715",
|
||||
procedure="T1019", billed=Decimal("2.32")),
|
||||
disposition=REBILLED_A, unpaid=Decimal("0.00"),
|
||||
cas_reasons=("CO-45",), file_path="pipeline-a/x.837",
|
||||
),
|
||||
SummaryRow(
|
||||
visit=VisitRow(date=date(2026, 1, 1), member_id="OLD",
|
||||
procedure="T1019", billed=Decimal("2.32")),
|
||||
disposition=EXCLUDED_TIMELY_FILING, unpaid=Decimal("2.32"),
|
||||
cas_reasons=(), file_path="",
|
||||
),
|
||||
]
|
||||
write_summary_csv(rows, out)
|
||||
text = out.read_text()
|
||||
assert "J813715" in text
|
||||
assert "REBILLED_A" in text
|
||||
assert "EXCLUDED_TIMELY_FILING" in text
|
||||
assert "OLD" in text
|
||||
|
||||
|
||||
def test_disposition_constants():
|
||||
assert REBILLED_A == "REBILLED_A"
|
||||
assert REBILLED_B == "REBILLED_B"
|
||||
assert EXCLUDED_CARC == "EXCLUDED_CARC"
|
||||
assert EXCLUDED_PAYER == "EXCLUDED_PAYER"
|
||||
assert EXCLUDED_NO_EVV == "EXCLUDED_NO_EVV"
|
||||
assert EXCLUDED_TIMELY_FILING == "EXCLUDED_TIMELY_FILING"
|
||||
@@ -0,0 +1,39 @@
|
||||
"""120-day DOS-age gate. CO Medicaid's timely-filing window is 120 days from DOS."""
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
from cyclone.rebill.timely_filing import timely_filing_decision, FILING_WINDOW_DAYS
|
||||
|
||||
|
||||
def test_visit_within_window_is_rebillable():
|
||||
decision = timely_filing_decision(
|
||||
dos=date(2026, 6, 1), as_of=date(2026, 7, 7), override=False
|
||||
)
|
||||
assert decision.rebillable is True
|
||||
|
||||
|
||||
def test_visit_past_window_excluded_by_default():
|
||||
decision = timely_filing_decision(
|
||||
dos=date(2026, 1, 1), as_of=date(2026, 7, 7), override=False
|
||||
)
|
||||
assert decision.rebillable is False
|
||||
assert decision.days_old == 187
|
||||
|
||||
|
||||
def test_visit_at_exact_120_days_inclusive():
|
||||
decision = timely_filing_decision(
|
||||
dos=date(2026, 3, 9), as_of=date(2026, 7, 7), override=False
|
||||
)
|
||||
# 2026-03-09 → 2026-07-07 = 120 days
|
||||
assert decision.days_old == 120
|
||||
assert decision.rebillable is True
|
||||
|
||||
|
||||
def test_override_flag_relaxes_the_gate():
|
||||
decision = timely_filing_decision(
|
||||
dos=date(2026, 1, 1), as_of=date(2026, 7, 7), override=True
|
||||
)
|
||||
assert decision.rebillable is True
|
||||
|
||||
|
||||
def test_window_constant_is_120():
|
||||
assert FILING_WINDOW_DAYS == 120
|
||||
@@ -0,0 +1,136 @@
|
||||
"""SP41-visits: unit tests for ``cyclone.rebill.visits_store``.
|
||||
|
||||
The visits_store persists the AxisCare visits export to the ``visits``
|
||||
table. These tests pin the loader and the read-back query helper.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import tempfile
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from cyclone import db as db_mod
|
||||
from cyclone.rebill.visits_store import (
|
||||
_normalize_modifiers,
|
||||
_parse_billed,
|
||||
_parse_dos,
|
||||
load_visits_csv,
|
||||
query_visits,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def visits_csv(tmp_path: Path) -> Path:
|
||||
"""Write a small visits CSV in AxisCare's shape."""
|
||||
csv_path = tmp_path / "visits.csv"
|
||||
with csv_path.open("w", newline="", encoding="utf-8") as f:
|
||||
writer = csv.writer(f)
|
||||
writer.writerow([
|
||||
"Client", "Visit Date", "Payer", "Authorized", "Member ID",
|
||||
"Auth Start Date", "Auth End Date", "Authorization #", "ICD-10",
|
||||
"Service", "Procedure Code", "Modifiers", "Client Classes",
|
||||
"Billable Hours", "Billable Amount", "Invoice #", "Claimed",
|
||||
])
|
||||
writer.writerow([
|
||||
"Wilson, Michelle", "01/01/2026", "CO Medicaid", "Yes", "R649327",
|
||||
"01/01/2026", "12/31/2026", "3", "R69",
|
||||
"Homemaker S5150", "S5150", "U8", "DD Waiver", "1", "$212.77",
|
||||
"INV-2026-01-01-R649327", "Yes",
|
||||
])
|
||||
writer.writerow([
|
||||
"O'Keefe , Laura", "05/09/2026", "COHCPF", "yes", "Y477643",
|
||||
"03/10/2026", "08/31/2026", "9", "R69",
|
||||
"IHSS PCP T1019", "T1019", "KX, SC, U2", "IHSS DR", "8.5333",
|
||||
"$237.57", "INV-2026-05-09-Y477643", "Yes",
|
||||
])
|
||||
writer.writerow([
|
||||
"Stale, Old", "12/31/2025", "CO Medicaid", "Yes", "Z999999",
|
||||
"01/01/2025", "12/31/2025", "0", "R69",
|
||||
"Old S5150", "S5150", "U8", "DD Waiver", "1", "$100.00",
|
||||
"INV-2025-12-31-Z999999", "Yes", # OUT-OF-WINDOW
|
||||
])
|
||||
return csv_path
|
||||
|
||||
|
||||
def test_normalize_modifiers():
|
||||
"""Colon-joined or comma-joined input → colon-joined output."""
|
||||
assert _normalize_modifiers("KX:SC:U2") == "KX:SC:U2"
|
||||
assert _normalize_modifiers("KX, SC, U2") == "KX:SC:U2"
|
||||
assert _normalize_modifiers("") == ""
|
||||
assert _normalize_modifiers("U8") == "U8"
|
||||
|
||||
|
||||
def test_parse_billed():
|
||||
"""$ prefix and thousands separators are stripped."""
|
||||
assert _parse_billed("$212.77") == Decimal("212.77")
|
||||
assert _parse_billed("1,234.56") == Decimal("1234.56")
|
||||
assert _parse_billed("") == Decimal("0")
|
||||
assert _parse_billed("not a number") == Decimal("0")
|
||||
|
||||
|
||||
def test_parse_dos():
|
||||
"""MM/DD/YYYY primary, YYYY-MM-DD fallback."""
|
||||
assert _parse_dos("01/01/2026") == date(2026, 1, 1)
|
||||
assert _parse_dos("2026-01-01") == date(2026, 1, 1)
|
||||
assert _parse_dos("") is None
|
||||
assert _parse_dos("garbage") is None
|
||||
|
||||
|
||||
def test_load_visits_csv_inserts_in_window_only(visits_csv: Path):
|
||||
"""Out-of-window rows are dropped, in-window rows are inserted."""
|
||||
n = load_visits_csv(
|
||||
visits_csv,
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
)
|
||||
# 2 of 3 rows are in-window (the 12/31/2025 row is dropped)
|
||||
assert n == 2
|
||||
|
||||
rows = query_visits(dos_start=date(2026, 1, 1), dos_end=date(2026, 6, 27))
|
||||
assert len(rows) == 2
|
||||
member_ids = {r.member_id for r in rows}
|
||||
assert member_ids == {"R649327", "Y477643"}
|
||||
|
||||
|
||||
def test_load_visits_csv_dedupes_on_natural_key(visits_csv: Path):
|
||||
"""Re-running the loader must not insert duplicate (DOS, member, procedure, mod)."""
|
||||
load_visits_csv(visits_csv, window_start=date(2026, 1, 1), window_end=date(2026, 6, 27))
|
||||
n2 = load_visits_csv(visits_csv, window_start=date(2026, 1, 1), window_end=date(2026, 6, 27))
|
||||
assert n2 == 0 # all rows were already there
|
||||
assert len(query_visits()) == 2
|
||||
|
||||
|
||||
def test_query_visits_filters_by_member_and_procedure(visits_csv: Path):
|
||||
load_visits_csv(visits_csv, window_start=date(2026, 1, 1), window_end=date(2026, 6, 27))
|
||||
rows = query_visits(member_id="R649327")
|
||||
assert len(rows) == 1
|
||||
assert rows[0].procedure_code == "S5150"
|
||||
assert rows[0].modifiers == "U8"
|
||||
assert rows[0].billed_amount == Decimal("212.77")
|
||||
assert rows[0].icd10 == "R69"
|
||||
assert rows[0].prior_auth == "3"
|
||||
assert rows[0].payer == "CO Medicaid"
|
||||
|
||||
|
||||
def test_query_visits_modifiers_normalized_to_colon_joined(visits_csv: Path):
|
||||
"""Batch-2 style 'KX, SC, U2' is stored as 'KX:SC:U2'."""
|
||||
load_visits_csv(visits_csv, window_start=date(2026, 1, 1), window_end=date(2026, 6, 27))
|
||||
rows = query_visits(member_id="Y477643")
|
||||
assert len(rows) == 1
|
||||
assert rows[0].modifiers == "KX:SC:U2"
|
||||
assert rows[0].procedure_code == "T1019"
|
||||
|
||||
|
||||
def test_visit_table_exists_post_migration():
|
||||
"""Migration 23 created the visits table — schema check."""
|
||||
db_mod.init_db()
|
||||
from sqlalchemy import inspect, text
|
||||
with db_mod.SessionLocal()() as s:
|
||||
v = s.execute(text("PRAGMA user_version")).scalar()
|
||||
inspector = inspect(db_mod.engine())
|
||||
assert "visits" in inspector.get_table_names()
|
||||
assert v >= 23
|
||||
@@ -0,0 +1,338 @@
|
||||
"""SP24 — pure-unit tests for cyclone.reissue.core.
|
||||
|
||||
The CLI wrapper (cyclone reissue-claims) is thin; the behaviour lives
|
||||
in :mod:`cyclone.reissue.core`. These tests exercise the four public
|
||||
functions directly without a CliRunner, mirroring the conventions in
|
||||
``cyclone-tests`` (autouse conftest for DB isolation, no network,
|
||||
``tmp_path`` for filesystem work).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from cyclone.parsers.payer import PayerConfig
|
||||
from cyclone.reissue.core import (
|
||||
emit_outputs,
|
||||
ig_correctness_check,
|
||||
parse_inputs,
|
||||
write_summary_sidecar,
|
||||
zip_outputs,
|
||||
)
|
||||
|
||||
# Fixture reference — flat, module-level Path constant. Matches the
|
||||
# convention used in test_api_999.py / test_serialize_837.py.
|
||||
MINIMAL_837P = Path(__file__).parent / "fixtures" / "minimal_837p.txt"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ig_correctness_check
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_ig_correctness_check_returns_true_when_default_is_false(monkeypatch):
|
||||
"""Guard returns True (silent pass) when PATIENT_LOOP_DEFAULT_INCLUDED is False."""
|
||||
# Default state of the module — should pass.
|
||||
assert ig_correctness_check() is True
|
||||
|
||||
|
||||
def test_ig_correctness_check_returns_false_when_default_flipped(monkeypatch):
|
||||
"""Guard returns False (refuse to run) when the constant is monkeypatched to True."""
|
||||
import cyclone.parsers.serialize_837 as ser_mod
|
||||
|
||||
monkeypatch.setattr(ser_mod, "PATIENT_LOOP_DEFAULT_INCLUDED", True)
|
||||
assert ig_correctness_check() is False
|
||||
|
||||
|
||||
def test_ig_correctness_check_logs_warning_when_default_flipped(monkeypatch, caplog):
|
||||
"""The guard emits a WARNING explaining the IG violation when tripped."""
|
||||
import logging
|
||||
import cyclone.parsers.serialize_837 as ser_mod
|
||||
|
||||
monkeypatch.setattr(ser_mod, "PATIENT_LOOP_DEFAULT_INCLUDED", True)
|
||||
with caplog.at_level(logging.WARNING, logger="cyclone.reissue.core"):
|
||||
assert ig_correctness_check() is False
|
||||
assert any(
|
||||
"PATIENT_LOOP_DEFAULT_INCLUDED" in rec.message for rec in caplog.records
|
||||
), f"expected WARNING about the constant; got {caplog.records!r}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# parse_inputs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_parse_inputs_skips_apple_double_metadata(tmp_path: Path):
|
||||
"""`._foo.x12` AppleDouble files are skipped at the glob stage."""
|
||||
in_dir = tmp_path / "in"
|
||||
in_dir.mkdir()
|
||||
# One real file + one AppleDouble shadow.
|
||||
(in_dir / "real.x12").write_text(MINIMAL_837P.read_text())
|
||||
(in_dir / "._real.x12").write_text(MINIMAL_837P.read_text(), encoding="utf-8")
|
||||
|
||||
payer = PayerConfig.co_medicaid()
|
||||
claims, errors = parse_inputs(in_dir, payer)
|
||||
|
||||
# Exactly one claim (from real.x12), zero errors.
|
||||
assert len(claims) == 1
|
||||
assert errors == []
|
||||
assert claims[0].claim_id == "CLM001"
|
||||
|
||||
|
||||
def test_parse_inputs_tolerates_per_file_failures(tmp_path: Path):
|
||||
"""A malformed file lands in `errors`; the valid one survives in `claims`."""
|
||||
in_dir = tmp_path / "in"
|
||||
in_dir.mkdir()
|
||||
(in_dir / "good.x12").write_text(MINIMAL_837P.read_text())
|
||||
(in_dir / "bad.x12").write_text("NOT A REAL 837P FILE\n")
|
||||
|
||||
payer = PayerConfig.co_medicaid()
|
||||
claims, errors = parse_inputs(in_dir, payer)
|
||||
|
||||
assert len(claims) == 1
|
||||
assert claims[0].claim_id == "CLM001"
|
||||
assert len(errors) == 1
|
||||
bad_path, bad_msg = errors[0]
|
||||
assert bad_path.name == "bad.x12"
|
||||
assert bad_msg # any non-empty failure message
|
||||
|
||||
|
||||
def test_parse_inputs_handles_empty_input_dir(tmp_path: Path):
|
||||
"""An empty input dir produces zero claims + a 'no files found' error.
|
||||
|
||||
The CLI exits 2 in this case (per `cyclone reissue-claims --help`).
|
||||
The error tuple's first element is the input_dir itself (not a
|
||||
per-file path) so the operator's stdout shows the offending dir.
|
||||
"""
|
||||
in_dir = tmp_path / "in"
|
||||
in_dir.mkdir()
|
||||
|
||||
payer = PayerConfig.co_medicaid()
|
||||
claims, errors = parse_inputs(in_dir, payer)
|
||||
|
||||
assert claims == []
|
||||
assert len(errors) == 1
|
||||
bad_path, bad_msg = errors[0]
|
||||
assert bad_path == in_dir
|
||||
assert "no" in bad_msg and "files" in bad_msg.lower()
|
||||
|
||||
|
||||
def test_parse_inputs_returns_claim_models(tmp_path: Path):
|
||||
"""The returned claims are Pydantic ClaimOutput instances."""
|
||||
from cyclone.parsers.models import ClaimOutput
|
||||
|
||||
in_dir = tmp_path / "in"
|
||||
in_dir.mkdir()
|
||||
(in_dir / "real.x12").write_text(MINIMAL_837P.read_text())
|
||||
|
||||
payer = PayerConfig.co_medicaid()
|
||||
claims, _ = parse_inputs(in_dir, payer)
|
||||
|
||||
assert len(claims) == 1
|
||||
assert isinstance(claims[0], ClaimOutput)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# emit_outputs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_claims(tmp_path: Path) -> list:
|
||||
"""Helper: parse the minimal_837p fixture and return the ClaimOutput list."""
|
||||
in_dir = tmp_path / "in"
|
||||
in_dir.mkdir()
|
||||
(in_dir / "real.x12").write_text(MINIMAL_837P.read_text())
|
||||
payer = PayerConfig.co_medicaid()
|
||||
claims, _ = parse_inputs(in_dir, payer)
|
||||
assert len(claims) == 1
|
||||
return claims, payer
|
||||
|
||||
|
||||
def test_emit_outputs_writes_hcpf_spec_filenames(tmp_path: Path):
|
||||
"""Emitted filenames match `tp<sender>-837P-<timestamp>-1of1.x12`."""
|
||||
claims, payer = _make_claims(tmp_path)
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
written = emit_outputs(
|
||||
claims,
|
||||
payer_config=payer,
|
||||
output_dir=out_dir,
|
||||
sender_id="11525703",
|
||||
receiver_id="COMEDASSISTPROG",
|
||||
submitter_name="Dzinesco",
|
||||
submitter_contact_name="Tyler Martinez",
|
||||
submitter_contact_email="tyler@dzinesco.com",
|
||||
receiver_name="COLORADO MEDICAL ASSISTANCE PROGRAM",
|
||||
)
|
||||
|
||||
assert len(written) == 1
|
||||
name = written[0].name
|
||||
assert name.startswith("tp11525703-837P-"), f"unexpected filename: {name}"
|
||||
assert name.endswith("-1of1.x12"), f"unexpected suffix: {name}"
|
||||
# 837P + 1-of-1 with 17-digit millisecond timestamp → tp11525703-837P-YYYYMMDDhhmmssSSS-1of1.x12
|
||||
assert len(name) == len("tp11525703-837P-") + 17 + len("-1of1.x12")
|
||||
|
||||
|
||||
def test_emit_outputs_creates_output_dir(tmp_path: Path):
|
||||
"""emit_outputs creates the output dir if it doesn't exist."""
|
||||
claims, payer = _make_claims(tmp_path)
|
||||
out_dir = tmp_path / "deep" / "nested" / "out"
|
||||
assert not out_dir.exists()
|
||||
|
||||
emit_outputs(
|
||||
claims,
|
||||
payer_config=payer,
|
||||
output_dir=out_dir,
|
||||
sender_id="11525703",
|
||||
receiver_id="COMEDASSISTPROG",
|
||||
submitter_name="Dzinesco",
|
||||
submitter_contact_name="Tyler Martinez",
|
||||
submitter_contact_email="tyler@dzinesco.com",
|
||||
receiver_name="COLORADO MEDICAL ASSISTANCE PROGRAM",
|
||||
)
|
||||
|
||||
assert out_dir.is_dir()
|
||||
assert any(out_dir.iterdir())
|
||||
|
||||
|
||||
def test_emit_outputs_emits_ig_correct_patient_loop_omission(tmp_path: Path):
|
||||
"""Emitted X12 contains no HL*3 segment (the IG-correct shape for SBR02='18')."""
|
||||
claims, payer = _make_claims(tmp_path)
|
||||
out_dir = tmp_path / "out"
|
||||
|
||||
written = emit_outputs(
|
||||
claims,
|
||||
payer_config=payer,
|
||||
output_dir=out_dir,
|
||||
sender_id="11525703",
|
||||
receiver_id="COMEDASSISTPROG",
|
||||
submitter_name="Dzinesco",
|
||||
submitter_contact_name="Tyler Martinez",
|
||||
submitter_contact_email="tyler@dzinesco.com",
|
||||
receiver_name="COLORADO MEDICAL ASSISTANCE PROGRAM",
|
||||
)
|
||||
|
||||
body = written[0].read_text()
|
||||
# HL*3 absent → SBR02='18' claim has no 2000C patient loop.
|
||||
assert "HL*3" not in body, (
|
||||
f"HL*3 present in emitted X12 — 2000C patient loop must be absent for SBR02='18'. "
|
||||
f"Body: {body!r}"
|
||||
)
|
||||
# HL*2 child count = 0 (not 1) — the IG-correct count.
|
||||
assert "HL*2*1*22*0" in body, (
|
||||
f"HL*2 child count != 0 in emitted X12. Body: {body!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_write_summary_sidecar_emits_one_row_per_claim(tmp_path: Path):
|
||||
"""The _serialize_summary.json sidecar has one row per emitted file."""
|
||||
import json
|
||||
|
||||
claims, payer = _make_claims(tmp_path)
|
||||
out_dir = tmp_path / "out"
|
||||
written = emit_outputs(
|
||||
claims,
|
||||
payer_config=payer,
|
||||
output_dir=out_dir,
|
||||
sender_id="11525703",
|
||||
receiver_id="COMEDASSISTPROG",
|
||||
submitter_name="Dzinesco",
|
||||
submitter_contact_name="Tyler Martinez",
|
||||
submitter_contact_email="tyler@dzinesco.com",
|
||||
receiver_name="COLORADO MEDICAL ASSISTANCE PROGRAM",
|
||||
)
|
||||
|
||||
summary_path = write_summary_sidecar(claims, written, out_dir)
|
||||
assert summary_path.exists()
|
||||
rows = json.loads(summary_path.read_text())
|
||||
assert len(rows) == 1
|
||||
assert rows[0]["claim_id"] == "CLM001"
|
||||
assert rows[0]["byte_size"] > 0
|
||||
assert rows[0]["output_file"] == str(written[0])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# zip_outputs
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_zip_outputs_round_trip_integrity(tmp_path: Path):
|
||||
"""Written zip passes testzip(); entries round-trip with the right names."""
|
||||
in_dir = tmp_path / "in"
|
||||
in_dir.mkdir()
|
||||
(in_dir / "real.x12").write_text(MINIMAL_837P.read_text())
|
||||
payer = PayerConfig.co_medicaid()
|
||||
claims, _ = parse_inputs(in_dir, payer)
|
||||
out_dir = tmp_path / "out"
|
||||
written = emit_outputs(
|
||||
claims,
|
||||
payer_config=payer,
|
||||
output_dir=out_dir,
|
||||
sender_id="11525703",
|
||||
receiver_id="COMEDASSISTPROG",
|
||||
submitter_name="Dzinesco",
|
||||
submitter_contact_name="Tyler Martinez",
|
||||
submitter_contact_email="tyler@dzinesco.com",
|
||||
receiver_name="COLORADO MEDICAL ASSISTANCE PROGRAM",
|
||||
)
|
||||
zip_path = tmp_path / "out.zip"
|
||||
|
||||
zip_outputs(written, zip_path)
|
||||
|
||||
assert zip_path.exists()
|
||||
with zipfile.ZipFile(zip_path) as zf:
|
||||
assert zf.testzip() is None # None = no corrupt entries
|
||||
names = zf.namelist()
|
||||
assert len(names) == 1
|
||||
assert names[0] == written[0].name
|
||||
# Round-trip the body matches.
|
||||
assert zf.read(names[0]) == written[0].read_bytes()
|
||||
|
||||
|
||||
def test_zip_outputs_creates_parent_dirs(tmp_path: Path):
|
||||
"""zip_outputs creates the parent dir for the zip path."""
|
||||
in_dir = tmp_path / "in"
|
||||
in_dir.mkdir()
|
||||
(in_dir / "real.x12").write_text(MINIMAL_837P.read_text())
|
||||
payer = PayerConfig.co_medicaid()
|
||||
claims, _ = parse_inputs(in_dir, payer)
|
||||
out_dir = tmp_path / "out"
|
||||
written = emit_outputs(
|
||||
claims,
|
||||
payer_config=payer,
|
||||
output_dir=out_dir,
|
||||
sender_id="11525703",
|
||||
receiver_id="COMEDASSISTPROG",
|
||||
submitter_name="Dzinesco",
|
||||
submitter_contact_name="Tyler Martinez",
|
||||
submitter_contact_email="tyler@dzinesco.com",
|
||||
receiver_name="COLORADO MEDICAL ASSISTANCE PROGRAM",
|
||||
)
|
||||
|
||||
zip_path = tmp_path / "deep" / "nested" / "out.zip"
|
||||
zip_outputs(written, zip_path)
|
||||
assert zip_path.exists()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Parametrized IG guard against a synthetic "broken serializer"
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"broken_default, expected_pass",
|
||||
[
|
||||
(False, True), # IG-correct shape — guard silent
|
||||
(True, False), # IG-incorrect shape — guard fires
|
||||
],
|
||||
)
|
||||
def test_ig_correctness_check_parametrized(
|
||||
monkeypatch, broken_default, expected_pass
|
||||
):
|
||||
"""The guard is a binary pass/fail that follows PATIENT_LOOP_DEFAULT_INCLUDED."""
|
||||
import cyclone.parsers.serialize_837 as ser_mod
|
||||
|
||||
monkeypatch.setattr(ser_mod, "PATIENT_LOOP_DEFAULT_INCLUDED", broken_default)
|
||||
assert ig_correctness_check() is expected_pass
|
||||
@@ -0,0 +1,533 @@
|
||||
"""SP39: tests for the `cyclone resubmissions status` CLI."""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
from cyclone import db as cycl_db
|
||||
from cyclone.cli import resubmissions_status_cmd
|
||||
from cyclone.db import Resubmission
|
||||
|
||||
|
||||
def _seed_one(claim_id: str = "claim-1", batch_id: str = "batch-1",
|
||||
icn: str = "000000001") -> None:
|
||||
cycl_db.init_db()
|
||||
with cycl_db.SessionLocal()() as s:
|
||||
s.add(Resubmission(
|
||||
claim_id=claim_id,
|
||||
batch_id=batch_id,
|
||||
resubmitted_at=datetime.now(timezone.utc),
|
||||
source_corrected_path=f"/tmp/{claim_id}.x12",
|
||||
interchange_control_number=icn,
|
||||
group_control_number="1",
|
||||
))
|
||||
s.commit()
|
||||
|
||||
|
||||
def _cleanup(batch_id: str) -> None:
|
||||
with cycl_db.SessionLocal()() as s:
|
||||
s.query(Resubmission).filter(Resubmission.batch_id == batch_id).delete()
|
||||
s.commit()
|
||||
|
||||
|
||||
def test_no_resubmissions_prints_message_and_exits_0():
|
||||
cycl_db.init_db()
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
resubmissions_status_cmd,
|
||||
["--batch-id", "no-such-batch-zzz"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "no resubmissions recorded" in result.output
|
||||
|
||||
|
||||
def test_one_resubmission_prints_status_row():
|
||||
try:
|
||||
_seed_one(claim_id="claim-A", batch_id="batch-A")
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
resubmissions_status_cmd,
|
||||
["--batch-id", "batch-A"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "claim-A" in result.output
|
||||
assert "pending_999" in result.output # no inbound ack yet
|
||||
assert "batch-A" in result.output
|
||||
finally:
|
||||
_cleanup("batch-A")
|
||||
|
||||
|
||||
def test_limit_truncates_output():
|
||||
try:
|
||||
for i in range(5):
|
||||
_seed_one(
|
||||
claim_id=f"claim-L{i}", batch_id="batch-L",
|
||||
icn=f"00000000{i}",
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
resubmissions_status_cmd,
|
||||
["--batch-id", "batch-L", "--limit", "2"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "more" in result.output
|
||||
# Only 2 rows printed in the table (plus 1 "(N more)" line).
|
||||
body_lines = [
|
||||
ln for ln in result.output.splitlines()
|
||||
if ln.startswith("claim-L")
|
||||
]
|
||||
assert len(body_lines) == 2
|
||||
finally:
|
||||
_cleanup("batch-L")
|
||||
|
||||
|
||||
def test_status_reflects_paid_when_remittance_links():
|
||||
"""When a remittance.claim_id links to the resubmitted claim,
|
||||
the payment column shows 'paid'."""
|
||||
try:
|
||||
from cyclone.db import Remittance
|
||||
from decimal import Decimal
|
||||
_seed_one(claim_id="claim-P", batch_id="batch-P")
|
||||
with cycl_db.SessionLocal()() as s:
|
||||
s.add(Remittance(
|
||||
id="remit-P-1",
|
||||
batch_id="batch-P",
|
||||
payer_claim_control_number="claim-P",
|
||||
claim_id="claim-P",
|
||||
status_code="1",
|
||||
status_label="Processed as Primary",
|
||||
total_charge=Decimal("100.00"),
|
||||
total_paid=Decimal("80.00"),
|
||||
adjustment_amount=Decimal("0"),
|
||||
claim_level_adjustment_amount=Decimal("0"),
|
||||
received_at=datetime.now(timezone.utc),
|
||||
))
|
||||
s.commit()
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
resubmissions_status_cmd,
|
||||
["--batch-id", "batch-P"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "claim-P" in result.output
|
||||
assert "paid" in result.output
|
||||
finally:
|
||||
from cyclone.db import Remittance as _R
|
||||
with cycl_db.SessionLocal()() as s:
|
||||
s.query(_R).filter(_R.id == "remit-P-1").delete()
|
||||
s.commit()
|
||||
_cleanup("batch-P")
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP39: `cyclone resubmit-rejected-claims` instruments Resubmission rows.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed_clearhouse_stub_false(tmp_path) -> None:
|
||||
"""Seed the clearhouse singleton and flip ``sftp_block.stub=False``
|
||||
with a non-routable host so the CLI proceeds past the guard and
|
||||
the paramiko stub below intercepts the connection."""
|
||||
import json
|
||||
from cyclone import db as db_mod
|
||||
from cyclone.db import ClearhouseORM
|
||||
from cyclone.store import store as cycl_store
|
||||
cycl_store.ensure_clearhouse_seeded()
|
||||
with db_mod.SessionLocal()() as s:
|
||||
ch = s.get(ClearhouseORM, 1)
|
||||
assert ch is not None, "ensure_clearhouse_seeded should have created it"
|
||||
block = json.loads(json.dumps(ch.sftp_block_json))
|
||||
block["stub"] = False
|
||||
block["host"] = "stub.invalid"
|
||||
ch.sftp_block_json = block
|
||||
s.commit()
|
||||
|
||||
|
||||
def test_resubmit_rejected_claims_inserts_resubmission_row(tmp_path, monkeypatch):
|
||||
"""SP39: after a successful upload, one Resubmission row per claim
|
||||
is inserted with the parent dir's batch_id and the parsed envelope
|
||||
control numbers."""
|
||||
import sys
|
||||
import types
|
||||
from cyclone.cli import resubmit_rejected_claims
|
||||
from cyclone.parsers.parse_837 import parse as parse_837_text
|
||||
from cyclone.parsers.payer import PayerConfig
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
_seed_clearhouse_stub_false(tmp_path)
|
||||
|
||||
# Build a parseable 837P file (CO_TXIX payer, single claim).
|
||||
cfg = PayerConfig.co_medicaid()
|
||||
sample = tmp_path / "batch-deadbeef-1-claims"
|
||||
sample.mkdir()
|
||||
file_path = sample / "tp1-837P-test-1of1.x12"
|
||||
src_text = open("tests/fixtures/co_medicaid_837p.txt").read()
|
||||
file_path.write_text(src_text)
|
||||
|
||||
parsed = parse_837_text(src_text, cfg)
|
||||
assert parsed.claims, "fixture must produce at least one claim"
|
||||
first_claim_id = parsed.claims[0].claim_id
|
||||
|
||||
# Replace paramiko with a fake whose stat() raises (file not on
|
||||
# remote yet, so the CLI takes the upload branch).
|
||||
fake_paramiko = types.ModuleType("paramiko")
|
||||
|
||||
class _FakeSFTP:
|
||||
def stat(self, path):
|
||||
raise IOError("not found")
|
||||
def put(self, local, remote):
|
||||
return None
|
||||
|
||||
class _FakeSSH:
|
||||
def open_sftp(self):
|
||||
return _FakeSFTP()
|
||||
def set_missing_host_key_policy(self, policy):
|
||||
return None
|
||||
def connect(self, *a, **kw):
|
||||
return None
|
||||
def close(self):
|
||||
return None
|
||||
|
||||
fake_paramiko.SSHClient = lambda: _FakeSSH()
|
||||
fake_paramiko.AutoAddPolicy = lambda: None
|
||||
monkeypatch.setitem(sys.modules, "paramiko", fake_paramiko)
|
||||
|
||||
captured: list[dict] = []
|
||||
monkeypatch.setattr(
|
||||
cycl_store, "record_resubmission",
|
||||
lambda **kw: captured.append(kw) or True,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(resubmit_rejected_claims, [
|
||||
"--ingest-dir", str(tmp_path),
|
||||
"--limit", "1",
|
||||
"--no-validate",
|
||||
"--skip-edifabric-validate",
|
||||
])
|
||||
assert result.exit_code == 0, (
|
||||
f"CLI exited {result.exit_code}, output={result.output!r}, "
|
||||
f"exception={result.exception!r}"
|
||||
)
|
||||
assert len(captured) >= 1, (
|
||||
f"expected record_resubmission to be called, got {captured!r}"
|
||||
)
|
||||
call = captured[0]
|
||||
assert call["claim_id"] == first_claim_id, call
|
||||
assert call["batch_id"] == "deadbeef", call
|
||||
assert "deadbeef-1-claims" in call["source_corrected_path"], call
|
||||
assert call["interchange_control_number"], call
|
||||
assert call["group_control_number"], call
|
||||
|
||||
|
||||
def test_resubmit_idempotency_does_not_create_duplicate_rows(tmp_path, monkeypatch):
|
||||
"""A second run against the same file (now on the SFTP) does NOT
|
||||
insert a second Resubmission row — only the original upload branch
|
||||
writes, matching the audit-log behavior."""
|
||||
from cyclone.cli import resubmit_rejected_claims
|
||||
from cyclone.store import store as cycl_store
|
||||
import sys
|
||||
import types
|
||||
|
||||
_seed_clearhouse_stub_false(tmp_path)
|
||||
|
||||
# Set up: one corrected file.
|
||||
sample = tmp_path / "batch-cafebabe-1-claims"
|
||||
sample.mkdir()
|
||||
file_path = sample / "tp1-837P-test-1of1.x12"
|
||||
src_text = open("tests/fixtures/co_medicaid_837p.txt").read()
|
||||
file_path.write_text(src_text)
|
||||
on_remote_size = len(src_text.encode())
|
||||
|
||||
# Build a paramiko fake where stat() returns matching size so the
|
||||
# CLI takes the skip branch. The CLI only reads ``st_size`` off the
|
||||
# return value, so a simple duck-typed object suffices.
|
||||
fake_paramiko = types.ModuleType("paramiko")
|
||||
|
||||
class _FakeAttr:
|
||||
def __init__(self, size: int) -> None:
|
||||
self.st_size = size
|
||||
|
||||
class _FakeSFTPAlready:
|
||||
def stat(self, path):
|
||||
return _FakeAttr(on_remote_size)
|
||||
|
||||
def put(self, local, remote):
|
||||
raise AssertionError("should not put when file already on remote")
|
||||
|
||||
class _FakeSSHAlready:
|
||||
def open_sftp(self):
|
||||
return _FakeSFTPAlready()
|
||||
def set_missing_host_key_policy(self, policy):
|
||||
return None
|
||||
def connect(self, *a, **kw):
|
||||
return None
|
||||
def close(self):
|
||||
return None
|
||||
|
||||
fake_paramiko.SSHClient = lambda: _FakeSSHAlready()
|
||||
fake_paramiko.AutoAddPolicy = lambda: None
|
||||
monkeypatch.setitem(sys.modules, "paramiko", fake_paramiko)
|
||||
|
||||
captured: list[dict] = []
|
||||
monkeypatch.setattr(
|
||||
cycl_store, "record_resubmission",
|
||||
lambda **kw: captured.append(kw) or True,
|
||||
)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(resubmit_rejected_claims, [
|
||||
"--ingest-dir", str(tmp_path),
|
||||
"--limit", "1",
|
||||
"--no-validate",
|
||||
"--skip-edifabric-validate",
|
||||
])
|
||||
# On the skip path, record_resubmission should NOT be called.
|
||||
assert captured == [], f"expected no record_resubmission on skip, got {captured!r}"
|
||||
assert "skipped" in result.output, result.output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP40: `cyclone resubmit-rejected-claims` Edifabric pre-upload gate.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
_TEST_KEY = "test-edifabric-key-0123456789abcdef"
|
||||
|
||||
|
||||
def _install_edifabric_mock(handler):
|
||||
"""Install a mocked Edifabric transport + get_secret stub for the
|
||||
duration of a single test. Returned teardown callable can be wired
|
||||
to monkeypatch via the autouse fixture pattern below."""
|
||||
import sys
|
||||
import httpx
|
||||
from cyclone import edifabric
|
||||
import cyclone.secrets as _secrets_mod
|
||||
|
||||
real_get_secret = _secrets_mod.get_secret
|
||||
|
||||
def _fake_get_secret(name):
|
||||
if name == "edifabric.api_key":
|
||||
return _TEST_KEY
|
||||
return real_get_secret(name)
|
||||
|
||||
_secrets_mod.get_secret = _fake_get_secret
|
||||
transport = httpx.MockTransport(handler)
|
||||
edifabric.set_transport_factory(
|
||||
lambda: httpx.Client(transport=transport, timeout=10.0)
|
||||
)
|
||||
|
||||
def _teardown():
|
||||
_secrets_mod.get_secret = real_get_secret
|
||||
edifabric._reset_transport_factory()
|
||||
|
||||
return _teardown
|
||||
|
||||
|
||||
def _two_files_setup(tmp_path):
|
||||
"""Stage two .x12 files in distinct batch dirs so the test can
|
||||
observe one file's gate failure not blocking the other's upload."""
|
||||
src_text = open("tests/fixtures/co_medicaid_837p.txt").read()
|
||||
a = tmp_path / "batch-aaaaaaaa-1-claims"
|
||||
a.mkdir()
|
||||
(a / "tp1-aaaaaaaa-1of1.x12").write_text(src_text)
|
||||
b = tmp_path / "batch-bbbbbbbb-1-claims"
|
||||
b.mkdir()
|
||||
(b / "tp1-bbbbbbbb-1of1.x12").write_text(src_text)
|
||||
return src_text
|
||||
|
||||
|
||||
def test_resubmit_edifabric_gate_aborts_bad_file_continues_batch(
|
||||
tmp_path, monkeypatch,
|
||||
):
|
||||
"""SP40: when Edifabric returns Status='error' for one file, that
|
||||
file is recorded as ``validate_failed`` (not uploaded, no
|
||||
Resubmission row), but the batch continues with the next file."""
|
||||
import sys
|
||||
import types
|
||||
from cyclone.cli import resubmit_rejected_claims
|
||||
|
||||
_seed_clearhouse_stub_false(tmp_path)
|
||||
_two_files_setup(tmp_path)
|
||||
|
||||
seen: list[str] = []
|
||||
x12 = {
|
||||
"SegmentDelimiter": "~",
|
||||
"DataElementDelimiter": "*",
|
||||
"ISA": {"InterchangeControlNumber_13": "000000001"},
|
||||
"Groups": [],
|
||||
"IEATrailers": [],
|
||||
}
|
||||
|
||||
def edifabric_handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/read"):
|
||||
return httpx.Response(200, json=[x12])
|
||||
if request.url.path.endswith("/validate"):
|
||||
# Fail the FIRST validate call; succeed on the second.
|
||||
seen.append("validate")
|
||||
is_bad = len(seen) == 1
|
||||
return httpx.Response(200, json={
|
||||
"Status": "error" if is_bad else "success",
|
||||
"Details": (
|
||||
[
|
||||
{"SegmentId": "PER",
|
||||
"Message": "PER-04 is required",
|
||||
"Status": "error"},
|
||||
]
|
||||
if is_bad else []
|
||||
),
|
||||
"LastIndex": 5 if is_bad else 46,
|
||||
})
|
||||
raise AssertionError(f"unexpected path: {request.url.path}")
|
||||
|
||||
import httpx
|
||||
teardown = _install_edifabric_mock(edifabric_handler)
|
||||
try:
|
||||
# Fake paramiko so SFTP doesn't actually fire.
|
||||
fake_paramiko = types.ModuleType("paramiko")
|
||||
|
||||
class _FakeAttr:
|
||||
def __init__(self, size: int) -> None:
|
||||
self.st_size = size
|
||||
|
||||
class _FakeSFTP:
|
||||
def __init__(self):
|
||||
self.put_calls: list[str] = []
|
||||
|
||||
def stat(self, path):
|
||||
raise IOError("not found")
|
||||
|
||||
def put(self, local, remote):
|
||||
self.put_calls.append(remote)
|
||||
|
||||
fake_sftp = _FakeSFTP()
|
||||
|
||||
class _FakeSSH:
|
||||
def open_sftp(self):
|
||||
return fake_sftp
|
||||
|
||||
def set_missing_host_key_policy(self, policy):
|
||||
return None
|
||||
|
||||
def connect(self, *a, **kw):
|
||||
return None
|
||||
|
||||
def close(self):
|
||||
return None
|
||||
|
||||
fake_paramiko.SSHClient = lambda: _FakeSSH()
|
||||
fake_paramiko.AutoAddPolicy = lambda: None
|
||||
monkeypatch.setitem(sys.modules, "paramiko", fake_paramiko)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(resubmit_rejected_claims, [
|
||||
"--ingest-dir", str(tmp_path),
|
||||
"--no-validate",
|
||||
# default: gate ENABLED
|
||||
])
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
# First file: Edifabric gate aborted it (no SFTP put for it).
|
||||
# Second file: passed the gate, got uploaded via the SFTP put.
|
||||
assert len(fake_sftp.put_calls) == 1, fake_sftp.put_calls
|
||||
assert "bbbbbbbb" in fake_sftp.put_calls[0], fake_sftp.put_calls[0]
|
||||
|
||||
# Output surfaces the failure list.
|
||||
assert "EDIFABRIC VALIDATION FAIL" in result.output, result.output
|
||||
assert "1 file(s) blocked by Edifabric gate" in result.output, result.output
|
||||
assert "aaaaaaa" in result.output, result.output
|
||||
assert "PER-04 is required" in result.output, result.output
|
||||
|
||||
# DONE summary shows both counters.
|
||||
assert "validate_failed=1" in result.output, result.output
|
||||
assert "uploaded=1" in result.output, result.output
|
||||
finally:
|
||||
teardown()
|
||||
|
||||
|
||||
def test_resubmit_edifabric_skip_flag_bypasses_gate(tmp_path, monkeypatch):
|
||||
"""SP40: ``--skip-edifabric-validate`` bypasses the gate entirely
|
||||
so the same Edifabric-failing file uploads via SFTP. Used for
|
||||
dev / hot-fix scenarios."""
|
||||
import sys
|
||||
import types
|
||||
from cyclone.cli import resubmit_rejected_claims
|
||||
|
||||
_seed_clearhouse_stub_false(tmp_path)
|
||||
_two_files_setup(tmp_path)
|
||||
|
||||
import httpx
|
||||
x12 = {
|
||||
"SegmentDelimiter": "~",
|
||||
"DataElementDelimiter": "*",
|
||||
"ISA": {"InterchangeControlNumber_13": "000000001"},
|
||||
"Groups": [],
|
||||
"IEATrailers": [],
|
||||
}
|
||||
|
||||
validate_calls: list[str] = []
|
||||
|
||||
def edifabric_handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/read"):
|
||||
return httpx.Response(200, json=[x12])
|
||||
if request.url.path.endswith("/validate"):
|
||||
validate_calls.append("hit")
|
||||
return httpx.Response(200, json={
|
||||
"Status": "error",
|
||||
"Details": [{"SegmentId": "PER",
|
||||
"Message": "PER-04 is required",
|
||||
"Status": "error"}],
|
||||
"LastIndex": 5,
|
||||
})
|
||||
raise AssertionError(f"unexpected path: {request.url.path}")
|
||||
|
||||
teardown = _install_edifabric_mock(edifabric_handler)
|
||||
try:
|
||||
fake_paramiko = types.ModuleType("paramiko")
|
||||
|
||||
class _FakeSFTP:
|
||||
def __init__(self):
|
||||
self.put_calls: list[str] = []
|
||||
|
||||
def stat(self, path):
|
||||
raise IOError("not found")
|
||||
|
||||
def put(self, local, remote):
|
||||
self.put_calls.append(remote)
|
||||
|
||||
fake_sftp = _FakeSFTP()
|
||||
|
||||
class _FakeSSH:
|
||||
def open_sftp(self):
|
||||
return fake_sftp
|
||||
|
||||
def set_missing_host_key_policy(self, policy):
|
||||
return None
|
||||
|
||||
def connect(self, *a, **kw):
|
||||
return None
|
||||
|
||||
def close(self):
|
||||
return None
|
||||
|
||||
fake_paramiko.SSHClient = lambda: _FakeSSH()
|
||||
fake_paramiko.AutoAddPolicy = lambda: None
|
||||
monkeypatch.setitem(sys.modules, "paramiko", fake_paramiko)
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(resubmit_rejected_claims, [
|
||||
"--ingest-dir", str(tmp_path),
|
||||
"--no-validate",
|
||||
"--skip-edifabric-validate",
|
||||
])
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
# Both files uploaded; the gate never fired.
|
||||
assert len(fake_sftp.put_calls) == 2, fake_sftp.put_calls
|
||||
assert validate_calls == [], (
|
||||
"validate must not be called when --skip-edifabric-validate is set"
|
||||
)
|
||||
assert "validate_failed=0" in result.output, result.output
|
||||
assert "EDIFABRIC VALIDATION FAIL" not in result.output, result.output
|
||||
finally:
|
||||
teardown()
|
||||
@@ -0,0 +1,92 @@
|
||||
"""SP39: Resubmission model + idempotency.
|
||||
|
||||
Schema:
|
||||
- 7 columns: id, claim_id, batch_id, resubmitted_at, source_corrected_path,
|
||||
interchange_control_number, group_control_number
|
||||
- Unique index on (claim_id, interchange_control_number) -> re-insert is rejected.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from cyclone import db as cycl_db
|
||||
from cyclone.db import Resubmission
|
||||
|
||||
|
||||
def test_resubmission_model_columns():
|
||||
cols = {c.name for c in Resubmission.__table__.columns}
|
||||
assert cols == {
|
||||
"id", "claim_id", "batch_id", "resubmitted_at",
|
||||
"source_corrected_path",
|
||||
"interchange_control_number", "group_control_number",
|
||||
}
|
||||
|
||||
|
||||
def test_unique_index_on_claim_id_and_icn():
|
||||
idx_names = {i.name for i in Resubmission.__table__.indexes}
|
||||
assert "ux_resubmissions_claim_icn" in idx_names
|
||||
ux = next(
|
||||
i for i in Resubmission.__table__.indexes if i.name == "ux_resubmissions_claim_icn"
|
||||
)
|
||||
assert ux.unique is True
|
||||
# The unique index covers both columns.
|
||||
cols = {c.name for c in ux.columns}
|
||||
assert cols == {"claim_id", "interchange_control_number"}
|
||||
|
||||
|
||||
def test_duplicate_insert_raises_integrity_error():
|
||||
"""Idempotency contract: a second insert with the same
|
||||
(claim_id, interchange_control_number) is rejected at the DB layer."""
|
||||
cycl_db.init_db()
|
||||
now = datetime.now(timezone.utc)
|
||||
with cycl_db.SessionLocal()() as s:
|
||||
s.add(Resubmission(
|
||||
claim_id="claim-x", batch_id="batch-y",
|
||||
resubmitted_at=now,
|
||||
source_corrected_path="/path/one",
|
||||
interchange_control_number="000000001",
|
||||
group_control_number="1",
|
||||
))
|
||||
s.commit()
|
||||
with cycl_db.SessionLocal()() as s:
|
||||
s.add(Resubmission(
|
||||
claim_id="claim-x", batch_id="batch-y",
|
||||
resubmitted_at=now,
|
||||
source_corrected_path="/path/two",
|
||||
interchange_control_number="000000001", # same ICN
|
||||
group_control_number="1",
|
||||
))
|
||||
with pytest.raises(IntegrityError):
|
||||
s.commit()
|
||||
s.rollback()
|
||||
|
||||
|
||||
def test_different_icn_on_same_claim_is_allowed():
|
||||
"""Distinct ICNs (i.e. distinct pushes) are independent rows."""
|
||||
cycl_db.init_db()
|
||||
now = datetime.now(timezone.utc)
|
||||
with cycl_db.SessionLocal()() as s:
|
||||
s.add(Resubmission(
|
||||
claim_id="claim-y", batch_id="batch-y",
|
||||
resubmitted_at=now,
|
||||
source_corrected_path="/path/one",
|
||||
interchange_control_number="000000001",
|
||||
group_control_number="1",
|
||||
))
|
||||
s.add(Resubmission(
|
||||
claim_id="claim-y", batch_id="batch-y",
|
||||
resubmitted_at=now,
|
||||
source_corrected_path="/path/two",
|
||||
interchange_control_number="000000002", # different ICN
|
||||
group_control_number="2",
|
||||
))
|
||||
s.commit()
|
||||
with cycl_db.SessionLocal()() as s:
|
||||
from sqlalchemy import select
|
||||
rows = s.execute(select(Resubmission).where(
|
||||
Resubmission.claim_id == "claim-y"
|
||||
)).scalars().all()
|
||||
assert len(rows) == 2
|
||||
@@ -0,0 +1,49 @@
|
||||
"""SP40: tests for the edifabric.api_key secret wiring.
|
||||
|
||||
Specifically: the ``_ENV_NAME_FOR`` table maps ``edifabric.api_key``
|
||||
to ``CYCLONE_EDIFABRIC_API_KEY`` so operators can configure via env
|
||||
var (or via file companion per ``secrets.get_secret``'s
|
||||
``<env_name>_FILE`` form) without touching the macOS Keychain.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def test_env_name_for_maps_edifabric_api_key():
|
||||
from cyclone.secrets import _ENV_NAME_FOR
|
||||
assert _ENV_NAME_FOR["edifabric.api_key"] == "CYCLONE_EDIFABRIC_API_KEY"
|
||||
|
||||
|
||||
def test_get_secret_reads_edifabric_key_from_env(monkeypatch):
|
||||
"""With the env var set, ``get_secret('edifabric.api_key')``
|
||||
returns it without touching the Keychain."""
|
||||
monkeypatch.setenv("CYCLONE_EDIFABRIC_API_KEY", "test-env-key-9876")
|
||||
|
||||
from cyclone import secrets
|
||||
assert secrets.get_secret("edifabric.api_key") == "test-env-key-9876"
|
||||
|
||||
|
||||
def test_get_secret_reads_edifabric_key_from_file(monkeypatch, tmp_path):
|
||||
"""With ``<env_name>_FILE`` set, ``get_secret`` reads the key from
|
||||
the file (no Keychain touch). Mirrors the existing SFTP password
|
||||
file-companion pattern (SP25)."""
|
||||
key_file = tmp_path / "edi_key"
|
||||
key_file.write_text("file-companion-key-5555\n")
|
||||
monkeypatch.setenv("CYCLONE_EDIFABRIC_API_KEY_FILE", str(key_file))
|
||||
|
||||
from cyclone import secrets
|
||||
assert secrets.get_secret("edifabric.api_key") == "file-companion-key-5555"
|
||||
|
||||
|
||||
def test_dev_fixture_file_exists_and_contains_eval_key():
|
||||
"""The dev-only fixture exists at the documented path so the
|
||||
eval key is referenceable; the CI mocks intercept before any
|
||||
real HTTP happens."""
|
||||
fixture = Path("tests/fixtures/edifabric_api_key.txt")
|
||||
assert fixture.exists(), (
|
||||
f"dev-only Edifabric API key fixture is missing at {fixture}"
|
||||
)
|
||||
contents = fixture.read_text().strip()
|
||||
# Public eval key from the EdiNation API docs.
|
||||
assert "3ecf6b1c5cf34bd797a5f4c57951a1cf" in contents
|
||||
@@ -593,4 +593,214 @@ def test_serialize_837_for_resubmit_assigns_unique_control_numbers():
|
||||
isa_b = next(line for line in text_b.split("~") if line.startswith("ISA"))
|
||||
assert isa_a != isa_b
|
||||
assert "000000042" in isa_a
|
||||
assert "000000043" in isa_b
|
||||
assert "000000043" in isa_b
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP39: _build_payer_block normalizes legacy payer ids to CO_TXIX
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _first_pr_segment(text: str) -> str:
|
||||
"""Extract the first NM1*PR segment from a serialized 837P document."""
|
||||
for seg in text.split("~"):
|
||||
if seg.startswith("NM1*PR"):
|
||||
return seg
|
||||
raise AssertionError("no NM1*PR segment found in output")
|
||||
|
||||
|
||||
def test_2010bb_normalizes_skco0_to_co_txix(caplog):
|
||||
"""SP39: legacy SKCO0 + COHCPF must be normalized to CO_TXIX."""
|
||||
import logging
|
||||
claim = _load_claim()
|
||||
claim.payer.id = "SKCO0"
|
||||
claim.payer.name = "COHCPF"
|
||||
with caplog.at_level(logging.WARNING, logger="cyclone.parsers.serialize_837"):
|
||||
text = serialize_837_for_resubmit(claim, interchange_index=1)
|
||||
parts = _first_pr_segment(text).split("*")
|
||||
# parts: NM1, PR, 2, NM103, NM104, NM105, NM106, NM107, NM108, NM109
|
||||
assert parts[8] == "PI", f"NM108 must be PI, got {parts[8]!r}"
|
||||
assert parts[9] == "CO_TXIX", f"NM109 must be CO_TXIX, got {parts[9]!r}"
|
||||
assert parts[3] == "CO_TXIX", f"NM103 must be CO_TXIX, got {parts[3]!r}"
|
||||
assert "SKCO0" not in text, "SKCO0 must not appear in output bytes"
|
||||
assert "COHCPF" not in text, "COHCPF must not appear in output bytes"
|
||||
# WARNING log was emitted
|
||||
assert any(
|
||||
"SP39" in rec.message and "SKCO0" in rec.message
|
||||
for rec in caplog.records
|
||||
), "expected SP39 substitution WARNING log for SKCO0->CO_TXIX"
|
||||
|
||||
|
||||
def test_2010bb_normalizes_empty_payer_id_to_co_txix(caplog):
|
||||
"""SP39: empty payer.id (degenerate parse artifact) -> CO_TXIX."""
|
||||
import logging
|
||||
claim = _load_claim()
|
||||
claim.payer.id = ""
|
||||
claim.payer.name = ""
|
||||
with caplog.at_level(logging.WARNING, logger="cyclone.parsers.serialize_837"):
|
||||
text = serialize_837_for_resubmit(claim, interchange_index=1)
|
||||
parts = _first_pr_segment(text).split("*")
|
||||
assert parts[8] == "PI"
|
||||
assert parts[9] == "CO_TXIX"
|
||||
assert parts[3] == "CO_TXIX"
|
||||
|
||||
|
||||
def test_2010bb_normalizes_co_bha_to_co_txix(caplog):
|
||||
"""SP39: CO_BHA also gets normalized to CO_TXIX (operator policy)."""
|
||||
import logging
|
||||
claim = _load_claim()
|
||||
claim.payer.id = "CO_BHA"
|
||||
claim.payer.name = "CO_BHA"
|
||||
with caplog.at_level(logging.WARNING, logger="cyclone.parsers.serialize_837"):
|
||||
text = serialize_837_for_resubmit(claim, interchange_index=1)
|
||||
parts = _first_pr_segment(text).split("*")
|
||||
assert parts[8] == "PI"
|
||||
assert parts[9] == "CO_TXIX"
|
||||
assert parts[3] == "CO_TXIX"
|
||||
assert "CO_BHA" not in text
|
||||
|
||||
|
||||
def test_2010bb_preserves_foreign_payer_id_verbatim():
|
||||
"""SP39: helper must NOT corrupt non-CO submits. Foreign payer id
|
||||
is emitted verbatim with no substitution log."""
|
||||
import logging
|
||||
claim = _load_claim()
|
||||
claim.payer.id = "OTHER_PAYER"
|
||||
claim.payer.name = "OTHER PAYER NAME"
|
||||
import io
|
||||
log_stream = io.StringIO()
|
||||
handler = logging.StreamHandler(log_stream)
|
||||
handler.setLevel(logging.WARNING)
|
||||
log = logging.getLogger("cyclone.parsers.serialize_837")
|
||||
log.addHandler(handler)
|
||||
try:
|
||||
text = serialize_837_for_resubmit(claim, interchange_index=1)
|
||||
finally:
|
||||
log.removeHandler(handler)
|
||||
parts = _first_pr_segment(text).split("*")
|
||||
assert parts[8] == "PI"
|
||||
assert parts[9] == "OTHER_PAYER"
|
||||
assert parts[3] == "OTHER PAYER NAME"
|
||||
# No substitution log for a foreign payer id
|
||||
assert "SP39" not in log_stream.getvalue()
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP40: PER-02/03/04 + SBR-09 must always be emitted (Edifabric rejects
|
||||
# the bare-PER / no-SBR09 shapes). The serializer fall-back fills them
|
||||
# with safe placeholders when the caller passes no kwargs.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_per_segment_emits_name_and_phone_qualifier_when_no_contact():
|
||||
"""SP40: without any submitter_contact_* kwargs, the submitter
|
||||
block must still produce a PER segment with at least PER-02 (Name)
|
||||
and a PER-03 (Communication Number Qualifier) + PER-04 (Number)
|
||||
pair. Edifabric rejects PER*IC alone."""
|
||||
claim = _load_claim()
|
||||
text = serialize_837_for_resubmit(claim, interchange_index=42)
|
||||
per_line = next(s for s in text.split("~") if s.startswith("PER"))
|
||||
parts = per_line.split("*")
|
||||
# PER*IC*<Name>*<Qual>*<Number>
|
||||
assert parts[1] == "IC"
|
||||
assert parts[2], f"PER-02 (Name) is required, got empty; line={per_line!r}"
|
||||
assert parts[3] in {"TE", "EM", "FX"}, (
|
||||
f"PER-03 must be a Communication Number Qualifier (TE/EM/FX); "
|
||||
f"got {parts[3]!r}; line={per_line!r}"
|
||||
)
|
||||
assert parts[4], f"PER-04 (Number) is required, got empty; line={per_line!r}"
|
||||
|
||||
|
||||
def test_sbr_segment_emits_claim_filing_indicator_default_mc():
|
||||
"""SP40: SBR-09 (Claim Filing Indicator Code) must always be
|
||||
emitted. Default is 'MC' (Medicaid) when the caller passes no
|
||||
claim_filing_indicator_code kwarg. Edifabric rejects the bare
|
||||
SBR*P*18******* shape."""
|
||||
claim = _load_claim()
|
||||
text = serialize_837_for_resubmit(claim, interchange_index=42)
|
||||
sbr_line = next(s for s in text.split("~") if s.startswith("SBR"))
|
||||
parts = sbr_line.split("*")
|
||||
# SBR*P*18*******MC → index 9 is SBR-09
|
||||
assert len(parts) >= 10, f"SBR must have 10 elements (with SBR-09), got {len(parts)}: {sbr_line!r}"
|
||||
assert parts[9] == "MC", (
|
||||
f"SBR-09 must default to 'MC' (Medicaid), got {parts[9]!r}; line={sbr_line!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_sbr_segment_respects_explicit_claim_filing_indicator():
|
||||
"""SP40: explicit claim_filing_indicator_code kwargs win over the
|
||||
default — preserves the existing caller-facing behavior for
|
||||
non-CO payers (e.g. '16' for Medicare Part B)."""
|
||||
claim = _load_claim()
|
||||
text = serialize_837_for_resubmit(
|
||||
claim, interchange_index=42,
|
||||
claim_filing_indicator_code="16",
|
||||
)
|
||||
sbr_line = next(s for s in text.split("~") if s.startswith("SBR"))
|
||||
parts = sbr_line.split("*")
|
||||
assert parts[9] == "16", f"SBR-09 should reflect explicit kwarg, got {parts[9]!r}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP24 — IG-correctness regression test
|
||||
#
|
||||
# The 2000C Patient Hierarchical Level (HL*3 → PAT → NM1*QC) MUST be
|
||||
# absent when SBR02 == "18" (Self-pay, the CO-Medicaid IHSS workflow).
|
||||
# This test pins the PATIENT_LOOP_DEFAULT_INCLUDED constant to False so
|
||||
# a future refactor cannot silently reintroduce the broken pattern that
|
||||
# surfaced in the 2026-07-08 Edifabric 999 audit.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_serialize_837_patient_loop_default_is_false():
|
||||
"""SP24 regression: PATIENT_LOOP_DEFAULT_INCLUDED must stay False.
|
||||
|
||||
Per X12 005010X222A1, the 2000C Patient Hierarchical Level
|
||||
(``HL*3 → PAT → NM1*QC``) is REQUIRED only when Patient != Subscriber
|
||||
(i.e. ``SBR02 != "18"``). When ``SBR02 == "18"`` (Self-pay, the
|
||||
CO-Medicaid IHSS workflow), the 2000C loop MUST be absent —
|
||||
otherwise Edifabric / pyX12 reject the file with
|
||||
``"2000C HL must be absent when 2000B SBR02 = '18'"``.
|
||||
|
||||
The serializer encodes this invariant as a module-level constant
|
||||
``PATIENT_LOOP_DEFAULT_INCLUDED``. Flipping the default back to
|
||||
``True`` would re-introduce the rejection on every self-pay claim
|
||||
in the dzinesco pipeline. This test pins the constant + verifies
|
||||
the helper actually honors it on a no-kwarg call.
|
||||
|
||||
Reference: ``docs/superpowers/specs/2026-07-08-cyclone-reissue-claims-design.md``
|
||||
"""
|
||||
from datetime import date as _date
|
||||
from cyclone.parsers.models import Subscriber
|
||||
from cyclone.parsers.serialize_837 import (
|
||||
PATIENT_LOOP_DEFAULT_INCLUDED,
|
||||
_build_subscriber_block,
|
||||
)
|
||||
|
||||
# 1. The constant itself is False — the IG-correct shape.
|
||||
assert PATIENT_LOOP_DEFAULT_INCLUDED is False, (
|
||||
f"PATIENT_LOOP_DEFAULT_INCLUDED must be False "
|
||||
f"(got {PATIENT_LOOP_DEFAULT_INCLUDED!r}); the IG-correct "
|
||||
f"serializer shape for SBR02='18' claims is 2000C-absent. "
|
||||
f"See the SP24 spec §2 Decision 2 for the regression story."
|
||||
)
|
||||
|
||||
# 2. Calling _build_subscriber_block with no kwarg honors the
|
||||
# constant — no HL*3 segment emitted, HL*2 child count = 0.
|
||||
sub = Subscriber(
|
||||
member_id="TEST123",
|
||||
first_name="Test",
|
||||
last_name="Patient",
|
||||
dob=_date(1990, 1, 1),
|
||||
gender="F",
|
||||
)
|
||||
out = _build_subscriber_block(sub, "MC")
|
||||
assert not any(seg.startswith("HL*3") for seg in out), (
|
||||
f"HL*3 emitted with the IG-correct default; segments={out!r}"
|
||||
)
|
||||
# HL*2 child_count must be 0, not 1.
|
||||
hl2_line = next(seg for seg in out if seg.startswith("HL*2"))
|
||||
hl2_parts = hl2_line.rstrip("~").split("*")
|
||||
assert hl2_parts[4] == "0", (
|
||||
f"HL*2 child_count must be 0 (no 2000C children); got {hl2_parts[4]!r}. "
|
||||
f"Line: {hl2_line!r}"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
"""End-to-end test for the thin scratch driver.
|
||||
|
||||
This test wires the shipped ``cyclone.edifabric`` MockTransport seam
|
||||
to return a clean ``Status: success`` OperationResult and asserts
|
||||
that the thin driver:
|
||||
|
||||
1. Calls ``select_spot_check_visits`` against an in-memory DB
|
||||
fixture (the same in-memory DB pattern as the pipeline tests).
|
||||
2. Calls ``write_spot_check_files`` to emit 837P files.
|
||||
3. Calls ``validate_spot_check_files`` (the committed module) and
|
||||
accepts the success response.
|
||||
4. Writes ``spot-check.json`` with 10 entries, each with
|
||||
``passed=true`` and a real ``result.Status == 'success'``.
|
||||
5. Writes ``spot-check-summary.txt`` with 10 PASS lines.
|
||||
6. Exits with code 0 and prints ``10/10 passed``.
|
||||
|
||||
The MockTransport here stands in for the live Edifabric API during
|
||||
the unit test (the live API is quota-blocked until next UTC
|
||||
midnight). When quota replenishes, the same driver runs unchanged
|
||||
against the live API and produces the same evidence shape.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
import cyclone.edifabric as edifabric
|
||||
import cyclone.secrets as _secrets
|
||||
from cyclone import db as db_mod
|
||||
from cyclone.rebill.visits_store import load_visits_csv
|
||||
|
||||
_REAL_GET_SECRET = _secrets.get_secret
|
||||
_TEST_KEY = "3ecf6b1c5cf34bd797a5f4c57951a1cf"
|
||||
|
||||
_X12_INTERCHANGE = {
|
||||
"SegmentDelimiter": "~",
|
||||
"DataElementDelimiter": "*",
|
||||
"ISA": {"InterchangeControlNumber_13": "000000001"},
|
||||
"Groups": [],
|
||||
"IEATrailers": [],
|
||||
}
|
||||
|
||||
DRIVER_PATH = Path(
|
||||
"/tmp/grok-goal-4336f8e1af6d/implementer/build_and_validate.py"
|
||||
)
|
||||
|
||||
|
||||
def _install_mock(handler, secrets_key: str | None = _TEST_KEY):
|
||||
"""Install a MockTransport + patched secrets for tests."""
|
||||
edifabric.set_transport_factory(
|
||||
lambda: httpx.Client(
|
||||
transport=httpx.MockTransport(handler), timeout=10.0
|
||||
)
|
||||
)
|
||||
if secrets_key is not None:
|
||||
def _fake(name: str):
|
||||
if name == "edifabric.api_key":
|
||||
return secrets_key
|
||||
return _REAL_GET_SECRET(name)
|
||||
_secrets.get_secret = _fake # type: ignore[assignment]
|
||||
|
||||
|
||||
def _restore():
|
||||
edifabric._reset_transport_factory()
|
||||
_secrets.get_secret = _REAL_GET_SECRET
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def visits_csv(tmp_path: Path) -> Path:
|
||||
csv_path = tmp_path / "visits.csv"
|
||||
with csv_path.open("w", newline="", encoding="utf-8") as f:
|
||||
writer = csv.writer(f)
|
||||
writer.writerow([
|
||||
"Client", "Visit Date", "Payer", "Authorized", "Member ID",
|
||||
"Auth Start Date", "Auth End Date", "Authorization #", "ICD-10",
|
||||
"Service", "Procedure Code", "Modifiers", "Client Classes",
|
||||
"Billable Hours", "Billable Amount", "Invoice #", "Claimed",
|
||||
])
|
||||
for i in range(12):
|
||||
day = (i % 27) + 1
|
||||
month = ((i // 27) % 6) + 1
|
||||
writer.writerow([
|
||||
f"Patient{i:02d}, Test", f"{month:02d}/{day:02d}/2026",
|
||||
"COHCPF", "Yes", f"R{i:06d}",
|
||||
"01/01/2026", "12/31/2026", "1", "R69",
|
||||
"S5150", "S5150", "U8", "DD", "1",
|
||||
f"${200 + i}.00", f"INV-{i}", "Yes",
|
||||
])
|
||||
return csv_path
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _always_restore():
|
||||
yield
|
||||
_restore()
|
||||
|
||||
|
||||
def _load_driver(scratch: Path):
|
||||
"""Load the scratch driver as a module with SCRATCH/OUT_DIR/
|
||||
EVIDENCE_JSON/EVIDENCE_TXT rebound to ``scratch``.
|
||||
|
||||
Returns the imported module (with .main() available).
|
||||
"""
|
||||
src = DRIVER_PATH.read_text()
|
||||
# Replace the constants block with scratch-scoped ones.
|
||||
for old, new in [
|
||||
('SCRATCH = Path("/tmp/grok-goal-4336f8e1af6d/implementer")',
|
||||
f'SCRATCH = Path("{scratch}")'),
|
||||
]:
|
||||
src = src.replace(old, new, 1)
|
||||
spec = importlib.util.spec_from_loader(
|
||||
"driver_under_test", loader=None,
|
||||
)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
exec(compile(src, str(DRIVER_PATH), "exec"), mod.__dict__)
|
||||
return mod
|
||||
|
||||
|
||||
def _seed_db(db_path: Path, visits_csv: Path) -> int:
|
||||
"""Seed a fresh sqlite DB at db_path with visits_csv."""
|
||||
if db_path.exists():
|
||||
db_path.unlink()
|
||||
os.environ["CYCLONE_DB_URL"] = f"sqlite:///{db_path}"
|
||||
db_mod._reset_for_tests()
|
||||
db_mod.init_db()
|
||||
return load_visits_csv(
|
||||
visits_csv,
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
)
|
||||
|
||||
|
||||
# --- Tests ----------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_driver_e2e_against_mock_live_edifabric_succeeds(
|
||||
tmp_path, visits_csv, capsys
|
||||
):
|
||||
"""End-to-end: driver + committed modules + mocked live success →
|
||||
exit 0 + '10/10 passed' + 10 live-success entries in evidence.
|
||||
"""
|
||||
db_path = tmp_path / "test.db"
|
||||
inserted = _seed_db(db_path, visits_csv)
|
||||
assert inserted == 12
|
||||
|
||||
scratch = tmp_path / "scratch"
|
||||
scratch.mkdir()
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/read"):
|
||||
return httpx.Response(200, json=[_X12_INTERCHANGE])
|
||||
if request.url.path.endswith("/validate"):
|
||||
return httpx.Response(200, json={
|
||||
"Status": "success",
|
||||
"Details": [],
|
||||
"LastIndex": 30,
|
||||
})
|
||||
raise AssertionError(f"unexpected path: {request.url.path}")
|
||||
|
||||
_install_mock(handler)
|
||||
driver = _load_driver(scratch)
|
||||
rc = driver.main()
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert rc == 0, (
|
||||
f"driver exited {rc}; expected 0\n"
|
||||
f"stdout: {captured.out}\nstderr: {captured.err}"
|
||||
)
|
||||
assert "10/10 passed" in captured.out, (
|
||||
f"driver stdout missing '10/10 passed'\n"
|
||||
f"stdout: {captured.out}"
|
||||
)
|
||||
|
||||
evidence_json = scratch / "spot-check.json"
|
||||
evidence_txt = scratch / "spot-check-summary.txt"
|
||||
assert evidence_json.exists(), "spot-check.json not written"
|
||||
entries = json.loads(evidence_json.read_text())
|
||||
assert len(entries) == 10, f"expected 10 entries, got {len(entries)}"
|
||||
for e in entries:
|
||||
assert e["passed"] is True
|
||||
assert e["result"] is not None
|
||||
assert e["result"]["Status"] == "success"
|
||||
assert e["transport_error"] is None
|
||||
|
||||
summary = evidence_txt.read_text().strip().splitlines()
|
||||
assert len(summary) == 10
|
||||
for line in summary:
|
||||
assert "\tPASS\t" in line
|
||||
assert "\tsuccess\t" in line
|
||||
|
||||
|
||||
def test_driver_e2e_against_403_quota_fails_loud(tmp_path, visits_csv, capsys):
|
||||
"""End-to-end: driver against mocked 403 quota → exit 1, NO '10/10 passed'."""
|
||||
db_path = tmp_path / "test.db"
|
||||
_seed_db(db_path, visits_csv)
|
||||
|
||||
scratch = tmp_path / "scratch"
|
||||
scratch.mkdir()
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(
|
||||
403,
|
||||
json={"statusCode": 403,
|
||||
"message": "Out of call volume quota."},
|
||||
)
|
||||
|
||||
_install_mock(handler)
|
||||
driver = _load_driver(scratch)
|
||||
rc = driver.main()
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert "10/10 passed" not in captured.out, (
|
||||
f"driver printed fake '10/10 passed' against 403 quota:\n"
|
||||
f"stdout: {captured.out}"
|
||||
)
|
||||
assert rc != 0, (
|
||||
f"driver exited 0 against 403 quota — should fail loud\n"
|
||||
f"stdout: {captured.out}\nstderr: {captured.err}"
|
||||
)
|
||||
|
||||
evidence_json = scratch / "spot-check.json"
|
||||
entries = json.loads(evidence_json.read_text())
|
||||
assert len(entries) == 10
|
||||
for e in entries:
|
||||
assert e["passed"] is False
|
||||
assert e["transport_error"]["status_code"] == 403
|
||||
assert e["result"] is None
|
||||
@@ -0,0 +1,556 @@
|
||||
"""SP41-goal: unit tests for the committed spot-check pipeline + validate
|
||||
modules.
|
||||
|
||||
The two modules are pure (no I/O outside the DB session + Edifabric
|
||||
HTTP seam) and unit-testable end-to-end:
|
||||
|
||||
- :mod:`cyclone.rebill.spot_check_pipeline` — generation (DB → files).
|
||||
- :mod:`cyclone.rebill.spot_check_validate` — validation (files →
|
||||
OperationResult, no fallback).
|
||||
|
||||
These tests pin the live-shape contract that the production driver
|
||||
exercises, so a regression in either module shows up here without
|
||||
having to run the live driver.
|
||||
|
||||
The 835 ingestion path under test reads raw ``*.x12`` files via
|
||||
``cyclone.rebill.parse_835_svc.load_in_window_svc_rows`` (NOT the
|
||||
DB's ``service_line_payments`` / ``cas_adjustments`` tables, which
|
||||
have no ``member_id`` at SVC scope). Tests that need SVC rows
|
||||
write minimal 835 fixtures into ``tmp_path`` and pass
|
||||
``ingest_dir=tmp_path``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
import cyclone.edifabric as edifabric
|
||||
import cyclone.secrets as _secrets
|
||||
from cyclone import db as db_mod
|
||||
from cyclone.rebill.spot_check_pipeline import (
|
||||
select_spot_check_visits,
|
||||
write_spot_check_files,
|
||||
)
|
||||
from cyclone.rebill.spot_check_validate import (
|
||||
is_edifabric_pass,
|
||||
validate_spot_check_files,
|
||||
)
|
||||
from cyclone.rebill.visits_store import load_visits_csv
|
||||
|
||||
_REAL_GET_SECRET = _secrets.get_secret
|
||||
_TEST_KEY = "3ecf6b1c5cf34bd797a5f4c57951a1cf"
|
||||
|
||||
_X12_INTERCHANGE = {
|
||||
"SegmentDelimiter": "~",
|
||||
"DataElementDelimiter": "*",
|
||||
"ISA": {"InterchangeControlNumber_13": "000000001"},
|
||||
"Groups": [],
|
||||
"IEATrailers": [],
|
||||
}
|
||||
|
||||
|
||||
# --- 835 fixture builder --------------------------------------------- #
|
||||
|
||||
|
||||
def _build_835_with_svc(
|
||||
*,
|
||||
member_id: str,
|
||||
claim_id: str,
|
||||
status: str,
|
||||
procedure: str,
|
||||
svc_date: date,
|
||||
charge: Decimal | str,
|
||||
paid: Decimal | str,
|
||||
cas: list[str] | None = None,
|
||||
pay_date: date | None = None,
|
||||
) -> str:
|
||||
"""Build a minimal valid 835 interchange with one SVC.
|
||||
|
||||
The shape matches what ``cyclone.rebill.parse_835_svc.parse_835_svc``
|
||||
walks: ISA/GS/ST header, BPR + TRN + DTM*405 (pay date), N1*PR + N1*PE
|
||||
(payer / payee), LX (loop), CLP (claim), NM1*QC (member), SVC + DTM*472
|
||||
(service date) + CAS (optional), SE/GE/IEA trailer.
|
||||
|
||||
``cas`` is a list of ``"GR-CODE"`` (e.g. ``["OA-18"]``).
|
||||
"""
|
||||
charge_s = str(Decimal(str(charge)))
|
||||
paid_s = str(Decimal(str(paid)))
|
||||
pay_dtm = (
|
||||
f"DTM*405*{pay_date.strftime('%Y%m%d')}~"
|
||||
if pay_date is not None
|
||||
else ""
|
||||
)
|
||||
cas_segment = ""
|
||||
if cas:
|
||||
cas_segment = "~".join(
|
||||
f"CAS*{c.replace('-', '*')}*0" for c in cas
|
||||
) + "~"
|
||||
# Procedure in SVC: "HC:T1019:U2" format that parse_835_svc splits on ":"
|
||||
svc_comp = f"HC:{procedure}"
|
||||
segs = [
|
||||
"ISA*00* *00* *ZZ*COMEDASSISTPROG*ZZ*11525703"
|
||||
f" *260202*0218*^*00501*200005613*0*P*:~",
|
||||
"GS*HP*COMEDASSISTPROG*11525703*20260202*021851*200005613*X*005010X221A1~",
|
||||
"ST*835*1001~",
|
||||
"BPR*I*0*C*ACH*CCP*01*075911603*DA*00000004556640373*1811725341"
|
||||
"**01*102103407*DA*8911987439*20260202~",
|
||||
"TRN*1*003049968*1811725341~",
|
||||
"REF*EV*1881068062~",
|
||||
pay_dtm,
|
||||
"N1*PR*CO_TXIX*XV*7912900843~",
|
||||
"N3*P.O. BOX 30~",
|
||||
"N4*DENVER*CO*80201~",
|
||||
"PER*CX*MEDICAID PROVIDER SERVICES*TE*8442352387~",
|
||||
"N1*PE*TOC, INC*XX*1881068062~",
|
||||
"REF*TJ*721587149~",
|
||||
"LX*1~",
|
||||
f"CLP*{claim_id}*{status}*{charge_s}*{paid_s}**MC*1234~",
|
||||
f"NM1*QC*1*Member*First****MR*{member_id}~",
|
||||
f"SVC*{svc_comp}*{charge_s}*{paid_s}**1~",
|
||||
f"DTM*472*{svc_date.strftime('%Y%m%d')}~",
|
||||
cas_segment,
|
||||
"SE*15*1001~",
|
||||
"GE*1*200005613~",
|
||||
"IEA*1*200005613~",
|
||||
]
|
||||
return "".join(segs)
|
||||
|
||||
|
||||
def _write_835(
|
||||
tmp_path: Path,
|
||||
name: str,
|
||||
content: str,
|
||||
) -> Path:
|
||||
"""Write a 835 fixture to ``tmp_path/<name>`` and return its path."""
|
||||
p = tmp_path / name
|
||||
p.write_text(content, encoding="ascii")
|
||||
return p
|
||||
|
||||
|
||||
# --- Visits-CSV fixture ---------------------------------------------- #
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def visits_csv(tmp_path: Path) -> Path:
|
||||
csv_path = tmp_path / "visits.csv"
|
||||
with csv_path.open("w", newline="", encoding="utf-8") as f:
|
||||
writer = csv.writer(f)
|
||||
writer.writerow([
|
||||
"Client", "Visit Date", "Payer", "Authorized", "Member ID",
|
||||
"Auth Start Date", "Auth End Date", "Authorization #", "ICD-10",
|
||||
"Service", "Procedure Code", "Modifiers", "Client Classes",
|
||||
"Billable Hours", "Billable Amount", "Invoice #", "Claimed",
|
||||
])
|
||||
writer.writerow([
|
||||
"Doe, Jane", "01/15/2026", "COHCPF", "Yes", "R111111",
|
||||
"01/01/2026", "12/31/2026", "1", "R69",
|
||||
"Homemaker S5150", "S5150", "U8", "DD Waiver", "1", "$300.00",
|
||||
"INV-2026-01-15-R111111", "Yes",
|
||||
])
|
||||
writer.writerow([
|
||||
"Roe, Richard", "02/20/2026", "COHCPF", "Yes", "R222222",
|
||||
"01/01/2026", "12/31/2026", "2", "R69",
|
||||
"IHSS T1019", "T1019", "KX, SC, U2", "IHSS MR", "1.5", "$250.00",
|
||||
"INV-2026-02-20-R222222", "Yes",
|
||||
])
|
||||
writer.writerow([
|
||||
"Old, Stale", "12/31/2025", "COHCPF", "Yes", "R999999",
|
||||
"01/01/2025", "12/31/2025", "0", "R69",
|
||||
"Old S5150", "S5150", "U8", "DD Waiver", "1", "$100.00",
|
||||
"INV-2025-12-31-R999999", "Yes", # OUT-OF-WINDOW
|
||||
])
|
||||
return csv_path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def loaded_db(tmp_path: Path, visits_csv: Path, monkeypatch):
|
||||
"""Load visits_csv into a fresh DB and return the session."""
|
||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path / 'test.db'}")
|
||||
db_mod._reset_for_tests()
|
||||
db_mod.init_db()
|
||||
inserted = load_visits_csv(
|
||||
visits_csv,
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
)
|
||||
assert inserted == 2 # the stale row is out-of-window
|
||||
SessionLocal = db_mod.SessionLocal()
|
||||
sess = SessionLocal()
|
||||
yield sess
|
||||
sess.close()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _restore():
|
||||
yield
|
||||
edifabric._reset_transport_factory()
|
||||
_secrets.get_secret = _REAL_GET_SECRET
|
||||
|
||||
|
||||
def _mock_secrets(patched_key: str):
|
||||
def _fake(name: str):
|
||||
if name == "edifabric.api_key":
|
||||
return patched_key
|
||||
return _REAL_GET_SECRET(name)
|
||||
_secrets.get_secret = _fake # type: ignore[assignment]
|
||||
|
||||
|
||||
def _install(handler):
|
||||
transport = httpx.MockTransport(handler)
|
||||
edifabric.set_transport_factory(
|
||||
lambda: httpx.Client(transport=transport, timeout=10.0)
|
||||
)
|
||||
|
||||
|
||||
# --- Pipeline tests --------------------------------------------------- #
|
||||
|
||||
|
||||
def test_select_spot_check_visits_returns_in_window_only(loaded_db, tmp_path):
|
||||
"""select_spot_check_visits honors the DOS window."""
|
||||
out = select_spot_check_visits(
|
||||
loaded_db,
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
n=10,
|
||||
ingest_dir=tmp_path,
|
||||
)
|
||||
assert len(out) == 2
|
||||
# The stale 2025 row was filtered out at the load step.
|
||||
dos_list = [v.dos for (v, _d, _c) in out]
|
||||
assert all(date(2026, 1, 1) <= d <= date(2026, 6, 27) for d in dos_list)
|
||||
|
||||
|
||||
def test_select_spot_check_visits_marks_not_in_835(loaded_db, tmp_path):
|
||||
"""With no SVC rows in ingest/, visits are NOT_IN_835 (rebill candidates)."""
|
||||
out = select_spot_check_visits(
|
||||
loaded_db,
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
n=10,
|
||||
ingest_dir=tmp_path,
|
||||
)
|
||||
dispositions = [d for (_v, d, _c) in out]
|
||||
assert dispositions == ["NOT_IN_835", "NOT_IN_835"]
|
||||
|
||||
|
||||
def test_select_spot_check_visits_n_truncates(loaded_db, tmp_path):
|
||||
"""n=1 returns the top 1 by billed_amount desc."""
|
||||
out = select_spot_check_visits(
|
||||
loaded_db,
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
n=1,
|
||||
ingest_dir=tmp_path,
|
||||
)
|
||||
assert len(out) == 1
|
||||
assert out[0][0].member_id == "R111111" # $300 > $250
|
||||
|
||||
|
||||
def test_select_spot_check_visits_oa18_excluded_member_scoped(
|
||||
loaded_db, tmp_path,
|
||||
):
|
||||
"""OA-18-only SVC matches are excluded as DENIED_DUPLICATE_NOISE.
|
||||
|
||||
Drives the SHIPPED code on the real path: a minimal 835 fixture
|
||||
in ``tmp_path`` with an OA-18 CAS for member R111111's SVC. R222222
|
||||
has no SVC match → NOT_IN_835. R111111 has a matched SVC whose
|
||||
CAS reasons are exactly ``{"OA-18"}`` → DENIED_DUPLICATE_NOISE
|
||||
→ excluded.
|
||||
"""
|
||||
# Empty CAS → R111111 becomes DENIED (kept), R222222 stays NOT_IN_835.
|
||||
no_cas_835 = _build_835_with_svc(
|
||||
member_id="R111111",
|
||||
claim_id="CLM-TEST-NOCAS",
|
||||
status="4", # denied
|
||||
procedure="S5150",
|
||||
svc_date=date(2026, 1, 15),
|
||||
charge=Decimal("300.00"),
|
||||
paid=Decimal("0.00"),
|
||||
cas=None,
|
||||
pay_date=date(2026, 1, 20),
|
||||
)
|
||||
_write_835(tmp_path, "no_cas.835", no_cas_835)
|
||||
|
||||
out = select_spot_check_visits(
|
||||
loaded_db,
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
n=10,
|
||||
ingest_dir=tmp_path,
|
||||
)
|
||||
dispositions = sorted(d for (_v, d, _c) in out)
|
||||
# R222222 (no SVC) → NOT_IN_835; R111111 (SVC hit, no CAS yet) → DENIED
|
||||
assert dispositions == ["DENIED", "NOT_IN_835"]
|
||||
|
||||
# Now add OA-18 CAS — R111111 flips to DENIED_DUPLICATE_NOISE.
|
||||
# Use a different filename so we don't collide with no_cas.835;
|
||||
# the walker re-parses each file independently.
|
||||
oa18_835 = _build_835_with_svc(
|
||||
member_id="R111111",
|
||||
claim_id="CLM-TEST-OA18",
|
||||
status="4", # denied
|
||||
procedure="S5150",
|
||||
svc_date=date(2026, 1, 15),
|
||||
charge=Decimal("300.00"),
|
||||
paid=Decimal("0.00"),
|
||||
cas=["OA-18"],
|
||||
pay_date=date(2026, 1, 20),
|
||||
)
|
||||
_write_835(tmp_path, "oa18.835", oa18_835)
|
||||
|
||||
out = select_spot_check_visits(
|
||||
loaded_db,
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
n=10,
|
||||
ingest_dir=tmp_path,
|
||||
)
|
||||
# R111111 now excluded (DENIED_DUPLICATE_NOISE); only R222222 remains.
|
||||
assert len(out) == 1
|
||||
assert out[0][0].member_id == "R222222"
|
||||
|
||||
|
||||
def test_select_spot_check_visits_member_isolation(tmp_path, monkeypatch):
|
||||
"""Two visits at the same (procedure, DOS) for DIFFERENT members:
|
||||
member A's SVC has OA-18-only CAS; member B's SVC has none.
|
||||
|
||||
On the correct (member_id, procedure, DOS) indexer, A is excluded
|
||||
and B stays as a candidate. On the previous (procedure, DOS) only
|
||||
indexer, A's CAS would bleed across to B and B would also be
|
||||
excluded. This is the structural fix the verifier demanded.
|
||||
"""
|
||||
# Visits for two members at the SAME (procedure, DOS).
|
||||
visits_csv = tmp_path / "visits.csv"
|
||||
with visits_csv.open("w", newline="", encoding="utf-8") as f:
|
||||
writer = csv.writer(f)
|
||||
writer.writerow([
|
||||
"Client", "Visit Date", "Payer", "Authorized", "Member ID",
|
||||
"Auth Start Date", "Auth End Date", "Authorization #", "ICD-10",
|
||||
"Service", "Procedure Code", "Modifiers", "Client Classes",
|
||||
"Billable Hours", "Billable Amount", "Invoice #", "Claimed",
|
||||
])
|
||||
# Member A at 2026-03-10, $400 — OA-18 candidate (will be excluded).
|
||||
writer.writerow([
|
||||
"Alpha, Anon", "03/10/2026", "COHCPF", "Yes", "R-ALPHA",
|
||||
"01/01/2026", "12/31/2026", "1", "R69",
|
||||
"HCPCS S5150", "S5150", "U8", "DD Waiver", "1", "$400.00",
|
||||
"INV-A", "Yes",
|
||||
])
|
||||
# Member B at SAME (procedure, DOS), $500 — DENIED candidate (kept).
|
||||
writer.writerow([
|
||||
"Beta, Anon", "03/10/2026", "COHCPF", "Yes", "R-BETA",
|
||||
"01/01/2026", "12/31/2026", "2", "R69",
|
||||
"HCPCS S5150", "S5150", "U8", "DD Waiver", "1", "$500.00",
|
||||
"INV-B", "Yes",
|
||||
])
|
||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path / 'test.db'}")
|
||||
db_mod._reset_for_tests()
|
||||
db_mod.init_db()
|
||||
assert load_visits_csv(visits_csv) == 2
|
||||
SessionLocal = db_mod.SessionLocal()
|
||||
sess = SessionLocal()
|
||||
|
||||
# Two 835s: one for R-ALPHA with OA-18-only CAS, one for R-BETA with no CAS.
|
||||
alpha_835 = _build_835_with_svc(
|
||||
member_id="R-ALPHA",
|
||||
claim_id="CLM-A",
|
||||
status="4",
|
||||
procedure="S5150",
|
||||
svc_date=date(2026, 3, 10),
|
||||
charge=Decimal("400.00"),
|
||||
paid=Decimal("0.00"),
|
||||
cas=["OA-18"],
|
||||
pay_date=date(2026, 3, 15),
|
||||
)
|
||||
beta_835 = _build_835_with_svc(
|
||||
member_id="R-BETA",
|
||||
claim_id="CLM-B",
|
||||
status="4",
|
||||
procedure="S5150",
|
||||
svc_date=date(2026, 3, 10),
|
||||
charge=Decimal("500.00"),
|
||||
paid=Decimal("0.00"),
|
||||
cas=None,
|
||||
pay_date=date(2026, 3, 15),
|
||||
)
|
||||
_write_835(tmp_path, "alpha.835", alpha_835)
|
||||
_write_835(tmp_path, "beta.835", beta_835)
|
||||
|
||||
out = select_spot_check_visits(
|
||||
sess,
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
n=10,
|
||||
ingest_dir=tmp_path,
|
||||
)
|
||||
|
||||
sess.close()
|
||||
|
||||
# Alpha is excluded as DENIED_DUPLICATE_NOISE; Beta is the sole
|
||||
# candidate. n=10 truncates to 1, by billed desc → Beta ($500).
|
||||
assert len(out) == 1
|
||||
assert out[0][0].member_id == "R-BETA"
|
||||
assert out[0][1] == "DENIED"
|
||||
|
||||
|
||||
def test_write_spot_check_files_emits_required_segments(tmp_path, loaded_db):
|
||||
"""Every emitted file passes the plan's literal grep."""
|
||||
out_dir = tmp_path / "spot-check-837Ps"
|
||||
visits = select_spot_check_visits(
|
||||
loaded_db,
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
n=10,
|
||||
ingest_dir=tmp_path,
|
||||
)
|
||||
written = write_spot_check_files(visits, out_dir)
|
||||
assert len(written) == 2
|
||||
plan_grep = re.compile(
|
||||
r"NM1\*41|NM1\*40|NM1\*QC|^CLM\*[A-Z]|SV1\*HC:|DTP\*472",
|
||||
re.MULTILINE,
|
||||
)
|
||||
for sc in written:
|
||||
content = sc.path.read_text()
|
||||
hits = plan_grep.findall(content)
|
||||
# 6 distinct segment classes.
|
||||
assert len(set(hits)) == 6, (
|
||||
f"{sc.path.name}: missing one of the 6 required segment "
|
||||
f"classes; got {set(hits)}"
|
||||
)
|
||||
# CLM01 must start with uppercase R.
|
||||
clm_line = next(
|
||||
(l for l in content.splitlines() if l.startswith("CLM*")), None
|
||||
)
|
||||
assert clm_line and clm_line.startswith("CLM*R"), clm_line
|
||||
|
||||
|
||||
def test_write_spot_check_files_segment_per_line(tmp_path, loaded_db):
|
||||
"""Files are written segment-per-line (matches prodfiles canonical shape)."""
|
||||
out_dir = tmp_path / "spot-check-837Ps"
|
||||
visits = select_spot_check_visits(
|
||||
loaded_db,
|
||||
window_start=date(2026, 1, 1),
|
||||
window_end=date(2026, 6, 27),
|
||||
n=10,
|
||||
ingest_dir=tmp_path,
|
||||
)
|
||||
written = write_spot_check_files(visits, out_dir)
|
||||
# Known segment names that may appear in a well-formed 837P.
|
||||
valid_segment_prefixes = {
|
||||
"ISA", "GS", "ST", "BHT", "NM1", "PER", "HL", "PRV", "N3", "N4",
|
||||
"REF", "SBR", "DMG", "PAT", "CLM", "HI", "LX", "SV1", "SV2",
|
||||
"SV3", "SV4", "SV5", "SV6", "SV7", "DTP", "SE", "GE", "IEA",
|
||||
}
|
||||
for sc in written:
|
||||
content = sc.path.read_text()
|
||||
# No tildes in the file body (only segment terminator is newline).
|
||||
assert "~" not in content, f"{sc.path.name} still contains ~"
|
||||
# Every line begins with a known segment name.
|
||||
for line in content.splitlines():
|
||||
prefix = line.split("*", 1)[0]
|
||||
assert prefix in valid_segment_prefixes, (
|
||||
f"unexpected segment prefix in {sc.path.name}: {line!r}"
|
||||
)
|
||||
|
||||
|
||||
# --- Validate tests --------------------------------------------------- #
|
||||
|
||||
|
||||
def test_is_edifabric_pass_true_on_success():
|
||||
assert is_edifabric_pass({"Status": "success", "Details": []}) is True
|
||||
|
||||
|
||||
def test_is_edifabric_pass_true_on_warning():
|
||||
assert is_edifabric_pass({"Status": "warning", "Details": []}) is True
|
||||
|
||||
|
||||
def test_is_edifabric_pass_false_on_error():
|
||||
assert is_edifabric_pass({"Status": "error", "Details": []}) is False
|
||||
|
||||
|
||||
def test_is_edifabric_pass_false_on_unknown():
|
||||
assert is_edifabric_pass({"Status": "fubar"}) is False
|
||||
|
||||
|
||||
def test_validate_spot_check_files_passes_on_success_status(tmp_path):
|
||||
"""validate_spot_check_files returns passed=True on Status='success'."""
|
||||
f = tmp_path / "test.x12"
|
||||
f.write_bytes(b"ISA*00*~ST*837*~SE*1*~IEA*1*~")
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/read"):
|
||||
return httpx.Response(200, json=[_X12_INTERCHANGE])
|
||||
if request.url.path.endswith("/validate"):
|
||||
return httpx.Response(
|
||||
200, json={"Status": "success", "Details": [], "LastIndex": 4},
|
||||
)
|
||||
raise AssertionError(f"unexpected path: {request.url.path}")
|
||||
|
||||
_mock_secrets(_TEST_KEY)
|
||||
_install(handler)
|
||||
out = validate_spot_check_files([f])
|
||||
assert len(out) == 1
|
||||
assert out[0]["passed"] is True
|
||||
assert out[0]["result"]["Status"] == "success"
|
||||
assert out[0]["transport_error"] is None
|
||||
|
||||
|
||||
def test_validate_spot_check_files_fails_on_error_status(tmp_path):
|
||||
"""validate_spot_check_files returns passed=False on Status='error'."""
|
||||
f = tmp_path / "test.x12"
|
||||
f.write_bytes(b"ISA*00*~ST*837*~SE*1*~IEA*1*~")
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/read"):
|
||||
return httpx.Response(200, json=[_X12_INTERCHANGE])
|
||||
if request.url.path.endswith("/validate"):
|
||||
return httpx.Response(200, json={
|
||||
"Status": "error",
|
||||
"Details": [{
|
||||
"SegmentId": "CLM", "Message": "missing elements",
|
||||
"Status": "error",
|
||||
}],
|
||||
"LastIndex": 2,
|
||||
})
|
||||
raise AssertionError(f"unexpected path: {request.url.path}")
|
||||
|
||||
_mock_secrets(_TEST_KEY)
|
||||
_install(handler)
|
||||
out = validate_spot_check_files([f])
|
||||
assert out[0]["passed"] is False
|
||||
assert "missing elements" in out[0]["issue"]
|
||||
|
||||
|
||||
def test_validate_spot_check_files_raises_on_403_quota(tmp_path):
|
||||
"""On 403 quota, raises EdifabricError — NO fallback to pyx12.
|
||||
|
||||
This is the structural fix: the previous scratch driver caught
|
||||
this error and fell back to a local validator, which silently
|
||||
marked passed=True without ever talking to Edifabric. The new
|
||||
module deliberately has no fallback.
|
||||
"""
|
||||
f = tmp_path / "test.x12"
|
||||
f.write_bytes(b"ISA*00*~ST*837*~SE*1*~IEA*1*~")
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/read"):
|
||||
return httpx.Response(
|
||||
403,
|
||||
json={"statusCode": 403,
|
||||
"message": "Out of call volume quota."},
|
||||
)
|
||||
raise AssertionError(f"unexpected path: {request.url.path}")
|
||||
|
||||
_mock_secrets(_TEST_KEY)
|
||||
_install(handler)
|
||||
with pytest.raises(edifabric.EdifabricError) as exc_info:
|
||||
validate_spot_check_files([f])
|
||||
assert exc_info.value.status_code == 403
|
||||
@@ -0,0 +1,167 @@
|
||||
"""SP25: defensive read-side stub for synthetic ``<synthetic:orphan-reconcile>``
|
||||
``Batch`` rows.
|
||||
|
||||
Background: ``CycloneStore.reconcile_orphan_st02s()`` (sp38) seeds ``Batch``
|
||||
rows for orphan 999 ACK ST02s so the envelope index can resolve
|
||||
``source_batch_id`` lookups. Those rows are inserted with
|
||||
``raw_result_json=NULL`` because no source 837 file was ever parsed
|
||||
for them. The read-side helper ``_row_to_record()`` in
|
||||
``store/batches.py`` was written assuming the column is always
|
||||
populated, so it raises a Pydantic ``ValidationError`` on ``summary:
|
||||
Field required`` the moment it sees a synthetic row — which makes
|
||||
``GET /api/batches?limit=5`` return HTTP 500.
|
||||
|
||||
This file pins the contract: a synthetic (or otherwise unparseable)
|
||||
``Batch`` row MUST still hydrate to a typed ``BatchRecord`` with an
|
||||
empty ``claims`` list and the ST02 echoed on ``result.summary.control_number``,
|
||||
so the dashboard "Recent batches" widget can render them as
|
||||
"synthetic: 0 claims" without the whole list blowing up.
|
||||
|
||||
Live state witnessed during the 2026-07-07 dashboard postmortem:
|
||||
the sp38 reconcile pass on the production DB seeded 4 such rows
|
||||
(SCN 991102986–989). Once the stub lands, ``/api/batches`` returns 200
|
||||
instead of 500 and the dashboard "Recent batches" panel surfaces the
|
||||
real underlying batches (the test runs).
|
||||
|
||||
Note: the rows inserted here are *ORM-level* fixtures (Batch rows with
|
||||
NULL raw_result_json, used to exercise the read-path's defensive
|
||||
deserialize). They are NOT synthetic EDI — the actual claim data for
|
||||
the recover-ingest path lives under ``ingest/`` and is replayed by the
|
||||
operator after this defensive stub ships.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from cyclone import db, store
|
||||
from cyclone.db import Batch
|
||||
from cyclone.store.records import BatchRecord837, BatchRecord835
|
||||
|
||||
|
||||
def _now():
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
def _insert_synthetic(s, *, kind: str, st02: str, filename: str = "<synthetic:orphan-reconcile>"):
|
||||
bid = f"syn-{st02}"
|
||||
s.add(Batch(
|
||||
id=bid,
|
||||
kind=kind,
|
||||
input_filename=filename,
|
||||
parsed_at=_now(),
|
||||
transaction_set_control_number=st02,
|
||||
raw_result_json=None,
|
||||
))
|
||||
s.commit()
|
||||
return bid
|
||||
|
||||
|
||||
def test_row_to_record_handles_null_raw_json_837p():
|
||||
"""837p synthetic row → stub BatchRecord837 with empty claims[] + SCN preserved."""
|
||||
s = db.SessionLocal()()
|
||||
try:
|
||||
_insert_synthetic(s, kind="837p", st02="991102986")
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
# No raw_result_json on the row → _row_to_record must NOT raise.
|
||||
rows = store.list_batches(limit=50)
|
||||
|
||||
syn = [r for r in rows if r.id == "syn-991102986"]
|
||||
assert len(syn) == 1
|
||||
rec = syn[0]
|
||||
assert isinstance(rec, BatchRecord837)
|
||||
assert rec.result.claims == []
|
||||
assert rec.result.summary.total_claims == 0
|
||||
assert rec.result.summary.control_number == "991102986"
|
||||
assert rec.result.summary.passed == 0
|
||||
assert rec.result.summary.failed == 0
|
||||
|
||||
|
||||
def test_row_to_record_handles_null_raw_json_835():
|
||||
"""835 synthetic row → stub BatchRecord835 with empty claims[].claims."""
|
||||
s = db.SessionLocal()()
|
||||
try:
|
||||
_insert_synthetic(s, kind="835", st02="991102987")
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
rows = store.list_batches(limit=50)
|
||||
syn = [r for r in rows if r.id == "syn-991102987"]
|
||||
assert len(syn) == 1
|
||||
rec = syn[0]
|
||||
assert isinstance(rec, BatchRecord835)
|
||||
assert rec.result.claims == []
|
||||
# summary control_number echoes ST02.
|
||||
assert rec.result.summary.control_number == "991102987"
|
||||
|
||||
|
||||
def test_row_to_record_handles_empty_string_raw_json():
|
||||
"""raw_result_json='' (empty string) gets the same defensive treatment."""
|
||||
s = db.SessionLocal()()
|
||||
try:
|
||||
bid = "syn-empty"
|
||||
s.add(Batch(
|
||||
id=bid, kind="837p",
|
||||
input_filename="<synthetic:orphan-reconcile>",
|
||||
parsed_at=_now(),
|
||||
transaction_set_control_number="991102988",
|
||||
raw_result_json="", # empty string instead of None
|
||||
))
|
||||
s.commit()
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
rows = store.list_batches(limit=50)
|
||||
syn = [r for r in rows if r.id == "syn-empty"]
|
||||
assert len(syn) == 1
|
||||
rec = syn[0]
|
||||
assert rec.result.claims == []
|
||||
assert rec.result.summary.total_claims == 0
|
||||
|
||||
|
||||
def test_row_to_record_normal_row_still_parses():
|
||||
"""Regression guard: a well-formed raw_result_json still deserializes fully.
|
||||
|
||||
Batch.raw_result_json is JSONText-backed, so we store a dict; the type
|
||||
decorator takes care of the JSON round-trip.
|
||||
"""
|
||||
payload = {
|
||||
"envelope": {
|
||||
"sender_id": "11525703",
|
||||
"receiver_id": "RECEIVERID",
|
||||
"control_number": "000000001",
|
||||
"transaction_date": "2026-07-07",
|
||||
"transaction_time": "120000",
|
||||
"implementation_guide": "005010X222A1",
|
||||
},
|
||||
"claims": [],
|
||||
"summary": {
|
||||
"input_file": "single-claim.x12",
|
||||
"control_number": "000000001",
|
||||
"transaction_date": "2026-07-07",
|
||||
"total_claims": 1,
|
||||
"passed": 1,
|
||||
"failed": 0,
|
||||
},
|
||||
}
|
||||
s = db.SessionLocal()()
|
||||
try:
|
||||
s.add(Batch(
|
||||
id="syn-normal",
|
||||
kind="837p",
|
||||
input_filename="single-claim.x12",
|
||||
parsed_at=_now(),
|
||||
transaction_set_control_number="991102977",
|
||||
raw_result_json=payload,
|
||||
))
|
||||
s.commit()
|
||||
finally:
|
||||
s.close()
|
||||
|
||||
rows = store.list_batches(limit=50)
|
||||
norm = [r for r in rows if r.id == "syn-normal"]
|
||||
assert len(norm) == 1
|
||||
rec = norm[0]
|
||||
assert rec.result.summary.total_claims == 1
|
||||
assert rec.result.summary.passed == 1
|
||||
@@ -0,0 +1,246 @@
|
||||
"""Claim-id dedup at the SFTP pre-flight stage."""
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from cyclone.submission.core import submit_file
|
||||
from cyclone.store.exceptions import DuplicateClaimError
|
||||
from cyclone.store.submission_dedup import (
|
||||
record_submission,
|
||||
check_duplicate,
|
||||
DEFAULT_WINDOW_DAYS,
|
||||
)
|
||||
|
||||
# Fixture file with a single CLM01 = "CLM001" claim (the minimal_837p
|
||||
# fixture is the source of truth; duplicated into submit-batch/single-claim.x12
|
||||
# but both byte-equal so we pick the canonical minimal path here).
|
||||
_FIXTURE = Path(__file__).parent / "fixtures" / "minimal_837p.txt"
|
||||
_FIXTURE_CLAIM_ID = "CLM001"
|
||||
|
||||
|
||||
def test_check_duplicate_returns_none_for_first_submission(tmp_path):
|
||||
db = tmp_path / "test.db"
|
||||
assert check_duplicate("ORIG-001", db_url=f"sqlite:///{db}") is None
|
||||
|
||||
|
||||
def test_check_duplicate_raises_within_window(tmp_path):
|
||||
db = tmp_path / "test.db"
|
||||
record_submission("ORIG-002", submitted_at=datetime(2026, 6, 1), db_url=f"sqlite:///{db}")
|
||||
try:
|
||||
check_duplicate("ORIG-002", db_url=f"sqlite:///{db}",
|
||||
now=datetime(2026, 6, 15))
|
||||
except DuplicateClaimError as e:
|
||||
assert e.claim_id == "ORIG-002"
|
||||
assert e.original_submission_at == datetime(2026, 6, 1)
|
||||
else:
|
||||
raise AssertionError("expected DuplicateClaimError")
|
||||
|
||||
|
||||
def test_check_duplicate_passes_after_window(tmp_path):
|
||||
db = tmp_path / "test.db"
|
||||
record_submission("ORIG-003", submitted_at=datetime(2026, 1, 1), db_url=f"sqlite:///{db}")
|
||||
# 100 days later — past the 30-day default window
|
||||
assert check_duplicate("ORIG-003", db_url=f"sqlite:///{db}",
|
||||
now=datetime(2026, 4, 11)) is None
|
||||
|
||||
|
||||
def test_default_window_is_30_days():
|
||||
assert DEFAULT_WINDOW_DAYS == 30
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# SP41 Task 9 integration: dedup guard wired into submit_file pre-flight.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_submit_file_raises_on_duplicate_claim_id(tmp_path):
|
||||
"""``submit_file`` raises ``DuplicateClaimError`` if any CLM01 in the
|
||||
file was submitted within the 30-day dedup window, and the DB write
|
||||
+ SFTP factory are both skipped.
|
||||
|
||||
Mirrors the DB-first, upload-second invariant from ``submit_file``:
|
||||
the dedup check sits between the payer-mismatch guard and the
|
||||
BatchRecord837 DB write, so a duplicate blocks both the DB write
|
||||
AND the SFTP upload. This test pins both invariants.
|
||||
|
||||
The conftest autouse fixture wires a fresh per-test SQLite DB at
|
||||
``tmp_path/test.db`` via the ``CYCLONE_DB_URL`` env var (so
|
||||
``cycl_db.SessionLocal()`` points there) — same engine the Batch
|
||||
write would use, same engine the dedup helper reads.
|
||||
"""
|
||||
# 1. Pre-populate the dedup table with the claim_id from the
|
||||
# fixture at a timestamp within the 30-day window. ``db_url``
|
||||
# is ignored inside the helper (signature-preserving shim) but
|
||||
# the argument is required by the current Task 8 signature, so
|
||||
# we pass the per-test DB URL for clarity.
|
||||
record_submission(
|
||||
_FIXTURE_CLAIM_ID,
|
||||
submitted_at=datetime.now(timezone.utc),
|
||||
db_url=f"sqlite:///{tmp_path}/test.db",
|
||||
)
|
||||
|
||||
# 2. Sentinel: if submit_file reaches the SFTP factory call,
|
||||
# fail loudly. ``submit_file`` raises DuplicateClaimError
|
||||
# BEFORE the factory is invoked, so the AssertionError here
|
||||
# only fires if the dedup guard is bypassed (regression).
|
||||
def _sftp_factory(_block): # pragma: no cover - regression only
|
||||
raise AssertionError(
|
||||
"SFTP factory must not be called when dedup blocks submit_file"
|
||||
)
|
||||
|
||||
sftp_block = MagicMock(stub=False, paths={"outbound": "/ToHPE"})
|
||||
|
||||
# 3. Expect the 409-class domain exception with structured
|
||||
# ``claim_id`` / ``original_submission_at`` attributes so the
|
||||
# operator can see which CLM01 tripped the guard.
|
||||
with pytest.raises(DuplicateClaimError) as exc_info:
|
||||
submit_file(
|
||||
_FIXTURE,
|
||||
sftp_block=sftp_block,
|
||||
actor="test",
|
||||
validate=True,
|
||||
sftp_client_factory=_sftp_factory,
|
||||
)
|
||||
assert exc_info.value.claim_id == _FIXTURE_CLAIM_ID
|
||||
assert exc_info.value.original_submission_at is not None
|
||||
|
||||
# 4. DB invariant: no Batch row was written (dedup blocks the
|
||||
# DB write, not just the SFTP upload). The autouse fixture
|
||||
# points cycl_db at tmp_path/test.db, so a fresh
|
||||
# ``db_mod.SessionLocal()()`` re-opens it.
|
||||
from cyclone import db as db_mod
|
||||
|
||||
with db_mod.SessionLocal()() as session:
|
||||
batch_count = session.query(db_mod.Batch).count()
|
||||
assert batch_count == 0, (
|
||||
"dedup guard must block the BatchRecord837 DB write; "
|
||||
f"found {batch_count} Batch row(s)"
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# SP41 Task 9 — router-level coverage: api_routers/submission.submit_batch
|
||||
# catches DuplicateClaimError BEFORE the generic Exception handler and
|
||||
# surfaces it as a per-file ``outcome="unexpected_error"`` row, with the
|
||||
# duplicated claim_id baked into ``error`` and ``batch_id=None`` (no DB
|
||||
# row written). The HTTP status code stays 200 — per-file failures live
|
||||
# in the JSON body, not in the status (see the top-of-module contract
|
||||
# in api_routers/submission.py).
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_submit_batch_per_file_duplicate_claim_classified_as_unexpected(
|
||||
tmp_path, monkeypatch,
|
||||
):
|
||||
"""Pre-record CLM001 (the fixture's only CLM01) then POST a batch
|
||||
containing one copy of the fixture. Asserts the router:
|
||||
|
||||
- keeps the HTTP status at 200 (per-file failures don't change status)
|
||||
- classifies the row as ``outcome == "unexpected_error"`` (the
|
||||
special-case handler in api_routers/submission.py)
|
||||
- includes the duplicated ``claim_id`` ("CLM001") in ``error``
|
||||
- sets ``batch_id is None`` (no BatchRecord837 row written)
|
||||
|
||||
Does NOT monkeypatch ``submit_file`` — the test exercises the real
|
||||
pre-flight raise path so the router's special-case catch is what
|
||||
produces the unexpected_error classification. submit_file raises
|
||||
DuplicateClaimError BEFORE the SFTP factory is invoked, so the
|
||||
paramiko factory is never reached even though the client fixture
|
||||
flips ``sftp_block.stub`` to ``False`` (no live MFT needed).
|
||||
"""
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from cyclone import db as db_mod
|
||||
from cyclone.api import app
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
# 1. Per-test DB + clearhouse (mirror test_api_submit_batch.py's
|
||||
# ``client`` fixture inline so this test is self-contained).
|
||||
# The autouse conftest already wired CYCLONE_DB_URL + init_db;
|
||||
# we just need to seed the clearhouse row and flip stub=False.
|
||||
db_mod._reset_for_tests()
|
||||
db_mod.init_db()
|
||||
cycl_store.ensure_clearhouse_seeded()
|
||||
ch = cycl_store.get_clearhouse()
|
||||
cycl_store.update_clearhouse(
|
||||
ch.model_copy(update={
|
||||
"sftp_block": ch.sftp_block.model_copy(update={"stub": False}),
|
||||
}),
|
||||
)
|
||||
|
||||
# 2. Pre-record the duplicate. CLM001 is the fixture's only CLM01;
|
||||
# ``record_submission`` is the same helper the dedup unit test
|
||||
# uses, so the window + exception shape match.
|
||||
record_submission(
|
||||
_FIXTURE_CLAIM_ID,
|
||||
submitted_at=datetime.now(timezone.utc),
|
||||
db_url=f"sqlite:///{tmp_path}/test.db",
|
||||
)
|
||||
|
||||
# 3. Stage a single batch-* dir containing one copy of the fixture.
|
||||
# Mirrors _stage_batch from test_api_submit_batch.py.
|
||||
batch_dir = tmp_path / "batch-dedup-claims"
|
||||
batch_dir.mkdir()
|
||||
(batch_dir / "dup-claim.x12").write_bytes(_FIXTURE.read_bytes())
|
||||
|
||||
# 4. Sentinel on submit_file's sftp_client_factory — submit_file
|
||||
# raises BEFORE the factory is invoked, so this AssertionError
|
||||
# only fires if the dedup guard is bypassed end-to-end. We
|
||||
# don't actually call submit_file directly here; the sentinel
|
||||
# is wired via monkeypatch on the function the router uses, so
|
||||
# if anything in the chain accidentally reaches the factory
|
||||
# call we'd see it. In practice submit_file raises and we never
|
||||
# get there — this is a regression tripwire only.
|
||||
def _sentinel_factory(_block): # pragma: no cover - regression only
|
||||
raise AssertionError(
|
||||
"SFTP factory must not be called: dedup blocks submit_file "
|
||||
"before the factory runs"
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
"cyclone.submission.core._default_sftp_factory", _sentinel_factory,
|
||||
)
|
||||
|
||||
# 5. Hit the endpoint. Per-file failures don't change the status
|
||||
# code → 200 even though one file was rejected.
|
||||
client = TestClient(app)
|
||||
resp = client.post(
|
||||
"/api/submit-batch",
|
||||
json={"ingest_dir": str(tmp_path), "validate": True},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
|
||||
# 6. Counts: 1 file total, classified as failed. (UNEXPECTED_ERROR
|
||||
# rolls into the ``failed`` counter in api_routers/submission.py
|
||||
# — see the counter branch at the bottom of the per-file loop.)
|
||||
assert body["submitted"] == 0
|
||||
assert body["skipped"] == 0
|
||||
assert body["failed"] == 1
|
||||
assert len(body["results"]) == 1
|
||||
|
||||
# 7. The single result row carries the special-case classification
|
||||
# (NOT a generic "runtime error: kaboom" — that's a different
|
||||
# test in test_api_submit_batch.py). The router's
|
||||
# DuplicateClaimError handler sets UNEXPECTED_ERROR + bakes the
|
||||
# claim_id into ``error``.
|
||||
row = body["results"][0]
|
||||
assert row["outcome"] == "unexpected_error"
|
||||
assert _FIXTURE_CLAIM_ID in row["error"], (
|
||||
f"expected claim_id {_FIXTURE_CLAIM_ID!r} in error string; "
|
||||
f"got {row['error']!r}"
|
||||
)
|
||||
# The router's handler does NOT construct a batch_id — the dedup
|
||||
# guard fires BEFORE the DB write, so no BatchRecord837 row exists.
|
||||
assert row["batch_id"] is None
|
||||
|
||||
# 8. DB invariant (defense in depth): even though we trust the
|
||||
# router-level assertion above, pin that no Batch row leaked.
|
||||
with db_mod.SessionLocal()() as session:
|
||||
batch_count = session.query(db_mod.Batch).count()
|
||||
assert batch_count == 0, (
|
||||
"dedup guard must block the BatchRecord837 DB write end-to-end; "
|
||||
f"found {batch_count} Batch row(s)"
|
||||
)
|
||||
@@ -0,0 +1,125 @@
|
||||
"""SP40: tests for `cyclone validate-837 <file>` CLI."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from cyclone import edifabric
|
||||
from cyclone.cli import validate_837_cmd
|
||||
|
||||
|
||||
_TEST_KEY = "test-edifabric-key-0123456789abcdef"
|
||||
|
||||
|
||||
def _make_client(handler):
|
||||
transport = httpx.MockTransport(handler)
|
||||
return httpx.Client(transport=transport, timeout=10.0)
|
||||
|
||||
|
||||
def _install(handler):
|
||||
edifabric.set_transport_factory(lambda: _make_client(handler))
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_transport():
|
||||
yield
|
||||
edifabric._reset_transport_factory()
|
||||
|
||||
|
||||
# --- happy path -------------------------------------------------------
|
||||
|
||||
|
||||
def test_validate_837_success_prints_status_and_exits_zero(tmp_path):
|
||||
"""A clean file with Status='success' from Edifabric exits 0 and
|
||||
prints the OperationResult summary."""
|
||||
fixture = Path("/home/tyler/dev/cyclone/backend/tests/fixtures/co_medicaid_837p.txt")
|
||||
sample = tmp_path / "ok.x12"
|
||||
sample.write_text(fixture.read_text())
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/read"):
|
||||
return httpx.Response(200, json=[{
|
||||
"SegmentDelimiter": "~",
|
||||
"DataElementDelimiter": "*",
|
||||
"ISA": {"InterchangeControlNumber_13": "000000001"},
|
||||
"Groups": [],
|
||||
"IEATrailers": [],
|
||||
}])
|
||||
if request.url.path.endswith("/validate"):
|
||||
return httpx.Response(200, json={
|
||||
"Status": "success",
|
||||
"Details": [],
|
||||
"LastIndex": 46,
|
||||
})
|
||||
raise AssertionError(f"unexpected path: {request.url.path}")
|
||||
|
||||
_install(handler)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(validate_837_cmd, [str(sample), "--api-key", _TEST_KEY])
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "Status: success" in result.output
|
||||
assert "Details: 0 item(s)" in result.output
|
||||
|
||||
|
||||
def test_validate_837_error_status_exits_two_and_prints_details(tmp_path):
|
||||
"""Status='error' from Edifabric exits 2 and prints the offending
|
||||
segment + message for each Details item."""
|
||||
sample = tmp_path / "bad.x12"
|
||||
sample.write_text("ISA*bad~")
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if request.url.path.endswith("/read"):
|
||||
return httpx.Response(200, json=[{
|
||||
"SegmentDelimiter": "~",
|
||||
"DataElementDelimiter": "*",
|
||||
"ISA": {"InterchangeControlNumber_13": "000000001"},
|
||||
"Groups": [],
|
||||
"IEATrailers": [],
|
||||
}])
|
||||
if request.url.path.endswith("/validate"):
|
||||
return httpx.Response(200, json={
|
||||
"Status": "error",
|
||||
"Details": [
|
||||
{"SegmentId": "PER", "Message": "PER-04 is required", "Status": "error"},
|
||||
{"SegmentId": "SBR", "Message": "SBR-09 is required", "Status": "error"},
|
||||
],
|
||||
"LastIndex": 5,
|
||||
})
|
||||
raise AssertionError(f"unexpected path: {request.url.path}")
|
||||
|
||||
_install(handler)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(validate_837_cmd, [str(sample), "--api-key", _TEST_KEY])
|
||||
assert result.exit_code == 2, result.output
|
||||
assert "Status: error" in result.output
|
||||
assert "PER: PER-04 is required" in result.output
|
||||
assert "SBR: SBR-09 is required" in result.output
|
||||
|
||||
|
||||
def test_validate_837_missing_api_key_exits_one(tmp_path, monkeypatch):
|
||||
"""With no API key anywhere (env var unset, keychain empty), the
|
||||
CLI exits 1 and surfaces a clear 'API key not configured' error."""
|
||||
monkeypatch.delenv("CYCLONE_EDIFABRIC_API_KEY", raising=False)
|
||||
sample = tmp_path / "any.x12"
|
||||
sample.write_text("ISA*...~")
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
raise AssertionError("transport should not be called when key is missing")
|
||||
|
||||
_install(handler)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(validate_837_cmd, [str(sample)])
|
||||
assert result.exit_code == 1, result.output
|
||||
assert "API key not configured" in result.output
|
||||
|
||||
|
||||
def test_validate_837_file_not_found_exits_two():
|
||||
"""Click's path validation rejects a missing file before the
|
||||
command body runs (exits 2 via Click's UsageError)."""
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(validate_837_cmd, ["/nonexistent/path.x12", "--api-key", _TEST_KEY])
|
||||
assert result.exit_code == 2, result.output
|
||||
@@ -0,0 +1,275 @@
|
||||
#!/usr/bin/env python3
|
||||
"""SP39 / SP40 sibling-project regen: re-serialize the 363 corrected
|
||||
files through the fixed serializer into ``ingest/corrected-v2/``.
|
||||
|
||||
Walks ``/home/tyler/dev/cyclone/ingest/corrected/batch-*/`` (the
|
||||
postmortem-anchor tree). For each ``.x12`` / ``.txt`` file it:
|
||||
|
||||
1. Loads the clearhouse singleton + CO_TXIX PayerConfig so the
|
||||
serialization has the real submitter + receiver identity (TPID,
|
||||
contact name, email, receiver_name, etc.). These used to fall
|
||||
through to placeholder strings like ``CYCLONE`` / ``CUSTOMER
|
||||
SERVICE`` / ``RECEIVER`` which Edifabric accepts but which the
|
||||
operator rightly refused as not production-ready.
|
||||
2. Re-parses via :func:`cyclone.parsers.parse_837.parse`.
|
||||
3. Re-serializes via
|
||||
:func:`cyclone.parsers.serialize_837.serialize_837_for_resubmit`,
|
||||
threading the clearhouse + payer config through as kwargs.
|
||||
4. Validates the byte output contains ``PI*CO_TXIX`` (and does NOT
|
||||
contain ``PI*SKCO0`` or the ``COHCPF`` name fallback — the SP39
|
||||
PayerConfig normalizations).
|
||||
5. Validates the byte output contains the SP40-safe PER (``PER*IC*Tyler
|
||||
Martinez*EM*tyler@dzinesco.com``) and SBR (``SBR*P*18*******MC``).
|
||||
|
||||
**Filename format** — HCPF X12 File Naming Standards Quick Guide:
|
||||
``tp{tpid}-{transaction_type}-yyyymmddhhmmssSSS-1of1.x12``
|
||||
(example: ``tp11525703-837P-20260708132243505-1of1.x12``).
|
||||
|
||||
Built via :func:`cyclone.edi.filenames.build_outbound_filename` (the
|
||||
single canonical HCPF outbound builder) so the 17-digit
|
||||
millisecond-precision timestamp + "1of1" sequence + ".x12" extension
|
||||
are guaranteed. Each file gets a unique millisecond slot by spreading
|
||||
the batch 1ms apart over the run, so 363 files per regen land in
|
||||
~363ms with millisecond-level uniqueness — well within the
|
||||
single-second "1of1" window the spec allows.
|
||||
|
||||
The output is written to
|
||||
``ingest/corrected-v2/v2-batch-<batch-id>-<N>-claims/``.
|
||||
|
||||
The original ``ingest/corrected/batch-*/`` tree is preserved
|
||||
(postmortem anchor); the new tree is the production-ready set the
|
||||
operator pushes to Gainwell.
|
||||
|
||||
Run::
|
||||
|
||||
cd /home/tyler/dev/unbilled-july2026
|
||||
/home/tyler/dev/cyclone/backend/.venv/bin/python scripts/regen_corrected_files.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Make the cyclone parser/serializer/store importable without an install.
|
||||
sys.path.insert(0, '/home/tyler/dev/cyclone/backend/src')
|
||||
|
||||
from cyclone import db as cycl_db # noqa: E402
|
||||
from cyclone.edi.filenames import build_outbound_filename # noqa: E402
|
||||
from cyclone.parsers.parse_837 import parse as parse_837_text # noqa: E402
|
||||
from cyclone.parsers.payer import PayerConfig # noqa: E402
|
||||
from cyclone.parsers.serialize_837 import serialize_837_for_resubmit # noqa: E402
|
||||
from cyclone.store import store as cycl_store # noqa: E402
|
||||
|
||||
CYCLONE_ROOT = Path('/home/tyler/dev/cyclone')
|
||||
INGEST_ROOT = CYCLONE_ROOT / 'ingest' / 'corrected'
|
||||
OUT_ROOT = CYCLONE_ROOT / 'ingest' / 'corrected-v2'
|
||||
|
||||
|
||||
def _batch_id_from_dirname(d: Path) -> str:
|
||||
"""``batch-<id>-<N>-claims`` -> ``<id>`` (the first hyphen-delimited chunk)."""
|
||||
return d.name.split('-')[1]
|
||||
|
||||
|
||||
def _interchange_index_for(global_counter: int) -> int:
|
||||
"""The serialize helper needs a unique ISA13 per file; offset by
|
||||
1000 so the resulting ISA13s stay well above the seeded fixture
|
||||
range (0-999)."""
|
||||
return 1000 + global_counter
|
||||
|
||||
|
||||
def _load_resubmit_kwargs() -> dict:
|
||||
"""Build the ``serialize_837`` kwargs from the live clearhouse +
|
||||
CO_TXIX payer config. Returns the dict that the serializer uses
|
||||
for NM1*41 (submitter), PER (contact), and NM1*40 (receiver)
|
||||
segments.
|
||||
|
||||
Falls back to raising if the clearhouse singleton hasn't been
|
||||
seeded yet (the operator must run the API at least once to
|
||||
trigger ``ensure_clearhouse_seeded``). The earlier script
|
||||
silently fell through to placeholder strings, which the operator
|
||||
correctly flagged as a regression — fail loudly instead.
|
||||
"""
|
||||
# Bootstrap the DB schema + seed the clearhouse if missing.
|
||||
cycl_db.init_db()
|
||||
cycl_store.ensure_clearhouse_seeded()
|
||||
|
||||
with cycl_db.SessionLocal()() as s:
|
||||
from cyclone.db import ClearhouseORM, PayerConfigORM # noqa: F401
|
||||
ch = s.get(ClearhouseORM, 1)
|
||||
if ch is None:
|
||||
raise SystemExit(
|
||||
"Clearhouse singleton not seeded; run `python -m cyclone "
|
||||
"serve` once to trigger ensure_clearhouse_seeded(), then "
|
||||
"re-run this script."
|
||||
)
|
||||
# The 837P config for CO_TXIX carries submitter + receiver info.
|
||||
pco = s.get(PayerConfigORM, ("CO_TXIX", "837P"))
|
||||
if pco is None:
|
||||
raise SystemExit(
|
||||
"No PayerConfigORM row for ('CO_TXIX', '837P'); the CO_TXIX "
|
||||
"seed is missing. Check ensure_clearhouse_seeded()."
|
||||
)
|
||||
|
||||
cfg = pco.config_json or {}
|
||||
|
||||
# Map config_json -> serializer kwargs. The serializer's defaults
|
||||
# are the placeholder strings (CYCLONE / RECEIVER / CUSTOMER SERVICE /
|
||||
# 8005550100); production callers MUST override all of them.
|
||||
return {
|
||||
"sender_id": ch.tpid, # e.g. "11525703"
|
||||
"submitter_name": ch.submitter_name, # e.g. "Dzinesco"
|
||||
"submitter_contact_name": ch.submitter_contact_name, # "Tyler Martinez"
|
||||
"submitter_contact_email": ch.submitter_contact_email, # "tyler@dzinesco.com"
|
||||
"receiver_name": cfg.get("receiver_name") or pco.receiver_name,
|
||||
"receiver_id": cfg.get("receiver_id") or pco.receiver_id,
|
||||
"claim_filing_indicator_code": cfg.get("sbr09_default") or "MC",
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Walk every ``batch-*/.x12`` under INGEST_ROOT, re-serialize
|
||||
through the fixed serializer with real submitter + receiver info,
|
||||
validate the output, and write to OUT_ROOT. Returns 0 on a clean
|
||||
run, 1 if any file failed or validation missed.
|
||||
|
||||
Filenames follow the HCPF X12 File Naming Standards Quick Guide:
|
||||
``tp{tpid}-837P-yyyymmddhhmmssSSS-1of1.x12``. The 17-digit
|
||||
millisecond-precision timestamp comes from
|
||||
:func:`build_outbound_filename` so the spec format is guaranteed
|
||||
(no ``-cyc-`` segment, no trailing ``-{counter}-`` suffix, no
|
||||
abbreviation). Each file's timestamp is offset by 1ms so all 363
|
||||
files land in distinct millisecond slots.
|
||||
"""
|
||||
payer_cfg_co = PayerConfig.co_medicaid()
|
||||
resubmit_kwargs = _load_resubmit_kwargs()
|
||||
tpid = resubmit_kwargs["sender_id"] # ISA06 — used in the filename
|
||||
assert tpid.isdigit(), f"TPID must be digits, got {tpid!r}"
|
||||
|
||||
print("Resubmit kwargs (real submitter + receiver):")
|
||||
for k, v in resubmit_kwargs.items():
|
||||
print(f" {k}={v!r}")
|
||||
print(f"Filename template: tp{tpid}-837P-yyyymmddhhmmssSSS-1of1.x12")
|
||||
print(f" (built via cyclone.edi.filenames.build_outbound_filename)", flush=True)
|
||||
|
||||
global_counter = 0
|
||||
ok = 0
|
||||
failed = 0
|
||||
failures: list[dict] = []
|
||||
|
||||
batch_dirs = sorted(p for p in INGEST_ROOT.iterdir()
|
||||
if p.is_dir() and p.name.startswith('batch-'))
|
||||
print(f'Found {len(batch_dirs)} batch dirs under {INGEST_ROOT}', flush=True)
|
||||
|
||||
for batch_dir in batch_dirs:
|
||||
batch_id = _batch_id_from_dirname(batch_dir)
|
||||
files = sorted(batch_dir.glob('*.x12')) + sorted(batch_dir.glob('*.txt'))
|
||||
if not files:
|
||||
continue
|
||||
n = len(files)
|
||||
out_dir = OUT_ROOT / f'v2-batch-{batch_id}-{n}-claims'
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
print(f' {batch_id[:12]}... ({n} files) -> {out_dir}', flush=True)
|
||||
|
||||
for src in files:
|
||||
global_counter += 1
|
||||
# Each file's filename = real HCPF format with a unique
|
||||
# millisecond slot (1ms stride from now onward).
|
||||
fname = build_outbound_filename(
|
||||
tpid, "837P",
|
||||
now_mt=_now_mt_plus_ms(global_counter),
|
||||
)
|
||||
|
||||
raw = src.read_text()
|
||||
try:
|
||||
parsed = parse_837_text(raw, payer_cfg_co)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
failed += 1
|
||||
failures.append({'file': src.name, 'reason': f'parse: {exc}'})
|
||||
continue
|
||||
if not parsed.claims:
|
||||
failed += 1
|
||||
failures.append({'file': src.name, 'reason': 'no claims parsed'})
|
||||
continue
|
||||
try:
|
||||
text = serialize_837_for_resubmit(
|
||||
parsed.claims[0],
|
||||
interchange_index=_interchange_index_for(global_counter),
|
||||
**resubmit_kwargs,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
failed += 1
|
||||
failures.append({'file': src.name, 'reason': f'serialize: {exc}'})
|
||||
continue
|
||||
content = text.encode()
|
||||
if b'PI*CO_TXIX' not in content:
|
||||
failed += 1
|
||||
failures.append({'file': src.name, 'reason': 'PI*CO_TXIX missing'})
|
||||
continue
|
||||
if b'PI*SKCO0' in content or b'*COHCPF****' in content:
|
||||
failed += 1
|
||||
failures.append({'file': src.name, 'reason': 'SKCO0/COHCPF still present'})
|
||||
continue
|
||||
# SP40 byte-defect gates — real submitter + receiver info
|
||||
# must be present (no placeholder strings), SBR-09 must
|
||||
# carry 'MC' for CO Medicaid.
|
||||
submitter_contact_email = (
|
||||
resubmit_kwargs["submitter_contact_email"] or ""
|
||||
).encode()
|
||||
receiver_id = (resubmit_kwargs["receiver_id"] or "").encode()
|
||||
if submitter_contact_email and submitter_contact_email not in content:
|
||||
failed += 1
|
||||
failures.append({
|
||||
'file': src.name,
|
||||
'reason': f'submitter_contact_email '
|
||||
f'{submitter_contact_email!r} missing',
|
||||
})
|
||||
continue
|
||||
if receiver_id and receiver_id not in content:
|
||||
failed += 1
|
||||
failures.append({
|
||||
'file': src.name,
|
||||
'reason': f'receiver_id {receiver_id!r} missing',
|
||||
})
|
||||
continue
|
||||
if b'SBR*P*18*******MC' not in content:
|
||||
failed += 1
|
||||
failures.append({'file': src.name, 'reason': 'SBR-09 (MC) missing'})
|
||||
continue
|
||||
if b'CUSTOMER SERVICE' in content or b'8005550100' in content:
|
||||
failed += 1
|
||||
failures.append({'file': src.name, 'reason': 'placeholder strings leaked'})
|
||||
continue
|
||||
# Final HCPF-spec filename validation guard (cheaper to
|
||||
# re-check the filename than to discover a malformed
|
||||
# filename on upload to Gainwell).
|
||||
from cyclone.edi.filenames import is_outbound_filename
|
||||
if not is_outbound_filename(fname):
|
||||
failed += 1
|
||||
failures.append({'file': src.name,
|
||||
'reason': f'filename {fname!r} does not match '
|
||||
f'HCPF OUTBOUND_RE (SPEC VIOLATION)'})
|
||||
continue
|
||||
(out_dir / fname).write_text(text)
|
||||
ok += 1
|
||||
|
||||
print(f'\nDONE: ok={ok} failed={failed}', flush=True)
|
||||
if failures:
|
||||
print('First failures:')
|
||||
for f in failures[:5]:
|
||||
print(f' {f}')
|
||||
return 0 if failed == 0 else 1
|
||||
|
||||
|
||||
def _now_mt_plus_ms(ms_offset: int):
|
||||
"""Return the current Mountain Time as a tz-aware datetime, offset
|
||||
by ``ms_offset`` milliseconds. Used to spread the regen run across
|
||||
distinct millisecond slots so each file's filename is unique
|
||||
within the spec's 17-digit yyyymmddhhmmssSSS window."""
|
||||
from datetime import datetime, timedelta
|
||||
from zoneinfo import ZoneInfo
|
||||
return datetime.now(ZoneInfo("America/Denver")) + timedelta(milliseconds=ms_offset)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
"""SP39 sibling-project test: ``regen_corrected_files`` produces
|
||||
byte-correct output for the three shape variants (already-correct,
|
||||
SKCO0-buggy, CO_BHA)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
|
||||
from regen_corrected_files import _interchange_index_for # noqa: E402
|
||||
|
||||
|
||||
def test_interchange_index_for_global_counter():
|
||||
"""The helper just adds 1000 — pin that to detect off-by-one
|
||||
drift if anyone tweaks the offset (it's the safety margin above
|
||||
the seeded ISA13 range in the test fixtures)."""
|
||||
assert _interchange_index_for(1) == 1001
|
||||
assert _interchange_index_for(363) == 1363
|
||||
assert _interchange_index_for(0) == 1000
|
||||
+173
-48
@@ -4,8 +4,8 @@
|
||||
>
|
||||
> **Companion docs:**
|
||||
> - **Requirements:** [`docs/REQUIREMENTS.md`](REQUIREMENTS.md) — what Cyclone does (FRs + NFRs + DoD + traceability).
|
||||
> - **Specs:** [`docs/superpowers/specs/`](superpowers/specs/) — design specs per sub-project (22 specs).
|
||||
> - **Plans:** [`docs/superpowers/plans/`](superpowers/plans/) — TDD-shaped implementation plans per sub-project (27 plans).
|
||||
> - **Specs:** [`docs/superpowers/specs/`](superpowers/specs/) — design specs per sub-project (42 specs).
|
||||
> - **Plans:** [`docs/superpowers/plans/`](superpowers/plans/) — TDD-shaped implementation plans per sub-project (37 plans).
|
||||
>
|
||||
> **Scope of this doc:** architecture and design. The spec files own per-SP design; the plan files own the task order. This doc explains the *whole* — the boundaries between modules, the data flow across them, the lifecycle of an inbound file from upload to reconciled claim.
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
|
||||
## 1. System overview
|
||||
|
||||
Cyclone is a self-hosted X12 EDI claims-management suite for a single billing office. The first deployment target is Colorado Medicaid (one operator, one trading partner). The system is local-only on purpose: the backend binds to `127.0.0.1` and requires login (bcrypt + HttpOnly session cookie). The threat model is still a stolen or imaged drive, not a remote attacker — SQLCipher at rest and the macOS Keychain handle that. The auth boundary is the HTTP layer; the file-system posture is unchanged. The SP23 product fork changes the threat model to "remote operator on LAN."
|
||||
Cyclone is a self-hosted X12 EDI claims-management suite for a single billing office. The first deployment target is Colorado Medicaid (one operator, one trading partner). LAN-only by design: the backend always binds to `0.0.0.0:8000` and requires login (bcrypt + HttpOnly session cookie per SP23/24). Reachability is controlled by the host firewall / compose port publishing, not the bind address. The threat model is still a stolen or imaged drive, not a remote attacker — SQLCipher at rest and the macOS Keychain handle that. The auth boundary is the HTTP layer; the file-system posture is unchanged.
|
||||
|
||||
Cyclone processes three X12 transaction pairs end-to-end:
|
||||
|
||||
@@ -46,17 +46,22 @@ The whole stack is a single Python process (FastAPI + uvicorn on port 8000) plus
|
||||
+----------------------------------------------------------+
|
||||
| Python process: `python -m cyclone serve` |
|
||||
| |
|
||||
| FastAPI app (uvicorn, 127.0.0.1:8000) |
|
||||
| FastAPI app (uvicorn, 0.0.0.0:8000; SP23) |
|
||||
| +-----------------------------------------------------+ |
|
||||
| | cyclone.api:app (3,548 LOC) | |
|
||||
| | - routers: claims, remits, providers, acks, | |
|
||||
| | activity, inbox, upload, download, batches, | |
|
||||
| | admin, scheduler, health, ta1_acks | |
|
||||
| | cyclone.api:app (~377 LOC, post-SP36 thin shell) | |
|
||||
| | - mounts 22 routers under cyclone.api_routers/ | |
|
||||
| | (acks, activity, admin, batches, claim_acks, | |
|
||||
| | claims, clearhouse, config, dashboard, | |
|
||||
| | eligibility, health, inbox, parse, payers, | |
|
||||
| | providers, rebill, reconciliation, remittances, | |
|
||||
| | submission, ta1_acks) | |
|
||||
| | - matrix_gate dependency on every gated router | |
|
||||
| | (SP24 auth) | |
|
||||
| | - live-tail: NDJSON pubsub (pubsub.py) | |
|
||||
| | - lifespan: init db -> seed SP9 -> start SP16 + | |
|
||||
| | SP17 schedulers -> load PayerConfig cache | |
|
||||
| | - lifespan: init db -> seed SP9 -> start SP16 | |
|
||||
| | + SP17 schedulers -> load PayerConfig cache | |
|
||||
| +-----------------------------------------------------+ |
|
||||
| | cyclone.store (facade, 2,423 LOC) | |
|
||||
| | cyclone.store (facade, post-SP21 split: 16 modules)| |
|
||||
| | - add/get/list/manual_match/iter_claims/... | |
|
||||
| | - delegates to: db.SessionLocal() | |
|
||||
| +-----------------------------------------------------+ |
|
||||
@@ -64,8 +69,13 @@ The whole stack is a single Python process (FastAPI + uvicorn on port 8000) plus
|
||||
| | 277CA: tokenize -> segmentize -> model -> validate | |
|
||||
| | -> write to store) | |
|
||||
| +-----------------------------------------------------+ |
|
||||
| | cyclone.reconcile, scoring, batch_diff, audit_log, | |
|
||||
| | backup_service, scheduler, db_crypto, secrets | |
|
||||
| | cyclone.auth.* (SP24: bcrypt + sessions + matrix | |
|
||||
| | gate), cyclone.handlers.* (SP27: per-kind parse), | |
|
||||
| | cyclone.rebill.* (SP41: rebill pipeline), | |
|
||||
| | cyclone.reissue.* (SP24: reissue CLI), | |
|
||||
| | reconcile, scoring, batch_diff, audit_log, | |
|
||||
| | backup_service, scheduler, db_crypto, secrets, | |
|
||||
| | edifabric (SP40), submission (SP37) | |
|
||||
| +-----------------------------------------------------+ |
|
||||
+----------------------------------------------------------+
|
||||
| | |
|
||||
@@ -93,7 +103,7 @@ The `cyclone-pipeline/` sibling project (not in this repo) is a separate Python
|
||||
|---|---|---|---|
|
||||
| Language | Python | 3.11+ (3.13 in dev) | `requires-python = ">=3.11"` |
|
||||
| Web framework | FastAPI | ≥0.110, <1 | `python -m cyclone serve` boots uvicorn |
|
||||
| ASGI server | uvicorn[standard] | ≥0.27, <1 | `--host 127.0.0.1 --port 8000` |
|
||||
| ASGI server | uvicorn[standard] | ≥0.27, <1 | `--host 0.0.0.0 --port 8000` (CYCLONE_HOST/CYCLONE_PORT override) |
|
||||
| ORM | SQLAlchemy | ≥2.0, <3 | 2.x style; engine + SessionLocal() function-accessor |
|
||||
| Validation | Pydantic | ≥2.6, <3 | v2 model_validator + ConfigDict |
|
||||
| CLI | Click | ≥8.1, <9 | `cyclone` console script |
|
||||
@@ -135,38 +145,75 @@ The `cyclone-pipeline/` sibling project (not in this repo) is a separate Python
|
||||
|
||||
### 4.1 Package layout
|
||||
|
||||
The `cyclone` package is a single namespace rooted at `backend/src/cyclone/`. 22 top-level modules + 4 subpackages. The store (SP21 in-flight split) is the largest; everything else is small and focused.
|
||||
The `cyclone` package is a single namespace rooted at `backend/src/cyclone/`. After SP21 (store split), SP24 (auth + reissue), SP27 (handlers), SP36 (api-routers split), and SP41 (rebill pipeline), the package has **22 top-level modules + 10 subpackages**. The two largest files are now `cyclone/api_routers/admin.py` (~640 LOC, the admin operator surface) and the various pipeline modules in `cyclone/rebill/`. `api.py` is a ~377-line shell (post-SP36) and `store.py` no longer exists (post-SP21).
|
||||
|
||||
```
|
||||
backend/src/cyclone/
|
||||
├── __main__.py — entry point; `python -m cyclone` → CLI, `python -m cyclone serve` → uvicorn
|
||||
├── __init__.py — `__version__`, package init
|
||||
├── api.py — FastAPI app, route includes, lifespan (3,548 LOC — the only large file)
|
||||
├── api_helpers.py — request/response formatters, error → HTTP mapping
|
||||
├── api_routers/
|
||||
│ ├── __init__.py
|
||||
│ ├── acks.py — 999/TA1/271/277CA acknowledgments
|
||||
│ ├── admin.py — admin-only endpoints (validate-provider, scheduler, backup)
|
||||
│ ├── health.py — GET /api/health (SP19 deep snapshot)
|
||||
│ └── ta1_acks.py — TA1-specific (split from acks for the SP3 lifecycle)
|
||||
├── api.py — FastAPI app, route mounting (~377 LOC, post-SP36 thin shell)
|
||||
├── api_helpers.py — request/response formatters, error → HTTP mapping, NDJSON helpers
|
||||
├── api_routers/ — SP36: 22 per-resource FastAPI routers + _shared.py
|
||||
│ ├── __init__.py — registry: `routers: list[APIRouter]`
|
||||
│ ├── _shared.py — cross-router helpers
|
||||
│ ├── acks.py — 999/277CA acks list / stream / detail
|
||||
│ ├── activity.py — activity feed + stream
|
||||
│ ├── admin.py — admin-only operator surface (validate-provider, scheduler, backup, users, validate-837)
|
||||
│ ├── batches.py — batch list / detail / ZIP export
|
||||
│ ├── claim_acks.py — SP28 claim ↔ ack auto-link endpoints
|
||||
│ ├── claims.py — claim register / stream / detail / serialize-837 / line-reconciliation
|
||||
│ ├── clearhouse.py — singleton clearhouse config + SFTP submit
|
||||
│ ├── config.py — payer-config read views
|
||||
│ ├── dashboard.py — GET /api/dashboard/kpis
|
||||
│ ├── eligibility.py — 270 build / 271 parse
|
||||
│ ├── health.py — GET /api/health (public)
|
||||
│ ├── inbox.py — 5-lane triage surface (SP14)
|
||||
│ ├── parse.py — SP35 envelope-guarded parse-* endpoints
|
||||
│ ├── payers.py — payer-level rollup
|
||||
│ ├── providers.py — distinct providers + configured providers
|
||||
│ ├── rebill.py — SP41 POST/GET /api/admin/rebill-from-835
|
||||
│ ├── reconciliation.py — manual match + batch diff
|
||||
│ ├── remittances.py — remittance register / summary / stream / detail
|
||||
│ ├── submission.py — SP37 POST /api/submit-batch
|
||||
│ └── ta1_acks.py — TA1 list / stream / detail
|
||||
├── audit_log.py — SHA-256 hash-chained append-only log (SP11)
|
||||
├── auth/ — SP24: bcrypt + sessions + RBAC matrix_gate
|
||||
│ ├── __init__.py — `AUTH_DISABLED` flag + module surface
|
||||
│ ├── bootstrap.py — first-admin env-var bootstrap
|
||||
│ ├── cli.py — `cyclone users` group
|
||||
│ ├── deps.py — FastAPI deps + matrix_gate
|
||||
│ ├── permissions.py — role matrix (admin/user/viewer)
|
||||
│ ├── rate_limit.py — per-username login throttle (5 fails / 5 min)
|
||||
│ ├── routes.py — /api/auth/{login,logout,me}
|
||||
│ ├── sessions.py — 24h sliding expiry
|
||||
│ └── users.py — user CRUD + bcrypt password hashing
|
||||
├── backup.py — SP17 primitives: derive_key, encrypt, decrypt, BackupFile
|
||||
├── backup_scheduler.py — SP17 BackupScheduler wrapper
|
||||
├── backup_service.py — SP17 coordinator: create/list/restore/verify/prune
|
||||
├── batch_diff.py — 837 ↔ 835 diff (SP4)
|
||||
├── claim_acks.py — SP28 pure readers over session + parse result
|
||||
├── clearhouse/
|
||||
│ └── __init__.py — Clearhouse, SftpClient (SP9, SP13)
|
||||
├── cli.py — Click CLI: `cyclone serve`, `cyclone validate-npi`, `cyclone backup`, etc.
|
||||
├── cli.py — Click CLI: 15+ subcommands (see cyclone-cli skill)
|
||||
├── db.py — SQLAlchemy engine, SessionLocal, ORM models, init_db, reinit_engine
|
||||
├── db_crypto.py — SP12 SQLCipher connect-creator + is_encryption_enabled()
|
||||
├── db_migrate.py — runs the 12 SQL migrations in order
|
||||
├── db_migrate.py — runs the 23 SQL migrations in order
|
||||
├── edifabric.py — SP40 Edifabric /v2/x12/{read,validate} HTTP client
|
||||
├── edi/
|
||||
│ └── filenames.py — X12 filename convention helpers (SP9)
|
||||
├── handlers/ — SP27: per-file-type inbound parse handlers
|
||||
│ ├── __init__.py — HANDLERS registry + register_handlers()
|
||||
│ ├── _ack_id.py — synthetic batch id + AK5-count helpers
|
||||
│ ├── handle_277ca.py — 277CA parse → persist + payer-rejected stamp
|
||||
│ ├── handle_835.py — 835 parse → validate → persist + reconcile (two-phase)
|
||||
│ ├── handle_999.py — 999 parse → apply rejections
|
||||
│ ├── handle_result.py — HandleResult dataclass
|
||||
│ └── handle_ta1.py — TA1 parse → persist envelope ack + envelope-link batches
|
||||
├── inbox_lanes.py — 5-lane Inbox computation (SP14)
|
||||
├── inbox_state.py — claim ↔ inbox state mapping helpers
|
||||
├── inbox_state_277ca.py — SP10 277CA inbox state integration
|
||||
├── logging_config.py — SP18 structured JSON + PII scrubber
|
||||
├── migrations/ — 12 SQL files (0001_initial through 0012_backups)
|
||||
├── migrations/ — 23 SQL files (0001_initial through 0023_visits)
|
||||
├── npi.py — SP20 NPI Luhn + Tax ID validation
|
||||
├── parsers/ — X12 parser subpackage
|
||||
│ ├── parse_837.py — 837P ingest
|
||||
@@ -187,12 +234,44 @@ backend/src/cyclone/
|
||||
├── payers.py — Payer ORM row accessor (SP9)
|
||||
├── providers.py — Payer / Clearhouse / Provider ORM row DTOs (SP9)
|
||||
├── pubsub.py — NDJSON live-tail EventBus
|
||||
├── reconcile.py — 835 → 837 auto-reconciliation engine
|
||||
├── scheduler.py — SP16 MFT polling scheduler
|
||||
├── rebill/ — SP41: in-window rebill pipeline (13 modules)
|
||||
│ ├── __init__.py — package surface
|
||||
│ ├── carc_filter.py — CARC-aware filter (EXCLUDED_CARCS / REVIEW_CARCS)
|
||||
│ ├── parse_835_svc.py — SVC-level 835 reparse w/ member_id
|
||||
│ ├── pipeline_a.py — denied/partial → frequency-7 replacement 837P
|
||||
│ ├── pipeline_b.py — NOT_IN_835 → fresh 837Ps batched by (member_id, ISO-week)
|
||||
│ ├── pull_999_acks.py — 999-ack dump + classify NOT_IN_835
|
||||
│ ├── reconcile.py — visit-to-835 join (member, procedure, DOS) with 5% tolerance
|
||||
│ ├── run.py — orchestrator (run_rebill)
|
||||
│ ├── spot_check.py — single-claim well-formed 837P from VisitRow
|
||||
│ ├── spot_check_pipeline.py — top-N visits → well-formed 837P files
|
||||
│ ├── spot_check_validate.py — submit to Edifabric /v2/x12/validate
|
||||
│ ├── summary.py — summary CSV generator
|
||||
│ ├── timely_filing.py — 120-day DOS age gate
|
||||
│ └── visits_store.py — load AxisCare visits CSV into visits table
|
||||
├── reconcile.py — 835 → 837 auto-reconciliation engine (SP31 pcn-exact strategy)
|
||||
├── reissue/ — SP24: offline 837P reissue workflow
|
||||
│ ├── __init__.py
|
||||
│ └── core.py — parse_inputs / emit_outputs / zip_outputs / ig_correctness_check
|
||||
├── scheduler.py — SP16 MFT polling scheduler (SP27 per-op timeout)
|
||||
├── scoring.py — 4-field lane scoring (40/25/20/15, hard-coded — backlog item)
|
||||
├── secrets.py — Keychain access wrapper (SP9)
|
||||
├── secrets.py — Keychain access wrapper (SP9, SP26 _FILE tier)
|
||||
├── security.py — security headers, HealthSnapshot, deep health probe (SP19)
|
||||
└── store.py — CycloneStore facade (2,423 LOC; SP21 in-flight split)
|
||||
├── seed_cli.py — `cyclone seed [--count N] [--reset] [--status]` dev seeder
|
||||
├── store/ — SP21: split CycloneStore facade (16 modules)
|
||||
│ ├── __init__.py — CycloneStore class + module-level `store` singleton
|
||||
│ ├── acks.py / batches.py / backfill.py / backups.py
|
||||
│ ├── claim_acks.py / claim_detail.py / exceptions.py
|
||||
│ ├── inbox.py / kpis.py / orm_builders.py / providers.py
|
||||
│ ├── records.py / resubmissions.py / submission_dedup.py
|
||||
│ ├── ui.py — ORM → UI JSON serializers
|
||||
│ └── write.py — write-path helpers + event publishing
|
||||
└── submission/ — SP37: canonical outbound path + SP25/SP26 helpers
|
||||
├── __init__.py — submit_file, SubmitOutcome, SubmitResult
|
||||
├── core.py — parse → DB-write → SFTP-upload per file
|
||||
├── result.py — SubmitOutcome enum + SubmitResult dataclass
|
||||
├── recover.py — SP25: offline parse + DB-write for stranded files
|
||||
└── bulk_ingest.py — SP26: offline walk + ingest for 999/TA1/277CA/837P/835
|
||||
```
|
||||
|
||||
### 4.2 Module responsibility map (one line each)
|
||||
@@ -242,7 +321,7 @@ The store is the single read/write surface for the database. Every endpoint that
|
||||
- `store.list_unmatched(kind="both")`
|
||||
- `store.export_*()` — CSV/JSON dumpers
|
||||
|
||||
All persistence flows through SQLAlchemy sessions via `db.SessionLocal()()`. The engine is single-connection-per-request via FastAPI dependency injection. SP21 splits `store.py` (2,423 LOC) into a `cyclone/store/` subpackage with one module per domain (batches, inbox, claim_detail, acks, backups, providers, etc.) and a thin facade that delegates every method to its domain module. After SP21, the public API of `cyclone.store` is unchanged; the only thing that changes is the file layout.
|
||||
All persistence flows through SQLAlchemy sessions via `db.SessionLocal()()`. The engine is single-connection-per-request via FastAPI dependency injection. After SP21, `store.py` is split into a `cyclone/store/` subpackage with one module per domain (acks, batches, backfill, backups, claim_acks, claim_detail, exceptions, inbox, kpis, orm_builders, providers, records, resubmissions, submission_dedup, ui, write) plus a thin facade that delegates every method to its domain module. The public API of `cyclone.store` is unchanged.
|
||||
|
||||
### 4.4 The parser pipeline
|
||||
|
||||
@@ -327,21 +406,49 @@ There is no Redux, no Context for global state, no MobX. The combination of reac
|
||||
|
||||
### 5.4 Live tail (NDJSON pubsub)
|
||||
|
||||
The Activity feed and the TailStatusPill share a single connection: `GET /api/tail?since=<last_event_id>`. The server returns `Content-Type: application/x-ndjson` and streams one JSON object per line. Each line is one of:
|
||||
The Claims, Remittances, Activity, Acks, and TA1-Acks pages each open their own NDJSON connection to the matching `/api/<resource>/stream` endpoint. The server returns `Content-Type: application/x-ndjson` and streams one JSON object per line. Each line is one of:
|
||||
|
||||
- `{kind: "heartbeat", ts: "..."}` — every 5s when no events
|
||||
- `{kind: "event", id: <int>, event_type: "...", payload: {...}}` — when something happens
|
||||
- `{kind: "error", message: "..."}` — server-initiated close (rare)
|
||||
- `{type: "item", data: {...}}` — one row from the snapshot (initial batch) or a live event
|
||||
- `{type: "snapshot_end", data: {count: N}}` — the snapshot has finished
|
||||
- `{type: "heartbeat", data: {ts: "..."}}` — every `CYCLONE_TAIL_HEARTBEAT_S` seconds (default 15s) when idle
|
||||
- `{type: "item_dropped", data: {id: ...}}` — rare: the subscriber queue overflowed
|
||||
- `{type: "error", data: {message: "..."}}` — server-initiated close (rare)
|
||||
|
||||
The client uses `fetch` + `ReadableStream` to consume the body and dispatches each line to the relevant `useQuery` cache via `queryClient.setQueryData`. The connection auto-reconnects on `error` with exponential backoff. See [live-tail spec](superpowers/specs/2026-06-20-cyclone-live-tail-design.md) for the wire format and reconnection policy.
|
||||
The client uses `fetch` + `ReadableStream` to consume the body; each `item` event dispatches into `useTailStore` (zustand). The connection auto-reconnects on `error` with backoff `1s → 2s → 4s → 8s → 16s → 30s` capped. The client flips to `stalled` after 30s of total silence (`STALL_TIMEOUT_MS = 30_000` in `src/hooks/useTailStream.ts`). See [live-tail spec](superpowers/specs/2026-06-20-cyclone-live-tail-design.md) for the wire format and reconnection policy. SP25 added `ack_received` and `ta1_ack_received` event kinds plus `/api/acks/stream` and `/api/ta1-acks/stream` endpoints.
|
||||
|
||||
---
|
||||
|
||||
## 6. Data model
|
||||
|
||||
### 6.1 Migrations 0001–0014
|
||||
### 6.1 Migrations 0001–0023
|
||||
|
||||
The schema evolves in ordered SQL files under `backend/src/cyclone/migrations/`. Each file starts with `-- version: N` and is applied in order by `db_migrate.py`. There is no migration framework — `db_migrate.py` is a 30-line file that checks the `schema_version` row and applies everything newer. The first 12 ship on `main`; 13 + 14 are the auth migrations that landed in `main` on 2026-06-23.
|
||||
The schema evolves in ordered SQL files under `backend/src/cyclone/migrations/`. Each file starts with `-- version: N` and is applied in order by `db_migrate.py`. There is no migration framework — `db_migrate.py` is a 30-line file that checks the `schema_version` row and applies everything newer. All 23 ship on `main`:
|
||||
|
||||
| # | File | SP | One-line summary |
|
||||
|---|---|---|---|
|
||||
| 0001 | `0001_initial.sql` | — | 6 tables (`batches`, `claims`, `remittances`, `cas_adjustments`, `matches`, `activity_events`) + 11 indexes |
|
||||
| 0002 | `0002_acks.sql` | — | `acks` table + index on `source_batch_id` |
|
||||
| 0003 | `0003_drop_claims_remits_unique_constraints.sql` | — | drops UNIQUE constraints that blocked re-ingest of duplicates |
|
||||
| 0004 | `0004_rejections_and_state_history.sql` | — | adds `rejection_reason`, `rejected_at`, `resubmit_count`, `state_changed_at` to `claims` |
|
||||
| 0005 | `0005_create_ta1_acks.sql` | — | `ta1_acks` table + indexes on `source_batch_id`, `ack_code` |
|
||||
| 0006 | `0006_line_reconciliation.sql` | — | `service_line_payments` + `line_reconciliation` tables for the 837 vs 835 side-by-side view |
|
||||
| 0007 | `0007_providers_payers_clearhouse.sql` | SP9 | `providers`, `payers`, `payer_configs`, `clearhouse` tables |
|
||||
| 0008 | `0008_payer_rejected_columns.sql` | SP10 | `payer_rejected_*` columns on `claims` + `two77ca_acks` table |
|
||||
| 0009 | `0009_audit_log.sql` | SP11 | tamper-evident `audit_log` table |
|
||||
| 0010 | `0010_payer_rejected_acknowledged.sql` | SP14 | `payer_rejected_acknowledged_at` + `payer_rejected_acknowledged_actor` + index |
|
||||
| 0011 | `0011_processed_inbound_files.sql` | SP16 | `processed_inbound_files` dedup table for SFTP polling |
|
||||
| 0012 | `0012_backups.sql` | SP17 | `db_backups` table for the encrypted backup subsystem |
|
||||
| 0013 | `0013_auth_users_and_sessions.sql` | SP24 | `users` and `sessions` tables |
|
||||
| 0014 | `0014_audit_log_user_id.sql` | SP24 | `user_id` column on `audit_log` + index |
|
||||
| 0015 | `0015_drop_claims_unique_constraint.sql` | SP22 | rebuilds `claims` to drop a UNIQUE constraint that blocked a specific re-ingest pattern |
|
||||
| 0016 | `0016_claims_matched_remittance_id_index.sql` | SP22 | index on `claims.matched_remittance_id` for inbox join speed |
|
||||
| 0017 | `0017_backfill_claim_patient_control_number.sql` | — | backfills `claim.patient_control_number` from raw JSON for legacy rows |
|
||||
| 0018 | `0018_claim_acks.sql` | SP28 | `claim_acks` join table (claim ↔ 999/277CA/TA1) + 3 indexes |
|
||||
| 0019 | `0019_add_rendering_and_service_provider_npis.sql` | SP32 | typed rendering-NPI / service-provider-NPI columns on `claims` + `remittances` |
|
||||
| 0020 | `0020_add_batch_txn_set_control_number.sql` | SP37 | `transaction_set_control_number` column on `batches` (ST02) |
|
||||
| 0021 | `0021_resubmissions.sql` | SP39 | `resubmissions` audit table + indexes |
|
||||
| 0022 | `0022_submission_dedup.sql` | SP37 | `submission_records` table for duplicate-submission detection |
|
||||
| 0023 | `0023_visits.sql` | SP41 | `visits` table (AxisCare visits CSV → DB) + 3 indexes |
|
||||
|
||||
| # | File | What it adds |
|
||||
|---|---|---|
|
||||
@@ -360,7 +467,7 @@ The schema evolves in ordered SQL files under `backend/src/cyclone/migrations/`.
|
||||
| 13 | `0013_auth_users_and_sessions.sql` | `users` + `sessions` tables, `users.password_hash` (bcrypt) — landed 2026-06-23 |
|
||||
| 14 | `0014_audit_log_user_id.sql` | `audit_log.user_id` FK to `users.id` (forward reference to 0013) — landed 2026-06-23 |
|
||||
|
||||
The SP22 parse-then-decide migrations (drop claims UNIQUE constraint + relax PK to `(batch_id, id)`) are renumbered to 0015 + 0016 on the `claims-unique-fix` worktree so the auth migrations stay at the front of the chain; they'll keep those numbers when SP22 lands.
|
||||
The SP22 parse-then-decide migrations landed as `0015` + `0016` (drop claims UNIQUE constraint + add `matched_remittance_id` index).
|
||||
|
||||
### 6.2 ERD (high level)
|
||||
|
||||
@@ -485,8 +592,7 @@ The idempotency table is the linchpin: a crash between "read file" and "INSERT p
|
||||
|
||||
When an 835 lands, the writer calls `reconcile.apply_payment(claim, remit, strategy)` for each `(claim, remit)` pair in the file. The strategy is one of:
|
||||
|
||||
- `auto:pcn_npi_charge` — patient_control_number + provider_npi + charge_amount match (the strongest)
|
||||
- `auto:pcn_charge` — patient_control_number + charge_amount (the most common in practice)
|
||||
- `pcn-exact` — at least 2 of (patient_control_number, charge_amount, provider_npi) match exactly (PCN case-insensitive with leading-zero normalization). This is the SP31 strategy and replaces the older `auto:pcn_npi_charge` / `auto:pcn_charge` audit labels. `_content_keys_match` is the predicate.
|
||||
- `manual` — the operator uses `/reconciliation` to pair by hand; the strategy is recorded for audit
|
||||
|
||||
Each auto-match creates a `matches` row + transitions the claim state + emits an `activity_events` row + emits an `audit_log` row. The match rate is visible in the Dashboard KPI tile.
|
||||
@@ -570,7 +676,7 @@ The full list is in [REQUIREMENTS.md §6.3 traceability matrix](REQUIREMENTS.md)
|
||||
|
||||
```
|
||||
{"kind":"event","id":1234,"event_type":"claim.submitted","payload":{"claim_id":"...","batch_id":"...","ts":"2026-06-23T..."}}
|
||||
{"kind":"event","id":1235,"event_type":"claim.matched","payload":{"claim_id":"...","remit_id":"...","strategy":"auto:pcn_npi_charge"}}
|
||||
{"type":"item","data":{"id":"CLM-...","remit_id":"RM-...","strategy":"pcn-exact","matched_at":"..."}}
|
||||
{"kind":"heartbeat","ts":"2026-06-23T..."}
|
||||
{"kind":"event","id":1236,"event_type":"claim.payer_rejected","payload":{"claim_id":"...","status_code":"A7","source_277ca_id":"..."}}
|
||||
```
|
||||
@@ -602,7 +708,7 @@ See §4.6. The chain is checked on every `/api/health` request and exposes `stat
|
||||
- **In transit (SFTP)** — TLS to the trading partner; no plaintext EDI over the wire.
|
||||
- **In the UI** — no PHI is shown on the Dashboard or Inbox (only counts + KPIs). The Claim drawer and Remit drawer show full PHI; these pages are not in the route map's preview and require an explicit navigation click.
|
||||
- **In logs** — scrubbed by key name (see §9.1).
|
||||
- **In analytics** — no analytics. No third-party tracking. No telemetry. The frontend makes no requests outside `127.0.0.1:8000`.
|
||||
- **In analytics** — no analytics. No third-party tracking. No telemetry. The frontend makes no requests outside the configured `VITE_API_BASE_URL` (default empty, dev proxy: `http://127.0.0.1:8000`).
|
||||
|
||||
### 9.4 Error model
|
||||
|
||||
@@ -689,9 +795,9 @@ npm run dev
|
||||
|
||||
The frontend reads its backend URL from `VITE_API_BASE_URL` (default empty). Create `.env.local` at the repo root with `VITE_API_BASE_URL=http://127.0.0.1:8000`. Without it, the UI falls back to its in-memory sample store via the `data` adapter (parses are disabled).
|
||||
|
||||
### 11.2 Optional Docker (SP23 — product fork)
|
||||
### 11.2 Docker (SP23 — shipped)
|
||||
|
||||
A Ubuntu + Docker + auth + RBAC + LAN-bind spec exists at [`docs/superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md`](superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md). This is a **product fork** awaiting user decision (per [REQUIREMENTS.md §6.2](REQUIREMENTS.md)). The v1 single-host single-operator posture assumes local-only; SP23 changes the threat model to "remote operator on LAN."
|
||||
The Ubuntu + Docker + auth + RBAC + LAN-bind product fork shipped as SP23 (`merge: SP23 Ubuntu Docker Deployment into main`, `07a7ecb`, 2026-06-23). The compose stack (`docker-compose.yml` + `docker-compose.override.yml`) plus `backend/Dockerfile` and `Dockerfile.frontend` are the canonical deploy shape. First admin is bootstrapped from `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD` env vars; compose refuses to start without both set. See [`docs/superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md`](superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md) for the design rationale.
|
||||
|
||||
### 11.3 Single-host "production"
|
||||
|
||||
@@ -703,7 +809,7 @@ For a single operator on a single machine, the dev deployment IS the production
|
||||
|
||||
The following are explicitly NOT built in v1, by design:
|
||||
|
||||
- **Multi-user / multi-host** — login is required (single-user model today; SP23 adds RBAC). LAN-bind, Docker, reverse proxy still out of scope. (SP23 covers the LAN-bind / Docker / RBAC product fork.)
|
||||
- **Multi-user / multi-host** — login is required (single-user model today; RBAC is in place per SP23 for admin / user / viewer). LAN-bind + Docker + reverse proxy + remote multi-user is the v2 model (SP23 covers LAN-bind / Docker / RBAC).
|
||||
- **Other trading partners** — only Colorado Medicaid via Gainwell is configured. The `PayerConfig` mechanism (SP9) supports adding more; the spec/plan/CLI work for that is on the backlog.
|
||||
- **837I / 837D / 834 / 820 / 275 / 278 / 276-277** — only 837P and 835 are first-class. Other transaction types are listed in [REQUIREMENTS.md §6.2](REQUIREMENTS.md) as backlog items.
|
||||
- **NPPES real-time lookup** — only local NPI checksum validation (SP20). The operator who wants real registry lookup has to wire it in later.
|
||||
@@ -747,11 +853,30 @@ All in `docs/superpowers/specs/`:
|
||||
- SP20 NPI validation: `2026-06-21-cyclone-npi-validation-design.md`
|
||||
- SP21 store split: `2026-06-21-cyclone-store-split-design.md`; universal drilldown: `2026-06-21-cyclone-universal-drilldown-design.md`
|
||||
- SP22 parse-decide: `2026-06-21-cyclone-parse-decide-workflow-design.md`; pipeline agent: `2026-06-21-cyclone-pipeline-agent-design.md`
|
||||
- SP23 (fork) Ubuntu + Docker: `2026-06-22-cyclone-ubuntu-docker-deployment-design.md`
|
||||
- SP23 Ubuntu + Docker + auth + RBAC + LAN-bind (shipped): `2026-06-22-cyclone-ubuntu-docker-deployment-design.md`
|
||||
- SP24 reissue-claims: `2026-07-08-cyclone-reissue-claims-design.md`
|
||||
- SP25 SFTP polling enablement: `2026-06-24-cyclone-sftp-polling-enablement-design.md`
|
||||
- SP25 ack live-tail: `2026-07-02-cyclone-ack-live-tail-design.md`
|
||||
- SP25 orphan data recovery: `2026-07-07-cyclone-orphan-data-recovery-design.md`
|
||||
- SP26 SFTP password file: `2026-06-24-cyclone-sftp-password-file-companion-design.md`
|
||||
- SP27 remittances architecture refactor: `2026-06-29-cyclone-remittances-architecture-refactor-design.md`
|
||||
- SP28 ack-claim auto-link: `2026-07-02-cyclone-ack-claim-auto-link-design.md`
|
||||
- SP29 Inbox 999-rejected drill: `2026-07-02-cyclone-999-rejected-drill-design.md`
|
||||
- SP30 Dashboard Recent batches widget: `2026-07-02-cyclone-dashboard-recent-batches-design.md`
|
||||
- SP31 835 strict content match: `2026-07-02-cyclone-835-strict-content-match-design.md`
|
||||
- SP32 rendering + service-provider NPI extraction: `2026-07-02-cyclone-rendering-npi-extraction-design.md`
|
||||
- SP33 CO TXIX payer fix: `2026-07-02-cyclone-co-txix-payer-fix-design.md`
|
||||
- SP35 parse input guards: `2026-07-06-cyclone-parse-input-guards-design.md`
|
||||
- SP36 api routers split: `2026-07-06-cyclone-api-routers-split-design.md`
|
||||
- SP37 submit-batch canonical flow: `2026-07-07-cyclone-submit-batch-canonical-flow-design.md`
|
||||
- SP38 orphan-ack housekeeping: `2026-07-07-cyclone-orphan-ack-housekeeping-design.md`
|
||||
- SP39 2010BB NM109 fix: `2026-07-07-cyclone-2010bb-nm109-fix-design.md`
|
||||
- SP40 Edifabric validation gate: `2026-07-07-cyclone-edifabric-validation-gate-design.md`
|
||||
- SP41 in-window rebill pipeline: `2026-07-07-cyclone-inwindow-rebill-pipeline-design.md`
|
||||
|
||||
### 13.3 Plans
|
||||
|
||||
All in `docs/superpowers/plans/`. 27 files — one per spec (plus the round-2 backfill for SP9–SP20 in the `2026-06-23-cyclone-sp*` naming).
|
||||
All in `docs/superpowers/plans/`. 37 files — one per shipped spec. SP34 (`999-acceptance-restore`) was de-scoped; SP31's `pcn-exact` content-match predicate supersedes the underlying race.
|
||||
|
||||
### 13.4 Reference docs
|
||||
|
||||
|
||||
+89
-51
@@ -25,7 +25,7 @@
|
||||
|
||||
Every requirement (§3, §4) has an ID; every ID is traceable to a sub-project (§6) which in turn points at a design spec and an implementation plan. The matrix at the end of §6 is the single index that ties everything together.
|
||||
|
||||
**Round history.** Round 1 (2026-06-23) produced this draft. Two independent reviewers (Reviewer A — components/data-model/deps/DoD; Reviewer B — traceability/consistency audit) read it against the actual codebase and surfaced 12 factual errors and 6 structural improvements. Round 2 (this version) applies every factual correction. Round 3 (pending) will re-verify reviewer agreement on the corrected doc.
|
||||
**Round history.** Round 1 (2026-06-23) produced this draft. Two independent reviewers (Reviewer A — components/data-model/deps/DoD; Reviewer B — traceability/consistency audit) read it against the actual codebase and surfaced 12 factual errors and 6 structural improvements. Round 2 (this version, dated 2026-07-08) applies every factual correction through SP41, including the post-SP23 doc alignment (LAN-bind + auth posture) and the SP21 store split + SP36 api split + SP41 rebill additions. Round 3 (pending) will re-verify reviewer agreement on the corrected doc.
|
||||
|
||||
---
|
||||
|
||||
@@ -33,18 +33,18 @@ Every requirement (§3, §4) has an ID; every ID is traceable to a sub-project (
|
||||
|
||||
A self-hosted EDI claims-management suite for a single billing office. Parses 837P professional claims and 835 ERA remittances (X12 005010X222A1 and 005010X221A1), with a local-only FastAPI backend and a React UI for browsing, filtering, and inspecting the parsed data.
|
||||
|
||||
- **Local-only on purpose:** binds `127.0.0.1`, requires login (auth boundary is HTTP; bcrypt + HttpOnly session cookie), no internet exposure, single operator, single machine, one trading partner (Colorado Medicaid, currently).
|
||||
- **LAN-only by design:** binds `0.0.0.0:8000`, requires login (auth boundary is HTTP; bcrypt + HttpOnly session cookie; first admin bootstrapped from `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD` env vars), reachability controlled by the host firewall / compose port publishing, single operator, one trading partner (Colorado Medicaid, currently). The dev/test escape hatch is `CYCLONE_AUTH_DISABLED=1`, which logs a WARNING at boot.
|
||||
- **Single-process:** the backend is one Python process; the frontend is one Vite-served React app; there is no message broker, no separate worker, no queue.
|
||||
- **Single source of truth:** SQLite at `~/.local/share/cyclone/cyclone.db` (overridable via `CYCLONE_DB_URL`); optionally encrypted at rest via SQLCipher (SP12).
|
||||
- **Threat model:** a process on the same host, plus a deliberate HTTP-layer login (no remote unauthenticated read or write). File-system threats (stolen/imaged drive) are still the primary concern; SQLCipher at rest + the macOS Keychain handle those. The SP23 product fork changes the threat model to "remote operator on LAN."
|
||||
- **Threat model:** a process on the same host, plus a deliberate HTTP-layer login (no remote unauthenticated read or write). File-system threats (stolen/imaged drive) are still the primary concern; SQLCipher at rest + the macOS Keychain handle those. SP23 expanded the threat model to "remote operator on LAN" by adding login + RBAC + Docker + Ubuntu packaging; the LAN-bind posture remains.
|
||||
|
||||
The Cyclone codebase has honored this contract end-to-end across 22 shipped sub-projects. Every design spec re-asserts the local-only / single-operator / single-payer / no-auth posture and lists (often at length) what is intentionally out of scope.
|
||||
The Cyclone codebase has honored this contract end-to-end across 41 shipped sub-projects. Every design spec re-asserts the LAN-only / single-operator / single-payer posture and lists (often at length) what is intentionally out of scope.
|
||||
|
||||
---
|
||||
|
||||
## 2. Scope boundaries
|
||||
|
||||
### 2.1 In scope (today, after SP1–SP22)
|
||||
### 2.1 In scope (today, after SP1–SP41)
|
||||
|
||||
| Capability | SP |
|
||||
|---|---|
|
||||
@@ -69,9 +69,16 @@ The Cyclone codebase has honored this contract end-to-end across 22 shipped sub-
|
||||
| Structured JSON logging + PII scrubber | SP18 |
|
||||
| Security hardening (body size limit, rate limit, security headers) | SP19 |
|
||||
| NPI Luhn checksum + Tax ID format validation | SP20 |
|
||||
| CycloneStore split (refactor; **in-flight on `refactor/store-split`**) | SP21 |
|
||||
| CycloneStore split (16-module subpackage; public API preserved via `cyclone.store` facade) | SP21 |
|
||||
| Parse-then-decide upload dedup (idempotency) | SP22 |
|
||||
| Pipeline automation agent (sibling project at `cyclone-pipeline/`) | SP22 |
|
||||
| HTTP-layer auth (bcrypt + HttpOnly session cookie) + RBAC roles + `CYCLONE_AUTH_DISABLED` escape hatch | SP23 |
|
||||
| Ubuntu + Docker + nginx + tini packaging | SP23 |
|
||||
| Audit-log `user_id` (every audit event records the acting user) | SP-auth |
|
||||
| `submission_dedup` table + claim-id dedup at SFTP pre-flight | SP22 + SP41 |
|
||||
| Resubmissions table + bulk ingest helpers | SP21 (resubmissions) |
|
||||
| In-window rebill pipeline (visit↔835 reconciliation + Pipeline A/B + summary CSV) | SP41 |
|
||||
| Live-tail NDJSON streams for `/api/acks/stream` and `/api/ta1-acks/stream` | SP-acks-tail |
|
||||
|
||||
### 2.2 Out of scope (today, by design)
|
||||
|
||||
@@ -140,18 +147,31 @@ Each requirement is `FR-NN`, traceable to one or more sub-projects. Verification
|
||||
| FR-32 | Reject oversize request bodies (413), rate-limit requests (429), and add default security headers via pure-ASGI middlewares; emit a tamper-evident `api.request_rejected` audit event on 413/429. | SP19 |
|
||||
| FR-33 | Surface a rich `GET /api/health` response — DB connectivity, MFT scheduler state, backup scheduler state, live pubsub subscriber counts, last batch id + timestamp. | SP19 (`api_routers/health.py:28-40`) |
|
||||
| FR-34 | Validate NPI with the CMS-published Luhn over `80840 + body` (warning-only — placeholder NPIs in test fixtures shouldn't block ingest). Reject Tax ID (EIN) with reserved prefixes (`00`, `07`, `80`–`89`). | SP20 |
|
||||
| FR-35 | Split `CycloneStore` into per-concern modules (`store_persistence`, `store_reconcile`, `store_queries`, `store_mappers`) without changing the public store API. | SP21 (**in-flight** on `refactor/store-split` branch) |
|
||||
| FR-36 | De-duplicate 837 / 835 uploads via a parse-then-decide workflow (`find_existing_batch_for_claim` / `find_existing_batch_for_remit`); relax the claims/remits PK to `(batch_id, id)` via migration 0014 to support re-parses into different batches. | SP22 (migration 0014 in worktree; not yet on `main`) |
|
||||
| FR-35 | Split `CycloneStore` into per-concern modules under `cyclone/store/` (16 modules, ~5,400 LOC total) without changing the public store API; the `cyclone.store` import surface still resolves to the same names. | SP21 (**Met**; merged on `main`; module list in §5.1) |
|
||||
| FR-36 | De-duplicate 837 / 835 uploads via a parse-then-decide workflow (`find_existing_batch_for_claim` / `find_existing_batch_for_remit`); relax the claims unique constraint via migration 0015 (a defensive drop of an inline UNIQUE that was a no-op against the current schema) to support re-parses into different batches. | SP22 (**Met**; migration 0015 on `main`) |
|
||||
| FR-37 | (Sibling project, not in this repo) Drive the full 7-phase round-trip — preflight → browser upload → parse verify → SFTP submit → TA1 wait → 999 wait → scan + report — with crash-safe resume, structured JSON logging, idempotency dedup, per-run folder. | SP22 (`cyclone-pipeline/`) |
|
||||
| FR-38 | Provide a `DrillStackProvider` + `DrillDrawerHeader` shell so every drillable cell across the app opens a consistent drawer or peek modal. | SP21 |
|
||||
|
||||
### 3.1 In-window rebill pipeline FRs (SP41)
|
||||
|
||||
The `FR-RB-*` IDs are SP41-specific functional requirements for the in-window rebill pipeline. They sit alongside `FR-1`..`FR-38` in §3 but use the `RB` prefix to indicate they cover the rebill sub-domain. Each row links to its implementation module under `cyclone.rebill.*` (or, for FR-RB-5, the store-side `cyclone.store.submission_dedup` helper that the pipeline integrates with).
|
||||
|
||||
| ID | Requirement | SP |
|
||||
|---|---|---|
|
||||
| FR-RB-1 | Visit-to-835 reconciliation on `(member_id, procedure, DOS)`. Match key plus a best-of-N across duplicate SVC rows (per-SVC charge tolerance within 5% to absorb legitimate AxisCare-vs-835 differences). Implemented in `cyclone.rebill.reconcile.reconcile_visits_to_835`. | SP41 |
|
||||
| FR-RB-2 | CARC-aware filter. Excluded reasons (CO-45 / CO-26 / CO-129) trip the `EXCLUDED_CARC` outcome; review reasons (PI-16, OA-18, etc.) produce `REBILLED_A` with `needs_review=true`. Implemented in `cyclone.rebill.carc_filter.decide_carc`. | SP41 |
|
||||
| FR-RB-3 | 120-day timely-filing gate with per-batch override. Default window = 120 days from DOS as of run date; `--override-filing/--no-override-filing` CLI flag (and matching API flag) relaxes the gate for past-window visits. Implemented in `cyclone.rebill.timely_filing.timely_filing_decision`. | SP41 |
|
||||
| FR-RB-4 | Pipeline A (frequency-7 replacement) for denied/partial visits + Pipeline B (member-week batched fresh 837Ps) for `NOT_IN_835` visits. Pipeline A emits a single 837P per visit with `CLM05-3 = '7'`; Pipeline B batches by `(member_id, ISO-week)`. Implemented in `cyclone.rebill.pipeline_a` and `cyclone.rebill.pipeline_b`. | SP41 |
|
||||
| FR-RB-5 | Claim-id dedup at SFTP pre-flight (30-day window, configurable). Implemented in `cyclone.store.submission_dedup.check_duplicate`; integrated into `cyclone.submission.submit_file` which raises `DuplicateClaimError` before any upload bytes are sent. | SP41 |
|
||||
| FR-RB-6 | Summary CSV with `REBILLED_A / REBILLED_B / EXCLUDED_CARC / EXCLUDED_TIMELY_FILING / EXCLUDED_PAYER / EXCLUDED_NO_EVV` rows plus a per-disposition tally. Implemented in `cyclone.rebill.summary.write_summary_csv` and surfaced by `cyclone.rebill.run.run_rebill`. | SP41 |
|
||||
|
||||
---
|
||||
|
||||
## 4. Non-functional requirements
|
||||
|
||||
| ID | NFR | Source / SP |
|
||||
|---|---|---|
|
||||
| NFR-1 | **Always bind 0.0.0.0 + auth.** Backend binds `0.0.0.0:8000` (overridable via `CYCLONE_HOST` / `CYCLONE_PORT`); CORS allowlist is exact (`http://localhost:5173`); login required (bcrypt + HttpOnly session cookie; first admin bootstrapped from env vars `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD`); dev/test escape hatch via `CYCLONE_AUTH_DISABLED=1` (logs WARNING at boot). Reachability is controlled by the host firewall / compose port publishing, not the bind address. | SP1 + auth (2026-06-23 merge) + SP24 (doc reconciliation) |
|
||||
| NFR-1 | **Always bind 0.0.0.0 + auth.** Backend binds `0.0.0.0:8000` (overridable via `CYCLONE_HOST` / `CYCLONE_PORT`); CORS allowlist is exact (`http://localhost:5173`); login required (bcrypt + HttpOnly session cookie; first admin bootstrapped from env vars `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD`); every API router declares `Depends(matrix_gate)` (`cyclone/auth/deps.py:matrix_gate`) so requests without a valid session are rejected; dev/test escape hatch via `CYCLONE_AUTH_DISABLED=1` (logs WARNING at boot). Reachability is controlled by the host firewall / compose port publishing, not the bind address. | SP1 + SP23 (auth + RBAC) |
|
||||
| NFR-2 | **Determinism.** Parser / serializer round-trip is guaranteed on 113 real prodfiles (`docs/prodfiles/claims/*.x12`); canonical fields, not byte-identity (called out in the SP8 spec). | SP8 |
|
||||
| NFR-3 | **Audit completeness.** Every reconciliation anomaly, every state transition, every 999/277CA reject writes an `ActivityEvent` (in `activity_events` table, un-chained; powers the `/activity` feed and inbox state transitions). | SP2 + SP10 |
|
||||
| NFR-4 | **Tamper-evidence.** A separate `audit_log` table (SP11) carries SHA-256 hash-chained rows for security-sensitive events (login attempts, rejections, key rotations, backup lifecycle, rejected requests). `GET /api/admin/audit-log/verify` detects any break. **The `activity_events` (NFR-3) and `audit_log` (NFR-4) tables are distinct** — different semantics, different audiences. New code that needs an audit row must decide which one to write to. | SP11 |
|
||||
@@ -165,7 +185,7 @@ Each requirement is `FR-NN`, traceable to one or more sub-projects. Verification
|
||||
| NFR-12 | **Idempotency.** Inbound MFT files are de-duplicated via `processed_inbound_files`; 837/835 uploads are de-duplicated via the parse-then-decide workflow with `(batch_id, claim_id)` / `(batch_id, remit_id)` relaxed PKs. | SP16 + SP22 |
|
||||
| NFR-13 | **Crash-safety.** Incoming files are processed inside per-file try/except so a bad file doesn't stop the MFT loop; the pipeline agent has crash-safe resume keyed by run id. | SP16 + SP22 |
|
||||
| NFR-14 | **Single-process.** No message broker, no separate worker, no queue; everything runs in one Python process. | (project-wide constraint) |
|
||||
| NFR-15 | **Test density.** Backend tests: **964 collected** (`pytest --collect-only`, 2026-06-23); frontend test files: **73**; prodfiles are exercised (113 + 19 + 1369), not just minimal fixtures. | (project-wide) |
|
||||
| NFR-15 | **Test density.** Backend tests: **1,624 collected** (`pytest --collect-only`, 2026-07-08); frontend test files: **85**; prodfiles are exercised (113 + 19 + 1369), not just minimal fixtures. | (project-wide) |
|
||||
| NFR-16 | **Documentation discipline.** Every shipped sub-project has a design spec + implementation plan + smoke test; a new engineer can read the project top-to-bottom. (SPs 9–16 ship without on-disk plan files — see §6.1 — but their `feat(spN)` commits are on `main` and carry their own commit-message plan.) | (project-wide) |
|
||||
| NFR-17 | **Fail-soft posture.** Reconciliation crashes don't lose the 835; the activity event records the failure. | SP2 |
|
||||
| NFR-18 | **Migration safety.** Migrations are forward-only via `PRAGMA user_version`; rollback procedures are documented per migration; the `user_version` runner is idempotent. **No checksum manifest exists today** (R-13 in §11). | SP2 + SP22 |
|
||||
@@ -180,14 +200,16 @@ Each requirement is `FR-NN`, traceable to one or more sub-projects. Verification
|
||||
|
||||
| Module | Owns | Notes |
|
||||
|---|---|---|
|
||||
| `cyclone.api` | HTTP routes, content negotiation, CORS allowlist, lifespan | 3,145 LOC (largest single file; partly split into `api_routers/` for 4 sub-routes) |
|
||||
| `cyclone.api` | HTTP routes, content negotiation, CORS allowlist, lifespan | 378 LOC; routes are mounted from `api_routers/` (22 sub-routers — see below) |
|
||||
| `cyclone.api_helpers` | NDJSON / content-negotiation / live-tail helpers | |
|
||||
| `cyclone.api_routers` | `acks`, `admin`, `health` (`/api/health`), `ta1_acks` sub-routers | Mounted at `api.py:270-273`; only 4 routes split out — bulk of routes still in `api.py` |
|
||||
| `cyclone.store` | Public store facade; persistence, reconciliation, queries, mappers | 2,172 LOC; SP21 split is in-flight on `refactor/store-split` |
|
||||
| `cyclone.api_routers` | Per-resource sub-routers mounted under `/api/...` | 22 modules: `acks`, `activity`, `admin`, `batches`, `claim_acks`, `claims`, `clearhouse`, `config`, `dashboard`, `eligibility`, `health`, `inbox`, `parse`, `payers`, `providers`, `rebill`, `reconciliation`, `remittances`, `submission`, `ta1_acks` (+ `_shared.py`). Every router declares `Depends(matrix_gate)` for auth. |
|
||||
| `cyclone.auth` | Login, logout, session cookie, RBAC roles, `matrix_gate` dependency | New in SP23; password hashing, session table, user roles |
|
||||
| `cyclone.handlers` | Shared FastAPI handlers (auth-aware JSON responses, audit-log injection, etc.) | Cross-cutting request glue |
|
||||
| `cyclone.store` (subpackage) | Public store facade; the split modules behind it | 16 modules (`__init__` facade + `acks`, `backfill`, `backups`, `batches`, `claim_acks`, `claim_detail`, `exceptions`, `inbox`, `kpis`, `orm_builders`, `providers`, `records`, `resubmissions`, `submission_dedup`, `ui`, `write`); public API re-exported from `cyclone.store` so callers see no breaking change |
|
||||
| `cyclone.db` | SQLAlchemy engine, session factory, ORM models (18 tables) | |
|
||||
| `cyclone.db_migrate` | `PRAGMA user_version` migration runner | |
|
||||
| `cyclone.db_crypto` | Optional SQLCipher encryption at rest; key rotation | `rotate_key()` is destructive (rewrites every page) |
|
||||
| `cyclone.audit_log` | Hash-chained `audit_log` (SP11, SHA-256 chain) | Distinct from `cyclone.activity_events` written by other modules |
|
||||
| `cyclone.audit_log` | Hash-chained `audit_log` (SP11, SHA-256 chain) | Distinct from `cyclone.activity_events` written by other modules; post-SP23 records `user_id` of the actor |
|
||||
| `cyclone.inbox_lanes` | The 5-lane inbox payload (Rejected / Payer-rejected / Candidates / Unmatched / Done today) | |
|
||||
| `cyclone.inbox_state` | 999 envelope reject → claim state transitions | |
|
||||
| `cyclone.inbox_state_277ca` | 277CA STC A4/A6/A7 → payer_rejected stamp (monotonic) | |
|
||||
@@ -197,16 +219,19 @@ Each requirement is `FR-NN`, traceable to one or more sub-projects. Verification
|
||||
| `cyclone.reconcile` | Pure-function 835→claim match + line-level match | No DB session inside; testable with fabricated ORM objects |
|
||||
| `cyclone.scoring` | 4-field weighted candidate scoring (40/25/20/15) | Weights hardcoded — see R-10 in §11 |
|
||||
| `cyclone.pubsub` | In-process EventBus (drop-oldest, per-kind fan-out) | In-process only; not thread-safe across asyncio event loops |
|
||||
| `cyclone.backup` | AES-256-GCM encrypted backups (orchestration + CLI) | 216 LOC |
|
||||
| `cyclone.backup_service` | Backup lifecycle (create / verify / restore) | 740 LOC |
|
||||
| `cyclone.backup_scheduler` | 24h backup autostart loop | 315 LOC |
|
||||
| `cyclone.backup` | AES-256-GCM encrypted backups (orchestration + CLI) | |
|
||||
| `cyclone.backup_service` | Backup lifecycle (create / verify / restore) | |
|
||||
| `cyclone.backup_scheduler` | 24h backup autostart loop | |
|
||||
| `cyclone.scheduler` | asyncio MFT polling loop (SP16) | Uses `processed_inbound_files` for idempotency |
|
||||
| `cyclone.security` | Per-IP rate-limit, body-size limit, security headers (`SecurityHeadersMiddleware`, `BodySizeLimitMiddleware`, `RateLimitMiddleware`) | `EXEMPT_PATHS` list at `:201` |
|
||||
| `cyclone.logging_config` | JSON formatter, PII scrubber | |
|
||||
| `cyclone.npi` | NPI Luhn + EIN format validators | |
|
||||
| `cyclone.batch_diff` | Batch Diff engine (powers `GET /api/batch-diff`) | 12 public functions/classes |
|
||||
| `cyclone.clearhouse` (`__init__.py`) | `SftpClient` (SP9 stub + SP13 paramiko), `InboundFile`, `make_client()` factory | 318 LOC; the SFTP home |
|
||||
| `cyclone.clearhouse` (`__init__.py`) | `SftpClient` (SP9 stub + SP13 paramiko), `InboundFile`, `make_client()` factory | The SFTP home |
|
||||
| `cyclone.edi` (`filenames.py`) | HCPF X12 File Naming Standards helpers | Used by SP13/SP16 outbound filename construction |
|
||||
| `cyclone.rebill` | In-window rebill pipeline: visit↔835 reconciliation, CARC filter, timely-filing gate, Pipeline A/B, summary CSV | New in SP41 |
|
||||
| `cyclone.reissue` | Reissue-claims pipeline (revised-claim regeneration + RUNBOOK entry) | SP24 |
|
||||
| `cyclone.submission` | Canonical `submit_file` helper shared by `cyclone submit-batch` CLI and `POST /api/submit-batch`; parse → DB-write → SFTP-upload → audit per file | SP37 |
|
||||
| `cyclone.cli` | Click CLI | |
|
||||
| `cyclone.__main__` | `python -m cyclone serve` | |
|
||||
| `cyclone.parsers/` | X12 tokenizer, models, validator, writers, 277CA, 999, TA1, 270, 271, CAS, seg | 25 `.py` modules |
|
||||
@@ -217,7 +242,7 @@ Each requirement is `FR-NN`, traceable to one or more sub-projects. Verification
|
||||
|
||||
| Directory | Owns |
|
||||
|---|---|
|
||||
| `src/pages/` | Acks, ActivityLog, BatchDiff, Batches, Claims, Dashboard, Inbox, Providers, Reconciliation, Remittances, Upload (11 pages) |
|
||||
| `src/pages/` | Acks, ActivityLog, BatchDiff, Batches, Claims, Dashboard, Inbox, Login, Providers, Reconciliation, Remittances, Upload (12 pages — Login added in SP23) |
|
||||
| `src/hooks/` | TanStack Query hooks, URL-state hooks, live-tail hooks, keyboard hooks, batch-export, search, drawer keyboard, count-up, provider detail, payer summary, parse, ack detail (35+ files) |
|
||||
| `src/lib/` | `api.ts`, `format.ts`, `utils.ts`, `csv.ts`, `download.ts`, `event-routing.ts`, `tail-stream.ts`, `inbox-api.ts` (all with paired `.test.ts`) |
|
||||
| `src/components/` | 3 drawer families (`ClaimDrawer/` ~15 files, `RemitDrawer/` ~10 files, `ProviderDrawer/` ~6, `AckDrawer/` 3); `inbox/` (10 files), `drill/` (10 files, includes `DrillStackProvider` + `DrillDrawerHeader`), `charts/` (4 files), `ui/` (18 shadcn primitives — radix + cva + tailwind-merge), `KeyboardCheatsheet/` (3 files), plus top-level ActivityFeed, AnimatedNumber, BatchDetail, BatchDiffView, BatchesList, ClaimCard837, DominantKpiCard, EditorialNote, ExportBar, ExportCsvButton, KpiCard, Layout, NewClaimDialog, PageHeader, SearchBar, SearchResults, Sidebar, Sparkline, StatusBadge, StatusPill, TailStatusPill, TickerTape |
|
||||
@@ -226,7 +251,7 @@ Each requirement is `FR-NN`, traceable to one or more sub-projects. Verification
|
||||
|
||||
### 5.2 Data model
|
||||
|
||||
**12 SQL migrations** in `backend/src/cyclone/migrations/` (verified on disk 2026-06-23):
|
||||
**23 SQL migrations** in `backend/src/cyclone/migrations/` (verified on disk 2026-07-08):
|
||||
|
||||
| # | File | Tables / columns added |
|
||||
|---|---|---|
|
||||
@@ -242,13 +267,17 @@ Each requirement is `FR-NN`, traceable to one or more sub-projects. Verification
|
||||
| 0010 | `0010_payer_rejected_acknowledged.sql` | `payer_rejected_acknowledged` column on `claims` |
|
||||
| 0011 | `0011_processed_inbound_files.sql` | `processed_inbound_files` (SP16 idempotency) |
|
||||
| 0012 | `0012_backups.sql` | `db_backups` (SP17) |
|
||||
|
||||
**In-flight on `claims-unique-fix` worktree (not yet on `main`):**
|
||||
|
||||
| # | File | Tables / columns added |
|
||||
|---|---|---|
|
||||
| 0013 | `0013_drop_claims_unique_constraint.sql` | Drop `claims` UNIQUE constraint |
|
||||
| 0014 | `0014_relax_claims_remits_pk.sql` | Relax PK to `(batch_id, id)` (SP22) |
|
||||
| 0013 | `0013_auth_users_and_sessions.sql` | `users`, `sessions`, RBAC role columns (SP23 auth) |
|
||||
| 0014 | `0014_audit_log_user_id.sql` | `audit_log.user_id` column + index (records acting user on every audit event) |
|
||||
| 0015 | `0015_drop_claims_unique_constraint.sql` | Defensive drop of the inline `UNIQUE(batch_id, patient_control_number)` on `claims` (SP22; table-recreate dance — no-op against current schema) |
|
||||
| 0016 | `0016_claims_matched_remittance_id_index.sql` | Index on `claims.matched_remittance_id` to speed inbox joins |
|
||||
| 0017 | `0017_backfill_claim_patient_control_number.sql` | Backfill `patient_control_number` from X12 CLM01 for legacy rows |
|
||||
| 0018 | `0018_claim_acks.sql` | `claim_acks` table (claim-level ACK metadata) |
|
||||
| 0019 | `0019_add_rendering_and_service_provider_npis.sql` | Rendering + service-provider NPI columns on `claims` |
|
||||
| 0020 | `0020_add_batch_txn_set_control_number.sql` | `batches.transaction_set_control_number` (ST/SE control number persistence) |
|
||||
| 0021 | `0021_resubmissions.sql` | `resubmissions` table (resubmission tracking + bulk-ingest helpers) |
|
||||
| 0022 | `0022_submission_dedup.sql` | `submission_dedup` table (claim-id dedup window at SFTP pre-flight, SP41) |
|
||||
| 0023 | `0023_visits.sql` | `visits` table (SP41 in-window rebill pipeline input) |
|
||||
|
||||
**ORM models** (declared in `backend/src/cyclone/db.py` — 18 classes total):
|
||||
|
||||
@@ -404,18 +433,20 @@ Each row's "Spec" / "Plan" point to the design spec and implementation plan on d
|
||||
| SP18 | Structured logging | [spec](superpowers/specs/2026-06-21-cyclone-structured-logging-design.md) | (no plan file on disk; feat(sp18) commits on `main`) | Met |
|
||||
| SP19 | Security hardening + health probe | [spec](superpowers/specs/2026-06-21-cyclone-security-hardening-design.md) | (no plan file on disk; feat(sp19) commits on `main`) | Met |
|
||||
| SP20 | NPI checksum + Tax ID format validation | [spec](superpowers/specs/2026-06-21-cyclone-npi-validation-design.md) | (no plan file on disk; feat(sp20) commits on `main`) | Met |
|
||||
| SP21 | CycloneStore split (in-flight) + Universal drilldown (shipped) | [split spec](superpowers/specs/2026-06-21-cyclone-store-split-design.md), [drilldown spec](superpowers/specs/2026-06-21-cyclone-universal-drilldown-design.md) | [store-split plan](superpowers/plans/2026-06-21-cyclone-store-split.md), [drilldown plan](superpowers/plans/2026-06-21-cyclone-universal-drilldown.md) | Drilldown **Met**; split **In-flight** (`refactor/store-split` branch, not yet on `main`) |
|
||||
| SP21 | CycloneStore split (16-module subpackage) + Universal drilldown | [split spec](superpowers/specs/2026-06-21-cyclone-store-split-design.md), [drilldown spec](superpowers/specs/2026-06-21-cyclone-universal-drilldown-design.md) | [store-split plan](superpowers/plans/2026-06-21-cyclone-store-split.md), [drilldown plan](superpowers/plans/2026-06-21-cyclone-universal-drilldown.md) | Met (split landed on `main`; public API re-exported from `cyclone.store` facade; drilldown used by every drawer) |
|
||||
| SP22 | Parse-decide workflow + Pipeline agent | [parse-decide spec](superpowers/specs/2026-06-21-cyclone-parse-decide-workflow-design.md), [pipeline spec](superpowers/specs/2026-06-21-cyclone-pipeline-agent-design.md) | [parse-decide plan](superpowers/plans/2026-06-21-cyclone-parse-decide-workflow.md), [pipeline plan](superpowers/plans/2026-06-21-cyclone-pipeline-agent.md) | Met (parse-decide on `main`; pipeline agent lives in `cyclone-pipeline/`) |
|
||||
| SP24 | Auth posture alignment (docs-only reconciliation with the auth work in `main`) | [spec](superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md) | [plan](superpowers/plans/2026-06-23-cyclone-auth-posture-alignment.md) | Met (no code paths change; only docs + `__main__.py` WARNING + 3 skills addenda) |
|
||||
| SP23 | Ubuntu + Docker + nginx + tini + auth + RBAC (LAN-bind product fork) | [spec](superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md) | (plan in the same docs directory; auth is the dominant change) | Met (merges `cyclone.auth.*`, every API router gated by `matrix_gate`, Login page added to the React app) |
|
||||
| SP24 | Reissue-claims (revised-claim regeneration + deprecation shim) | [spec](superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md) (docs alignment + reissue) | [plan](superpowers/plans/2026-06-23-cyclone-auth-posture-alignment.md) | Met (merge commit `c9588f7`; `cyclone.reissue` shipped) |
|
||||
| SP25–SP40 | Sub-projects landed between SP24 and SP41 (covering resubmissions, `claim_acks`, rendering/service NPI columns, transaction-set control numbers, additional live-tail streams, dashboards, rebill prerequisites) | (per-SP design specs + plans under `docs/superpowers/`) | (per-SP plans under `docs/superpowers/plans/`) | Met — see §6.1 acceptance table below |
|
||||
| SP41 | In-window rebill pipeline (`cyclone.rebill.*`) | [spec](superpowers/specs/2026-07-07-cyclone-inwindow-rebill-pipeline-design.md) | [plan](superpowers/plans/2026-07-07-cyclone-inwindow-rebill-pipeline.md) | Met (merge commit `309e6c0`; visit↔835 reconciliation, Pipeline A/B, summary CSV, claim-id dedup at SFTP pre-flight) |
|
||||
|
||||
### 6.2 Backlog (specs awaiting plans, or vice versa)
|
||||
|
||||
| Item | Status | Why |
|
||||
|---|---|---|
|
||||
| SP23 (candidate) | Spec-only; **product fork awaiting user decision** | `2026-06-22-cyclone-ubuntu-docker-deployment-design.md` (auth + Docker + RBAC + LAN-bind); see §11 R-1, R-3 |
|
||||
| Plan-file backfill for SP9, SP10, SP11, SP12, SP13, SP14, SP15, SP16, SP17, SP18, SP19, SP20 | Plan files missing on disk; `feat(spN)` commit messages substitute | Round-3 doc-backlog item |
|
||||
| Spec-file backfill for SP6 (workflow/inbox), SP11 (audit log), SP12 (SQLCipher), SP13 (paramiko), SP15 (key rotation), SP16 (MFT scheduler) | Specs missing on disk | Round-3 doc-backlog item |
|
||||
| Migration manifest (checksums for the 12 SQL files) | Open | Completeness review §3.1.18 |
|
||||
| Migration manifest (checksums for the 23 SQL files) | Open | Completeness review §3.1.18 |
|
||||
| Tunable score weights (move 40/25/20/15 from code to config) | Open | Completeness review §3.1.15 |
|
||||
| License file (`LICENSE`) | Open | Completeness review §3.2.28 |
|
||||
| `pre-commit` / `Makefile` / `CONTRIBUTING.md` / `.editorconfig` / `ruff` config | Open | Completeness review §3.2.30 |
|
||||
@@ -469,8 +500,8 @@ This matrix is the single index. Each row points to a requirement and where it's
|
||||
| FR-32 | SP19 | security-hardening | (none on disk) | `test_security.py` |
|
||||
| FR-33 | SP19 | security-hardening | (none on disk) | `test_api.py:33 test_health_endpoint`, `test_security.py:204-221` (health snapshot) |
|
||||
| FR-34 | SP20 | npi-validation | (none on disk) | `test_npi.py`, `test_api_validate_provider.py`, `test_cli_validate.py` |
|
||||
| FR-35 | SP21 | store-split | store-split | `test_store.py` (currently covers the un-split `store.py`; will cover the split modules once `refactor/store-split` lands) |
|
||||
| FR-36 | SP22 | parse-decide-workflow | parse-decide-workflow | `test_api_parse_persists.py`, `test_prodfiles_smoke.py`; migration 0014 in `claims-unique-fix` worktree |
|
||||
| FR-35 | SP21 | store-split | store-split | `tests/store/test_*.py` (per-module coverage of the split subpackage); `test_store.py` continues to exercise the public `cyclone.store` facade |
|
||||
| FR-36 | SP22 | parse-decide-workflow | parse-decide-workflow | `test_api_parse_persists.py`, `test_prodfiles_smoke.py`, migration 0015 (defensive drop of inline UNIQUE on `claims(batch_id, patient_control_number)`) |
|
||||
| FR-37 | SP22 | pipeline-agent | pipeline-agent | (lives in `cyclone-pipeline/`, not this repo) |
|
||||
| FR-38 | SP21 | universal-drilldown | universal-drilldown | `DrillStackProvider.test.tsx`, `useDrawerUrlState.test.ts`, plus per-drawer tests in `ClaimDrawer/`, `RemitDrawer/`, `ProviderDrawer/`, `AckDrawer/` |
|
||||
|
||||
@@ -575,12 +606,12 @@ These are the existing per-SP acceptance checklists as captured in their design
|
||||
```bash
|
||||
# Backend
|
||||
cd backend
|
||||
.venv/bin/pytest # 964 tests collected (2026-06-23)
|
||||
.venv/bin/pytest # 1,624 tests collected (2026-07-08)
|
||||
.venv/bin/python -m cyclone serve
|
||||
|
||||
# Frontend
|
||||
npm run typecheck
|
||||
npm test # 73 test files
|
||||
npm test # 85 test files
|
||||
npm run build
|
||||
|
||||
# Smoke (planned in SP23)
|
||||
@@ -615,7 +646,7 @@ Recorded explicitly so a new engineer can challenge any of them.
|
||||
| A8 | The X12 5010 transaction set Cyclone supports is sufficient for the operator's billing workflow. | Adding 837I / 837D / 278 / 276-277 / 277CA-round-trip / etc. is each a new SP. |
|
||||
| A9 | Local-only bind is sufficient — no remote access needed. | Would require auth (SP23), TLS, reverse proxy, RBAC. |
|
||||
| A10 | One Python process, no message broker, no separate worker. | A multi-worker deployment would require breaking thread-affinity (SP12/SP15 specifically). |
|
||||
| A11 | 964 backend tests + 73 frontend test files is the right test density for this codebase (2026-06-23). If we add more transaction types or surface area, this needs to grow proportionally; coverage gates should be enforced (round-3 backlog). | New transaction types or surface area without test growth → coverage drifts down. |
|
||||
| A11 | 1,624 backend tests + 85 frontend test files is the right test density for this codebase (2026-07-08). If we add more transaction types or surface area, this needs to grow proportionally; coverage gates should be enforced (round-3 backlog). | New transaction types or surface area without test growth → coverage drifts down. |
|
||||
| A12 | `cyclone-pipeline/` is a sibling project, not a sub-project of this repo. | The 7-phase round-trip agent is shipped in `cyclone-pipeline/`; if that project is abandoned, the manual operator workflow remains (upload via `/upload`, SFTP by hand, `check-835` not available). |
|
||||
|
||||
---
|
||||
@@ -626,9 +657,9 @@ Pulled forward from [`docs/reviews/2026-06-20-cyclone-completeness-review.md`](r
|
||||
|
||||
| ID | Risk / gap | Source | Status |
|
||||
|---|---|---|---|
|
||||
| R-1 | No app-layer auth (anyone on the host can read PHI). | Completeness review §3.1.4 + §3.2.25 | **Closed by SP24** — auth shipped via the merge of `origin/main` on 2026-06-23 (commits `a25504b`..`39ae988`, `cyclone.auth.*` + `matrix_gate` dependency on every router); SP24 reconciled the docs to match. SP23 still covers the LAN-bind / Docker / RBAC product fork. |
|
||||
| R-1 | No app-layer auth (anyone on the host can read PHI). | Completeness review §3.1.4 + §3.2.25 | **Closed by SP23** — auth shipped via the merge of `origin/main` on 2026-06-23 (commits `a25504b`..`39ae988`, `cyclone.auth.*` + `matrix_gate` dependency on every router); `users` / `sessions` tables live in migration 0013; every audit event records the acting `user_id` (migration 0014). |
|
||||
| R-2 | SQLite plaintext if SQLCipher not enabled. | Completeness review §3.1.1 | **Partial** — SP12 + SP15 ship the encryption option, but it falls back to plain if the Keychain entry is missing. |
|
||||
| R-3 | No Dockerfile / docker-compose.yml. | Completeness review §3.2.29 | **Open** — SP23 spec covers it but is awaiting user decision. |
|
||||
| R-3 | No Dockerfile / docker-compose.yml. | Completeness review §3.2.29 | **Closed by SP23** — Docker packaging is part of the LAN-bind product fork (Ubuntu base, `tini`, nginx reverse proxy, `docker-compose.yml`). |
|
||||
| R-4 | PayerConfig still partly in code (`co_medicaid()` factory at `cyclone/parsers/payer.py:57-58`). | Completeness review §3.1.9 | **Open** — SP9 externalized config but the factory remains as a fallback for tests and default-payer selection. |
|
||||
| R-5 | No vocabulary tables (ICD-10 / HCPCS / NDC). | Completeness review §3.1.11 | **Open** — explicitly out of scope per A6. |
|
||||
| R-6 | No COB / secondary-claim generator. | Completeness review §3.1.12 | **Open** — explicitly out of scope per §2.2. |
|
||||
@@ -636,9 +667,9 @@ Pulled forward from [`docs/reviews/2026-06-20-cyclone-completeness-review.md`](r
|
||||
| R-8 | No NPPES NPI lookup. | Completeness review §3.1.10 | **Open** — only Luhn checksum (SP20). |
|
||||
| R-9 | No 837I / 837D / 834 / 820 / 275 / 278 / 276-277 transaction types. | Completeness review §2.1 | **Open** — explicitly out of scope per §2.2. |
|
||||
| R-10 | Score weights hardcoded (`cyclone/scoring.py:42-46`). | Completeness review §3.1.15 | **Open** — not scheduled. |
|
||||
| R-11 | `cyclone/store.py` is a 2,172-line god-module. | Completeness review §3.1.17 | **In-flight** — SP21 spec + plan written; branch `refactor/store-split` exists; merge to `main` pending. |
|
||||
| R-12 | `cyclone/api.py` is a 3,145-line god-module (larger than `store.py`). | Completeness review §3.1.19 | **Partial** — `api_routers/` exists for 4 sub-routes (acks, admin, health, ta1_acks); bulk of routes still in `api.py`. |
|
||||
| R-13 | Migrations in flat dir without manifest. | Completeness review §3.1.18 | **Open** — no checksum list; 12 SQL files in `backend/src/cyclone/migrations/`. |
|
||||
| R-11 | `cyclone/store.py` is a 2,172-line god-module. | Completeness review §3.1.17 | **Closed by SP21** — `cyclone.store` is now a 16-module subpackage (`acks`, `backfill`, `backups`, `batches`, `claim_acks`, `claim_detail`, `exceptions`, `inbox`, `kpis`, `orm_builders`, `providers`, `records`, `resubmissions`, `submission_dedup`, `ui`, `write` + `__init__` facade); public API re-exported from `cyclone.store` so callers see no breaking change. |
|
||||
| R-12 | `cyclone/api.py` is a 3,145-line god-module (larger than `store.py`). | Completeness review §3.1.19 | **Closed by SP36** — `api.py` is now 378 LOC; 22 sub-routers live under `cyclone/api_routers/` (every router gated by `Depends(matrix_gate)`). |
|
||||
| R-13 | Migrations in flat dir without manifest. | Completeness review §3.1.18 | **Open** — no checksum list; 23 SQL files in `backend/src/cyclone/migrations/`. |
|
||||
| R-14 | No `pre-commit` / `Makefile` / `CONTRIBUTING.md` / `.editorconfig` / `ruff` config. | Completeness review §3.2.30 | **Open** — single-operator project. |
|
||||
| R-15 | No license file (`LICENSE`). | Completeness review §3.2.28 | **Open** — not scheduled. |
|
||||
| R-16 | PHI fixtures not flagged as PHI. | Completeness review §3.2.23 | **Open** — not scheduled. |
|
||||
@@ -649,10 +680,11 @@ Pulled forward from [`docs/reviews/2026-06-20-cyclone-completeness-review.md`](r
|
||||
| R-21 | No structured logging (before SP18). | Completeness review §3.1.5 | **Closed by SP18**. |
|
||||
| R-22 | No backup automation (before SP17). | Completeness review §3.1.3 | **Closed by SP17**. |
|
||||
| R-23 | `ActivityEvent` not tamper-evident (before SP11). | Completeness review §3.1.2 | **Closed by SP11** — but note: the tampered table is `audit_log`, distinct from `activity_events` (see NFR-3 / NFR-4). |
|
||||
| R-24 | SP23 product fork (auth + Docker + RBAC + LAN-bind). | §6.2 | **Awaiting user decision** — see §2.2 and §11 R-1. |
|
||||
| R-24 | SP23 product fork (auth + Docker + RBAC + LAN-bind). | §6.1 | **Closed by SP23** — auth, RBAC, Docker, nginx, tini, LAN-bind all shipped on `main`. |
|
||||
| R-25 | `vitest@^4.1.9` is unusually new (current stable line is 1.x/2.x). | §5.3 | **Open** — verify the version pin is intentional before the next `npm install`; fallback to `^2.x` if accidental. |
|
||||
| R-26 | No CI workflow (`.github/workflows/`). | §8.1 #3 | **Open** — release gating is manual today. |
|
||||
| R-27 | `backend/src/cyclone/workflow/__pycache__/` is dead bytecode. | §5.1 | **Open** — recommend deletion. |
|
||||
| R-28 | Test density drifted from 964 → 1,624 backend tests and 73 → 85 frontend test files. | §4 NFR-15 | **Open** — coverage gates (`--cov-fail-under=90`) still not enforced; counts tracked in NFR-15. |
|
||||
|
||||
---
|
||||
|
||||
@@ -681,7 +713,10 @@ All under [`docs/superpowers/specs/`](superpowers/specs/):
|
||||
- [2026-06-21-cyclone-store-split-design.md](superpowers/specs/2026-06-21-cyclone-store-split-design.md) — SP21 (split)
|
||||
- [2026-06-21-cyclone-structured-logging-design.md](superpowers/specs/2026-06-21-cyclone-structured-logging-design.md) — SP18
|
||||
- [2026-06-21-cyclone-universal-drilldown-design.md](superpowers/specs/2026-06-21-cyclone-universal-drilldown-design.md) — SP21 (drilldown)
|
||||
- [2026-06-22-cyclone-ubuntu-docker-deployment-design.md](superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md) — **SP23 candidate; awaiting plan + user product-fork decision**
|
||||
- [2026-06-22-cyclone-ubuntu-docker-deployment-design.md](superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md) — **SP23 (shipped)**: Ubuntu + Docker + nginx + tini + auth + RBAC LAN-bind product fork
|
||||
- [2026-06-23-cyclone-auth-posture-alignment-design.md](superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md) — **SP24 (shipped)**: reissue-claims + auth docs alignment
|
||||
- [2026-07-07-cyclone-inwindow-rebill-pipeline-design.md](superpowers/specs/2026-07-07-cyclone-inwindow-rebill-pipeline-design.md) — **SP41 (shipped)**: in-window rebill pipeline (`cyclone.rebill.*`)
|
||||
- [2026-07-08-cyclone-doc-pass-design.md](superpowers/specs/2026-07-08-cyclone-doc-pass-design.md) — **SP42 (in flight)**: this doc-pass (post-SP23–SP41 reconciliation)
|
||||
|
||||
### 12.2 Implementation plans
|
||||
|
||||
@@ -700,8 +735,11 @@ All under [`docs/superpowers/plans/`](superpowers/plans/):
|
||||
- [2026-06-21-cyclone-parse-decide-workflow.md](superpowers/plans/2026-06-21-cyclone-parse-decide-workflow.md) — SP22
|
||||
- [2026-06-21-cyclone-pipeline-agent.md](superpowers/plans/2026-06-21-cyclone-pipeline-agent.md) — SP22
|
||||
- [2026-06-21-cyclone-skill-catalog.md](superpowers/plans/2026-06-21-cyclone-skill-catalog.md) — documentation meta-project
|
||||
- [2026-06-21-cyclone-store-split.md](superpowers/plans/2026-06-21-cyclone-store-split.md) — SP21 (split, in-flight)
|
||||
- [2026-06-21-cyclone-store-split.md](superpowers/plans/2026-06-21-cyclone-store-split.md) — SP21 (split, shipped)
|
||||
- [2026-06-21-cyclone-universal-drilldown.md](superpowers/plans/2026-06-21-cyclone-universal-drilldown.md) — SP21 (drilldown)
|
||||
- [2026-06-23-cyclone-auth-posture-alignment.md](superpowers/plans/2026-06-23-cyclone-auth-posture-alignment.md) — SP24 (reissue-claims + auth docs alignment)
|
||||
- [2026-07-07-cyclone-inwindow-rebill-pipeline.md](superpowers/plans/2026-07-07-cyclone-inwindow-rebill-pipeline.md) — SP41 (in-window rebill pipeline)
|
||||
- [2026-07-08-cyclone-doc-pass.md](superpowers/plans/2026-07-08-cyclone-doc-pass.md) — SP42 (this doc-pass)
|
||||
|
||||
### 12.3 Reviews and reference
|
||||
|
||||
@@ -724,13 +762,13 @@ All under [`docs/superpowers/plans/`](superpowers/plans/):
|
||||
|
||||
For the two independent reviewers at the end of round 2 (re-verifying the corrections):
|
||||
|
||||
1. **Components** — Do §5.1 (backend modules) + §5.1 (frontend dirs) list every existing module that ships PHI or serves a state-changing route? Are the boundaries (DB / ORM / reconcile / store / api / pubsub / clearhouse / batch_diff / edi.filenames) drawn correctly? Can you, from §5.1 alone, draw the import graph?
|
||||
2. **Data model** — Are all 12 migrations listed in §5.2 with their purpose? Are the two worktree-only migrations (0013, 0014) flagged as in-flight? Are all 18 ORM models in §5.2? Is the `db_backups` (not `backups`) correction applied? Is the `Rejection` phantom removed? Is the `ActivityEvent` vs `AuditLog` distinction clear (NFR-3 vs NFR-4)?
|
||||
3. **Dependencies** — Are §5.3 runtime + tooling deps correct (no missing `pyyaml`, no missing `keyring`, no `httpx` listed as runtime, `paramiko` flagged as `[sftp]` extra, `sqlcipher3` flagged as `[sqlcipher]` extra, frontend test env is `happy-dom` not `jsdom`, vitest pinned at `^4.1.9` flagged as unusual)? Are OS-level deps (Keychain via `keyring`, SQLCipher C library, `tini`, `nginx` if SP23 ships) called out?
|
||||
4. **Definition of Done** — Is §8 testable? Project DoD #6 now points at the reviewer-agreement meta-criterion. Per-release + per-task DoDs each independently checkable? FR-7 / FR-33 traceability rows corrected to point at existing test files and the correct endpoint name?
|
||||
5. **Traceability** — Pick any FR-NN in §3. Can you trace it to an SP, a spec on disk (or an honest "no dedicated spec on disk" note), a plan on disk (or an honest "no plan file on disk; feat(spN) commits substitute" note), and at least one test file in `backend/tests/` or `src/`? If not, that's a remaining gap.
|
||||
6. **Assumptions** — Are §10 assumptions falsifiable? Could a new engineer challenge any of them and find the test that proves/disproves them? Is A2's reference to `keyring` accurate now?
|
||||
7. **Risks** — Are §11 risks current (R-11 corrected to 2,172; R-12 added; R-25/R-26/R-27 added)? Compare against `docs/reviews/2026-06-20-cyclone-completeness-review.md`. Is R-24 (SP23 fork) clearly flagged as awaiting user decision?
|
||||
1. **Components** — Do §5.1 (backend modules) + §5.1 (frontend dirs) list every existing module that ships PHI or serves a state-changing route? Are the boundaries (DB / ORM / reconcile / store / api / pubsub / clearhouse / batch_diff / edi.filenames / auth / handlers / submission / rebill / reissue / store/) drawn correctly? Can you, from §5.1 alone, draw the import graph?
|
||||
2. **Data model** — Are all 23 migrations listed in §5.2 with their purpose (no worktree-only entries)? Are all 18 ORM models in §5.2? Is the `db_backups` (not `backups`) correction applied? Is the `ActivityEvent` vs `AuditLog` distinction clear (NFR-3 vs NFR-4)?
|
||||
3. **Dependencies** — Are §5.3 runtime + tooling deps correct (no missing `pyyaml`, no missing `keyring`, no `httpx` listed as runtime, `paramiko` flagged as `[sftp]` extra, `sqlcipher3` flagged as `[sqlcipher]` extra, frontend test env is `happy-dom` not `jsdom`, vitest pinned at `^4.1.9` flagged as unusual)? Are OS-level deps (Keychain via `keyring`, SQLCipher C library, `tini`, `nginx` — SP23 shipped) called out?
|
||||
4. **Definition of Done** — Is §8 testable? Project DoD #6 now points at the reviewer-agreement meta-criterion. Per-release + per-task DoDs each independently checkable? FR-7 / FR-33 / FR-35 / FR-36 traceability rows corrected to point at existing test files and the correct endpoint names?
|
||||
5. **Traceability** — Pick any FR-NN in §3. Can you trace it to an SP, a spec on disk (or an honest "no dedicated spec on disk" note), a plan on disk (or an honest "no plan file on disk; feat(spN) commits substitute" note), and at least one test file in `backend/tests/` or `src/`? If not, that's a remaining gap. Note FR-RB-1..FR-RB-6 (SP41) and the auth-related rows point at the new subpackages.
|
||||
6. **Assumptions** — Are §10 assumptions falsifiable? Could a new engineer challenge any of them and find the test that proves/disproves them? Is A2's reference to `keyring` accurate now (Linux dispatch to Secret Service / kwallet, Windows via Credential Manager)?
|
||||
7. **Risks** — Are §11 risks current (R-1/R-3/R-11/R-12/R-24 closed by SP23/SP21/SP36; R-28 added for test-density drift)? Compare against `docs/reviews/2026-06-20-cyclone-completeness-review.md`.
|
||||
|
||||
If reviewers materially agree on all seven, round 2 closes; this doc becomes the project's new top-level entry point (linked from `README.md`).
|
||||
|
||||
|
||||
+271
@@ -233,6 +233,53 @@ sorted, with `._*` AppleDouble files skipped.
|
||||
- CLI exit 0 on completed runs (per-file failures counted, not bumped);
|
||||
2 on config-level failures (no clearhouse / stub mode / missing dir).
|
||||
|
||||
### Reissue claims (SP24)
|
||||
|
||||
For rebuilding raw 837P files into one IG-correct single-claim X12
|
||||
file per claim — e.g. after a 999 rejection that surfaced an
|
||||
in-scope serializer defect — use `cyclone reissue-claims`. This is
|
||||
the canonical offline-rebuild workflow; the pre-SP24 script
|
||||
`scripts/reissue_claims.py` is a deprecation shim that prints a
|
||||
warning at import time.
|
||||
|
||||
```bash
|
||||
cd /home/tyler/dev/cyclone/backend
|
||||
.venv/bin/python -m cyclone.cli reissue-claims \
|
||||
--input-dir /home/tyler/dev/cyclone/july8billing \
|
||||
--date 2026-07-08 \
|
||||
--output-root /home/tyler/dev/cyclone/dev/rebills \
|
||||
--pipeline initial \
|
||||
--zip-output xxxclaims.zip
|
||||
```
|
||||
|
||||
What it does:
|
||||
|
||||
1. Parse every `*.x12` / `*.txt` / `*.edi` under `--input-dir`.
|
||||
2. Validate each parsed claim; skip claims with hard errors.
|
||||
3. Serialize each surviving claim via the IG-correct serializer
|
||||
(no `HL*3` / `PAT` / `NM1*QC` when `SBR02 == "18"`).
|
||||
4. Write one X12 per claim under
|
||||
`--output-root/<date>/<pipeline>/` with HCPF-spec filenames.
|
||||
5. Optionally zip the output to `--zip-output`.
|
||||
|
||||
**Exit codes:** 0 success (per-file warnings are non-fatal); 1
|
||||
IG-correctness guard tripped (the serializer's `PATIENT_LOOP_DEFAULT_INCLUDED`
|
||||
constant is `True`) or unexpected error; 2 parse / validation failure
|
||||
on a file AND zero claims survived.
|
||||
|
||||
**IG-correctness guard.** On every invocation the CLI refuses to run
|
||||
if the serializer's default has been flipped back to `True`. This is
|
||||
the SP24 regression guard — flipping the constant would re-introduce
|
||||
the 2026-07-08 Edifabric 999 rejection ("2000C HL must be absent
|
||||
when 2000B SBR02 = '18'"). The guard's only job is to fail loud; it
|
||||
runs once per invocation and reads one module-level constant.
|
||||
|
||||
**No SFTP.** `reissue-claims` is a pure local workflow — no DB write,
|
||||
no SFTP upload, no Edifabric gate. To push the rebuilt files to
|
||||
Gainwell, use `cyclone submit-batch --ingest-dir <the emitted dir>`
|
||||
(see the "Submitting claims" section above) or move the zip manually
|
||||
per the "Manual SFTP mode" posture.
|
||||
|
||||
### Note on per-file parse CLIs
|
||||
|
||||
`parse-837` and `parse-835` exist as CLIs but only emit JSON files to
|
||||
@@ -247,6 +294,37 @@ from cyclone.parsers.parse_999 import parse as parse_999
|
||||
result = parse_999(open("path.999.x12").read())
|
||||
```
|
||||
|
||||
### SP39 — corrected-claim push + tracking
|
||||
|
||||
The corrected files now live under
|
||||
`ingest/corrected-v2/v2-batch-<batch-id>-<N>-claims/` instead of the
|
||||
SP33-era `ingest/corrected/batch-<id>-<N>-claims/` tree. The
|
||||
originals are preserved for postmortem; the new tree is the
|
||||
resubmission payload. Push workflow:
|
||||
|
||||
1. `cyclone resubmit-rejected-claims --ingest-dir /home/tyler/dev/cyclone/ingest/corrected-v2`
|
||||
(one push per batch dir; the CLI is idempotent on
|
||||
`(claim_id, interchange_control_number)` so re-running is safe).
|
||||
2. After Gainwell returns 999 acks, drop them into `ingest/` for
|
||||
the standard `cyclone pull-inbound` flow.
|
||||
3. Track the post-submission outcomes:
|
||||
`cyclone resubmissions status [--batch-id=<id>]` — shows
|
||||
`pending_999 / 999_accepted / 999_rejected / 277ca_accepted /
|
||||
paid / denied_again` per claim, joined against `claim_acks`
|
||||
(via the existing SP28/31 auto-link) and `remittances` (via
|
||||
CLP→claim).
|
||||
|
||||
What SP39 fixed in the serializer (2010BB loop, payer NM1 segment):
|
||||
|
||||
- Empty / `SKCO0` / `CO_BHA` payer.id → normalized to `CO_TXIX`
|
||||
(Gainwell's companion guide requires `CO_TXIX` for the 2010BB
|
||||
NM109; `SKCO0` and `CO_BHA` were silently accepted by Gainwell's
|
||||
pre-processor in late 2025 but their SET-level structural
|
||||
validator rejects them as of 2026-07).
|
||||
- Empty / `COHCPF` / `CO_BHA` payer.name → normalized to `CO_TXIX`.
|
||||
- Foreign payer IDs (anything else) pass through verbatim with a
|
||||
WARNING log per substitution.
|
||||
|
||||
### Switching from manual → real (and back)
|
||||
|
||||
When this host's IP is whitelisted with Gainwell's MFT admin, the
|
||||
@@ -291,3 +369,196 @@ curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
|
||||
```
|
||||
|
||||
To revert, set `stub: true` and `auth: {"method": "keychain", "secret_ref": "sftp.gainwell.password"}`.
|
||||
|
||||
### SP40 — Edifabric /v2/x12/validate integration
|
||||
|
||||
SP40 wires Edifabric's reference X12 validator into the push flow
|
||||
as a fail-closed pre-upload gate, plus a CLI / HTTP endpoint for
|
||||
ad-hoc validation of single files. The integration catches the
|
||||
same byte-level defects (`PER-04 required`, `SBR-09 required`,
|
||||
etc.) Edifabric flagged on the SP39 regen `ingest/corrected-v2/`.
|
||||
|
||||
**One-time secret setup** (operator-only, never commit):
|
||||
|
||||
```bash
|
||||
# Option A — macOS Keychain (preferred, paid tier key):
|
||||
cyclone secrets set edifabric.api_key <paid-tier-key>
|
||||
# → stores in Keychain under service='cyclone', account='edifabric.api_key'
|
||||
|
||||
# Option B — env var (CI / headless / per-process):
|
||||
export CYCLONE_EDIFABRIC_API_KEY="<paid-tier-key>"
|
||||
|
||||
# Option C — file companion (matches the SP25 SFTP-password pattern):
|
||||
export CYCLONE_EDIFABRIC_API_KEY_FILE=/path/to/edi-key.txt
|
||||
# file should be chmod 600, single-line, no trailing newline needed
|
||||
```
|
||||
|
||||
The dev-only `tests/fixtures/edifabric_api_key.txt` holds the
|
||||
free `EdiNation Developer API` provisional key from the public docs
|
||||
(`3ecf6b1c5cf34bd797a5f4c57951a1cf`); CI uses the mocked transport
|
||||
so the key is never actually transmitted.
|
||||
|
||||
**Single-file validation — CLI**
|
||||
|
||||
```bash
|
||||
# Status printout + OperationResult summary; exits 0 / 1 / 2.
|
||||
.venv/bin/python -m cyclone.cli validate-837 ingest/corrected-v2/v2-batch-<id>-<N>-claims/tp-<id>-1of1.x12
|
||||
```
|
||||
|
||||
Exit codes (matches the rest of the validator family):
|
||||
|
||||
- `0` — Edifabric returned `Status="success"` (warnings are non-blocking).
|
||||
- `2` — `Status="error"`: at least one `Details` item has
|
||||
`Status="error"`. The CLI prints the first 5 offending segment
|
||||
IDs + messages, then exits.
|
||||
- `1` — Edifabric client config error (no API key, or transport
|
||||
exception). The operator must fix `cyc secrets set` before retrying.
|
||||
|
||||
**Single-file validation — HTTP (auth-gated)**
|
||||
|
||||
```bash
|
||||
curl -b admin-cookies.txt -X POST \
|
||||
-F "file=@ingest/corrected-v2/v2-batch-.../tp-...-1of1.x12" \
|
||||
http://127.0.0.1:8000/api/admin/validate-837
|
||||
# → 200 with the raw OperationResult JSON (Status + Details + LastIndex)
|
||||
# Caller inspects Status to decide whether the file is acceptable.
|
||||
```
|
||||
|
||||
`POST /api/admin/validate-837` is matrix-gated (ADMIN_ONLY); 401
|
||||
without a session cookie, 403 with a viewer cookie.
|
||||
|
||||
**Pre-upload gate on resubmit-rejected-claims**
|
||||
|
||||
`cyclone resubmit-rejected-claims` runs Edifabric validation
|
||||
before every SFTP `put`. Any file that returns `Status="error"`
|
||||
is skipped (not uploaded, no `Resubmission` row inserted), and the
|
||||
batch continues with the remaining files. After the run:
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m cyclone.cli resubmit-rejected-claims \
|
||||
--ingest-dir /home/tyler/dev/cyclone/ingest/corrected-v2
|
||||
# ...
|
||||
# EDIFABRIC VALIDATION FAIL tp-cyc-deadbeef-1of1.x12: 2 error(s)
|
||||
# [error] PER: PER-04 is required
|
||||
# [error] SBR: SBR-09 is required
|
||||
# ...
|
||||
# DONE uploaded=361 skipped=0 failed=0 validated=363 validate_failed=2 ...
|
||||
#
|
||||
# 2 file(s) blocked by Edifabric gate:
|
||||
# tp-cyc-aaaaaaaa-1of1.x12
|
||||
# [error] PER: PER-04 is required
|
||||
# tp-cyc-bbbbbbbb-1of1.x12
|
||||
# [error] SBR: SBR-09 is required
|
||||
```
|
||||
|
||||
To bypass the gate (dev only — for hot-fix scenarios where
|
||||
Edifabric is the bottleneck):
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m cyclone.cli resubmit-rejected-claims \
|
||||
--ingest-dir ... \
|
||||
--skip-edifabric-validate
|
||||
```
|
||||
|
||||
**Expected OperationResult shape** (verbatim from EdiNation docs):
|
||||
|
||||
```json
|
||||
{
|
||||
"Status": "success" | "warning" | "error",
|
||||
"Details": [
|
||||
{"Index": 5, "SegmentId": "PER", "Value": "...",
|
||||
"Message": "PER-04 is required", "Status": "error"}
|
||||
],
|
||||
"LastIndex": 46
|
||||
}
|
||||
```
|
||||
|
||||
Warnings (`Status="warning"`) never block. Only `Details` items
|
||||
with `Status="error"` fail the gate.
|
||||
|
||||
**Troubleshooting**
|
||||
|
||||
- `API key not configured` → run
|
||||
`cyclone secrets set edifabric.api_key <key>` (or set
|
||||
`CYCLONE_EDIFABRIC_API_KEY_FILE`).
|
||||
- `502 Edifabric upstream failure` → check
|
||||
https://status.edifabric.com/ (or the API status page); the
|
||||
gate treats any 4xx/5xx from Edifabric as a hard fail for that
|
||||
file but continues the batch.
|
||||
- After fixing the byte defect, re-run
|
||||
`cyclone resubmit-rejected-claims` for the blocked file —
|
||||
idempotent on `(claim_id, interchange_control_number)`.
|
||||
|
||||
## Known historical drift — the 804 orphan 999s
|
||||
|
||||
The `acks` table may hold several hundred 999 acks whose source 837
|
||||
batch is not present in the `batches` table — the Inbox "Ack orphans"
|
||||
lane surfaces them at `GET /api/inbox/ack-orphans`. These are real
|
||||
production 999s (e.g. sender_id `COMEDASSISTPROG`) whose source 837s
|
||||
were submitted to HPE clearinghouse before the current
|
||||
`~/.local/share/cyclone/cyclone.db` snapshot was created. The source
|
||||
837s themselves were never re-ingested into the current DB, so the
|
||||
`claims` table has no rows that could link against them. SP37's
|
||||
canonical submit-batch flow captures ST02 going forward, so the
|
||||
orphan count is stable — not a forward-looking bug, just historical
|
||||
drift.
|
||||
|
||||
**You cannot auto-link these orphans.** The source 837s are not in
|
||||
`ingest/`, `backend/var/sftp/staging/`, or any local path — they
|
||||
were transmitted to HPE and never came back. Cyclone is downstream
|
||||
of the clearinghouse and does not retain copies of outbound 837s
|
||||
after SFTP ACK. Do not attempt to re-ingest from SFTP inbound —
|
||||
those files are the 999 acks themselves, not the source 837s.
|
||||
|
||||
### Triage path
|
||||
|
||||
1. **Inspect the Inbox > AckOrphansLane** in the UI. Each row is a
|
||||
999 ack with no resolvable claim. Sort by `parsed_at DESC` to
|
||||
see the most recent first; older orphans are less likely to be
|
||||
actionable.
|
||||
2. **Decide per row.** If the operator can identify the source 837
|
||||
outside Cyclone (e.g. a manual record of what was submitted that
|
||||
day), manually create a `claims` row + a `claim_acks` link via
|
||||
`POST /api/parse-837` followed by `POST /api/inbox/candidates/{remit_id}/match`
|
||||
or `POST /api/acks/.../match-claim`. If not, leave the orphan
|
||||
alone — it stays as valid audit history.
|
||||
|
||||
### Optional: seed synthetic batch rows (one-shot)
|
||||
|
||||
`cyclone ack-orphans reconcile` inserts a synthetic `batches` row
|
||||
for every orphan ST02 that doesn't already have one. The synthetic
|
||||
row is marked with `input_filename = '<synthetic:orphan-reconcile>'`
|
||||
so it's trivially distinguishable in queries. Future 999 acks for
|
||||
the same ST02s will resolve against the batch envelope index (so
|
||||
the operator can see "this 999 is for a known orphan source") but
|
||||
will not link to claims (because no claim rows exist).
|
||||
|
||||
```bash
|
||||
# Preview the reconcile without writing rows.
|
||||
cyclone ack-orphans reconcile --dry-run
|
||||
|
||||
# Insert the synthetic batch rows.
|
||||
cyclone ack-orphans reconcile
|
||||
```
|
||||
|
||||
Re-running `cyclone ack-orphans reconcile` after a successful pass
|
||||
is a no-op (idempotent). To inspect the per-ST02 breakdown at any
|
||||
time:
|
||||
|
||||
```bash
|
||||
cyclone ack-orphans status
|
||||
```
|
||||
|
||||
The status command prints a table with ST02, ACK COUNT, and HAS
|
||||
BATCH columns, ranked by ack count so the heaviest backlog
|
||||
surfaces first.
|
||||
|
||||
### What this is NOT
|
||||
|
||||
- **Not a backfill.** No `claims` rows are synthesized — the source
|
||||
data is gone.
|
||||
- **Not auto-runnable.** The reconcile CLI is operator-invoked only;
|
||||
it does not run on boot, in the SFTP polling scheduler, or via
|
||||
cron.
|
||||
- **Not a deletion.** The orphan 999s are valid audit history and
|
||||
must remain queryable.
|
||||
|
||||
@@ -127,8 +127,9 @@ assert the row count is unchanged after the second call.
|
||||
- [ ] `status` calls `cycl_store.find_ack_orphan_st02_summary()`,
|
||||
prints a table: `ST02 | ACK COUNT | HAS BATCH`. Total line at
|
||||
the bottom.
|
||||
- [ ] Exit 0 on success, exit 2 on DB error (matching `cyclone-cli`
|
||||
convention).
|
||||
- [ ] Exit 0 on success, exit 1 on DB error (matching the
|
||||
`cyclone-cli` convention: 1 = unexpected exception, including
|
||||
SQLAlchemy DB errors).
|
||||
|
||||
**RED test:** `tests/test_ack_orphans_cli.py::test_status_prints_table`
|
||||
using `CliRunner.invoke(["ack-orphans", "status"])`. Assert exit
|
||||
@@ -142,7 +143,8 @@ line.
|
||||
- [ ] Calls `cycl_store.reconcile_orphan_st02s(dry_run=dry_run)`,
|
||||
prints `Created: N synthetic batch rows. Skipped: M (already
|
||||
had a batch row).`.
|
||||
- [ ] Exit 0 on success, exit 2 on DB error.
|
||||
- [ ] Exit 0 on success, exit 1 on DB error (same convention as
|
||||
`status`).
|
||||
|
||||
**RED test:** `tests/test_ack_orphans_cli.py::test_reconcile_creates_synthetic_batches`
|
||||
— seed two orphan ST02s (one with batch, one without), invoke
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
# Orphan Data Recovery Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use
|
||||
> superpowers:subagent-driven-development (recommended) or
|
||||
> superpowers:executing-plans to implement this plan task-by-task. Steps
|
||||
> use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Recover stranded billing data from `ingest/`, fix the synthetic-batch read-side bomb, and verify the dashboard KPI tiles come back to life after the recovery run.
|
||||
|
||||
**Architecture:** Three surgical changes — one defensive read-side fallback in `store/batches.py`, one new offline CLI `cyclone recover-ingest`, one verification script. No new HTTP endpoints, no schema changes, no SFTP interaction. Uses `CycloneStore.add()` as the canonical write path so live-tail pages light up immediately.
|
||||
|
||||
**Tech Stack:** Python 3.11+, FastAPI/uvicorn (for FastAPI bootstrap helper to share `EventBus`), SQLAlchemy (existing models), Click (existing CLI), Pydantic v2 (existing models).
|
||||
|
||||
**Spec:** [`docs/superpowers/specs/2026-07-07-cyclone-orphan-data-recovery-design.md`](../specs/2026-07-07-cyclone-orphan-data-recovery-design.md)
|
||||
|
||||
---
|
||||
|
||||
## File structure
|
||||
|
||||
```
|
||||
backend/src/cyclone/
|
||||
store/batches.py # A: defensive stub in _row_to_record
|
||||
cli.py # B: add `cyclone recover-ingest`
|
||||
submission/
|
||||
recover.py # B: shared parse-then-store logic (new, small)
|
||||
backend/tests/
|
||||
test_store_batches_synthetic.py # A: unit (the defensive stub needs synthetic rows with NULL raw_result_json — these are ORM-level test fixtures for the read path, not EDI samples)
|
||||
test_api_batches_synthetic.py # A: integration
|
||||
test_cli_recover_ingest.py # B: cli + integration (uses existing co_medicaid_* fixtures; no new minimal_* EDI mirrors)
|
||||
backend/scripts/
|
||||
verify_dashboard_recovery.py # post-recovery sanity check (operator-only)
|
||||
docs/superpowers/specs/
|
||||
2026-07-07-cyclone-dashboard-mess-postmortem.md # deep-dive writeup
|
||||
```
|
||||
|
||||
## Task 0: Setup — branch + verify
|
||||
|
||||
- [ ] Confirm `main` is the working branch and is clean: `git status -sb`.
|
||||
- [ ] Create the branch: `git checkout -b sp25-orphan-data-recovery`.
|
||||
- [ ] Confirm DB file location: `sqlite3 ~/.local/share/cyclone/cyclone.db ".tables"` returns the 22 tables.
|
||||
|
||||
## Task 1: A — failing test for `_row_to_record` defensive stub
|
||||
|
||||
- [ ] Add `backend/tests/test_store_batches_synthetic.py` with two cases:
|
||||
1. `test_row_to_record_handles_null_raw_json` — insert a Batch with `raw_result_json=None, kind='837p', input_filename='<synthetic:orphan-reconcile>'`, call `store.list_batches()`, assert the stub returns a `BatchRecord` with `result.summary.total_claims==0` and empty `result.claims`.
|
||||
2. `test_row_to_record_handles_null_raw_json_835` — same but `kind='835'`, plus assert `result.claims==[]`.
|
||||
- [ ] Run `cd backend && .venv/bin/pytest tests/test_store_batches_synthetic.py -v` — both tests must fail (`ValidationError` on `summary`).
|
||||
|
||||
## Task 2: A — implement stub in `_row_to_record`
|
||||
|
||||
- [ ] In `backend/src/cyclone/store/batches.py`, replace the body of `_row_to_record` (lines ~47-79) with a defensive version:
|
||||
- If `row.raw_result_json` is None or `{}`, build a stub `ParseResult` (837p) with `envelope=None, claims=[], summary=BatchSummary(input_file=row.input_filename, control_number=row.transaction_set_control_number, total_claims=0, passed=0, failed=0)` and a `Date.today()`-based `transaction_date`, **or** a stub `ParseResult835` (835) with `envelope=Envelope(sender_id="",receiver_id="",control_number=row.transaction_set_control_number,transaction_date=date.today()), financial_info=FinancialInfo(... zeros ...), trace=ReassociationTrace(...), payer=Payer835(...), payee=Payee835(...), claims=[], summary=<minimal>`.
|
||||
- Otherwise behave as today.
|
||||
- [ ] Re-run `tests/test_store_batches_synthetic.py` — both tests now pass.
|
||||
- [ ] Run the full backend suite: `cd backend && .venv/bin/pytest -q`. Must stay green.
|
||||
|
||||
## Task 3: A — verify against the live DB
|
||||
|
||||
- [ ] Restart-free smoke: `curl -sS 'http://127.0.0.1:8000/api/batches?limit=5' | python -m json.tool` — expect 200, `items[].claimCount==0` for the 4 synthetic rows, `inputFilename=="<synthetic:orphan-reconcile>"`.
|
||||
|
||||
## Task 4: B — failing test for `recover-ingest` against real co_medicaid fixtures
|
||||
|
||||
- [ ] Add `backend/tests/test_cli_recover_ingest.py` with three cases. Test inputs come from existing `backend/tests/fixtures/co_medicaid_837p.txt` + `co_medicaid_835.txt` — real production EDI already in the repo. **No synthetic EDI fixtures are introduced.** The ingest/ files are reserved for the operator-only recovery run (Task 6), not unit-tested.
|
||||
1. `test_ingest_835_persists_remittance` — invoke `recover-ingest` via Click's `CliRunner` with `co_medicaid_835.txt`; assert a Remittance row was created with `total_paid > 0`, plus a `processed_inbound_files` row with `sftp_block_name='manual-recover'`.
|
||||
2. `test_ingest_837p_persists_claims` — invoke with `co_medicaid_837p.txt`; assert ≥1 Claim row + a Batch row that references it.
|
||||
3. `test_ingest_idempotent` — invoke twice with the same file; second run is a no-op (skip messages, no extra rows).
|
||||
- [ ] Run the test file: must fail (no `recover-ingest` command yet).
|
||||
|
||||
## Task 5: B — implement `cyclone recover-ingest`
|
||||
|
||||
- [ ] Add `backend/src/cyclone/submission/recover.py` with one function:
|
||||
- `def recover_file(path: Path, *, sftp_block_name: str = "manual-recover") -> dict` — detects kind from `ST01*` (837 → '837p'; 835 → '835'), calls the matching parser, constructs a `BatchRecord`, calls `CycloneStore.add(record, event_bus=local_bus)`, returns `{"file": str(path), "status": "ok"|"duplicate"|"failed", "batch_id": str|None, "error": str|None}`.
|
||||
- Wraps the dedup check around `processed_inbound_files` `(sftp_block_name, path.name)`.
|
||||
- [ ] In `backend/src/cyclone/cli.py`, add the Click command `recover-ingest`:
|
||||
- Options: `--file PATH` (repeatable), `--sftp-block-name` (default `manual-recover`).
|
||||
- Validates that each `--file` exists.
|
||||
- Calls `recover_file` per file; exits 0 if at least one ingested OK, 2 if a file parse failed, 1 otherwise.
|
||||
- [ ] Re-run `tests/test_cli_recover_ingest.py` — all three pass.
|
||||
- [ ] Run the full backend suite: `cd backend && .venv/bin/pytest -q`. Stays green.
|
||||
|
||||
## Task 6: C — line-reconciliation is automatic; verify with the live DB
|
||||
|
||||
- [ ] Capture the pre-recovery `/api/dashboard/kpis` snapshot into a file:
|
||||
- `curl -sS 'http://127.0.0.1:8000/api/dashboard/kpis' > /tmp/recovery_pre.json`.
|
||||
- [ ] Run the recovery against the 5 real files:
|
||||
- `.venv/bin/python -m cyclone.cli recover-ingest --file ingest/tp11525703-837P-20260701162932052-1of1.txt --file ingest/tp11525703-837P-20260701162935524-1of1.txt --file ingest/tp11525703-837P-20260701162938977-1of1.txt --file ingest/tp11525703-837P-20260701162942746-1of1.txt --file ingest/tp11525703-835_M019771179-20260706005516577-1of1.x12`.
|
||||
- Expect: 4× 837P → 4 new `Batch` rows + 338 `Claim` rows; 1× 835 → 1 new `Batch` + 1 `Remittance` + ~1,148 `service_line_payments` rows + N `Match` rows (auto-reconciled).
|
||||
- [ ] Capture the post-recovery `/api/dashboard/kpis` snapshot: `curl ... > /tmp/recovery_post.json`.
|
||||
- [ ] Open `backend/scripts/verify_dashboard_recovery.py` (write the script during this step):
|
||||
- Parses both JSON files; asserts `count_post >= count_pre + 338`, `billed_post >= $57,986`, `pending_post >= 339`.
|
||||
- Optionally prints the top-provisioners delta so the operator eyeballs the result.
|
||||
|
||||
## Task 7: write the postmortem doc
|
||||
|
||||
- [ ] Create `docs/superpowers/specs/2026-07-07-cyclone-dashboard-mess-postmortem.md`:
|
||||
- Header: `# 2026-07-07 Cyclone Dashboard postmortem`.
|
||||
- Sections: `## What we saw / ## Root cause / ## What we fixed / ## KPI values before & after / ## Open watch-items`.
|
||||
- Embed the before/after KPI JSON snapshots as fenced code blocks under `## KPI values before & after`.
|
||||
|
||||
## Task 8: final verification + merge
|
||||
|
||||
- [ ] `cd backend && .venv/bin/pytest -q` — green.
|
||||
- [ ] `npm test` in repo root — green.
|
||||
- [ ] `git status -sb` — clean working tree.
|
||||
- [ ] `git log --oneline -10` — at minimum: docs(spec), docs(plan), `feat(sp25):`, postmortem commit.
|
||||
- [ ] Commit the chain (the merge commit is the last one):
|
||||
- `git commit -m 'docs(spec): design for SP25 orphan data recovery (Step 1)'`
|
||||
- `git commit -m 'docs(plan): implementation plan for SP25 orphan data recovery (Step 1)'`
|
||||
- `git commit -am 'feat(sp25): defensive read-side stub for synthetic-batch rows + recover-ingest CLI (Steps 2-5)'`
|
||||
- `git commit -m 'docs: dashboard mess postmortem (Step 7)'`
|
||||
- [ ] Merge into `main` with `--no-ff`:
|
||||
- `git checkout main && git merge --no-ff sp25-orphan-data-recovery -m 'merge: SP25 orphan data recovery into main'`
|
||||
- Confirm a single atomic merge commit with all the above in its `git log`.
|
||||
@@ -0,0 +1,234 @@
|
||||
# Doc-Pass Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use
|
||||
> superpowers:subagent-driven-development (recommended) or
|
||||
> superpowers:executing-plans to implement this plan task-by-task. Steps
|
||||
> use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Reconcile CLAUDE.md, README.md, docs/ARCHITECTURE.md, docs/REQUIREMENTS.md, docs/RUNBOOK.md, and the eight `.superpowers/skills/*/SKILL.md` files so the prose matches the current implementation after SPs 23–41.
|
||||
**Architecture:** Pure `.md` edits. No code paths change; no API endpoints change; no test assertions change. One commit per file, prefixed `docs(sp42): <file> …`. The merge commit (`merge: SP42 doc-pass into main`) is the audit trail.
|
||||
**Tech Stack:** Markdown + `git` + `grep` for verification.
|
||||
**Spec:** [`docs/superpowers/specs/2026-07-08-cyclone-doc-pass-design.md`](../specs/2026-07-08-cyclone-doc-pass-design.md)
|
||||
|
||||
---
|
||||
|
||||
## File structure
|
||||
|
||||
```
|
||||
docs/superpowers/
|
||||
specs/2026-07-08-cyclone-doc-pass-design.md ← new
|
||||
plans/2026-07-08-cyclone-doc-pass.md ← new
|
||||
|
||||
CLAUDE.md ← edit (header + subpackage + LOC claims)
|
||||
README.md ← edit (lead paragraph + project layout)
|
||||
docs/ARCHITECTURE.md ← edit (§1, §2, §4.1, §4.3, §5.4, §6.2, §6.3, §7.4, §8.1, §11.2, §13.2)
|
||||
docs/REQUIREMENTS.md ← edit (§1, §2.2, §5.1, §5.2, §6.1, §6.2, §11, §12)
|
||||
docs/RUNBOOK.md ← edit (SP23 Docker stack → Docker stack)
|
||||
|
||||
.superpowers/skills/cyclone-spec/SKILL.md ← edit (As of this writing header)
|
||||
.superpowers/skills/cyclone-tests/SKILL.md ← edit (counts + SP-N pointer)
|
||||
.superpowers/skills/cyclone-store/SKILL.md ← edit (SP21 framing + line refs + module list)
|
||||
.superpowers/skills/cyclone-api-router/SKILL.md ← edit (4 → 22 routers + line refs)
|
||||
.superpowers/skills/cyclone-edi/SKILL.md ← edit (SP-N pointer)
|
||||
.superpowers/skills/cyclone-frontend-page/SKILL.md ← edit (counts + drawer list)
|
||||
.superpowers/skills/cyclone-cli/SKILL.md ← edit (Seven → 15+ subcommands)
|
||||
.superpowers/skills/cyclone-tail/SKILL.md ← edit (3 → 4 streaming pages + line refs)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 0: branch + spec/plan
|
||||
|
||||
- [x] Branch `sp42-doc-pass` from `main` (no worktree — work in current checkout).
|
||||
- [x] First commit on the branch: `docs(spec): SP42 doc-pass design` (commits the spec).
|
||||
- [x] Second commit: `docs(plan): SP42 doc-pass implementation plan` (commits this plan).
|
||||
|
||||
---
|
||||
|
||||
## Task 1: refresh `CLAUDE.md`
|
||||
|
||||
- [ ] Update line ~7 to read "LAN-only by design — always binds to `0.0.0.0:8000` and requires login…".
|
||||
- [ ] Update line ~97 ("SP22 used, next free is SP24") to "SP numbers are used through **SP41** (in-window rebill pipeline); the next free increment is **SP42**. A `sp42-doc-pass` worktree is checked out locally but not yet merged into `main`."
|
||||
- [ ] Update line ~121 (api.py / store.py LOC + "only large file" claim): replace the "3,548 LOC / 2,423 LOC / in flight" framing with post-SP21/SP36 reality. "The largest files now live under `cyclone/api_routers/` (22 routers under SP36) and `cyclone/store/` (the SP21-split facade in 16 modules). `api.py` is a ~377-line shell that wires the FastAPI app and mounts the routers."
|
||||
- [ ] Update line ~122 subpackage list: expand `api_routers/` to 22 routers; add `auth/` (SP24, 10 modules), `handlers/` (SP27, 6 modules), `rebill/` (SP41, 13 modules), `reissue/` (SP24, 2 modules), `store/` (SP21, 16 modules). Note that `workflow/` is dead bytecode.
|
||||
- [ ] Update line ~128 migration count: "23 SQL migrations under `migrations/` (0001_initial through 0023_visits)".
|
||||
- [ ] Drop the three phantom `parse-999` / `parse-ta1` / `parse-277ca` entries from the CLI block (lines 174–176) — they contradict the body text and were already flagged in CLAUDE.md's "Things that are easy to get wrong" section.
|
||||
- [ ] Commit: `docs(sp42): CLAUDE.md — refresh SP numbers, store split, subpackage list`.
|
||||
|
||||
---
|
||||
|
||||
## Task 2: refresh `README.md`
|
||||
|
||||
- [ ] Update lines 7–8: replace "Local-only on purpose: binds to `127.0.0.1`, no auth, no internet exposure." with "LAN-only by design — always binds to `0.0.0.0:8000` and requires login (bcrypt + HttpOnly session cookie; first admin bootstrapped from `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD`). Reachability is controlled by the host firewall / compose port publishing, not the bind address."
|
||||
- [ ] Update the project-layout block: replace `store.py` with `store/` (post-SP21 split); replace the `api.py` description with the thin-shell description; add `auth/`, `handlers/`, `rebill/`, `reissue/`, `store/`, `submission/`, `api_routers/` (22 routers).
|
||||
- [ ] Update the "Sub-projects 2 through 19 are shipped" claim to "Sub-projects 2 through 41 are shipped."
|
||||
- [ ] Update the CLI example block to drop `parse-999`, `parse-ta1`, `parse-277ca` (which don't exist as DB-writing CLIs — see CLAUDE.md "Inbound ingestion paths" note).
|
||||
- [ ] Commit: `docs(sp42): README.md — refresh auth/bind claim and project layout`.
|
||||
|
||||
---
|
||||
|
||||
## Task 3: refresh `docs/ARCHITECTURE.md`
|
||||
|
||||
- [ ] Update §1 line ~14: replace `127.0.0.1` with `0.0.0.0` and add the reachability + auth boundary sentence.
|
||||
- [ ] Update §2 process diagram: replace `api.py 3,548 LOC` with `api.py ~377 LOC (post-SP36 thin shell)`; replace the 4-router enumeration with the full 22-router list.
|
||||
- [ ] Update §4.1 package layout: add `auth/`, `handlers/`, `rebill/`, `reissue/`, `store/`, expand `api_routers/` to 22 routers.
|
||||
- [ ] Update §4.2 module map: add the missing subpackage rows.
|
||||
- [ ] Update §4.3 SP21 framing: present-tense ("After SP21, the public API of `cyclone.store` is unchanged…"); add the post-split module list reference.
|
||||
- [ ] Update §5.4: replace `/api/tail?since=…` with the canonical three-stream endpoints and replace `api.py:1357-2006` with `api_routers/{claims,remittances,activity}.py`.
|
||||
- [ ] Update §6.2 ERD: add `claim_acks`, `resubmissions`, `visits` tables.
|
||||
- [ ] Update §6.3 claim-state diagram: add `partial` (per REQUIREMENTS FR-6).
|
||||
- [ ] Update §7.4: replace `auto:pcn_npi_charge` / `auto:pcn_charge` with `pcn-exact` (SP31).
|
||||
- [ ] Update §8.1 endpoint enumeration with the canonical list (~85 routes after SPs 23–41).
|
||||
- [ ] Update §11.2: replace "SP23 product fork awaiting user decision" with "SP23 (shipped 2026-06-23 via `merge: SP23 Ubuntu Docker Deployment into main`)".
|
||||
- [ ] Update §13.2: replace "22 specs / 27 plans" with "42 specs / 37 plans"; add the missing SP25–SP41 spec entries.
|
||||
- [ ] Update §6.1 migration table from 12 to 23.
|
||||
- [ ] Commit: `docs(sp42): ARCHITECTURE.md — refresh post-SP23–41 package map, endpoints, SP23 status`.
|
||||
|
||||
---
|
||||
|
||||
## Task 4: refresh `docs/REQUIREMENTS.md`
|
||||
|
||||
- [ ] Update line ~49: "Sub-projects 2 through 41 are **shipped**."
|
||||
- [ ] Update §2.2 / §5.4 / §5.1: drop "SP23 would add" speculative framing; rewrite as past tense where SP23 shipped something.
|
||||
- [ ] Update §5.1 LOC claims (api.py 3,145 → ~377; store.py 2,172 → split into `cyclone/store/`).
|
||||
- [ ] Update §5.1 module list: add `auth`, `handlers`, `rebill`, `reissue`, `submission` subpackage rows; expand `store` row to its 16 modules.
|
||||
- [ ] Update §5.2 migration table: 12 → 23 with the full enumeration.
|
||||
- [ ] Update §6.1 SP table: add SP23 (Ubuntu+Docker, shipped) through SP41 (rebill pipeline, shipped) as separate rows.
|
||||
- [ ] Update §6.2 backlog: move SP23 from backlog to §6.1; remove the "SP23 candidate" row.
|
||||
- [ ] Update §11 R-1, R-3, R-11, R-12 to Closed with concrete merge-SHA evidence; update R-13 migration count to 23.
|
||||
- [ ] Update §12.1 / §12.2 spec-and-plan lists to include SP25–SP41.
|
||||
- [ ] Commit: `docs(sp42): REQUIREMENTS.md — refresh SP table, close SP23 risks, add SP25–41`.
|
||||
|
||||
---
|
||||
|
||||
## Task 5: refresh `docs/RUNBOOK.md`
|
||||
|
||||
- [ ] Update line ~120: replace "For the SP23 Docker stack, mount the MFT password as a file…" with "For the Docker stack, mount the MFT password as a file…" (SP23 is shipped; the wording should not read as conditional).
|
||||
- [ ] Audit any other "SP23 Docker stack" / "SP23 would" / "SP23 candidate" wording; rewrite as past tense where SP23 shipped the feature.
|
||||
- [ ] Commit: `docs(sp42): RUNBOOK.md — drop 'SP23 Docker stack' conditional framing`.
|
||||
|
||||
---
|
||||
|
||||
## Task 6: refresh `.superpowers/skills/cyclone-spec/SKILL.md`
|
||||
|
||||
- [ ] Update line ~16: "As of this writing: **42 specs** in `docs/superpowers/specs/`, **37 plans** in `docs/superpowers/plans/`, SP numbers used through **SP41** (in-window rebill pipeline). The next free increment is **SP42**."
|
||||
- [ ] Add a one-line note about the SP25 reuse anomaly (three SP25s: sftp-polling-enablement, ack-live-tail, orphan-data-recovery).
|
||||
- [ ] Commit: `docs(sp42): cyclone-spec skill — refresh SP-N framing`.
|
||||
|
||||
---
|
||||
|
||||
## Task 7: refresh `.superpowers/skills/cyclone-tests/SKILL.md`
|
||||
|
||||
- [ ] Update line ~7–8: refresh test counts (89 backend → 178; 13 fixtures → 22; 59 frontend → 130+); replace "next is SP24" with "next is SP42".
|
||||
- [ ] Commit: `docs(sp42): cyclone-tests skill — refresh counts and SP-N pointer`.
|
||||
|
||||
---
|
||||
|
||||
## Task 8: refresh `.superpowers/skills/cyclone-store/SKILL.md`
|
||||
|
||||
- [ ] Update line ~10: replace `backend/src/cyclone/store.py:882` with `backend/src/cyclone/store/__init__.py:148`.
|
||||
- [ ] Update line ~17: drop "single 2412-line module, SP21 in progress" framing; replace with the post-SP21 16-module enumeration + "Last merged SP: SP41; next free increment: SP42."
|
||||
- [ ] Update SP21 split module list (lines 60–70) to add the 5 missing modules: `backfill.py`, `claim_acks.py`, `kpis.py`, `resubmissions.py`, `submission_dedup.py`.
|
||||
- [ ] Update line ~86 (`store.py:898-1107`) → `store/write.py`.
|
||||
- [ ] Update line ~121 (`api.py:1380-1401`) → `api_routers/{claims,remittances,activity}.py`.
|
||||
- [ ] Update line ~136 (`api.py:1891,2002`) → `api_routers/{claims,remittances,activity}.py`.
|
||||
- [ ] Commit: `docs(sp42): cyclone-store skill — refresh SP21 framing, line refs, module list`.
|
||||
|
||||
---
|
||||
|
||||
## Task 9: refresh `.superpowers/skills/cyclone-api-router/SKILL.md`
|
||||
|
||||
- [ ] Update lines 14–18: replace "4 router modules" with the canonical 22-router list; replace "api.py ~3,548 LOC" with "api.py ~377 LOC (thin shell)".
|
||||
- [ ] Update lines 49–50: drop the "parse endpoints next refactor target" note (SP36 already split them).
|
||||
- [ ] Update lines 170–183 line refs (`api.py:246-256` etc.) → `api_routers/__init__.py`.
|
||||
- [ ] Commit: `docs(sp42): cyclone-api-router skill — refresh router count + line refs`.
|
||||
|
||||
---
|
||||
|
||||
## Task 10: refresh `.superpowers/skills/cyclone-edi/SKILL.md`
|
||||
|
||||
- [ ] Update line ~13: "The next increment is **SP42**" (was SP22). Add a one-line note about SP32 (rendering NPI), SP35 (parse input guards), SP39 (2010BB NM109), SP40 (Edifabric gate) having touched parser/validator territory.
|
||||
- [ ] Commit: `docs(sp42): cyclone-edi skill — refresh SP-N pointer`.
|
||||
|
||||
---
|
||||
|
||||
## Task 11: refresh `.superpowers/skills/cyclone-frontend-page/SKILL.md`
|
||||
|
||||
- [ ] Update line ~11: refresh counts (11 → 12 pages, ~30 → ~53 hooks, 2 → 4 drawers); replace "next is SP22" with "next is SP42".
|
||||
- [ ] Update drawer list at lines 18–20: add `ProviderDrawer/`, `AckDrawer/`.
|
||||
- [ ] Commit: `docs(sp42): cyclone-frontend-page skill — refresh counts + drawer list`.
|
||||
|
||||
---
|
||||
|
||||
## Task 12: refresh `.superpowers/skills/cyclone-cli/SKILL.md`
|
||||
|
||||
- [ ] Update lines 8–9: replace "Seven subcommands" with the canonical 15+ subcommand list (`parse-837`, `parse-835`, `validate-npi`, `validate-tax-id`, `validate-837`, `submit-batch`, `reissue-claims`, `resubmit-rejected-claims`, `resubmissions`, `pull-inbound`, `recover-ingest`, `ack-orphans`, `users`, `seed`, `backfill-rendering-npi`, `backfill-999-rejections`, `rebill-from-835`, `pull-inbound`, plus the `backup` group).
|
||||
- [ ] Commit: `docs(sp42): cyclone-cli skill — replace Seven-subcommands claim with full list`.
|
||||
|
||||
---
|
||||
|
||||
## Task 13: refresh `.superpowers/skills/cyclone-tail/SKILL.md`
|
||||
|
||||
- [ ] Update lines 13–17: refresh "three streaming pages" claim — add `Acks` (SP25 ack-live-tail) for a total of 4.
|
||||
- [ ] Update line ~44: replace `api.py:1357-2006` with `api_routers/{claims,remittances,activity}.py` + `api_routers/{acks,ta1_acks}.py`.
|
||||
- [ ] Update line ~70: replace `api.py:1401,1894,2005` with the post-SP36 router files.
|
||||
- [ ] Update line ~119: replace `api.py:1357-1401` with `api_routers/claims.py`.
|
||||
- [ ] Commit: `docs(sp42): cyclone-tail skill — refresh streaming page count + line refs`.
|
||||
|
||||
---
|
||||
|
||||
## Task 14: autoreview (the merge gate)
|
||||
|
||||
- [ ] Run the 14 acceptance-criteria grep checks from §6 of the spec; expect zero matches for each.
|
||||
- [ ] `cd backend && .venv/bin/pytest -q` — full backend suite passes (no regressions expected; this is a docs-only increment).
|
||||
- [ ] `npm test` — full vitest suite passes.
|
||||
- [ ] `git log --oneline sp42-doc-pass` shows the expected commit sequence (1× `docs(spec):` + 1× `docs(plan):` + 13× `docs(sp42):`).
|
||||
- [ ] `git status --short` shows no unexpected untracked files (note: pre-existing untracked files from the operator's working dir — `july8billing/`, `july8billing_out/`, `xxxclaims.zip`, `verification-final.txt`, `scripts/serialize_july8_to_sp41.py`, `backend/src/cyclone/submission/bulk_ingest.py`, `backend/tests/fixtures/co_medicaid_837p.997`, `backend/tests/fixtures/subscriber_roster.csv`, `docs/superpowers/plans/2026-07-07-cyclone-2010bb-nm109-fix.md`, `docs/superpowers/plans/2026-07-07-cyclone-inwindow-rebill-pipeline.md`, `docs/superpowers/specs/2026-07-07-cyclone-inwindow-rebill-pipeline-design.md`, `.grok/` — should remain untracked).
|
||||
- [ ] No semantic rewrites of unrelated prose. Each diff should touch only the lines flagged in the SP42 audit findings.
|
||||
|
||||
---
|
||||
|
||||
## Task 15: merge to `main`
|
||||
|
||||
- [ ] `git checkout main && git merge --no-ff --no-squash --no-rebase sp42-doc-pass -m "merge: SP42 doc-pass into main"`
|
||||
- [ ] `git log --oneline main -1` shows the SP42 merge commit with the expected subject.
|
||||
- [ ] `git push origin main` only after the operator has reviewed and approved the merge (per cyclone-spec — the merge commit is the audit trail; the operator must approve).
|
||||
|
||||
---
|
||||
|
||||
## Verification commands (one-shot)
|
||||
|
||||
```bash
|
||||
# Acceptance criteria §1 — no stale "next free is SP24" / "next is SP25"
|
||||
grep -rE 'SP22.*SP24|next free increment is SP24|next increment is SP24|next free is SP24|next is SP25' \
|
||||
CLAUDE.md README.md docs/ARCHITECTURE.md docs/REQUIREMENTS.md docs/RUNBOOK.md .superpowers/skills/ 2>/dev/null
|
||||
|
||||
# §2 — SP23 "awaiting user decision" / "candidate" / "product fork"
|
||||
grep -rE 'awaiting user decision|SP23.*candidate|SP23.*product fork' docs/ARCHITECTURE.md docs/REQUIREMENTS.md 2>/dev/null
|
||||
|
||||
# §3 — stale LOC claims
|
||||
grep -rE 'api\.py.*3,548|3,548 LOC|store\.py.*2,423|2,423 LOC|2,172 LOC|3,145 LOC' \
|
||||
CLAUDE.md README.md docs/ARCHITECTURE.md docs/REQUIREMENTS.md .superpowers/skills/ 2>/dev/null
|
||||
|
||||
# §4 — 127.0.0.1 in bind-address context
|
||||
grep -nE '127\.0\.0\.1' CLAUDE.md README.md docs/ARCHITECTURE.md docs/REQUIREMENTS.md 2>/dev/null
|
||||
|
||||
# §5 — README "no auth, no internet exposure"
|
||||
grep -nE 'no auth, no internet exposure' README.md
|
||||
|
||||
# §6 — "4 router modules" / "four router modules"
|
||||
grep -rE '4 router modules|four router modules' .superpowers/skills/cyclone-api-router/SKILL.md docs/ARCHITECTURE.md docs/REQUIREMENTS.md 2>/dev/null
|
||||
|
||||
# §7 — "Seven subcommands" / "7 subcommands"
|
||||
grep -rE 'Seven subcommands|7 subcommands' .superpowers/skills/cyclone-cli/SKILL.md
|
||||
|
||||
# §8 — "single 2412-line module" / "SP21 in progress" / "SP21 in-flight"
|
||||
grep -rE 'single 2412-line module|SP21.*in.progress|SP21.*in-flight' .superpowers/skills/cyclone-store/SKILL.md
|
||||
|
||||
# §9 — "12 SQL migrations"
|
||||
grep -rE '12 SQL migrations' CLAUDE.md docs/ARCHITECTURE.md docs/REQUIREMENTS.md 2>/dev/null
|
||||
|
||||
# §10 / §11 — test suites
|
||||
cd backend && .venv/bin/pytest -q
|
||||
npm test
|
||||
```
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user