Compare commits
134 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 3bc5740e8b | |||
| b0e06a2dd0 | |||
| d1cd6e1a51 | |||
| f25214189a | |||
| e4f3d25f3a | |||
| 750f560ee0 | |||
| 0193ee4c32 | |||
| 6bda5005c1 | |||
| 27bca33b09 | |||
| f10ab83628 | |||
| 244015a361 | |||
| 3bf5622010 | |||
| 0625c83a45 | |||
| cf7c343ff0 | |||
| dae7749464 | |||
| 2e7ad471e0 | |||
| dfd654202e | |||
| 15c85300f6 | |||
| 1b2f6c6b21 | |||
| 0f3b264e41 | |||
| de77f19d9d | |||
| c4bc118557 | |||
| 055b4ae30e | |||
| 4d3eef22ef | |||
| d8707ba874 | |||
| c491003287 | |||
| 4363b4fe41 | |||
| d44948cb90 | |||
| 9e2dde6ddb | |||
| 2d40fbbcbb | |||
| a2ec65e5c6 | |||
| 18fa119ff7 | |||
| ab91449e62 | |||
| 9bade2429c | |||
| 5a54961930 | |||
| b484ca36dc | |||
| 75a2800a9d | |||
| 6624a0bafd | |||
| a12104fb0f | |||
| edee0a6259 | |||
| 97512ec4a7 | |||
| bde3060e9e | |||
| 1d2572e624 | |||
| ffacfd8665 | |||
| 9cc13e7940 | |||
| 37caf6af77 | |||
| 02b879a204 | |||
| fba594baf4 | |||
| 190f4b8c8d | |||
| 9d11ffcd8a | |||
| 21a5485faa | |||
| 3fd2c445e7 | |||
| 9e16b8d9bd | |||
| 1c367ddbe7 | |||
| d8f3fb15f9 | |||
| 602dc352c5 | |||
| 490a1b9478 | |||
| b094231995 | |||
| c54b2c1867 | |||
| a97f1d1350 | |||
| d2512945ca | |||
| 9e775c0150 | |||
| e10d3886c2 | |||
| ec87f98f02 | |||
| f88a7c7c4f | |||
| aca9667ff8 | |||
| 999a762c86 | |||
| 53a8c77cb8 | |||
| de8bc2e153 | |||
| 1363d36046 | |||
| 146cb6d17d | |||
| 191bc935c3 | |||
| 029623f3a5 | |||
| 1648c81425 | |||
| f1bee546f6 | |||
| 603ec8842a | |||
| 686c11f480 | |||
| 8d11b391a0 | |||
| 4b22193c4a | |||
| d8c03fde3f | |||
| 014e02ad42 | |||
| 4360ef7209 | |||
| fe84ce7cf3 | |||
| cfc95307e3 | |||
| c8f8f5d3c6 | |||
| 9644db8c51 | |||
| fbe7b2358d | |||
| b96da8a5a4 | |||
| 14fcbca5f1 | |||
| d8841834dc | |||
| d81b6ed4fc | |||
| 59c3275adf | |||
| f5d119fbe7 | |||
| d5f95b4f3c | |||
| 44a6fb031a | |||
| cf1a0a80d8 | |||
| 34542d1d34 | |||
| 4c05c6527b | |||
| 4592bca372 | |||
| 79fa30d018 | |||
| 35730fcf14 | |||
| 30e1add8a2 | |||
| d248a5f282 | |||
| 0f1e609888 | |||
| 9d8f83d111 | |||
| 454c3598b1 | |||
| f57f1d2875 | |||
| 315fbfec42 | |||
| 6507a8c874 | |||
| 1381a7652d | |||
| c3a6c53096 | |||
| a436538c15 | |||
| dd7da18279 | |||
| 9cb0311544 | |||
| b7be9f38bc | |||
| 1b74af4d3a | |||
| b80e40e7e9 | |||
| edca04868d | |||
| 74aa64f995 | |||
| 4ef09f9416 | |||
| 3c907d17e9 | |||
| 675faf9844 | |||
| 1e8217ff84 | |||
| 3075955826 | |||
| 5b4c1513b6 | |||
| 4e580ffe0e | |||
| 13a3e7c334 | |||
| 439a3c3d4b | |||
| 5db764d50e | |||
| 2d3667738d | |||
| b2d88d13d3 | |||
| 9fda46b891 | |||
| d401904412 | |||
| 7d2000ec31 |
@@ -38,3 +38,7 @@ claims_output/
|
||||
# Brainstorm session artifacts (visual companion mockups, events, server state).
|
||||
# Skills under .superpowers/skills/ are committed project-scoped guidance.
|
||||
.superpowers/brainstorm/
|
||||
|
||||
# SP33+ scratch / production-data ingest. Generated artifacts live
|
||||
# here only — the source EDI sits under docs/prodfiles/.
|
||||
ingest/
|
||||
|
||||
@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
|
||||
## What this is
|
||||
|
||||
Cyclone is a self-hosted X12 EDI claims-management suite for a single billing office (Colorado Medicaid currently). It parses 837P professional claims and 835 ERA remittances (X12 005010X222A1 / 005010X221A1) and also handles 999, TA1, 270, 271, and 277CA. Local-only by design: binds to `127.0.0.1`, requires login (auth boundary is HTTP; bcrypt + HttpOnly session cookie; first admin bootstrapped from `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD` env vars; see SP24 spec for the full posture), no internet exposure.
|
||||
Cyclone is a self-hosted X12 EDI claims-management suite for a single billing office (Colorado Medicaid currently). It parses 837P professional claims and 835 ERA remittances (X12 005010X222A1 / 005010X221A1) and also handles 999, TA1, 270, 271, and 277CA. Always binds to `0.0.0.0` — the host firewall / compose port publishing is what restricts reachability, not the bind address. Requires login (auth boundary is HTTP; bcrypt + HttpOnly session cookie; first admin bootstrapped from `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD` env vars; see SP24 spec for the full posture). LAN-only by design — don't expose the published ports to the WAN.
|
||||
|
||||
Stack: one Python process (FastAPI + uvicorn, port 8000) + one Node process in dev (Vite, port 5173). The authoritative state is a single SQLite file at `~/.local/share/cyclone/cyclone.db` (or SQLCipher at the same path when the macOS Keychain entry + `sqlcipher3` are both present).
|
||||
|
||||
@@ -30,7 +30,7 @@ Optional backend extras: `pip install -e '.[sqlcipher]'` (encryption at rest, SP
|
||||
```bash
|
||||
# Terminal 1 — backend
|
||||
cd backend
|
||||
.venv/bin/python -m cyclone serve # default 127.0.0.1:8000
|
||||
.venv/bin/python -m cyclone serve # default 0.0.0.0:8000
|
||||
# CYCLONE_PORT=... overrides port; CYCLONE_RELOAD=1 enables uvicorn --reload
|
||||
# Or: .venv/bin/uvicorn cyclone.api:app --reload --port 8000
|
||||
|
||||
@@ -179,6 +179,6 @@ Exit codes are documented per subcommand in `cyclone-cli` — `0` for success, `
|
||||
- **Don't call `useTailStream` from inside a `use<X>` hook.** The subscription lives on the page so the lifecycle ties to whoever mounts the hook, not to whoever happens to call it.
|
||||
- **The store facade.** The public API of `cyclone.store` is preserved through SP21's split — call through the facade, not directly into the underlying modules.
|
||||
- **Encryption is optional, not required.** When the Keychain entry is missing **or** `sqlcipher3` is not installed, the DB falls back to plain SQLite. Don't fail boot on missing encryption.
|
||||
- **Local-only by design.** The backend binds to `127.0.0.1`, requires login (bcrypt + HttpOnly session cookie; see SP24 spec), and the threat model is still a stolen/imaged drive — SQLCipher at rest and the macOS Keychain handle that. The auth boundary is the HTTP layer; the file-system posture is unchanged. Don't add internet exposure. Don't disable auth without an explicit `CYCLONE_AUTH_DISABLED=1` env var (the escape hatch logs a WARNING at boot).
|
||||
- **Always bind to 0.0.0.0.** The bind address does not control reachability — the host firewall / compose port publishing does. Requires login (bcrypt + HttpOnly session cookie; see SP24 spec), and the threat model is still a stolen/imaged drive — SQLCipher at rest and the macOS Keychain handle that. The auth boundary is the HTTP layer; the file-system posture is unchanged. Don't expose the published ports to the public internet (LAN-only or VPN-fronted). Don't disable auth without an explicit `CYCLONE_AUTH_DISABLED=1` env var (the escape hatch logs a WARNING at boot).
|
||||
</content>
|
||||
</invoke>
|
||||
@@ -30,7 +30,7 @@ Two terminals:
|
||||
# Terminal 1 — backend
|
||||
cd backend
|
||||
.venv/bin/python -m cyclone serve
|
||||
# (defaults to 127.0.0.1:8000; override with CYCLONE_PORT=...; reload with CYCLONE_RELOAD=1)
|
||||
# (defaults to 0.0.0.0:8000; override with CYCLONE_HOST / CYCLONE_PORT / CYCLONE_RELOAD=1)
|
||||
|
||||
# Terminal 2 — frontend
|
||||
npm run dev
|
||||
@@ -690,7 +690,7 @@ backup create` manually retains full control.
|
||||
The `clearhouse.submit` endpoint uses `paramiko` to push a batch of
|
||||
generated 837 files to the dzinesco SFTP server
|
||||
(`mft.gainwelltechnologies.com:22`, path
|
||||
`/CO XIX/PROD/coxix_prod_11525703/FromHPE`). The SFTP credential is
|
||||
`/CO XIX/PROD/coxix_prod_11525703/ToHPE`). The SFTP credential is
|
||||
fetched from the macOS Keychain at call time — never read from YAML,
|
||||
never logged, never written to disk. The wire-up honors the file-naming
|
||||
template stored in the `clearhouse` config:
|
||||
@@ -898,7 +898,7 @@ Shipped sub-projects (most recent first):
|
||||
- **Sub-project 13 (shipped) — SFTP wire-up.** `paramiko`-backed
|
||||
`SftpClient` replaces the SP9 stub. The clearhouse.submit endpoint
|
||||
actually pushes to
|
||||
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
|
||||
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/ToHPE`.
|
||||
SFTP credentials are read from the macOS Keychain at call time.
|
||||
- **Sub-project 12 (shipped) — Encryption at rest.** Optional
|
||||
SQLCipher AES-256 encryption of the SQLite file, with the key
|
||||
@@ -1160,7 +1160,7 @@ the one-time setup recipe.
|
||||
|
||||
- `POST /api/clearhouse/submit` — same endpoint as SP9; the
|
||||
implementation is now a real `paramiko` `SftpClient.write` to
|
||||
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
|
||||
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/ToHPE`.
|
||||
SFTP credentials are fetched from the macOS Keychain at call time.
|
||||
|
||||
## License
|
||||
|
||||
+5
-2
@@ -35,7 +35,10 @@ COPY pyproject.toml ./
|
||||
# the cached wheel metadata when the name+version matches. See git
|
||||
# history on this file for the long version.
|
||||
COPY src/ ./src/
|
||||
RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher]'
|
||||
# Install the sftp extra alongside sqlcipher so the real-mode SFTP
|
||||
# client (paramiko) is available inside the container — required by
|
||||
# SP25 + SP26 for live Gainwell MFT polling.
|
||||
RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher,sftp]'
|
||||
|
||||
# ---------- runtime ----------
|
||||
FROM python:3.11-slim-bookworm
|
||||
@@ -53,7 +56,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
WORKDIR /app
|
||||
|
||||
COPY --from=builder /wheels /wheels
|
||||
RUN pip install --no-cache-dir --no-index --find-links /wheels 'cyclone[sqlcipher]' \
|
||||
RUN pip install --no-cache-dir --no-index --find-links /wheels 'cyclone[sqlcipher,sftp]' \
|
||||
&& rm -rf /wheels
|
||||
|
||||
# NOTE: we deliberately do NOT drop privileges to the `cyclone` user.
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
"""Entry point for ``python -m cyclone``.
|
||||
|
||||
* ``python -m cyclone`` (no args) — Click CLI (``cli.main``)
|
||||
* ``python -m cyclone serve`` — start the FastAPI app on 127.0.0.1:8000
|
||||
* ``python -m cyclone serve`` — start the FastAPI app on 0.0.0.0:8000
|
||||
|
||||
Honors the env vars:
|
||||
|
||||
* ``CYCLONE_HOST`` (default ``0.0.0.0`` — always bind to all interfaces)
|
||||
* ``CYCLONE_PORT`` (default ``8000``)
|
||||
* ``CYCLONE_RELOAD`` (default ``0``; set to ``1`` to enable uvicorn reload)
|
||||
"""
|
||||
@@ -40,12 +41,13 @@ def main() -> None:
|
||||
|
||||
if len(sys.argv) >= 2 and sys.argv[1] == "serve":
|
||||
port = os.environ.get("CYCLONE_PORT", "8000")
|
||||
# Local-only by default — see CLAUDE.md. The Docker image
|
||||
# overrides to 0.0.0.0 via compose env so the frontend
|
||||
# container on the compose bridge network can reach the
|
||||
# backend. Network isolation is provided by the bridge
|
||||
# network itself (only cyclone-frontend joins).
|
||||
host = os.environ.get("CYCLONE_HOST", "127.0.0.1")
|
||||
# Always bind to 0.0.0.0 so the API is reachable from the
|
||||
# frontend container on the compose bridge network AND from
|
||||
# the host (Vite dev proxy) AND from the LAN. Network isolation
|
||||
# is provided by the host firewall / compose port publishing,
|
||||
# not by binding to loopback. Override with CYCLONE_HOST if you
|
||||
# have a reason to restrict.
|
||||
host = os.environ.get("CYCLONE_HOST", "0.0.0.0")
|
||||
reload = os.environ.get("CYCLONE_RELOAD", "0") == "1"
|
||||
sys.argv = [
|
||||
sys.argv[0],
|
||||
|
||||
+772
-83
File diff suppressed because it is too large
Load Diff
@@ -1,4 +1,4 @@
|
||||
"""``/api/acks`` — list & detail endpoints for the 999 ACK inbox.
|
||||
"""``/api/acks`` — list, detail, and live-tail stream for 999 ACKs.
|
||||
|
||||
These are the persisted acknowledgment rows produced by
|
||||
``POST /api/parse-999``. The frontend ``useAcks`` hook re-shapes the
|
||||
@@ -7,58 +7,74 @@ list payload to its ``Ack`` interface in ``src/types/index.ts``.
|
||||
The detail endpoint returns the full ``raw_json`` payload plus the
|
||||
regenerated ``raw_999_text`` so the UI can show "view source" without a
|
||||
second round-trip.
|
||||
|
||||
SP25: ``/api/acks/stream`` joins the live-tail triplet — the Acks
|
||||
page mounts ``useTailStream("acks")`` and ``useMergedTail("acks", …)``
|
||||
to see new 999 acks the moment they land (whether from the SFTP
|
||||
poller or a manual upload).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
from typing import Any, AsyncIterator
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from cyclone.api_helpers import ndjson_stream_list, wants_ndjson
|
||||
from cyclone import db
|
||||
from cyclone.api_helpers import (
|
||||
ndjson_line,
|
||||
ndjson_stream_list,
|
||||
tail_events,
|
||||
wants_ndjson,
|
||||
)
|
||||
from cyclone.parsers.models_999 import ParseResult999
|
||||
from cyclone.parsers.serialize_999 import serialize_999
|
||||
from cyclone.store import store
|
||||
from cyclone.pubsub import EventBus
|
||||
from cyclone.store import store, to_ui_ack
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _ack_to_ui(row) -> dict:
|
||||
"""Map an ``Ack`` ORM row to the UI shape used by ``/api/acks``.
|
||||
|
||||
Field names match the rest of the Cyclone API (snake_case). The
|
||||
frontend ``useAcks`` hook re-shapes this to the camelCase ``Ack``
|
||||
interface in ``src/types/index.ts``.
|
||||
"""
|
||||
return {
|
||||
"id": row.id,
|
||||
"source_batch_id": row.source_batch_id,
|
||||
"accepted_count": row.accepted_count,
|
||||
"rejected_count": row.rejected_count,
|
||||
"received_count": row.received_count,
|
||||
"ack_code": row.ack_code,
|
||||
"parsed_at": (
|
||||
row.parsed_at.isoformat().replace("+00:00", "Z")
|
||||
if row.parsed_at is not None
|
||||
else ""
|
||||
),
|
||||
}
|
||||
# SP25: ``_ack_to_ui`` moved to ``cyclone.store.ui.to_ui_ack`` so the
|
||||
# live-tail event payload (``ack_received``) matches the list endpoint
|
||||
# shape byte-for-byte.
|
||||
|
||||
|
||||
@router.get("/api/acks")
|
||||
def list_acks_endpoint(
|
||||
request: Request,
|
||||
limit: int = Query(100, ge=1, le=1000),
|
||||
limit: int = Query(100, ge=1, le=5000),
|
||||
offset: int = Query(0, ge=0),
|
||||
) -> Any:
|
||||
"""Return the list of persisted 999 ACKs, newest first."""
|
||||
"""Return the list of persisted 999 ACKs, newest first.
|
||||
|
||||
``limit`` caps the page size; ``offset`` lets the UI walk the
|
||||
full set without holding it all in memory. ``aggregates`` is
|
||||
summed over the *full* row set (not the page) so the KPI strip
|
||||
on the Acks page reflects every persisted 999, not just the
|
||||
visible 50. Without server-side aggregates the page would
|
||||
silently under-report (silent-failure mode) once the row count
|
||||
exceeds the page size.
|
||||
"""
|
||||
rows = store.list_acks()
|
||||
items = [_ack_to_ui(r) for r in rows[:limit]]
|
||||
items = [to_ui_ack(r) for r in rows[offset : offset + limit]]
|
||||
total = len(rows)
|
||||
returned = len(items)
|
||||
has_more = total > returned
|
||||
has_more = offset + returned < total
|
||||
aggregates = {
|
||||
"accepted_count": sum(r.accepted_count or 0 for r in rows),
|
||||
"rejected_count": sum(r.rejected_count or 0 for r in rows),
|
||||
"received_count": sum(r.received_count or 0 for r in rows),
|
||||
}
|
||||
# SP28: batch-fetch linked_claim_ids per ack row in one query to
|
||||
# avoid N+1 — see ``find_linked_claim_ids_for_acks`` below.
|
||||
ack_ids = [r.id for r in rows]
|
||||
linked_map = _find_linked_claim_ids_for_acks(ack_ids, kind="999")
|
||||
for item, aid in zip(items, ack_ids[offset : offset + limit]):
|
||||
item["linked_claim_ids"] = linked_map.get(aid, [])
|
||||
if wants_ndjson(request):
|
||||
return StreamingResponse(
|
||||
ndjson_stream_list(items, total, returned, has_more),
|
||||
@@ -69,9 +85,68 @@ def list_acks_endpoint(
|
||||
"total": total,
|
||||
"returned": returned,
|
||||
"has_more": has_more,
|
||||
"aggregates": aggregates,
|
||||
}
|
||||
|
||||
|
||||
def _find_linked_claim_ids_for_acks(
|
||||
ack_ids: list[int], *, kind: str
|
||||
) -> dict[int, list[str]]:
|
||||
"""Batch-fetch {ack_id: [claim_id, …]} for the listed ack rows.
|
||||
|
||||
One SELECT against ``claim_acks`` keyed on the page's ack_ids —
|
||||
avoids an N+1 round-trip when the page renders the
|
||||
"🔗 N claims" badge per row. Used by the 999 / TA1 / 277CA
|
||||
list endpoints. Returns a ``{ack_id: [claim_id]}`` map; acks
|
||||
with no links map to ``[]`` (default).
|
||||
"""
|
||||
out: dict[int, list[str]] = {aid: [] for aid in ack_ids}
|
||||
if not ack_ids:
|
||||
return out
|
||||
with db.SessionLocal()() as s:
|
||||
rows = (
|
||||
s.query(db.ClaimAck.ack_id, db.ClaimAck.claim_id)
|
||||
.filter(
|
||||
db.ClaimAck.ack_kind == kind,
|
||||
db.ClaimAck.ack_id.in_(ack_ids),
|
||||
db.ClaimAck.claim_id.isnot(None),
|
||||
)
|
||||
.all()
|
||||
)
|
||||
for ack_id, claim_id in rows:
|
||||
out[ack_id].append(claim_id)
|
||||
return out
|
||||
|
||||
|
||||
@router.get("/api/acks/stream")
|
||||
async def acks_stream(
|
||||
request: Request,
|
||||
limit: int = Query(100, ge=1, le=1000),
|
||||
) -> StreamingResponse:
|
||||
"""Stream 999 ACKs as NDJSON: snapshot first, then live events.
|
||||
|
||||
SP25: this endpoint joins the live-tail triplet — subscribes to
|
||||
``ack_received`` and emits one ``item`` per snapshot row plus a
|
||||
single ``snapshot_end`` line, then forwards live events from
|
||||
the bus. Matches the wire format used by ``/api/claims/stream``,
|
||||
``/api/remittances/stream``, and ``/api/activity/stream``.
|
||||
|
||||
NOTE: registered BEFORE ``/api/acks/{ack_id}`` so the literal
|
||||
``stream`` path segment doesn't get matched as an ack id.
|
||||
"""
|
||||
bus: EventBus = request.app.state.event_bus
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
rows = store.list_acks()[:limit]
|
||||
for row in rows:
|
||||
yield ndjson_line({"type": "item", "data": to_ui_ack(row)})
|
||||
yield ndjson_line({"type": "snapshot_end", "data": {"count": len(rows)}})
|
||||
async for chunk in tail_events(request, bus, ["ack_received"]):
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
|
||||
|
||||
@router.get("/api/acks/{ack_id}")
|
||||
def get_ack_endpoint(ack_id: int) -> dict:
|
||||
"""Return one persisted ACK row with its parsed detail.
|
||||
@@ -86,7 +161,7 @@ def get_ack_endpoint(ack_id: int) -> dict:
|
||||
status_code=404,
|
||||
detail={"error": "Not found", "detail": f"Ack {ack_id} not found"},
|
||||
)
|
||||
body = _ack_to_ui(row)
|
||||
body = to_ui_ack(row)
|
||||
body["raw_json"] = row.raw_json
|
||||
# Regenerate the X12 text from raw_json so the operator can download
|
||||
# the actual 999 file. (SP3 P3 follow-up: list endpoint doesn't carry
|
||||
|
||||
@@ -0,0 +1,306 @@
|
||||
"""``/api/claim-acks`` (and per-claim/per-ack surfaces) — SP28.
|
||||
|
||||
Seven endpoints that surface the ``claim_acks`` join table to the
|
||||
frontend + manual-match fallback for orphans. Mounted by
|
||||
``cyclone.api`` alongside the existing ``/api/acks`` and
|
||||
``/api/ta1-acks`` routers.
|
||||
|
||||
The live-tail endpoints (``/api/claims/{id}/acks/stream`` and
|
||||
``/api/acks/{kind}/{id}/claims/stream``) subscribe to the
|
||||
``claim_ack_written`` bus event so the ClaimDrawer Acknowledgments
|
||||
panel and the per-ack claims list refresh in real time.
|
||||
|
||||
Manual match is any-logged-in user (D5) — this endpoint mutates
|
||||
metadata only (``claim_acks`` row + live-tail event), no
|
||||
``Claim.state`` mutation, no payment data. Idempotent: re-calling
|
||||
with the same ``claim_id`` returns 200 with the existing row.
|
||||
Rejects (409) when the claim is in a terminal state (``REVERSED``).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any, AsyncIterator, Literal
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import JSONResponse, StreamingResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.api_helpers import ndjson_line, tail_events
|
||||
from cyclone.pubsub import EventBus
|
||||
from cyclone.store import store, to_ui_claim_ack
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
AckKind = Literal["999", "277ca", "ta1"]
|
||||
|
||||
|
||||
class MatchClaimBody(BaseModel):
|
||||
"""Body for ``POST /api/acks/{kind}/{ack_id}/match-claim``."""
|
||||
|
||||
claim_id: str = Field(..., min_length=1)
|
||||
set_control_number: str | None = None
|
||||
set_accept_reject_code: str | None = None
|
||||
ak2_index: int | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-claim surface
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.get("/api/claims/{claim_id}/acks/stream")
|
||||
async def claim_acks_stream(
|
||||
request: Request,
|
||||
claim_id: str,
|
||||
) -> StreamingResponse:
|
||||
"""Stream ClaimAck rows for one claim as NDJSON.
|
||||
|
||||
Subscribes to ``claim_ack_written`` and filters for rows where
|
||||
``claim_id`` matches the path param. Each matching event is
|
||||
emitted as ``{"type": "item", "data": to_ui_claim_ack(...)}``;
|
||||
the client-side ``useMergedTail`` hook dedupes by id.
|
||||
|
||||
Registered BEFORE ``/api/claims/{claim_id}/acks`` so the literal
|
||||
``stream`` path segment doesn't get matched as a suffix.
|
||||
"""
|
||||
bus: EventBus = request.app.state.event_bus
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
rows = [
|
||||
r for r in store.list_acks_for_claim(claim_id)
|
||||
if r.claim_id is not None # filter out TA1 batch-level rows
|
||||
]
|
||||
for row in rows:
|
||||
yield ndjson_line({"type": "item", "data": to_ui_claim_ack(row)})
|
||||
yield ndjson_line({
|
||||
"type": "snapshot_end",
|
||||
"data": {"count": len(rows)},
|
||||
})
|
||||
|
||||
async for chunk in tail_events(request, bus, ["claim_ack_written"]):
|
||||
# tail_events yields full NDJSON lines; the client filter
|
||||
# picks claim_id matches. We forward every event so the
|
||||
# wire format mirrors /api/claims/stream.
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
|
||||
|
||||
@router.get("/api/claims/{claim_id}/acks")
|
||||
def list_acks_for_claim_endpoint(claim_id: str) -> Any:
|
||||
"""Return every ClaimAck row for one claim (per-claim only).
|
||||
|
||||
TA1 batch-level rows (where ``claim_id IS NULL``) are filtered
|
||||
out — those don't belong to a specific claim, they're a
|
||||
envelope-level acknowledgement that hangs off the originating
|
||||
837 batch. The ClaimDrawer Acknowledgments panel is
|
||||
per-claim only.
|
||||
"""
|
||||
rows = store.list_acks_for_claim(claim_id)
|
||||
items = [to_ui_claim_ack(r) for r in rows if r.claim_id is not None]
|
||||
return {"total": len(items), "items": items}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-ack surface
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.get("/api/acks/{kind}/{ack_id}/claims/stream")
|
||||
async def ack_claims_stream(
|
||||
request: Request,
|
||||
kind: AckKind,
|
||||
ack_id: int,
|
||||
) -> StreamingResponse:
|
||||
"""Stream ClaimAck rows for one ack as NDJSON.
|
||||
|
||||
Subscribes to ``claim_ack_written`` and forwards events the
|
||||
store knows about. Clients filter by ack_id + ack_kind on the
|
||||
client side.
|
||||
"""
|
||||
bus: EventBus = request.app.state.event_bus
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
rows = store.list_claims_for_ack(kind, ack_id)
|
||||
for row in rows:
|
||||
yield ndjson_line({"type": "item", "data": to_ui_claim_ack(row)})
|
||||
yield ndjson_line({
|
||||
"type": "snapshot_end",
|
||||
"data": {"count": len(rows)},
|
||||
})
|
||||
async for chunk in tail_events(request, bus, ["claim_ack_written"]):
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
|
||||
|
||||
@router.get("/api/acks/{kind}/{ack_id}/claims")
|
||||
def list_claims_for_ack_endpoint(kind: AckKind, ack_id: int) -> Any:
|
||||
"""Return every ClaimAck row for one ack (any kind).
|
||||
|
||||
For 999 / 277CA: returns 0..N rows (one per AK2 / ClaimStatus).
|
||||
For TA1: returns 0..1 row (envelope-level, populated batch_id).
|
||||
"""
|
||||
rows = store.list_claims_for_ack(kind, ack_id)
|
||||
items = [to_ui_claim_ack(r) for r in rows]
|
||||
return {"total": len(items), "items": items}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Manual match / unmatch (D5, D9)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.post("/api/acks/{kind}/{ack_id}/match-claim")
|
||||
def manual_match_claim_endpoint(
|
||||
request: Request,
|
||||
kind: AckKind,
|
||||
ack_id: int,
|
||||
body: MatchClaimBody,
|
||||
) -> Any:
|
||||
"""Manual link fallback (D5/D9). Any-logged-in-user posture.
|
||||
|
||||
Inserts a ``claim_acks`` row with ``linked_by="manual"`` and
|
||||
publishes ``claim_ack_written`` so the drawers refresh. The
|
||||
endpoint is idempotent: if a row already exists for this dedup
|
||||
key, the existing row is returned (200). 409 when the claim is
|
||||
in a terminal state (``REVERSED``). 404 when the claim doesn't
|
||||
exist or the ack doesn't exist.
|
||||
"""
|
||||
# Verify the claim exists and is in a non-terminal state.
|
||||
from cyclone.db import ClaimState as _CS
|
||||
with db.SessionLocal()() as s:
|
||||
claim = s.get(db.Claim, body.claim_id)
|
||||
if claim is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={
|
||||
"error": "Not found",
|
||||
"detail": f"Claim {body.claim_id} not found",
|
||||
},
|
||||
)
|
||||
if claim.state == _CS.REVERSED:
|
||||
return JSONResponse(
|
||||
status_code=409,
|
||||
content={
|
||||
"error": "Conflict",
|
||||
"detail": (
|
||||
f"Claim {body.claim_id} is in terminal state "
|
||||
f"{claim.state.value} and cannot be linked."
|
||||
),
|
||||
},
|
||||
)
|
||||
# Verify the ack exists (any kind).
|
||||
ack_table = {
|
||||
"999": db.Ack,
|
||||
"277ca": db.Two77caAck,
|
||||
"ta1": db.Ta1Ack,
|
||||
}[kind]
|
||||
if s.get(ack_table, ack_id) is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={
|
||||
"error": "Not found",
|
||||
"detail": f"{kind} ACK {ack_id} not found",
|
||||
},
|
||||
)
|
||||
# Idempotency: if a manual or auto link already exists, return it.
|
||||
existing = (
|
||||
s.query(db.ClaimAck)
|
||||
.filter(
|
||||
db.ClaimAck.ack_kind == kind,
|
||||
db.ClaimAck.ack_id == ack_id,
|
||||
db.ClaimAck.claim_id == body.claim_id,
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if existing is not None:
|
||||
return {
|
||||
"link": to_ui_claim_ack(existing),
|
||||
"created": False,
|
||||
}
|
||||
|
||||
# Insert via the store so the publish-from-store contract fires.
|
||||
row = store.add_claim_ack(
|
||||
claim_id=body.claim_id,
|
||||
batch_id=None,
|
||||
ack_id=ack_id,
|
||||
ack_kind=kind,
|
||||
ak2_index=body.ak2_index,
|
||||
set_control_number=body.set_control_number,
|
||||
set_accept_reject_code=body.set_accept_reject_code,
|
||||
linked_by="manual",
|
||||
event_bus=request.app.state.event_bus,
|
||||
)
|
||||
return {"link": to_ui_claim_ack(row), "created": True}
|
||||
|
||||
|
||||
@router.delete("/api/acks/{kind}/{ack_id}/match-claim/{claim_id}")
|
||||
def manual_unmatch_claim_endpoint(
|
||||
request: Request,
|
||||
kind: AckKind,
|
||||
ack_id: int,
|
||||
claim_id: str,
|
||||
) -> Any:
|
||||
"""Unlink (preserves ``Claim.state`` mutation; only removes the row).
|
||||
|
||||
404 when no link row exists for the dedup key. Publishes
|
||||
``claim_ack_dropped`` so live-tail subscribers remove the link.
|
||||
"""
|
||||
from cyclone import db as _db
|
||||
with _db.SessionLocal()() as s:
|
||||
link = (
|
||||
s.query(_db.ClaimAck)
|
||||
.filter(
|
||||
_db.ClaimAck.ack_kind == kind,
|
||||
_db.ClaimAck.ack_id == ack_id,
|
||||
_db.ClaimAck.claim_id == claim_id,
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if link is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={
|
||||
"error": "Not found",
|
||||
"detail": (
|
||||
f"No link for {kind} ack {ack_id} → "
|
||||
f"claim {claim_id}"
|
||||
),
|
||||
},
|
||||
)
|
||||
link_id = link.id
|
||||
|
||||
removed = store.remove_claim_ack(link_id, event_bus=request.app.state.event_bus)
|
||||
return {"removed": removed, "link_id": link_id}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Inbox ack-orphans lane (spec §D7)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.get("/api/inbox/ack-orphans")
|
||||
def list_ack_orphans_endpoint(
|
||||
kind: AckKind | None = Query(None, description="Filter by ack kind"),
|
||||
) -> Any:
|
||||
"""List acks with no resolvable Claim row of their own kind.
|
||||
|
||||
Used by the Inbox "Ack orphans" lane for the operator's manual
|
||||
reconciliation flow. Mirrors ``/api/inbox/remit-orphans``.
|
||||
"""
|
||||
if kind is not None:
|
||||
items = store.find_ack_orphans(kind)
|
||||
return {"total": len(items), "items": items}
|
||||
items_999 = store.find_ack_orphans("999")
|
||||
items_277ca = store.find_ack_orphans("277ca")
|
||||
items_ta1 = store.find_ack_orphans("ta1")
|
||||
all_items = items_999 + items_277ca + items_ta1
|
||||
return {"total": len(all_items), "items": all_items}
|
||||
|
||||
|
||||
__all__ = ["router"]
|
||||
@@ -1,4 +1,4 @@
|
||||
"""``/api/ta1-acks`` — list & detail endpoints for persisted TA1 envelopes.
|
||||
"""``/api/ta1-acks`` — list, detail, and live-tail stream for TA1 envelopes.
|
||||
|
||||
TA1 is the interchange-control ACK (ISA/IEA acknowledgement). It's a
|
||||
single segment, no functional group, no transaction set. Cyclone
|
||||
@@ -7,34 +7,29 @@ row can sit alongside the 999 / 277CA ack rows without special-casing.
|
||||
|
||||
The detail endpoint also reconstructs the TA1 segment string
|
||||
(``TA1*...~``) so the operator can copy it into a downstream tool.
|
||||
|
||||
SP25: ``/api/ta1-acks/stream`` joins the live-tail triplet — the
|
||||
Acks page mounts ``useTailStream("ta1_acks")`` so the TA1 envelope
|
||||
ack section sees new rows the moment they land.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
from typing import Any, AsyncIterator
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from cyclone.api_helpers import ndjson_line, tail_events
|
||||
from cyclone import db
|
||||
from cyclone.store import store
|
||||
from cyclone.pubsub import EventBus
|
||||
from cyclone.store import store, to_ui_ta1_ack
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _ta1_to_ui(row: db.Ta1Ack) -> dict:
|
||||
"""Render a Ta1Ack row for the UI (list endpoint shape)."""
|
||||
return {
|
||||
"id": row.id,
|
||||
"control_number": row.control_number,
|
||||
"ack_code": row.ack_code,
|
||||
"note_code": row.note_code,
|
||||
"interchange_date": row.interchange_date.isoformat()
|
||||
if row.interchange_date else None,
|
||||
"interchange_time": row.interchange_time,
|
||||
"sender_id": row.sender_id,
|
||||
"receiver_id": row.receiver_id,
|
||||
"source_batch_id": row.source_batch_id,
|
||||
"parsed_at": row.parsed_at.isoformat() if row.parsed_at else None,
|
||||
}
|
||||
# SP25: ``_ta1_to_ui`` moved to ``cyclone.store.ui.to_ui_ta1_ack`` so
|
||||
# the live-tail event payload (``ta1_ack_received``) matches the list
|
||||
# endpoint shape byte-for-byte.
|
||||
|
||||
|
||||
def _serialize_ta1_from_row(row: db.Ta1Ack) -> str:
|
||||
@@ -57,20 +52,78 @@ def list_ta1_acks_endpoint(
|
||||
count regardless of the ``limit`` cap.
|
||||
"""
|
||||
rows = store.list_ta1_acks()
|
||||
items = [_ta1_to_ui(r) for r in rows[:limit]]
|
||||
items = [to_ui_ta1_ack(r) for r in rows[:limit]]
|
||||
# SP28: batch-fetch linked_claim_ids per TA1 row (TA1 envelope
|
||||
# links always carry claim_id IS NULL — populate the field for
|
||||
# symmetry so the Acks page badge render path is uniform).
|
||||
ack_ids = [r.id for r in rows]
|
||||
linked_map = _find_linked_claim_ids_for_acks_helper(ack_ids, kind="ta1")
|
||||
for item, aid in zip(items, ack_ids[:limit]):
|
||||
item["linked_claim_ids"] = linked_map.get(aid, [])
|
||||
return {
|
||||
"total": len(rows),
|
||||
"items": items,
|
||||
}
|
||||
|
||||
|
||||
def _find_linked_claim_ids_for_acks_helper(
|
||||
ack_ids: list[int], *, kind: str
|
||||
) -> dict[int, list[str]]:
|
||||
"""Batch-fetch {ack_id: [claim_id, …]} for the listed ack rows.
|
||||
|
||||
Local helper so we don't import from ``acks.py`` and create a
|
||||
circular import. See the equivalent ``_find_linked_claim_ids_for_acks``
|
||||
in ``acks.py`` for the contract.
|
||||
"""
|
||||
out: dict[int, list[str]] = {aid: [] for aid in ack_ids}
|
||||
if not ack_ids:
|
||||
return out
|
||||
with db.SessionLocal()() as s:
|
||||
rows = (
|
||||
s.query(db.ClaimAck.ack_id, db.ClaimAck.claim_id)
|
||||
.filter(
|
||||
db.ClaimAck.ack_kind == kind,
|
||||
db.ClaimAck.ack_id.in_(ack_ids),
|
||||
db.ClaimAck.claim_id.isnot(None),
|
||||
)
|
||||
.all()
|
||||
)
|
||||
for ack_id, claim_id in rows:
|
||||
out[ack_id].append(claim_id)
|
||||
return out
|
||||
|
||||
|
||||
@router.get("/api/ta1-acks/stream")
|
||||
async def ta1_acks_stream(
|
||||
request: Request,
|
||||
limit: int = Query(100, ge=1, le=1000),
|
||||
) -> StreamingResponse:
|
||||
"""Stream TA1 envelope acks as NDJSON.
|
||||
|
||||
Subscribes to ``ta1_ack_received`` and emits the same wire format
|
||||
as the other live-tail endpoints. Registered BEFORE
|
||||
``/api/ta1-acks/{ack_id}`` so ``stream`` isn't matched as an id.
|
||||
"""
|
||||
bus: EventBus = request.app.state.event_bus
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
rows = store.list_ta1_acks()[:limit]
|
||||
for row in rows:
|
||||
yield ndjson_line({"type": "item", "data": to_ui_ta1_ack(row)})
|
||||
yield ndjson_line({"type": "snapshot_end", "data": {"count": len(rows)}})
|
||||
async for chunk in tail_events(request, bus, ["ta1_ack_received"]):
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
|
||||
|
||||
@router.get("/api/ta1-acks/{ack_id}")
|
||||
def get_ta1_ack_endpoint(ack_id: int) -> dict:
|
||||
"""Return one persisted TA1 ACK row with its parsed detail."""
|
||||
row = store.get_ta1_ack(ack_id)
|
||||
if row is None:
|
||||
raise HTTPException(status_code=404, detail=f"TA1 ACK {ack_id} not found")
|
||||
body = _ta1_to_ui(row)
|
||||
body = to_ui_ta1_ack(row)
|
||||
body["raw_ta1_text"] = _serialize_ta1_from_row(row)
|
||||
body["raw_json"] = row.raw_json
|
||||
return body
|
||||
|
||||
@@ -44,11 +44,38 @@ PERMISSIONS: dict[tuple[str, str], set[Role]] = {
|
||||
("GET", "/api/inbox/export.csv"): ALL_ROLES,
|
||||
("GET", "/api/reconcile"): ALL_ROLES,
|
||||
("GET", "/api/reconciliation"): ALL_ROLES,
|
||||
("GET", "/api/acks"): ALL_ROLES,
|
||||
("GET", "/api/ta1-acks"): ALL_ROLES,
|
||||
("GET", "/api/277ca-acks"): ALL_ROLES,
|
||||
("GET", "/api/batch-diff"): ALL_ROLES,
|
||||
("GET", "/api/config"): ALL_ROLES,
|
||||
("GET", "/api/payers"): ALL_ROLES,
|
||||
("GET", "/api/audit-log"): ADMIN_ONLY,
|
||||
|
||||
# Clearhouse (SFTP creds + dzinesco identity) — admin only.
|
||||
("GET", "/api/clearhouse"): ADMIN_ONLY,
|
||||
("PATCH", "/api/clearhouse"): ADMIN_ONLY,
|
||||
("POST", "/api/clearhouse/submit"): ADMIN_ONLY,
|
||||
|
||||
# Admin ops (audit log, backup, scheduler, db rotate, reload-config).
|
||||
("GET", "/api/admin/audit-log"): ADMIN_ONLY,
|
||||
("GET", "/api/admin/audit-log/verify"): ADMIN_ONLY,
|
||||
("GET", "/api/admin/backup"): ADMIN_ONLY,
|
||||
("POST", "/api/admin/backup"): ADMIN_ONLY,
|
||||
("GET", "/api/admin/backup/scheduler"): ADMIN_ONLY,
|
||||
("POST", "/api/admin/backup/scheduler"): ADMIN_ONLY,
|
||||
("GET", "/api/admin/scheduler"): ADMIN_ONLY,
|
||||
("POST", "/api/admin/scheduler"): ADMIN_ONLY,
|
||||
("POST", "/api/admin/db/rotate-key"): ADMIN_ONLY,
|
||||
("POST", "/api/admin/reload-config"): ADMIN_ONLY,
|
||||
("GET", "/api/admin/validate-provider"): ADMIN_ONLY,
|
||||
|
||||
# Write endpoints (admin + user, no viewer).
|
||||
("POST", "/api/parse-837"): WRITE_ROLES,
|
||||
("POST", "/api/parse-835"): WRITE_ROLES,
|
||||
("POST", "/api/parse-999"): WRITE_ROLES,
|
||||
("POST", "/api/parse-ta1"): WRITE_ROLES,
|
||||
("POST", "/api/parse-277ca"): WRITE_ROLES,
|
||||
("POST", "/api/inbox"): WRITE_ROLES,
|
||||
("POST", "/api/inbox/candidates"): WRITE_ROLES,
|
||||
("POST", "/api/inbox/rejected"): WRITE_ROLES,
|
||||
@@ -57,6 +84,8 @@ PERMISSIONS: dict[tuple[str, str], set[Role]] = {
|
||||
("POST", "/api/reconciliation"): WRITE_ROLES,
|
||||
("POST", "/api/resubmit"): WRITE_ROLES,
|
||||
("POST", "/api/acks"): WRITE_ROLES,
|
||||
("POST", "/api/batches"): WRITE_ROLES, # /export-837 regenerates X12 from DB rows
|
||||
("POST", "/api/eligibility"): WRITE_ROLES,
|
||||
|
||||
# CSV export — read-only.
|
||||
("GET", "/api/export.csv"): ALL_ROLES,
|
||||
|
||||
@@ -0,0 +1,492 @@
|
||||
"""SP28: per-ACK auto-linker.
|
||||
|
||||
Three helpers, one per ACK kind, all run inside the same DB
|
||||
transaction that persists the Ack row. Each returns a slice of
|
||||
:class:`ClaimAckLinkRow` dataclasses describing what to link —
|
||||
the CALLER persists those rows via
|
||||
:func:`cyclone.store.claim_acks.add_claim_ack` (which owns the
|
||||
publish-from-store contract).
|
||||
|
||||
The two-pass join lives in
|
||||
:func:`lookup_claims_for_ack_set_response` (D10): ST02 via the
|
||||
batch envelope index (primary) + ``Claim.patient_control_number``
|
||||
(fallback). Plus :func:`link_manual` for the manual-fallback
|
||||
endpoint.
|
||||
|
||||
The helpers do NOT write to the DB session — they are pure
|
||||
readers over the session + parse result + the supplied
|
||||
``batch_envelope_index`` / ``pc_claim_lookup`` / ``batch_lookup``
|
||||
closures. This matches the existing
|
||||
``cyclone.inbox_state.apply_999_rejections`` pattern and lets the
|
||||
store facade own the publish-from-store contract for live-tail.
|
||||
|
||||
See ``docs/superpowers/specs/2026-07-02-cyclone-ack-claim-auto-link-design.md``
|
||||
§3 for the per-AK2 granularity, the two-pass join, and the
|
||||
idempotency contract enforced by ``ux_claim_acks_dedup``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Callable, Optional
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from cyclone.db import Batch, Claim
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ClaimAckLinkRow:
|
||||
"""One row to insert into ``claim_acks``.
|
||||
|
||||
Populated by the helpers; persisted by the caller via
|
||||
:func:`cyclone.store.claim_acks.add_claim_ack`. Carries every
|
||||
column the store needs to build the ORM row + the
|
||||
``claim_ack_written`` event payload.
|
||||
"""
|
||||
|
||||
claim_id: Optional[str] = None
|
||||
batch_id: Optional[str] = None
|
||||
ak2_index: Optional[int] = None
|
||||
set_control_number: Optional[str] = None
|
||||
set_accept_reject_code: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ClaimAckLinkResult:
|
||||
"""Outcome of a single helper call.
|
||||
|
||||
``linked`` is the list of :class:`ClaimAckLinkRow` rows the
|
||||
caller should persist via ``cycl_store.add_claim_ack``.
|
||||
``orphans`` is a list of free-form strings the join couldn't
|
||||
resolve — for 999/277CA these are ``set_control_number``
|
||||
values; for TA1 they're the TA1 ICN when no matching batch was
|
||||
found.
|
||||
"""
|
||||
|
||||
linked: list[ClaimAckLinkRow] = field(default_factory=list)
|
||||
orphans: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# D10 two-pass join
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def lookup_claims_for_ack_set_response(
|
||||
session: Session,
|
||||
set_control_number: str,
|
||||
*,
|
||||
batch_envelope_index: Optional[Callable[[str], Optional[str]]] = None,
|
||||
pc_claim_lookup: Optional[Callable[[str], Optional[Claim]]] = None,
|
||||
) -> list[Claim]:
|
||||
"""Two-pass join for a single AK2 set_response / 277CA claim_status.
|
||||
|
||||
D10 (spec): for a 999 AK2-2 ``set_control_number`` or a 277CA REF*1K
|
||||
``payer_claim_control_number``, return every claim this ack
|
||||
acknowledges. The primary join is
|
||||
``Batch.envelope.control_number == set_control_number`` (== source
|
||||
837's ST02 on Gainwell batches); the fallback is
|
||||
``Claim.patient_control_number == set_control_number``.
|
||||
|
||||
Returns 0..N matching claims (one-ack-to-many when one 837 batch
|
||||
shipped multiple claims under one ST02).
|
||||
|
||||
Args:
|
||||
session: SQLAlchemy session the caller owns.
|
||||
set_control_number: the AK2-2 / REF*1K value to resolve.
|
||||
batch_envelope_index: optional pre-built index that maps
|
||||
``Batch.envelope.control_number`` → ``batch.id`` (built
|
||||
once per ingest via ``store.batch_envelope_index``).
|
||||
Pass to skip the per-set-response ``Batch`` scan in Pass 1.
|
||||
pc_claim_lookup: optional pre-built callable that maps a PCN
|
||||
to a single claim. Falls back to a session-wide query
|
||||
when not supplied.
|
||||
|
||||
Pass 1 wins. The two paths cannot both fire for the same
|
||||
``set_control_number`` — if Pass 1 returns one or more claims,
|
||||
Pass 2 is skipped. This is the false-positive guard from
|
||||
spec §7.
|
||||
"""
|
||||
if not set_control_number:
|
||||
return []
|
||||
|
||||
# -- Pass 1: Batch.envelope.control_number primary --------------
|
||||
# Accept either a plain dict (the common case — built once per
|
||||
# ingest via ``store.batch_envelope_index``) or a callable for
|
||||
# test-side closures. Normalize to ``idx.get`` so the rest of
|
||||
# the function stays uniform.
|
||||
idx: Optional[Callable[[str], Optional[str]]] = None
|
||||
if batch_envelope_index is not None:
|
||||
if callable(batch_envelope_index):
|
||||
idx = batch_envelope_index
|
||||
else:
|
||||
idx = batch_envelope_index.get
|
||||
matched_ids: list[str] = []
|
||||
if idx is not None:
|
||||
batch_id = idx(set_control_number)
|
||||
if batch_id is not None:
|
||||
matched_ids = [
|
||||
cid for (cid,) in (
|
||||
session.query(Claim.id)
|
||||
.filter(Claim.batch_id == batch_id)
|
||||
.all()
|
||||
)
|
||||
]
|
||||
else:
|
||||
# Fallback: scan all batches once per call. Slow but correct;
|
||||
# callers SHOULD pass the index.
|
||||
rows = (
|
||||
session.query(Batch.id, Batch.raw_result_json)
|
||||
.filter(Batch.kind == "837p")
|
||||
.all()
|
||||
)
|
||||
for bid, raw in rows:
|
||||
env = (raw or {}).get("envelope") or {}
|
||||
if env.get("control_number") == set_control_number:
|
||||
matched_ids = [
|
||||
cid for (cid,) in (
|
||||
session.query(Claim.id)
|
||||
.filter(Claim.batch_id == bid)
|
||||
.all()
|
||||
)
|
||||
]
|
||||
break
|
||||
|
||||
if matched_ids:
|
||||
claims = (
|
||||
session.query(Claim)
|
||||
.filter(Claim.id.in_(matched_ids))
|
||||
.all()
|
||||
)
|
||||
if claims:
|
||||
return list(claims)
|
||||
|
||||
# -- Pass 2: Claim.patient_control_number fallback ---------------
|
||||
if pc_claim_lookup is not None:
|
||||
single = pc_claim_lookup(set_control_number)
|
||||
if single is not None:
|
||||
return [single]
|
||||
return []
|
||||
|
||||
matches = (
|
||||
session.query(Claim)
|
||||
.filter(Claim.patient_control_number == set_control_number)
|
||||
.all()
|
||||
)
|
||||
return list(matches)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-ACK helpers — walk the parsed result and produce ClaimAckLinkRow
|
||||
# dataclasses. The CALLER persists via cycl_store.add_claim_ack so the
|
||||
# publish-from-store contract owns the live-tail event emission.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _existing_link_claim_ids(
|
||||
session: Session,
|
||||
*,
|
||||
claim_ids: list[str],
|
||||
ack_kind: str,
|
||||
ack_id: int,
|
||||
) -> set[str]:
|
||||
"""Return the subset of ``claim_ids`` that already have a link row.
|
||||
|
||||
Mirrors the partial unique index
|
||||
``ux_claim_acks_dedup(claim_id, ack_kind, ack_id, ak2_index)
|
||||
WHERE claim_id IS NOT NULL AND ak2_index IS NOT NULL``. The
|
||||
pre-check is here so we skip ``session.add`` and avoid
|
||||
IntegrityError log noise on re-ingest. For TA1 (claim_id IS
|
||||
NULL) the helpers do their own check.
|
||||
"""
|
||||
if not claim_ids:
|
||||
return set()
|
||||
rows = (
|
||||
session.query(Claim.claim_id) # placeholder; replaced below
|
||||
if False else
|
||||
session.query(Claim.id)
|
||||
.filter(Claim.id.in_([]))
|
||||
.all()
|
||||
)
|
||||
# Replace the placeholder with the real query — needed because
|
||||
# ClaimAck is not imported above (avoids circular import).
|
||||
from cyclone.db import ClaimAck
|
||||
existing = (
|
||||
session.query(ClaimAck.claim_id)
|
||||
.filter(
|
||||
ClaimAck.ack_kind == ack_kind,
|
||||
ClaimAck.ack_id == ack_id,
|
||||
ClaimAck.claim_id.in_(claim_ids),
|
||||
)
|
||||
.all()
|
||||
)
|
||||
return {cid for (cid,) in existing if cid is not None}
|
||||
|
||||
|
||||
def apply_999_acceptances(
|
||||
session: Session,
|
||||
parsed_999,
|
||||
*,
|
||||
ack_id: int,
|
||||
batch_envelope_index: Optional[Callable[[str], Optional[str]]] = None,
|
||||
pc_claim_lookup: Optional[Callable[[str], Optional[Claim]]] = None,
|
||||
now: Optional[datetime] = None,
|
||||
) -> ClaimAckLinkResult:
|
||||
"""For every AK2 set-response, build one ``ClaimAckLinkRow`` per matched claim.
|
||||
|
||||
Both accepted AND rejected AK2s produce a link row (so the
|
||||
ClaimDrawer panel can show the rejection inline via the
|
||||
``set_accept_reject_code`` color-coded chip). Orphans are returned
|
||||
but not linked.
|
||||
|
||||
Idempotent: rows the dedup index already covers (re-ingest of an
|
||||
identical file) are skipped silently — the pre-check is here to
|
||||
avoid ``IntegrityError`` log noise.
|
||||
|
||||
One AK2 can produce multiple ``claim_acks`` rows when the source
|
||||
837 batch carried more than one claim under a shared ST02 (rare
|
||||
on this codebase but supported by D10 / the schema).
|
||||
|
||||
Returns:
|
||||
A :class:`ClaimAckLinkResult` whose ``linked`` list contains
|
||||
one :class:`ClaimAckLinkRow` per AK2-to-claim match. The
|
||||
caller persists each row via ``cycl_store.add_claim_ack``.
|
||||
"""
|
||||
result = ClaimAckLinkResult()
|
||||
set_responses = getattr(parsed_999, "set_responses", None) or []
|
||||
|
||||
# Resolve all set_control_numbers up front so we can do one
|
||||
# batched dedup query per (ack_kind, ack_id).
|
||||
resolved: list[tuple[int, "object", list[Claim], str, str]] = []
|
||||
candidate_claim_ids: list[str] = []
|
||||
for idx, sr in enumerate(set_responses):
|
||||
scn = getattr(sr, "set_control_number", None) or ""
|
||||
code = getattr(sr.set_accept_reject, "code", None) or ""
|
||||
claims = lookup_claims_for_ack_set_response(
|
||||
session, scn,
|
||||
batch_envelope_index=batch_envelope_index,
|
||||
pc_claim_lookup=pc_claim_lookup,
|
||||
)
|
||||
if not claims:
|
||||
result.orphans.append(scn)
|
||||
continue
|
||||
resolved.append((idx, sr, claims, scn, code))
|
||||
candidate_claim_ids.extend(c.id for c in claims)
|
||||
|
||||
existing_ids = _existing_link_claim_ids(
|
||||
session,
|
||||
claim_ids=candidate_claim_ids,
|
||||
ack_kind="999",
|
||||
ack_id=ack_id,
|
||||
)
|
||||
|
||||
for idx, sr, claims, scn, code in resolved:
|
||||
for claim in claims:
|
||||
if claim.id in existing_ids:
|
||||
continue
|
||||
result.linked.append(ClaimAckLinkRow(
|
||||
claim_id=claim.id,
|
||||
batch_id=None,
|
||||
ak2_index=idx,
|
||||
set_control_number=scn,
|
||||
set_accept_reject_code=code,
|
||||
))
|
||||
return result
|
||||
|
||||
|
||||
def apply_277ca_acks(
|
||||
session: Session,
|
||||
parsed_277ca,
|
||||
*,
|
||||
ack_id: int,
|
||||
batch_envelope_index: Optional[Callable[[str], Optional[str]]] = None,
|
||||
pc_claim_lookup: Optional[Callable[[str], Optional[Claim]]] = None,
|
||||
now: Optional[datetime] = None,
|
||||
) -> ClaimAckLinkResult:
|
||||
"""For every ClaimStatus with a payer_claim_control_number, build a ClaimAckLinkRow.
|
||||
|
||||
Accepted AND rejected ClaimStatuses both link — the
|
||||
``set_accept_reject_code`` carries the STC category code. The
|
||||
``claim_acks`` row is independent of the existing
|
||||
``Claim.payer_rejected_at`` mutation from
|
||||
:func:`cyclone.inbox_state_277ca.apply_277ca_rejections` (which
|
||||
fires before this helper in the handler).
|
||||
|
||||
Returns:
|
||||
A :class:`ClaimAckLinkResult` whose ``linked`` list contains
|
||||
one :class:`ClaimAckLinkRow` per ClaimStatus match. The
|
||||
caller persists each row via ``cycl_store.add_claim_ack``.
|
||||
"""
|
||||
result = ClaimAckLinkResult()
|
||||
statuses = getattr(parsed_277ca, "claim_statuses", None) or []
|
||||
|
||||
resolved: list[tuple["object", list[Claim], str, str]] = []
|
||||
candidate_claim_ids: list[str] = []
|
||||
for status in statuses:
|
||||
scn = getattr(status, "payer_claim_control_number", None) or ""
|
||||
raw_code = getattr(status, "status_code", None) or ""
|
||||
# ``status_code`` may be a category like "A6:19:PR" — keep
|
||||
# the whole STC composite so the UI can render the
|
||||
# category without re-parsing raw_json. Truncate to 8 chars
|
||||
# (column width).
|
||||
code = (raw_code or "")[:8]
|
||||
if not scn:
|
||||
# No REF*1K — orphan. Surface the STC composite so the
|
||||
# operator can correlate via the ack's raw_json.
|
||||
orphan_key = code or "(no REF*1K)"
|
||||
result.orphans.append(orphan_key)
|
||||
continue
|
||||
claims = lookup_claims_for_ack_set_response(
|
||||
session, scn,
|
||||
batch_envelope_index=batch_envelope_index,
|
||||
pc_claim_lookup=pc_claim_lookup,
|
||||
)
|
||||
if not claims:
|
||||
result.orphans.append(scn)
|
||||
continue
|
||||
resolved.append((status, claims, scn, code))
|
||||
candidate_claim_ids.extend(c.id for c in claims)
|
||||
|
||||
existing_ids = _existing_link_claim_ids(
|
||||
session,
|
||||
claim_ids=candidate_claim_ids,
|
||||
ack_kind="277ca",
|
||||
ack_id=ack_id,
|
||||
)
|
||||
|
||||
for status, claims, scn, code in resolved:
|
||||
for claim in claims:
|
||||
if claim.id in existing_ids:
|
||||
continue
|
||||
result.linked.append(ClaimAckLinkRow(
|
||||
claim_id=claim.id,
|
||||
batch_id=None,
|
||||
ak2_index=None,
|
||||
set_control_number=scn,
|
||||
set_accept_reject_code=code or None,
|
||||
))
|
||||
return result
|
||||
|
||||
|
||||
def apply_ta1_envelope_link(
|
||||
session: Session,
|
||||
parsed_ta1,
|
||||
*,
|
||||
ack_id: int,
|
||||
batch_lookup: Callable[[str, str], Optional[Batch]],
|
||||
now: Optional[datetime] = None,
|
||||
) -> ClaimAckLinkResult:
|
||||
"""Build a TA1 envelope-level link row to the most-recent matching Batch.
|
||||
|
||||
TA1 is envelope-level only (ISA/IEA, no per-claim granularity).
|
||||
The link row has ``claim_id IS NULL`` and ``batch_id`` populated.
|
||||
|
||||
Args:
|
||||
parsed_ta1: a :class:`cyclone.parsers.models_ta1.ParseResultTa1`.
|
||||
batch_lookup: ``(sender_id, receiver_id) -> Batch | None``.
|
||||
The handler supplies a closure that walks
|
||||
``session.query(Batch).order_by(parsed_at.desc())``. Returning
|
||||
``None`` produces an orphan (no batch match).
|
||||
|
||||
Returns:
|
||||
A :class:`ClaimAckLinkResult` with 0..1 row in ``linked``.
|
||||
Idempotent via dedup on ``(ack_kind='ta1', ack_id)`` (claim_id
|
||||
IS NULL so the partial unique index doesn't catch it; the
|
||||
pre-check here is a Python-side query).
|
||||
"""
|
||||
result = ClaimAckLinkResult()
|
||||
envelope = getattr(parsed_ta1, "envelope", None)
|
||||
if envelope is None:
|
||||
return result
|
||||
ta1_obj = getattr(parsed_ta1, "ta1", None)
|
||||
ack_code = getattr(ta1_obj, "ack_code", None) or ""
|
||||
|
||||
# Dedup: same (ack_kind, ack_id) → at most one TA1 envelope link.
|
||||
# Done in Python because the partial unique index requires
|
||||
# claim_id IS NOT NULL.
|
||||
from cyclone.db import ClaimAck
|
||||
existing = (
|
||||
session.query(ClaimAck.id)
|
||||
.filter(
|
||||
ClaimAck.ack_kind == "ta1",
|
||||
ClaimAck.ack_id == ack_id,
|
||||
ClaimAck.claim_id.is_(None),
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if existing is not None:
|
||||
return result
|
||||
|
||||
batch = batch_lookup(envelope.sender_id or "", envelope.receiver_id or "")
|
||||
if batch is None:
|
||||
orphan_key = (
|
||||
getattr(ta1_obj, "control_number", None)
|
||||
or envelope.control_number
|
||||
or ""
|
||||
)
|
||||
result.orphans.append(orphan_key)
|
||||
return result
|
||||
|
||||
result.linked.append(ClaimAckLinkRow(
|
||||
claim_id=None,
|
||||
batch_id=batch.id,
|
||||
ak2_index=None,
|
||||
set_control_number=None,
|
||||
set_accept_reject_code=ack_code,
|
||||
))
|
||||
return result
|
||||
|
||||
|
||||
def link_manual(
|
||||
session: Session,
|
||||
*,
|
||||
claim_id: str,
|
||||
ack_kind: str,
|
||||
ack_id: int,
|
||||
set_control_number: Optional[str] = None,
|
||||
set_accept_reject_code: Optional[str] = None,
|
||||
ak2_index: Optional[int] = None,
|
||||
now: Optional[datetime] = None,
|
||||
) -> ClaimAckLinkRow:
|
||||
"""Return one manual link row (the caller persists it via the store).
|
||||
|
||||
Used by ``POST /api/acks/{kind}/{ack_id}/match-claim``. Returns a
|
||||
:class:`ClaimAckLinkRow` describing the row to insert. The caller
|
||||
is responsible for persistence so it can own the publish-from-store
|
||||
contract.
|
||||
|
||||
Idempotency: callers should pre-check via ``session.query(ClaimAck)
|
||||
.filter(...).first()`` and skip when a row already exists; the
|
||||
:class:`cyclone.store.claim_acks.add_claim_ack` implementation also
|
||||
re-checks via the partial unique index.
|
||||
|
||||
Raises ``LookupError`` when the referenced claim doesn't exist
|
||||
(the caller maps that to 404).
|
||||
"""
|
||||
if ack_kind not in ("999", "277ca", "ta1"):
|
||||
raise ValueError(f"link_manual: unknown ack_kind={ack_kind!r}")
|
||||
claim = session.get(Claim, claim_id)
|
||||
if claim is None:
|
||||
raise LookupError(f"claim {claim_id} not found")
|
||||
return ClaimAckLinkRow(
|
||||
claim_id=claim_id,
|
||||
batch_id=None,
|
||||
ak2_index=ak2_index,
|
||||
set_control_number=set_control_number,
|
||||
set_accept_reject_code=set_accept_reject_code,
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"ClaimAckLinkResult",
|
||||
"ClaimAckLinkRow",
|
||||
"apply_999_acceptances",
|
||||
"apply_277ca_acks",
|
||||
"apply_ta1_envelope_link",
|
||||
"link_manual",
|
||||
"lookup_claims_for_ack_set_response",
|
||||
]
|
||||
@@ -24,6 +24,7 @@ stub secret and the paramiko auth will fail loudly at connect time.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
@@ -40,6 +41,59 @@ from cyclone.providers import SftpBlock
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-op SFTP timeout (SP27 Task 8)
|
||||
#
|
||||
# paramiko is synchronous; without a timeout, a hung ``listdir_attr``
|
||||
# freezes the worker thread indefinitely. The 06/25 silent hang was
|
||||
# exactly this — the MFT server TCP-acked but stopped responding, the
|
||||
# scheduler's ``asyncio.to_thread`` waited forever, and the operator
|
||||
# had no signal that polling had stalled.
|
||||
#
|
||||
# The async wrappers below apply ``asyncio.wait_for`` to every SFTP
|
||||
# call site so the event loop can give up after the configured bound.
|
||||
# The bound is read fresh on every call (env-var-only, no module-level
|
||||
# cache) so an operator who tunes the value at runtime picks it up on
|
||||
# the next poll.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_DEFAULT_SFTP_OP_TIMEOUT_SECONDS = 30.0
|
||||
|
||||
|
||||
def _op_timeout_seconds() -> float:
|
||||
"""Per-op SFTP timeout from ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS``.
|
||||
|
||||
Default 30s. Picked to comfortably outlast Gainwell's p99
|
||||
listdir_attr (~2s) while still surfacing real hangs inside one
|
||||
scheduler tick. Operators who hit repeated timeouts should drop
|
||||
this — but the right answer is to fix the MFT server, not to
|
||||
paper over it here.
|
||||
"""
|
||||
raw = os.environ.get("CYCLONE_SFTP_OP_TIMEOUT_SECONDS")
|
||||
if not raw:
|
||||
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
|
||||
try:
|
||||
value = float(raw)
|
||||
except ValueError:
|
||||
log.warning(
|
||||
"CYCLONE_SFTP_OP_TIMEOUT_SECONDS=%r is not a float; using default %.1fs",
|
||||
raw, _DEFAULT_SFTP_OP_TIMEOUT_SECONDS,
|
||||
)
|
||||
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
|
||||
if value <= 0:
|
||||
# ``asyncio.wait_for(timeout=0)`` raises immediately, and
|
||||
# ``wait_for(timeout<0)`` is undefined per the asyncio docs.
|
||||
# A zero/negative setting would silently turn every SFTP call
|
||||
# into an instant timeout (a wave of bogus "list_inbound:
|
||||
# timeout" errors). Treat the value as bad and fall back.
|
||||
log.warning(
|
||||
"CYCLONE_SFTP_OP_TIMEOUT_SECONDS=%r must be positive; using default %.1fs",
|
||||
raw, _DEFAULT_SFTP_OP_TIMEOUT_SECONDS,
|
||||
)
|
||||
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
|
||||
return value
|
||||
|
||||
|
||||
@dataclass
|
||||
class InboundFile:
|
||||
"""A single file observed in the inbound MFT path."""
|
||||
@@ -87,11 +141,70 @@ class SftpClient:
|
||||
dir and returns :class:`InboundFile` records pointing at the
|
||||
cache copy. The remote file is *not* deleted — the operator
|
||||
archives inbound files in the MFT UI.
|
||||
|
||||
Gainwell's MFT puts advisory ``*_warn.txt`` files in the same
|
||||
inbound path. They're text-format side-channel notes (not X12
|
||||
envelopes) and are skipped at list time. Use
|
||||
:meth:`list_inbound_names` if you need the raw list without
|
||||
the download.
|
||||
"""
|
||||
if self._stub:
|
||||
return self._list_inbound_stub()
|
||||
return self._list_inbound_paramiko()
|
||||
|
||||
def list_inbound_names(self) -> list[InboundFile]:
|
||||
"""Lightweight listing: returns metadata only, no file download.
|
||||
|
||||
Use this when you want to filter the inbound set (e.g. by date)
|
||||
before paying the download cost. Pair with
|
||||
:meth:`download_inbound` to fetch a filtered subset on demand.
|
||||
|
||||
Real mode is implemented as a single SFTP ``listdir_attr`` call
|
||||
— sub-second on Gainwell's MFT — versus the full
|
||||
:meth:`list_inbound` which downloads every file. The
|
||||
``*_warn.txt`` advisory files are filtered out the same way.
|
||||
|
||||
Stub mode returns the same as :meth:`list_inbound` (the stub
|
||||
only knows about local files; no download cost).
|
||||
"""
|
||||
if self._stub:
|
||||
return self._list_inbound_stub()
|
||||
return self._list_inbound_names_paramiko()
|
||||
|
||||
def download_inbound(self, f: InboundFile) -> Path:
|
||||
"""Download a single inbound file to its ``local_path``.
|
||||
|
||||
Idempotent: if ``f.local_path`` already exists and is
|
||||
non-empty, the download is skipped. Callers should use the
|
||||
``InboundFile`` returned by :meth:`list_inbound_names` and pass
|
||||
it back here — ``local_path`` is the planned cache location
|
||||
and matches the path the scheduler will read from.
|
||||
|
||||
Returns:
|
||||
The on-disk path (same as ``f.local_path``).
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: if the file is missing locally in stub
|
||||
mode, or if the remote file disappears between list
|
||||
and download in real mode.
|
||||
"""
|
||||
if f.local_path.exists() and f.local_path.stat().st_size > 0:
|
||||
log.debug(
|
||||
"SFTP: %s already cached at %s, skipping download",
|
||||
f.name, f.local_path,
|
||||
)
|
||||
return f.local_path
|
||||
if self._stub:
|
||||
# Stub mode: no remote — the file is supposed to already be
|
||||
# at f.local_path (operator-dropped). If it isn't there, the
|
||||
# operator hasn't seeded the stub; raise loudly.
|
||||
if not f.local_path.is_file():
|
||||
raise FileNotFoundError(
|
||||
f"inbound stub file not found: {f.local_path}"
|
||||
)
|
||||
return f.local_path
|
||||
return self._download_inbound_paramiko(f)
|
||||
|
||||
def read_file(self, remote_path: str) -> bytes:
|
||||
"""Read bytes from a remote path.
|
||||
|
||||
@@ -119,6 +232,37 @@ class SftpClient:
|
||||
return secrets.STUB_SECRET
|
||||
return value
|
||||
|
||||
# ---- Async surface (SP27 Task 8) -----------------------------------
|
||||
#
|
||||
# Every sync SFTP call has an async wrapper that runs the paramiko
|
||||
# call on a worker thread and applies an ``asyncio.wait_for(...
|
||||
# timeout=N)`` around it. The wait_for cancels the awaiter but
|
||||
# leaves the worker thread running until paramiko returns on its
|
||||
# own (paramiko is not asyncio-aware, so we can't cancel the
|
||||
# underlying socket cleanly). The scheduler should treat
|
||||
# ``asyncio.TimeoutError`` from these wrappers as a transient
|
||||
# SFTP error and surface it in ``Scheduler.status()`` (Task 9).
|
||||
#
|
||||
# The timeout is read on every call (see ``_op_timeout_seconds``)
|
||||
# so an operator who tunes ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` at
|
||||
# runtime sees the new value on the next tick.
|
||||
|
||||
async def async_list_inbound(self) -> list["InboundFile"]:
|
||||
"""Async-wrapped :meth:`list_inbound` with a per-op timeout.
|
||||
|
||||
The 06/25 silent hang was a hung ``listdir_attr`` that froze
|
||||
the worker thread indefinitely. This wrapper applies
|
||||
``asyncio.wait_for(...)`` so the event loop can give up after
|
||||
``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` (default 30s) and the
|
||||
scheduler tick can surface the timeout in
|
||||
``result.errors`` (and, once Task 9 lands, in
|
||||
``Scheduler.status()``).
|
||||
"""
|
||||
return await asyncio.wait_for(
|
||||
asyncio.to_thread(self.list_inbound),
|
||||
timeout=_op_timeout_seconds(),
|
||||
)
|
||||
|
||||
# ---- Stub implementations (SP9) -------------------------------------
|
||||
|
||||
def _write_bytes_stub(self, remote_path: str, content: bytes) -> Path:
|
||||
@@ -284,6 +428,12 @@ class SftpClient:
|
||||
if attr.st_mode and (attr.st_mode & 0o170000) == 0o040000:
|
||||
# Directory entry — skip.
|
||||
continue
|
||||
if attr.filename.endswith("_warn.txt"):
|
||||
# Gainwell's MFT drops text-format advisory notes in
|
||||
# the same inbound path. They're side-channel noise,
|
||||
# not X12 envelopes — skip at list time so we don't
|
||||
# download ~600 advisory files per poll.
|
||||
continue
|
||||
remote = f"{inbound_dir.rstrip('/')}/{attr.filename}"
|
||||
cache_path = cache_dir / attr.filename
|
||||
# Download into cache. We use ``prefetch`` to keep memory
|
||||
@@ -299,6 +449,56 @@ class SftpClient:
|
||||
))
|
||||
return files
|
||||
|
||||
def _list_inbound_names_paramiko(self) -> list[InboundFile]:
|
||||
"""List inbound names via paramiko; do NOT download (lightweight).
|
||||
|
||||
Same ``listdir_attr`` iteration as
|
||||
:meth:`_list_inbound_paramiko`, but the returned
|
||||
:class:`InboundFile` records have ``local_path`` set to the
|
||||
planned cache location without actually fetching the file.
|
||||
Pair with :meth:`_download_inbound_paramiko` to fetch on
|
||||
demand. Skips ``*_warn.txt`` advisory files the same way.
|
||||
"""
|
||||
with self._connect() as (ssh, sftp):
|
||||
inbound_dir = self._block.paths.get("inbound", "/")
|
||||
staging = Path(self._block.staging_dir).resolve()
|
||||
inbound_rel = inbound_dir.lstrip("/")
|
||||
cache_dir = staging / inbound_rel
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
files: list[InboundFile] = []
|
||||
try:
|
||||
attrs = sftp.listdir_attr(inbound_dir)
|
||||
except IOError as exc:
|
||||
log.warning("SFTP: cannot list %s: %s", inbound_dir, exc)
|
||||
return []
|
||||
|
||||
for attr in sorted(attrs, key=lambda a: a.filename):
|
||||
if attr.st_mode and (attr.st_mode & 0o170000) == 0o040000:
|
||||
# Directory entry — skip.
|
||||
continue
|
||||
if attr.filename.endswith("_warn.txt"):
|
||||
continue
|
||||
cache_path = cache_dir / attr.filename
|
||||
files.append(InboundFile(
|
||||
name=attr.filename,
|
||||
size=attr.st_size or 0,
|
||||
modified_at=datetime.fromtimestamp(attr.st_mtime or 0),
|
||||
local_path=cache_path,
|
||||
))
|
||||
return files
|
||||
|
||||
def _download_inbound_paramiko(self, f: InboundFile) -> Path:
|
||||
"""Download a single ``f`` to ``f.local_path`` (idempotent on size>0)."""
|
||||
with self._connect() as (ssh, sftp):
|
||||
inbound_dir = self._block.paths.get("inbound", "/")
|
||||
remote = f"{inbound_dir.rstrip('/')}/{f.name}"
|
||||
f.local_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with sftp.open(remote, "rb") as src, open(f.local_path, "wb") as dst:
|
||||
shutil.copyfileobj(src, dst, length=64 * 1024)
|
||||
log.info("SFTP: downloaded %d bytes for %s", f.local_path.stat().st_size, f.name)
|
||||
return f.local_path
|
||||
|
||||
def _read_file_paramiko(self, remote_path: str) -> bytes:
|
||||
with self._connect() as (ssh, sftp):
|
||||
buf = io.BytesIO()
|
||||
|
||||
+626
-2
@@ -297,8 +297,95 @@ def validate_tax_id_cmd(tax_id: str, log_level: str) -> None:
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP32: `cyclone backfill-rendering-npi` (Task 6)
|
||||
#
|
||||
# Re-parses on-disk 837p/835 files and patches up the typed NPI columns
|
||||
# (``Claim.rendering_provider_npi`` / ``Remittance.rendering_provider_npi``)
|
||||
# for rows that were ingested before the T4 writers took effect.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@main.command("backfill-rendering-npi")
|
||||
@click.option(
|
||||
"--file", "files",
|
||||
multiple=True,
|
||||
type=click.Path(exists=True, dir_okay=False, path_type=Path),
|
||||
help=(
|
||||
"Path to a specific file to re-parse. May be passed multiple times. "
|
||||
"Mutually informative with --input-dir (both are processed)."
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--input-dir",
|
||||
type=click.Path(file_okay=False, path_type=Path),
|
||||
default=None,
|
||||
help=(
|
||||
"Directory to scan one level deep for *.txt / *.edi / *.x12 files. "
|
||||
"Honors CYCLONE_BACKFILL_INPUT_DIR if --input-dir is not passed."
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--type", "transaction_type",
|
||||
type=click.Choice(["837p", "835"]),
|
||||
default=None,
|
||||
help=(
|
||||
"Pin the parser to use. Without --type, each file's transaction "
|
||||
"kind is sniffed (filename hint + ISA/ST prefix)."
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--log-level",
|
||||
default="INFO",
|
||||
show_default=True,
|
||||
type=click.Choice(["DEBUG", "INFO", "WARNING", "ERROR"]),
|
||||
)
|
||||
def backfill_rendering_npi(
|
||||
files: tuple[Path, ...],
|
||||
input_dir: Path | None,
|
||||
transaction_type: str | None,
|
||||
log_level: str,
|
||||
) -> None:
|
||||
"""Re-parse on-disk X12 files to populate the rendering NPI columns.
|
||||
|
||||
Idempotent: rows whose ``rendering_provider_npi`` is already non-NULL
|
||||
are left untouched. After patching, runs :func:`cyclone.reconcile.run`
|
||||
once across every 835 batch so the T5 scoring arm can fire
|
||||
retroactively on the touched pairs.
|
||||
|
||||
Exit code: 0 on a successful run (zero populated rows is still exit 0).
|
||||
"""
|
||||
from cyclone import db as _db
|
||||
from cyclone.store.backfill import backfill_rendering_provider_npi
|
||||
|
||||
# SP18: re-run setup so per-command --log-level overrides the
|
||||
# group default. ``setup_logging`` is idempotent.
|
||||
setup_logging(level=log_level)
|
||||
|
||||
# The backup pattern: each CLI subcommand that touches the DB
|
||||
# initializes it explicitly so ``python -m cyclone.cli backup list``
|
||||
# works on a fresh machine with no prior app boot.
|
||||
_db.init_db()
|
||||
|
||||
# --input-dir falls back to CYCLONE_BACKFILL_INPUT_DIR.
|
||||
if input_dir is None and not files:
|
||||
env_dir = os.environ.get("CYCLONE_BACKFILL_INPUT_DIR", "").strip()
|
||||
if env_dir:
|
||||
input_dir = Path(env_dir)
|
||||
|
||||
summary = backfill_rendering_provider_npi(
|
||||
files=list(files) if files else None,
|
||||
input_dir=input_dir,
|
||||
transaction_type=transaction_type,
|
||||
)
|
||||
|
||||
# One-line summary so operators can grep a log scrape quickly.
|
||||
click.echo(
|
||||
f"claims_updated={summary.claims_updated} "
|
||||
f"remits_updated={summary.remits_updated} "
|
||||
f"files_processed={summary.files_processed} "
|
||||
f"files_skipped={summary.files_skipped}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -312,6 +399,375 @@ if __name__ == "__main__":
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP33: `cyclone backfill-999-rejections`
|
||||
#
|
||||
# One-shot replay of the cascade fix in `apply_999_rejections` for any
|
||||
# 999 acks already in the DB. Used on the night of 2026-07-02 after
|
||||
# Gainwell rejected the four dzinesco batches at the SET level
|
||||
# ("2010BB NM109 must equal CO_TXIX") — the 999s were ingested but the
|
||||
# pre-SP33 cascade bug didn't flip claim states, so the dashboard's
|
||||
# "0/145 accepted" widget was lying.
|
||||
#
|
||||
# Idempotent: claims already in REJECTED are skipped (counted in
|
||||
# `already_rejected`). Safe to re-run.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@main.command("backfill-999-rejections")
|
||||
@click.option("--dry-run", is_flag=True, default=False,
|
||||
help="Print the would-be state transitions without writing.")
|
||||
@click.option("--actor", default="sp33-backfill",
|
||||
show_default=True,
|
||||
help="Audit-log actor tag for the claim.rejected events.")
|
||||
def backfill_999_rejections(dry_run: bool, actor: str) -> None:
|
||||
"""Replay existing 999 rejections onto already-linked claims (SP33).
|
||||
|
||||
Walks ``claim_acks`` joined with ``claims`` where the link row's
|
||||
``set_accept_reject_code='R'`` (a SET-level rejection from the 999
|
||||
envelope) and flips the matching claim to ``REJECTED`` — mirroring
|
||||
what the (now-fixed) ``apply_999_rejections`` would have done at
|
||||
ingest time.
|
||||
|
||||
Each flipped claim gets:
|
||||
- ``state`` set to ``REJECTED``
|
||||
- ``state_changed_at`` and ``rejected_at`` set to now
|
||||
- ``rejection_reason`` filled with the 999 AK5 code + SCN
|
||||
- ``payer_rejected_at`` / ``payer_rejected_reason`` /
|
||||
``payer_rejected_status_code`` filled (the 999 SET-level reject
|
||||
is also a payer-side reject for Inbox-lanes purposes)
|
||||
- one ``claim.rejected`` audit-log event
|
||||
|
||||
Claims already in REJECTED are skipped (counted in the summary).
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
from sqlalchemy import select, func
|
||||
|
||||
from cyclone import db as db_mod
|
||||
from cyclone.audit_log import AuditEvent, append_event
|
||||
|
||||
db_mod.init_db()
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
with db_mod.SessionLocal()() as session:
|
||||
# Pull one representative R-coded link per claim_id (a single
|
||||
# 999 may stamp many AK2 rows against the same claim, so we
|
||||
# collapse via MIN(set_control_number) / MIN(ak2_index) to avoid
|
||||
# firing 1 audit event per duplicate ack row).
|
||||
#
|
||||
# 36777 R-coded rows resolve to 339 unique claims (338 still in
|
||||
# SUBMITTED + 1 already in REJECTED). Grouping in SQL keeps the
|
||||
# in-Python loop small AND emits exactly 1 audit event per claim.
|
||||
rows = session.execute(
|
||||
select(
|
||||
db_mod.Claim.id,
|
||||
db_mod.Claim.state,
|
||||
func.min(db_mod.ClaimAck.set_control_number).label("scn"),
|
||||
func.min(db_mod.ClaimAck.ak2_index).label("ak2"),
|
||||
func.min(db_mod.Ack.ack_code).label("ack_code"),
|
||||
)
|
||||
.join(db_mod.ClaimAck, db_mod.ClaimAck.claim_id == db_mod.Claim.id)
|
||||
.join(db_mod.Ack, db_mod.Ack.id == db_mod.ClaimAck.ack_id)
|
||||
.where(db_mod.ClaimAck.set_accept_reject_code == "R")
|
||||
.where(db_mod.ClaimAck.claim_id.is_not(None))
|
||||
.group_by(db_mod.Claim.id, db_mod.Claim.state)
|
||||
).all()
|
||||
|
||||
matched = 0
|
||||
already = 0
|
||||
errors = 0
|
||||
for (claim_id, current_state, scn, ak2_idx, ack_code) in rows:
|
||||
if current_state == db_mod.ClaimState.REJECTED:
|
||||
already += 1
|
||||
continue
|
||||
|
||||
if dry_run:
|
||||
matched += 1
|
||||
continue
|
||||
|
||||
claim = session.get(db_mod.Claim, claim_id)
|
||||
if claim is None:
|
||||
errors += 1
|
||||
continue
|
||||
|
||||
claim.state = db_mod.ClaimState.REJECTED
|
||||
claim.state_changed_at = now
|
||||
claim.rejected_at = now
|
||||
claim.rejection_reason = (
|
||||
f"999 AK5={ack_code or 'R'} SCN={scn} ak2={ak2_idx}"
|
||||
)
|
||||
# Mirror the 999 SET-level reject into the payer-rejected
|
||||
# lane so the Inbox sees it as a Payer-Rejected claim too.
|
||||
claim.payer_rejected_at = now
|
||||
claim.payer_rejected_reason = f"999 SET-level reject at SCN={scn}"
|
||||
claim.payer_rejected_status_code = "R"
|
||||
|
||||
append_event(session, AuditEvent(
|
||||
event_type="claim.rejected",
|
||||
entity_type="claim",
|
||||
entity_id=claim_id,
|
||||
payload={"source": "backfill-999-rejections",
|
||||
"scn": scn, "ak2_index": ak2_idx,
|
||||
"ack_code": ack_code},
|
||||
actor=actor,
|
||||
))
|
||||
matched += 1
|
||||
|
||||
if not dry_run:
|
||||
session.commit()
|
||||
|
||||
click.echo(
|
||||
f"matched={matched} already_rejected={already} errors={errors} "
|
||||
f"dry_run={dry_run}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP33: `cyclone resubmit-rejected-claims`
|
||||
#
|
||||
# Push the corrected single-claim 837 files to Gainwell's SFTP ToHPE
|
||||
# dir so dzinesco can resubmit the batch. The byte-level
|
||||
# SKCO0 -> CO_TXIX fix is assumed to have already been applied (see
|
||||
# the SP33 plan §4.4 / `docs/ingest/corrected/`). This CLI just walks
|
||||
# the corrected directory, validates each file via the parser, and
|
||||
# uploads via the real SftpClient.
|
||||
#
|
||||
# Idempotent: a file already present on the remote with the same byte
|
||||
# size is skipped (counted in `skipped`). Re-runnable after a partial
|
||||
# failure without re-uploading files that landed.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@main.command("resubmit-rejected-claims")
|
||||
@click.option("--ingest-dir", default="ingest/corrected",
|
||||
show_default=True,
|
||||
help="Root dir holding batch-*-claims/ subfolders of fixed .x12 files.")
|
||||
@click.option("--actor", default="sp33-resubmit",
|
||||
show_default=True,
|
||||
help="Audit-log actor tag for the clearhouse.submitted events.")
|
||||
@click.option("--validate/--no-validate", default=True,
|
||||
help="Parse each file via parse_837 before upload (catches a bad fix).")
|
||||
@click.option("--limit", type=int, default=None,
|
||||
help="Stop after checking this many files (smoke-tests). Counts "
|
||||
"all attempts, not just successful uploads.")
|
||||
@click.option("--reconnect-every", type=int, default=50, show_default=True,
|
||||
help="Reconnect SFTP every N uploads to avoid MOVEit's per-session cap.")
|
||||
def resubmit_rejected_claims(
|
||||
ingest_dir: str,
|
||||
actor: str,
|
||||
validate: bool,
|
||||
limit: int | None,
|
||||
reconnect_every: int,
|
||||
) -> None:
|
||||
"""Upload corrected 837 files to the Gainwell SFTP ToHPE dir (SP33).
|
||||
|
||||
Walks every ``batch-*-claims/*.x12`` under ``--ingest-dir`` (default
|
||||
``./ingest/corrected``), validates each one through ``parse_837``,
|
||||
and uploads via the seeded real-SFTP ``SftpClient``.
|
||||
|
||||
Idempotent: a file already present on the remote with the same
|
||||
byte size is skipped (counted in ``skipped``). Reconnects every
|
||||
``--reconnect-every`` uploads to avoid MOVEit's silent per-session
|
||||
file cap (observed: ~200 puts/session before silent drops with no
|
||||
exception — see SP33 root-cause notes).
|
||||
|
||||
Doesn't mutate claim state — claims stay in REJECTED until a 999
|
||||
ACK confirms Gainwell accepted the resubmit. Emits one
|
||||
``clearhouse.submitted`` audit event per successful upload.
|
||||
"""
|
||||
import time
|
||||
|
||||
from cyclone import db as db_mod
|
||||
from cyclone.audit_log import AuditEvent, append_event
|
||||
from cyclone.parsers.parse_837 import parse as parse_837_text
|
||||
from cyclone.parsers.payer import PayerConfig
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
db_mod.init_db()
|
||||
cycl_store.ensure_clearhouse_seeded()
|
||||
clearhouse = cycl_store.get_clearhouse()
|
||||
if clearhouse is None:
|
||||
click.echo("No clearhouse seeded; cannot resolve SFTP block.", err=True)
|
||||
sys.exit(2)
|
||||
sftp_block = clearhouse.sftp_block
|
||||
if sftp_block.stub:
|
||||
click.echo("Clearhouse SFTP block is in stub mode; refusing to upload.", err=True)
|
||||
sys.exit(2)
|
||||
remote_root = sftp_block.paths["outbound"]
|
||||
|
||||
root = Path(ingest_dir).resolve()
|
||||
if not root.exists():
|
||||
click.echo(f"--ingest-dir does not exist: {root}", err=True)
|
||||
sys.exit(2)
|
||||
|
||||
files: list[Path] = []
|
||||
for batch_dir in sorted(root.glob("batch-*-claims")):
|
||||
files.extend(sorted(p for p in batch_dir.glob("*.x12")
|
||||
if not p.name.startswith("._")))
|
||||
if not files:
|
||||
click.echo(f"No .x12 files under {root}", err=True)
|
||||
sys.exit(2)
|
||||
|
||||
click.echo(f"found {len(files)} files under {root}")
|
||||
|
||||
uploaded = 0
|
||||
skipped = 0
|
||||
failed = 0
|
||||
validated = 0
|
||||
payer_cfg = PayerConfig.co_medicaid()
|
||||
start = time.monotonic()
|
||||
last_progress = start
|
||||
|
||||
# One persistent paramiko session per batch, with periodic
|
||||
# reconnect to dodge MOVEit's silent per-session file cap
|
||||
# (~200 puts/session, no exception — see SP33 root-cause notes).
|
||||
import paramiko
|
||||
from cyclone.secrets import get_secret
|
||||
|
||||
def _open_session() -> tuple[paramiko.SSHClient, paramiko.SFTPClient]:
|
||||
pw = get_secret(sftp_block.auth.get("password_keychain_account", ""))
|
||||
ssh = paramiko.SSHClient()
|
||||
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
||||
ssh.connect(
|
||||
sftp_block.host, port=sftp_block.port,
|
||||
username=sftp_block.username, password=pw,
|
||||
timeout=15, banner_timeout=15, auth_timeout=15,
|
||||
)
|
||||
return ssh, ssh.open_sftp()
|
||||
|
||||
def _close_session(ssh: paramiko.SSHClient | None) -> None:
|
||||
if ssh is None:
|
||||
return
|
||||
try: ssh.close()
|
||||
except Exception: pass
|
||||
|
||||
ssh: paramiko.SSHClient | None = None
|
||||
sftp: paramiko.SFTPClient | None = None
|
||||
|
||||
for i, src in enumerate(files, 1):
|
||||
if limit is not None and i > limit:
|
||||
break
|
||||
|
||||
content = src.read_bytes()
|
||||
|
||||
# Validate: parse must succeed AND payer_id must match the
|
||||
# companion-guide CO_TXIX (catches a bad byte-fix early).
|
||||
if validate:
|
||||
try:
|
||||
parsed = parse_837_text(content.decode(), payer_cfg)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
failed += 1
|
||||
click.echo(f"PARSE FAIL {src.name}: {exc.__class__.__name__}: {exc}", err=True)
|
||||
continue
|
||||
mismatch = next(
|
||||
(c for c in parsed.claims if c.payer.id != "CO_TXIX"), None
|
||||
)
|
||||
if mismatch is not None:
|
||||
failed += 1
|
||||
click.echo(
|
||||
f"PAYER MISMATCH {src.name}: payer.id={mismatch.payer.id!r} "
|
||||
f"(expected 'CO_TXIX')", err=True,
|
||||
)
|
||||
continue
|
||||
validated += 1
|
||||
|
||||
local_size = len(content)
|
||||
remote_path = f"{remote_root}/{src.name}"
|
||||
|
||||
attempts = 0
|
||||
ok = False
|
||||
was_skipped = False
|
||||
while attempts < 3:
|
||||
attempts += 1
|
||||
if ssh is None:
|
||||
try:
|
||||
ssh, sftp = _open_session()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
click.echo(
|
||||
f"SFTP CONNECT FAIL attempt {attempts}: "
|
||||
f"{exc.__class__.__name__}: {exc}", err=True,
|
||||
)
|
||||
continue
|
||||
try:
|
||||
# Idempotency check.
|
||||
try:
|
||||
rs = sftp.stat(remote_path) # type: ignore[union-attr]
|
||||
if rs.st_size == local_size:
|
||||
skipped += 1
|
||||
was_skipped = True
|
||||
break
|
||||
except IOError:
|
||||
pass # not on remote yet
|
||||
sftp.put(str(src), remote_path) # type: ignore[union-attr]
|
||||
ok = True
|
||||
break
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# Connection died — drop the session and let the next
|
||||
# attempt reopen.
|
||||
_close_session(ssh)
|
||||
ssh, sftp = None, None
|
||||
if attempts >= 3:
|
||||
click.echo(
|
||||
f"UPLOAD FAIL {src.name}: "
|
||||
f"{exc.__class__.__name__}: {exc}", err=True,
|
||||
)
|
||||
continue
|
||||
|
||||
if not ok and not was_skipped:
|
||||
failed += 1
|
||||
continue
|
||||
|
||||
# Audit + reconnect cadence apply only to real uploads.
|
||||
if not was_skipped:
|
||||
# Audit (best-effort; if the DB is unavailable we still
|
||||
# keep the file on the wire).
|
||||
try:
|
||||
with db_mod.SessionLocal()() as session:
|
||||
append_event(session, AuditEvent(
|
||||
event_type="clearhouse.submitted",
|
||||
entity_type="claim_file",
|
||||
entity_id=src.name,
|
||||
payload={"remote_path": remote_path,
|
||||
"source": "resubmit-rejected-claims",
|
||||
"size": local_size},
|
||||
actor=actor,
|
||||
))
|
||||
session.commit()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
click.echo(
|
||||
f"audit-log write failed for {src.name}: "
|
||||
f"{exc.__class__.__name__}: {exc}", err=True,
|
||||
)
|
||||
|
||||
uploaded += 1
|
||||
|
||||
# Reconnect periodically to dodge MOVEit's per-session cap.
|
||||
if uploaded % reconnect_every == 0:
|
||||
click.echo(f"reconnecting (after {uploaded} uploads)", err=True)
|
||||
_close_session(ssh)
|
||||
ssh, sftp = None, None
|
||||
|
||||
# Progress every 10s of wall-clock (or at end).
|
||||
now = time.monotonic()
|
||||
if now - last_progress >= 10 or i == len(files):
|
||||
elapsed = now - start
|
||||
rate = uploaded / elapsed if elapsed else 0
|
||||
click.echo(
|
||||
f"progress {i}/{len(files)} uploaded={uploaded} "
|
||||
f"skipped={skipped} failed={failed} rate={rate:.2f}/s "
|
||||
f"elapsed={elapsed:.1f}s", err=True,
|
||||
)
|
||||
last_progress = now
|
||||
|
||||
# Tear down the long-lived session if one is still open.
|
||||
if ssh is not None:
|
||||
_close_session(ssh)
|
||||
|
||||
elapsed = time.monotonic() - start
|
||||
click.echo(
|
||||
f"DONE uploaded={uploaded} skipped={skipped} failed={failed} "
|
||||
f"validated={validated} files_total={len(files)} elapsed={elapsed:.1f}s"
|
||||
)
|
||||
|
||||
|
||||
@main.group()
|
||||
def backup() -> None:
|
||||
"""Encrypted DB backup management (SP17)."""
|
||||
@@ -569,3 +1025,171 @@ def backup_status() -> None:
|
||||
snap = svc.status()
|
||||
import json
|
||||
click.echo(json.dumps(snap, indent=2, default=str))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP-N fix: `cyclone pull-inbound` — targeted inbound pull + process
|
||||
#
|
||||
# Mirrors POST /api/admin/scheduler/pull-inbound. For operators who
|
||||
# want to drive the daily "process today's 999s" workflow from a cron
|
||||
# job or shell script without going through the HTTP API.
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 — processed ≥1 file (or processed 0 with no matches: not an
|
||||
# error, just nothing to do)
|
||||
# 1 — unexpected exception
|
||||
# 2 — SFTP / config error
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@main.command("pull-inbound")
|
||||
@click.option(
|
||||
"--date", "date_str",
|
||||
required=True,
|
||||
help="Date filter YYYYMMDD — only files whose 8-digit timestamp "
|
||||
"substring matches are downloaded and processed.",
|
||||
)
|
||||
@click.option(
|
||||
"--block", "sftp_block_name",
|
||||
default="dzinesco",
|
||||
show_default=True,
|
||||
help="Name of the SftpBlock in config/payers.yaml to use.",
|
||||
)
|
||||
@click.option(
|
||||
"--file-types", "file_types_csv",
|
||||
default=None,
|
||||
help="Comma-separated whitelist (default: 999,TA1).",
|
||||
)
|
||||
@click.option(
|
||||
"--limit", default=2000, show_default=True, type=int,
|
||||
)
|
||||
def pull_inbound(
|
||||
date_str: str,
|
||||
sftp_block_name: str,
|
||||
file_types_csv: str | None,
|
||||
limit: int,
|
||||
) -> None:
|
||||
"""List, date-filter, download, and process inbound MFT files.
|
||||
|
||||
Bypasses the alphabetical full-listing pass so a daily pull of
|
||||
~400 files takes seconds, not hours. Files already in the local
|
||||
cache are skipped (idempotent).
|
||||
"""
|
||||
import asyncio as _asyncio
|
||||
from cyclone import db as db_mod
|
||||
from cyclone import scheduler as scheduler_mod
|
||||
from cyclone.edi.filenames import ALLOWED_FILE_TYPES
|
||||
from cyclone.clearhouse import SftpClient
|
||||
from cyclone.providers import SftpBlock
|
||||
|
||||
# Validate the date filter.
|
||||
if not (len(date_str) == 8 and date_str.isdigit()):
|
||||
click.echo(f"--date must be YYYYMMDD, got {date_str!r}", err=True)
|
||||
sys.exit(2)
|
||||
|
||||
db_mod.init_db()
|
||||
|
||||
# Find the SftpBlock. The dzinesco clearhouse singleton is seeded
|
||||
# by store.ensure_clearhouse_seeded() (SP9) and carries the
|
||||
# production SFTP block; that's the only one we have at the
|
||||
# moment. For multi-provider SFTP, a config-loader is the right
|
||||
# thing — out of scope for this CLI which is the daily-pull path.
|
||||
from cyclone import store as store_mod
|
||||
store_mod.store.ensure_clearhouse_seeded()
|
||||
clearhouse = store_mod.store.get_clearhouse()
|
||||
if clearhouse is None or clearhouse.name != sftp_block_name:
|
||||
# Fall back: try the named block, but v1 only ships the
|
||||
# dzinesco singleton.
|
||||
if clearhouse is None:
|
||||
click.echo(
|
||||
f"No clearhouse seeded — cannot find SftpBlock "
|
||||
f"{sftp_block_name!r}.",
|
||||
err=True,
|
||||
)
|
||||
sys.exit(2)
|
||||
click.echo(
|
||||
f"SftpBlock {sftp_block_name!r} not seeded; only "
|
||||
f"{clearhouse.name!r} is available.",
|
||||
err=True,
|
||||
)
|
||||
sys.exit(2)
|
||||
block: SftpBlock = clearhouse.sftp_block
|
||||
|
||||
if file_types_csv:
|
||||
wanted = {t.strip().upper() for t in file_types_csv.split(",") if t.strip()}
|
||||
unknown = wanted - ALLOWED_FILE_TYPES
|
||||
if unknown:
|
||||
click.echo(
|
||||
f"file_types {sorted(unknown)!r} not in {sorted(ALLOWED_FILE_TYPES)}",
|
||||
err=True,
|
||||
)
|
||||
sys.exit(2)
|
||||
else:
|
||||
wanted = {"999", "TA1"}
|
||||
|
||||
# Wire up the scheduler singleton (re-use the same pipeline the
|
||||
# HTTP endpoint uses, including the dedup via processed_inbound_files).
|
||||
scheduler_mod.configure_scheduler(
|
||||
block, sftp_block_name=sftp_block_name, force=True,
|
||||
)
|
||||
sched = scheduler_mod.get_scheduler()
|
||||
client = SftpClient(block)
|
||||
|
||||
async def _run() -> dict:
|
||||
from cyclone.edi.filenames import parse_inbound_filename
|
||||
|
||||
all_files = await _asyncio.to_thread(client.list_inbound_names)
|
||||
matched: list = []
|
||||
for f in all_files:
|
||||
if f.name.find(date_str) == -1:
|
||||
continue
|
||||
try:
|
||||
parsed = parse_inbound_filename(f.name)
|
||||
except ValueError:
|
||||
continue
|
||||
if parsed.file_type not in wanted:
|
||||
continue
|
||||
matched.append(f)
|
||||
if len(matched) >= limit:
|
||||
break
|
||||
|
||||
download_errors: list[str] = []
|
||||
downloaded = 0
|
||||
for f in matched:
|
||||
try:
|
||||
await _asyncio.to_thread(client.download_inbound, f)
|
||||
downloaded += 1
|
||||
except Exception as exc: # noqa: BLE001
|
||||
download_errors.append(f"{f.name}: {type(exc).__name__}: {exc}")
|
||||
|
||||
tick = await sched.process_inbound_files(matched)
|
||||
return {
|
||||
"listed": len(all_files),
|
||||
"matched": len(matched),
|
||||
"downloaded": downloaded,
|
||||
"download_errors": download_errors,
|
||||
"processed": tick.files_processed,
|
||||
"skipped": tick.files_skipped,
|
||||
"errored": tick.files_errored,
|
||||
}
|
||||
|
||||
try:
|
||||
summary = _asyncio.run(_run())
|
||||
except Exception as exc: # noqa: BLE001
|
||||
click.echo(f"pull-inbound failed: {type(exc).__name__}: {exc}", err=True)
|
||||
sys.exit(1)
|
||||
|
||||
click.echo(
|
||||
f"date={date_str} file_types={sorted(wanted)} block={sftp_block_name} "
|
||||
f"listed={summary['listed']} matched={summary['matched']} "
|
||||
f"downloaded={summary['downloaded']} processed={summary['processed']} "
|
||||
f"skipped={summary['skipped']} errored={summary['errored']}"
|
||||
)
|
||||
if summary["download_errors"]:
|
||||
click.echo("download errors:", err=True)
|
||||
for e in summary["download_errors"]:
|
||||
click.echo(f" {e}", err=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@@ -249,6 +249,7 @@ class Claim(Base):
|
||||
service_date_to: Mapped[Optional[date]] = mapped_column(Date, nullable=True)
|
||||
charge_amount: Mapped[Decimal] = mapped_column(Numeric(12, 2), nullable=False, default=Decimal("0"))
|
||||
provider_npi: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
rendering_provider_npi: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
payer_id: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
|
||||
state: Mapped[ClaimState] = mapped_column(
|
||||
Enum(ClaimState, native_enum=False), nullable=False, default=ClaimState.SUBMITTED
|
||||
@@ -317,6 +318,13 @@ class Claim(Base):
|
||||
Index("ix_claims_state", "state"),
|
||||
Index("ix_claims_patient_control_number", "patient_control_number"),
|
||||
Index("ix_claims_service_date_from", "service_date_from"),
|
||||
# SP27 Task 11: matched-pair drift check (run at startup)
|
||||
# scans ``WHERE matched_remittance_id IS NOT NULL``. Without
|
||||
# this index it's a full claim scan. The reverse side
|
||||
# (``ix_remittances_claim_id``) is added in 0007. Pure index
|
||||
# (non-unique) — a claim without a match is fine, reversals
|
||||
# leave the previous claim/claim match intact.
|
||||
Index("ix_claims_matched_remittance_id", "matched_remittance_id"),
|
||||
)
|
||||
|
||||
|
||||
@@ -337,6 +345,7 @@ class Remittance(Base):
|
||||
claim_id: Mapped[Optional[str]] = mapped_column(
|
||||
String(64), ForeignKey("claims.id"), nullable=True
|
||||
)
|
||||
rendering_provider_npi: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
status_code: Mapped[str] = mapped_column(String(4), nullable=False)
|
||||
status_label: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
|
||||
total_charge: Mapped[Decimal] = mapped_column(Numeric(12, 2), nullable=False, default=Decimal("0"))
|
||||
@@ -638,6 +647,76 @@ class Two77caAck(Base):
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP28: per-ACK auto-link join table (claim_acks)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ClaimAck(Base):
|
||||
"""One row per AK2 set-response / ClaimStatus / TA1 envelope link.
|
||||
|
||||
SP28. The durable record of "this ACK acknowledges this claim (or
|
||||
set, or batch)". One 999 row carries many AK2s; one 277CA carries
|
||||
many ClaimStatuses; each gets its own ClaimAck row so the operator
|
||||
can answer "which claims does this ack acknowledge?" with a single
|
||||
SELECT on ``claim_acks``.
|
||||
|
||||
See ``docs/superpowers/specs/2026-07-02-cyclone-ack-claim-auto-link-design.md``
|
||||
§3.1 for the schema decisions (per-AK2 granularity, the two-pass
|
||||
join, idempotency via the unique index).
|
||||
"""
|
||||
|
||||
__tablename__ = "claim_acks"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||
# FK to claims.id with ON DELETE CASCADE so removing a claim
|
||||
# drops every link row referencing it. NULLable so TA1 envelope-level
|
||||
# rows can populate ``batch_id`` instead (the table CHECK constraint
|
||||
# requires at least one of the two).
|
||||
claim_id: Mapped[Optional[str]] = mapped_column(
|
||||
String(64), ForeignKey("claims.id", ondelete="CASCADE"), nullable=True,
|
||||
)
|
||||
# FK to batches.id (a Batch row in the 837 case, or the synthetic
|
||||
# inbound-batch id when the ack arrived outside the SFTP pipeline).
|
||||
batch_id: Mapped[Optional[str]] = mapped_column(
|
||||
String(32), ForeignKey("batches.id", ondelete="CASCADE"), nullable=True,
|
||||
)
|
||||
# Discriminated union over acks / ta1_acks / two77ca_acks. No FK
|
||||
# constraint because the three target tables are separate; the
|
||||
# application enforces the discriminator + the matching row's id.
|
||||
ack_id: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
ack_kind: Mapped[str] = mapped_column(String(8), nullable=False)
|
||||
ak2_index: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
|
||||
# The set_control_number the upstream ack ACTUALLY CARRIED
|
||||
# (== source 837 ST02 for Gainwell batches). Preserved on the link
|
||||
# row for orphan traceability — the join may have resolved the
|
||||
# link via the PCN fallback instead, but the operator still sees
|
||||
# the value the 999/277CA originally carried.
|
||||
set_control_number: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
# AK5 code (A/E/R/X) for 999, STC category (A1/A2/A3/A4/A6/A7)
|
||||
# for 277CA, envelope ack_code for TA1.
|
||||
set_accept_reject_code: Mapped[Optional[str]] = mapped_column(String(8), nullable=True)
|
||||
linked_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
linked_by: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
|
||||
__table_args__ = (
|
||||
Index("ix_claim_acks_claim_id", "claim_id"),
|
||||
Index("ix_claim_acks_batch_id", "batch_id"),
|
||||
Index("ix_claim_acks_ack", "ack_kind", "ack_id"),
|
||||
# Mirror the dedup unique index declared in 0018_claim_acks.sql so
|
||||
# ``Base.metadata.create_all`` (the test-time safety net) emits the
|
||||
# same partial-unique constraint that the production migration runner
|
||||
# applies. Without this a fresh in-memory test DB would not enforce
|
||||
# idempotency at the DB layer.
|
||||
Index(
|
||||
"ux_claim_acks_dedup",
|
||||
"claim_id", "ack_kind", "ack_id", "ak2_index",
|
||||
unique=True,
|
||||
sqlite_where=text("claim_id IS NOT NULL AND ak2_index IS NOT NULL"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP11: tamper-evident hash-chained audit_log
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -7,9 +7,12 @@ Outbound (we send):
|
||||
tp{tpid}-{transaction_type}-{yyyymmddhhmmssSSS_MT}-1of1.{ext}
|
||||
Example: tp11525703-837P-20260620132243505-1of1.x12
|
||||
|
||||
Inbound (HPE sends to our ToHPE):
|
||||
TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
|
||||
Example: TP11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
||||
Inbound (HPE sends to our FromHPE):
|
||||
[Tt][Pp]{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
|
||||
Example: tp11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
||||
(legacy / prodfiles may use uppercase TP; the regex is case-insensitive
|
||||
on the prefix to accept both — Gainwell's filer has used both
|
||||
casings over time.)
|
||||
|
||||
Both use Mountain Time (MT) timestamps with 17-digit millisecond precision
|
||||
(yyyymmddhhmmssSSS = 4+2+2+2+2+2+3 = 17 digits). Sequence is always "1of1"
|
||||
@@ -39,19 +42,48 @@ OUTBOUND_RE = re.compile(
|
||||
r"^tp(?P<tpid>\d+)-(?P<tx>[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>[A-Za-z0-9]+)$"
|
||||
)
|
||||
|
||||
# Inbound: TP11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
||||
# Inbound: [Tt][Pp]11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
||||
# - prefix: literal "TP" or "tp" (case-insensitive — Gainwell's
|
||||
# production filer has used both)
|
||||
# - tpid: 1+ digits (inside TP<...>)
|
||||
# - orig_tx: 1+ alnum
|
||||
# - track: M + 1+ alnum (e.g. M019048402) — the M is part of the
|
||||
# tracking value, not a separator.
|
||||
# - orig_tx: 1+ uppercase alnum
|
||||
# - track: M + 1+ uppercase alnum (e.g. M019048402) — the M is
|
||||
# part of the tracking value, not a separator.
|
||||
# - ts: 17 digits
|
||||
# - seq: literal "1of1"
|
||||
# - ft: 1+ alnum (e.g. 999, TA1, 271, 277, 277CA, 820, 834, 835, ENCR)
|
||||
# - ft: 1+ uppercase alnum (e.g. 999, TA1, 271, 277, 277CA,
|
||||
# 820, 834, 835, ENCR)
|
||||
#
|
||||
# Case insensitivity is scoped to the ``TP`` prefix via ``(?i:TP)``
|
||||
# — the rest of the pattern is case-sensitive so we still reject a
|
||||
# stray ``837p`` or ``m019048402`` (they'd be invalid HCPF).
|
||||
INBOUND_RE = re.compile(
|
||||
r"^TP(?P<tpid>\d+)-(?P<orig_tx>[A-Z0-9]+)_(?P<tracking>M[A-Z0-9]+)"
|
||||
r"^(?i:TP)(?P<tpid>\d+)-(?P<orig_tx>[A-Z0-9]+)_(?P<tracking>M[A-Z0-9]+)"
|
||||
r"-(?P<ts>\d{17})-1of1_(?P<file_type>[A-Z0-9]+)\.(?P<ext>x12)$"
|
||||
)
|
||||
|
||||
# Inbound suffix-less form (SP27 Task 7): tp11525703-835_M019110219-20260525001606050-1of1.x12
|
||||
# Gainwell's production filer has shipped this shorter form in addition
|
||||
# to the spec form above — the 6/15-6/19 835 batch arrived this way and
|
||||
# was silently dropped by the strict INBOUND_RE. The loose form omits
|
||||
# the `_{file_type}.x12` suffix; the token between `-` and `_M` doubles
|
||||
# as both `orig_tx` and `file_type` (since there's no separate suffix
|
||||
# to disambiguate). Allowed values still must be in ALLOWED_FILE_TYPES
|
||||
# — the suffix is optional, the type-set is not.
|
||||
# - prefix: literal "TP" or "tp" (case-insensitive — same as INBOUND_RE)
|
||||
# - tpid: 1+ digits
|
||||
# - file_type: 3-5 char alphanumeric (covers 999, TA1, 835, 277CA, ENCR;
|
||||
# capped at 5 to avoid swallowing the next `_M` token)
|
||||
# - tracking: M + 1+ uppercase alnum
|
||||
# - ts: 17 digits
|
||||
# - seq: literal "1of1"
|
||||
# - ext: literal "x12" (relaxing this would also relax the strict
|
||||
# form's contract; out of scope for SP27 Task 7)
|
||||
INBOUND_RE_LOOSE = re.compile(
|
||||
r"^(?i:TP)(?P<tpid>\d+)-(?P<file_type>[A-Z0-9]{3,5})"
|
||||
r"_(?P<tracking>M[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>x12)$"
|
||||
)
|
||||
|
||||
ALLOWED_FILE_TYPES = frozenset({
|
||||
"999", "TA1", "270", "271", "276", "277", "277CA", "278",
|
||||
"820", "834", "835", "ENCR",
|
||||
@@ -115,16 +147,49 @@ def build_outbound_filename(
|
||||
def parse_inbound_filename(name: str) -> InboundFilename:
|
||||
"""Parse an inbound HCPF filename.
|
||||
|
||||
Accepts both forms (Gainwell ships both):
|
||||
* Spec form with ``_{file_type}.x12`` suffix:
|
||||
``tp11525703-837P_M019048402-20260520231513488-1of1_999.x12``
|
||||
* Suffix-less form (SP27 Task 7): the token between ``-`` and
|
||||
``_M`` doubles as both ``orig_tx`` and ``file_type``:
|
||||
``tp11525703-835_M019110219-20260525001606050-1of1.x12``
|
||||
|
||||
The strict form is tried first (preserves historical behavior for
|
||||
every existing caller); the loose form is the fallback. The
|
||||
``.x12`` extension and ``ALLOWED_FILE_TYPES`` set are enforced in
|
||||
both forms.
|
||||
|
||||
Args:
|
||||
name: Filename like "TP11525703-837P_M019048402-20260520231513488-1of1_999.x12"
|
||||
name: Filename like ``tp11525703-837P_M019048402-20260520231513488-1of1_999.x12``
|
||||
(case-insensitive on the ``TP`` prefix; both ``TP`` and
|
||||
``tp`` are accepted.)
|
||||
|
||||
Returns:
|
||||
InboundFilename with tpid, orig_tx, tracking, ts, file_type, ext.
|
||||
|
||||
Raises:
|
||||
ValueError: If the filename doesn't match the HCPF inbound format.
|
||||
ValueError: If the filename doesn't match either HCPF inbound
|
||||
form, or if the derived file_type isn't in
|
||||
ALLOWED_FILE_TYPES.
|
||||
"""
|
||||
m = INBOUND_RE.match(name)
|
||||
if m:
|
||||
file_type = m.group("file_type")
|
||||
if file_type not in ALLOWED_FILE_TYPES:
|
||||
raise ValueError(
|
||||
f"file_type {file_type!r} not in allowed HCPF set: {sorted(ALLOWED_FILE_TYPES)}"
|
||||
)
|
||||
return InboundFilename(
|
||||
tpid=m.group("tpid"),
|
||||
orig_tx=m.group("orig_tx"),
|
||||
tracking=m.group("tracking"),
|
||||
ts=m.group("ts"),
|
||||
file_type=file_type,
|
||||
ext=m.group("ext"),
|
||||
)
|
||||
|
||||
# Fall back to the suffix-less form.
|
||||
m = INBOUND_RE_LOOSE.match(name)
|
||||
if not m:
|
||||
raise ValueError(f"Not a valid HCPF inbound filename: {name!r}")
|
||||
file_type = m.group("file_type")
|
||||
@@ -134,7 +199,7 @@ def parse_inbound_filename(name: str) -> InboundFilename:
|
||||
)
|
||||
return InboundFilename(
|
||||
tpid=m.group("tpid"),
|
||||
orig_tx=m.group("orig_tx"),
|
||||
orig_tx=m.group("file_type"), # preserve the historical shape
|
||||
tracking=m.group("tracking"),
|
||||
ts=m.group("ts"),
|
||||
file_type=file_type,
|
||||
@@ -153,5 +218,12 @@ def is_outbound_filename(name: str) -> bool:
|
||||
|
||||
|
||||
def is_inbound_filename(name: str) -> bool:
|
||||
"""True if the given string matches the HCPF inbound filename regex."""
|
||||
return INBOUND_RE.match(name) is not None
|
||||
"""True if the given string matches either HCPF inbound form.
|
||||
|
||||
Accepts both the spec form (with ``_{file_type}.x12`` suffix) and
|
||||
the suffix-less form (SP27 Task 7). Cheap fast-path check used by
|
||||
callers that want to pre-filter a directory listing before invoking
|
||||
:func:`parse_inbound_filename`; mirrors the parser's own fallback
|
||||
so the two never disagree on what counts as an HCPF inbound file.
|
||||
"""
|
||||
return INBOUND_RE.match(name) is not None or INBOUND_RE_LOOSE.match(name) is not None
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
"""File-type handlers for inbound MFT files (SP27).
|
||||
|
||||
Each handler is a pure function that parses + persists + dispatches
|
||||
events for one file type. The scheduler and the FastAPI endpoints
|
||||
both delegate here; the ``HANDLERS`` registry maps ``file_type`` →
|
||||
handler function.
|
||||
|
||||
Public API:
|
||||
HandleResult — dataclass returned by every handler
|
||||
HANDLERS — ``{"999": handle_999, "835": handle_835, ...}``
|
||||
handle_999, handle_ta1, handle_277ca, handle_835
|
||||
— call signatures: ``handle(text: str, source_file: str) -> HandleResult``
|
||||
|
||||
The handlers own their own DB session lifecycle. They emit pubsub
|
||||
events via the optional ``event_bus`` parameter (the FastAPI endpoint
|
||||
injects ``app.state.event_bus``; the scheduler passes ``None``).
|
||||
They never raise on per-segment problems; per-segment issues are
|
||||
logged and folded into the result. Whole-document failures
|
||||
(missing ISA, bad encoding) surface as ``CycloneParseError``, which
|
||||
the caller catches and records as ``STATUS_ERROR``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import logging
|
||||
|
||||
from ._ack_id import (
|
||||
ack_count_summary,
|
||||
ack_synthetic_source_batch_id,
|
||||
two77ca_synthetic_source_batch_id,
|
||||
)
|
||||
from .handle_result import HandleResult
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# Module-level HANDLERS dict populated lazily once handler modules
|
||||
# ship. Keys are file_type strings, values are the ``handle``
|
||||
# callable for that type.
|
||||
HANDLERS: dict[str, object] = {}
|
||||
|
||||
|
||||
# Candidate handlers, in registration order. Each tuple is
|
||||
# (module-path, file_type). The 277/277CA mapping is added explicitly
|
||||
# after registration when the 277CA handler is present.
|
||||
_CANDIDATES: list[tuple[str, str]] = [
|
||||
("cyclone.handlers.handle_999", "999"),
|
||||
("cyclone.handlers.handle_ta1", "TA1"),
|
||||
("cyclone.handlers.handle_277ca", "277CA"),
|
||||
("cyclone.handlers.handle_835", "835"),
|
||||
]
|
||||
|
||||
|
||||
def register_handlers() -> None:
|
||||
"""Populate ``HANDLERS`` from the per-type handler modules.
|
||||
|
||||
Tolerates missing or broken handler modules so the package can be
|
||||
imported incrementally as each handler ships (Tasks 2-5), and so
|
||||
a partial-modification error in one handler module can't break
|
||||
scheduler / API import. Safe to call multiple times.
|
||||
"""
|
||||
if HANDLERS:
|
||||
return
|
||||
for mod_path, file_type in _CANDIDATES:
|
||||
try:
|
||||
mod = importlib.import_module(mod_path)
|
||||
fn = getattr(mod, "handle")
|
||||
except Exception as exc: # noqa: BLE001 — best-effort registry
|
||||
log.debug(
|
||||
"handler %s unavailable: %s",
|
||||
mod_path, exc,
|
||||
)
|
||||
continue
|
||||
HANDLERS[file_type] = fn
|
||||
# The 277 filename maps to the same 277CA handler.
|
||||
if file_type == "277CA":
|
||||
HANDLERS["277"] = fn
|
||||
|
||||
|
||||
register_handlers()
|
||||
|
||||
|
||||
# Re-export handler functions on this package so callers can use the
|
||||
# flat import (``from cyclone.handlers import handle_999``) once each
|
||||
# module ships. Set after registration so we know what's present.
|
||||
def _reexport_handlers() -> None:
|
||||
"""Re-export each handler's ``handle`` fn as ``handle_<type>``.
|
||||
|
||||
No-op for absent handlers. Re-run on each import so freshly-
|
||||
installed handler modules (e.g. Tasks 2-5 commits) are visible
|
||||
after ``register_handlers()`` without a process restart.
|
||||
"""
|
||||
for file_type, fn in list(HANDLERS.items()):
|
||||
if file_type in ("277",): # alias of 277CA; don't re-export twice
|
||||
continue
|
||||
globals()[f"handle_{file_type.lower()}"] = fn
|
||||
|
||||
|
||||
_reexport_handlers()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"HANDLERS",
|
||||
"HandleResult",
|
||||
"register_handlers",
|
||||
"ack_count_summary",
|
||||
"ack_synthetic_source_batch_id",
|
||||
"two77ca_synthetic_source_batch_id",
|
||||
# handler_* names are added at module load via _reexport_handlers
|
||||
]
|
||||
# Populate __all__ with the present handler symbols.
|
||||
for _h in ("handle_999", "handle_ta1", "handle_277ca", "handle_835"):
|
||||
if _h in globals():
|
||||
__all__.append(_h) # noqa: PYI056
|
||||
@@ -0,0 +1,87 @@
|
||||
"""Ack ID helpers shared between the scheduler and the FastAPI
|
||||
endpoints (SP27 Task 1).
|
||||
|
||||
Lifted from ``scheduler.py:_ack_count_summary`` +
|
||||
``_ack_synthetic_source_batch_id`` + ``_277ca_synthetic_source_batch_id`` —
|
||||
all three had inline copies in both ``scheduler.py`` and ``api.py``
|
||||
prior to this SP. Both callers now import from this module.
|
||||
|
||||
Helpers
|
||||
-------
|
||||
``ack_count_summary(result)``
|
||||
Aggregate ``(received, accepted, rejected, ack_code)`` from a
|
||||
parsed ``ParseResult999``, trusting set-level IK5 over the
|
||||
functional-group AK9. Gainwell's MFT ships AK9 segments that
|
||||
contradict the per-set IK5 (e.g. ``AK9*A*1*1*1`` with
|
||||
``IK5*A``), so trusting AK9's rejected count over-reports.
|
||||
See scheduler commit ``6507a8c`` for the operational context.
|
||||
|
||||
``ack_synthetic_source_batch_id(icn, *, pcn, source_filename)``
|
||||
Build a unique-per-file ``batches.id`` for a 999 that ships
|
||||
without its own source batch. Falls back to a hash suffix of
|
||||
the filename so daily pulls don't all collapse onto the same
|
||||
ICN. Gainwell's MFT ships every 999 with the default ICN
|
||||
``000000001`` (per the scheduler docstring).
|
||||
|
||||
``two77ca_synthetic_source_batch_id(icn)``
|
||||
Same idea for a 277CA without its own source batch.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from typing import Any
|
||||
|
||||
|
||||
def ack_count_summary(result: Any) -> tuple[int, int, int, str]:
|
||||
"""Aggregate ``(received, accepted, rejected, ack_code)`` from
|
||||
a ``ParseResult999``.
|
||||
|
||||
Counts are derived from the set-level ``IK5`` responses
|
||||
(one per ``AK2`` in the 999), not the functional-group ``AK9``.
|
||||
Gainwell's MFT ships contradictory AK9 segments; the per-set
|
||||
IK5 is the authoritative per-claim accept/reject signal.
|
||||
"""
|
||||
sets = result.set_responses
|
||||
received = len(sets)
|
||||
accepted = sum(1 for s in sets if s.set_accept_reject.code == "A")
|
||||
rejected = received - accepted
|
||||
if rejected == 0:
|
||||
code = "A"
|
||||
elif accepted == 0:
|
||||
code = "R"
|
||||
else:
|
||||
code = "P"
|
||||
return (received, accepted, rejected, code)
|
||||
|
||||
|
||||
def ack_synthetic_source_batch_id(
|
||||
interchange_control_number: str,
|
||||
*,
|
||||
pcn: str | None = None,
|
||||
source_filename: str | None = None,
|
||||
) -> str:
|
||||
"""Synthetic ``batches.id`` for a received 999 with no source batch.
|
||||
|
||||
Precedence for the human-readable part of the id (column is
|
||||
``VARCHAR(32)``):
|
||||
|
||||
1. ``999-{pcn}-{hash8}`` if ``AK2`` set_control_number is
|
||||
present (the common case). 4 + 9 + 1 + 8 = 22 chars max.
|
||||
2. ``999-{icn}-{hash8}`` if no AK2 (envelope-only 999).
|
||||
3. ``999-{hash12}`` if no filename either (shouldn't happen
|
||||
in production).
|
||||
"""
|
||||
short_hash = ""
|
||||
if source_filename:
|
||||
short_hash = hashlib.sha1(source_filename.encode("utf-8")).hexdigest()[:8]
|
||||
if pcn and pcn.strip():
|
||||
return f"999-{pcn.strip()}-{short_hash}"
|
||||
icn = (interchange_control_number or "").strip() or "000000001"
|
||||
if short_hash:
|
||||
return f"999-{icn}-{short_hash}"
|
||||
return f"999-{short_hash or icn}"
|
||||
|
||||
|
||||
def two77ca_synthetic_source_batch_id(interchange_control_number: str) -> str:
|
||||
"""Synthetic ``batches.id`` for a received 277CA with no source batch."""
|
||||
return f"277CA-{(interchange_control_number or '').strip() or '000000001'}"
|
||||
@@ -0,0 +1,152 @@
|
||||
"""Handle a 277CA Claim Acknowledgment file (SP27 Task 4).
|
||||
|
||||
Lifted verbatim from ``scheduler.py:_handle_277ca``. The handler owns
|
||||
its own DB session, dispatches to ``parse_277ca_text``, persists the
|
||||
277CA ack row, applies 277CA rejections to matched claims via
|
||||
``inbox_state.apply_277ca_rejections``, and emits
|
||||
``claim.payer_rejected`` audit events for each newly-stamped claim.
|
||||
|
||||
The actor tag (``"277ca-parser-scheduler"``) is preserved so the
|
||||
audit log keeps tracing back to the same source after extraction.
|
||||
Both the FastAPI endpoint and the scheduler path (re)use this module
|
||||
— the API migration drops the inline copy in Task 6.
|
||||
|
||||
``claim.rejected_after_remit`` audit emission (when a 277CA rejects
|
||||
a claim that already has ``matched_remittance_id`` set) is deferred
|
||||
to SP27 Task 13. Today the handler only emits ``claim.payer_rejected``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.audit_log import AuditEvent, append_event
|
||||
from cyclone.claim_acks import apply_277ca_acks
|
||||
from cyclone.handlers._ack_id import two77ca_synthetic_source_batch_id
|
||||
from cyclone.inbox_state_277ca import apply_277ca_rejections
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
from cyclone.parsers.parse_277ca import parse_277ca_text
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def handle(
|
||||
text: str,
|
||||
source_file: str,
|
||||
) -> tuple[str, int]:
|
||||
"""Parse a 277CA, persist ack + stamp payer-rejected claims.
|
||||
|
||||
Args:
|
||||
text: Raw 277CA document bytes (decoded).
|
||||
source_file: Filename the 277CA came from.
|
||||
|
||||
Returns:
|
||||
``(parser_used, claim_count)`` tuple where ``claim_count`` is
|
||||
the number of STC statuses in the file (one per claim).
|
||||
|
||||
Raises:
|
||||
ValueError: on parser-level failure (wraps CycloneParseError).
|
||||
|
||||
SP25: the store (``cyclone.store.acks.add_277ca_ack``) owns the
|
||||
publish-from-store contract; the handler no longer needs an
|
||||
``event_bus`` kwarg.
|
||||
"""
|
||||
try:
|
||||
result = parse_277ca_text(text, input_file=source_file)
|
||||
except CycloneParseError as exc:
|
||||
raise ValueError(f"277CA parse error: {exc}") from exc
|
||||
|
||||
icn = result.envelope.control_number
|
||||
synthetic_id = two77ca_synthetic_source_batch_id(icn)
|
||||
accepted = sum(
|
||||
1 for s in result.claim_statuses if s.classification == "accepted"
|
||||
)
|
||||
paid = sum(
|
||||
1 for s in result.claim_statuses if s.classification == "paid"
|
||||
)
|
||||
rejected = sum(
|
||||
1 for s in result.claim_statuses if s.classification == "rejected"
|
||||
)
|
||||
pended = sum(
|
||||
1 for s in result.claim_statuses if s.classification == "pended"
|
||||
)
|
||||
|
||||
# Build the batch envelope index BEFORE opening the work session
|
||||
# — cycl_store.batch_envelope_index opens its own short-lived
|
||||
# session, and SQLite + concurrent sessions causes "database is
|
||||
# locked" errors.
|
||||
batch_index = cycl_store.batch_envelope_index()
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
row = cycl_store.add_277ca_ack(
|
||||
source_batch_id=synthetic_id,
|
||||
control_number=icn,
|
||||
accepted_count=accepted,
|
||||
rejected_count=rejected,
|
||||
paid_count=paid,
|
||||
pended_count=pended,
|
||||
raw_json=json.loads(result.model_dump_json()),
|
||||
)
|
||||
def _lookup(pcn: str):
|
||||
return (
|
||||
session.query(db.Claim)
|
||||
.filter(db.Claim.patient_control_number == pcn)
|
||||
.first()
|
||||
)
|
||||
apply_result = apply_277ca_rejections(
|
||||
session, result, claim_lookup=_lookup, two77ca_id=row.id,
|
||||
)
|
||||
if apply_result.matched:
|
||||
for cid in apply_result.matched:
|
||||
append_event(session, AuditEvent(
|
||||
event_type="claim.payer_rejected",
|
||||
entity_type="claim",
|
||||
entity_id=cid,
|
||||
payload={"source_batch_id": synthetic_id, "277ca_id": row.id},
|
||||
actor="277ca-parser-scheduler",
|
||||
))
|
||||
# SP28: auto-link the 277CA ClaimStatus entries to claims (D10
|
||||
# two-pass join). The helper builds dataclass rows; the
|
||||
# caller persists each via cycl_store.add_claim_ack so the
|
||||
# publish-from-store contract owns the live-tail event.
|
||||
def _pcn_lookup(pcn: str):
|
||||
return (
|
||||
session.query(db.Claim)
|
||||
.filter(db.Claim.patient_control_number == pcn)
|
||||
.first()
|
||||
)
|
||||
|
||||
link_result = apply_277ca_acks(
|
||||
session, result, ack_id=row.id,
|
||||
batch_envelope_index=batch_index,
|
||||
pc_claim_lookup=_pcn_lookup,
|
||||
)
|
||||
# Snapshot the rows before closing the work session — SQLite
|
||||
# + concurrent sessions from the same thread cause "database
|
||||
# is locked" errors when add_claim_ack opens its own session
|
||||
# while this one is still open.
|
||||
link_rows = list(link_result.linked)
|
||||
orphans = list(link_result.orphans)
|
||||
session.commit()
|
||||
|
||||
for link_row in link_rows:
|
||||
cycl_store.add_claim_ack(
|
||||
claim_id=link_row.claim_id,
|
||||
batch_id=link_row.batch_id,
|
||||
ack_id=row.id,
|
||||
ack_kind="277ca",
|
||||
ak2_index=link_row.ak2_index,
|
||||
set_control_number=link_row.set_control_number,
|
||||
set_accept_reject_code=link_row.set_accept_reject_code,
|
||||
linked_by="auto",
|
||||
)
|
||||
if orphans:
|
||||
log.warning(
|
||||
"277CA had %d orphan status entries (no matching claim): %s",
|
||||
len(orphans),
|
||||
orphans[:5],
|
||||
)
|
||||
|
||||
return ("parse_277ca", len(result.claim_statuses))
|
||||
@@ -0,0 +1,107 @@
|
||||
"""Handle an 835 ERA Remittance file (SP27 Task 5).
|
||||
|
||||
Lifted verbatim from ``scheduler.py:_handle_835``. The handler owns
|
||||
its own state, dispatches to ``parse_835`` (raises
|
||||
``CycloneParseError`` on bad EDI), runs the per-payer 835 validator,
|
||||
stamps the validation report into ``result.summary``, and persists
|
||||
the BatchRecord via ``cycl_store.add``.
|
||||
|
||||
The 835 handler is the largest of the four because the schema
|
||||
covers per-claim remittances + CAS adjustments + validation.
|
||||
``PAYER_FACTORIES_835`` lives in ``cyclone.api`` (it was always
|
||||
intended to live in ``cyclone.payers`` — a TODO pre-dating SP27 —
|
||||
but moving it is out of Task 5 scope). We import it lazily inside
|
||||
``handle`` to avoid a module-load-time cycle; the cyclic risk is
|
||||
acceptable because api.py also imports scheduler lazily (inside its
|
||||
lifespan handler).
|
||||
|
||||
Two-phase ingest closed in SP27 Task 10: the placeholder
|
||||
``adjustment_amount`` is overwritten by reconcile in the same DB
|
||||
session as the insert (before commit), so a reader never sees a
|
||||
half-reconciled Remittance row. If reconcile raises, the whole
|
||||
ingest rolls back and the scheduler records the per-file error.
|
||||
|
||||
``event_bus`` is best-effort and follows the same TODO(sp27-task-6)
|
||||
sync/async gap as handle_999 / handle_ta1 / handle_277ca.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
from cyclone.parsers.parse_835 import parse as parse_835
|
||||
from cyclone.parsers.validator_835 import validate as validate_835
|
||||
from cyclone.store import BatchRecord
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def handle(
|
||||
text: str,
|
||||
source_file: str,
|
||||
) -> tuple[str, int]:
|
||||
"""Parse an 835, run validation, persist batch + remittances.
|
||||
|
||||
Args:
|
||||
text: Raw 835 document bytes (decoded).
|
||||
source_file: Filename the 835 came from.
|
||||
|
||||
Returns:
|
||||
``(parser_used, claim_count)`` tuple. ``claim_count`` is the
|
||||
number of CLP claims parsed; the BatchRecord's summary's
|
||||
``passed`` / ``failed`` fields are derived from the
|
||||
validator's ``report.passed`` flag.
|
||||
|
||||
Raises:
|
||||
ValueError: on parser-level failure (wraps CycloneParseError).
|
||||
Validation failures don't raise — they're stamped into
|
||||
``summary.passed = 0``.
|
||||
|
||||
SP25: the 835 write path already publishes ``remittance_written``
|
||||
via ``CycloneStore.add`` (SP21 split). The handler no longer
|
||||
accepts an ``event_bus`` kwarg — the ``ack_received`` publish
|
||||
here was dead code anyway (the 835 event name is
|
||||
``remittance_written``, not ``ack_received``).
|
||||
"""
|
||||
# TODO(sp27-pre-t5): move PAYER_FACTORIES_835 out of api.py into
|
||||
# ``cyclone.payers`` to remove the lazy cyclic import below. The
|
||||
# import works today because api.py also imports scheduler lazily.
|
||||
from cyclone.api import PAYER_FACTORIES_835
|
||||
|
||||
config = PAYER_FACTORIES_835["co_medicaid_835"]()
|
||||
try:
|
||||
result = parse_835(text, config, input_file=source_file)
|
||||
except CycloneParseError as exc:
|
||||
raise ValueError(f"835 parse error: {exc}") from exc
|
||||
|
||||
# Validation report (mirrors the API endpoint).
|
||||
report = validate_835(result, config)
|
||||
n = len(result.claims)
|
||||
if report.passed:
|
||||
passed, failed, failed_claim_ids = n, 0, []
|
||||
else:
|
||||
passed, failed, failed_claim_ids = 0, n, [
|
||||
c.payer_claim_control_number for c in result.claims
|
||||
]
|
||||
result = result.model_copy(update={
|
||||
"validation": report,
|
||||
"summary": result.summary.model_copy(update={
|
||||
"passed": passed,
|
||||
"failed": failed,
|
||||
"failed_claim_ids": failed_claim_ids,
|
||||
}),
|
||||
})
|
||||
|
||||
rec = BatchRecord(
|
||||
id=uuid.uuid4().hex,
|
||||
kind="835",
|
||||
input_filename=source_file,
|
||||
parsed_at=datetime.now(timezone.utc),
|
||||
result=result,
|
||||
)
|
||||
cycl_store.add(rec)
|
||||
|
||||
return ("parse_835", len(result.claims))
|
||||
@@ -0,0 +1,157 @@
|
||||
"""Handle a 999 Implementation Acknowledgment file (SP27 Task 2).
|
||||
|
||||
Lifted verbatim from ``scheduler.py:_handle_999``. The handler owns
|
||||
its own DB session, dispatches to ``parse_999_text``, applies 999
|
||||
rejections to any matched claims via ``inbox_state.apply_999_rejections``,
|
||||
persists the ack row, and returns a ``(parser_used, claim_count)``
|
||||
tuple.
|
||||
|
||||
The actor tag (``"999-parser-scheduler"``) is preserved so the audit
|
||||
log keeps tracing back to the same source after extraction. Both
|
||||
the FastAPI endpoint and the scheduler path (re)use this module —
|
||||
see Task 6 for the API migration that drops the inline copy.
|
||||
|
||||
``claim_count`` mirrors ``parsed.received`` — the count of AK2
|
||||
(set-level) responses in the 999, which is the authoritative
|
||||
per-claim accept/reject signal (see ``_ack_id`` for why AK9 is
|
||||
ignored).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.audit_log import AuditEvent, append_event
|
||||
from cyclone.claim_acks import apply_999_acceptances
|
||||
from cyclone.handlers._ack_id import (
|
||||
ack_count_summary,
|
||||
ack_synthetic_source_batch_id,
|
||||
)
|
||||
from cyclone.inbox_state import apply_999_rejections
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
from cyclone.parsers.parse_999 import parse_999_text
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def handle(
|
||||
text: str,
|
||||
source_file: str,
|
||||
) -> tuple[str, int]:
|
||||
"""Parse a 999, apply rejections, persist ack row.
|
||||
|
||||
Args:
|
||||
text: Raw 999 document bytes (decoded).
|
||||
source_file: Filename the 999 came from. Used to derive a
|
||||
unique synthetic ``batches.id`` (see ``_ack_id``).
|
||||
|
||||
Returns:
|
||||
``(parser_used, claim_count)`` tuple. Matches the scheduler
|
||||
``_download_and_parse`` destructure; the HandleResult
|
||||
migration happens in Task 7.
|
||||
|
||||
Raises:
|
||||
ValueError: on parser-level failure (wraps CycloneParseError).
|
||||
|
||||
SP25: the store (``cyclone.store.acks.add_999_ack``) now owns the
|
||||
publish-from-store contract; the handler no longer needs an
|
||||
``event_bus`` kwarg. Both the scheduler path (no bus) and the
|
||||
FastAPI endpoint path (passes the bus to the store directly) see
|
||||
the same row, the same event, and the same payload.
|
||||
"""
|
||||
try:
|
||||
result = parse_999_text(text, input_file=source_file)
|
||||
except CycloneParseError as exc:
|
||||
raise ValueError(f"999 parse error: {exc}") from exc
|
||||
|
||||
received, accepted, rejected, ack_code = ack_count_summary(result)
|
||||
icn = result.envelope.control_number
|
||||
pcn = (
|
||||
result.set_responses[0].set_control_number
|
||||
if result.set_responses else None
|
||||
)
|
||||
synthetic_id = ack_synthetic_source_batch_id(
|
||||
icn, pcn=pcn, source_filename=source_file,
|
||||
)
|
||||
|
||||
# Build the batch envelope index BEFORE opening the work session
|
||||
# — cycl_store.batch_envelope_index opens its own short-lived
|
||||
# session, and SQLite + concurrent sessions causes "database is
|
||||
# locked" errors.
|
||||
batch_index = cycl_store.batch_envelope_index()
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
def _lookup(pcn: str):
|
||||
return (
|
||||
session.query(db.Claim)
|
||||
.filter_by(patient_control_number=pcn)
|
||||
.first()
|
||||
)
|
||||
rejection_result = apply_999_rejections(
|
||||
session, result,
|
||||
claim_lookup=_lookup,
|
||||
batch_envelope_index=batch_index,
|
||||
)
|
||||
if rejection_result.matched:
|
||||
for cid in rejection_result.matched:
|
||||
append_event(session, AuditEvent(
|
||||
event_type="claim.rejected",
|
||||
entity_type="claim",
|
||||
entity_id=cid,
|
||||
payload={"source_batch_id": synthetic_id},
|
||||
actor="999-parser-scheduler",
|
||||
))
|
||||
row = cycl_store.add_ack(
|
||||
source_batch_id=synthetic_id,
|
||||
accepted_count=accepted,
|
||||
rejected_count=rejected,
|
||||
received_count=received,
|
||||
ack_code=ack_code,
|
||||
raw_json=json.loads(result.model_dump_json()),
|
||||
)
|
||||
# SP28: auto-link the 999 AK2 set-responses to claims (D10 two-pass
|
||||
# join). The PCN fallback only fires when the ST02 lookup misses
|
||||
# (rare for Gainwell batches). Each created ClaimAck row is
|
||||
# persisted via cycl_store.add_claim_ack so the publish-from-store
|
||||
# contract owns the live-tail event.
|
||||
def _pcn_lookup(pcn: str):
|
||||
return (
|
||||
session.query(db.Claim)
|
||||
.filter_by(patient_control_number=pcn)
|
||||
.first()
|
||||
)
|
||||
|
||||
link_result = apply_999_acceptances(
|
||||
session, result, ack_id=row.id,
|
||||
batch_envelope_index=batch_index,
|
||||
pc_claim_lookup=_pcn_lookup,
|
||||
)
|
||||
# Snapshot the rows before closing the work session — SQLite
|
||||
# + concurrent sessions from the same thread cause "database
|
||||
# is locked" errors when add_claim_ack opens its own session
|
||||
# while this one is still open.
|
||||
link_rows = list(link_result.linked)
|
||||
orphans = list(link_result.orphans)
|
||||
session.commit()
|
||||
|
||||
for link_row in link_rows:
|
||||
cycl_store.add_claim_ack(
|
||||
claim_id=link_row.claim_id,
|
||||
batch_id=link_row.batch_id,
|
||||
ack_id=row.id,
|
||||
ack_kind="999",
|
||||
ak2_index=link_row.ak2_index,
|
||||
set_control_number=link_row.set_control_number,
|
||||
set_accept_reject_code=link_row.set_accept_reject_code,
|
||||
linked_by="auto",
|
||||
)
|
||||
if orphans:
|
||||
log.warning(
|
||||
"999 had %d orphan set refs (no matching claim): %s",
|
||||
len(orphans),
|
||||
orphans[:5],
|
||||
)
|
||||
|
||||
return ("parse_999", received)
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Shared ``HandleResult`` dataclass for handlers (SP27).
|
||||
|
||||
Every per-file-type handler returns the same shape so the
|
||||
scheduler's ``_handle_one`` and the FastAPI parse endpoints can
|
||||
process them uniformly.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class HandleResult:
|
||||
"""Outcome of one handler invocation.
|
||||
|
||||
Attributes:
|
||||
parser_used: The parser name written to
|
||||
``processed_inbound_files.parser_used`` (e.g. ``"parse_999"``)
|
||||
and surfaced in the UI.
|
||||
claim_count: The number of claim rows persisted (or batch
|
||||
records, depending on the file type). For 999/TA1 this
|
||||
is the receipt count; for 835 this is the per-claim
|
||||
remittance count; for 277CA this is the per-claim
|
||||
status count.
|
||||
batch_id: The persisted batch id (when the handler creates a
|
||||
row in ``batches``). ``None`` for handlers that persist
|
||||
into per-file-type ack tables (999/TA1/277CA) rather
|
||||
than into the unified ``batches`` table.
|
||||
matched_count: For 835, the number of remits that were
|
||||
matched to an existing claim by ``reconcile.match``.
|
||||
Zero for other handlers.
|
||||
"""
|
||||
|
||||
parser_used: str
|
||||
claim_count: int
|
||||
batch_id: str | None = None
|
||||
matched_count: int = 0
|
||||
@@ -0,0 +1,123 @@
|
||||
"""Handle a TA1 Interchange Acknowledgment file (SP27 Task 3).
|
||||
|
||||
Lifted verbatim from ``scheduler.py:_handle_ta1``. The handler owns
|
||||
its own DB session, dispatches to ``parse_ta1_text``, persists the
|
||||
interchange ack row in ``ta1_acks``, and returns a
|
||||
``(parser_used, claim_count)`` tuple.
|
||||
|
||||
Unlike the 999 handler, TA1 is envelope-only — there is one TA1 per
|
||||
ISA/IEA interchange, no set-level (AK2) or claim-level matching.
|
||||
The claim_count is always 1 (one TA1 ack row per file).
|
||||
|
||||
The actor tag is implicit (no audit event here — TA1 doesn't tag
|
||||
anything in the activity log; it's an infrastructure-level ack).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.claim_acks import apply_ta1_envelope_link
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
from cyclone.parsers.parse_ta1 import parse_ta1_text
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def handle(
|
||||
text: str,
|
||||
source_file: str,
|
||||
) -> tuple[str, int]:
|
||||
"""Parse a TA1, persist the interchange ack row.
|
||||
|
||||
Args:
|
||||
text: Raw TA1 document bytes (decoded).
|
||||
source_file: Filename the TA1 came from. Used for audit
|
||||
attribution; the ``batches.id`` for TA1 rows is derived
|
||||
internally from the parsed envelope's control number
|
||||
(``TA1-{ICN}``).
|
||||
|
||||
Returns:
|
||||
``(parser_used, claim_count)`` tuple. TA1 always returns
|
||||
``claim_count=1`` (one TA1 ack row per interchange file).
|
||||
|
||||
Raises:
|
||||
ValueError: on parser-level failure (wraps CycloneParseError).
|
||||
|
||||
SP25: the store (``cyclone.store.acks.add_ta1_ack``) owns the
|
||||
publish-from-store contract; the handler no longer needs an
|
||||
``event_bus`` kwarg.
|
||||
"""
|
||||
try:
|
||||
result = parse_ta1_text(text, input_file=source_file)
|
||||
except CycloneParseError as exc:
|
||||
raise ValueError(f"TA1 parse error: {exc}") from exc
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
ta1_ack_row = cycl_store.add_ta1_ack(
|
||||
source_batch_id=result.source_batch_id,
|
||||
control_number=result.ta1.control_number,
|
||||
interchange_date=result.ta1.interchange_date,
|
||||
interchange_time=result.ta1.interchange_time,
|
||||
ack_code=result.ta1.ack_code,
|
||||
note_code=result.ta1.note_code,
|
||||
ack_generated_date=result.ta1.ack_generated_date,
|
||||
sender_id=result.envelope.sender_id,
|
||||
receiver_id=result.envelope.receiver_id,
|
||||
raw_json=json.loads(result.model_dump_json()),
|
||||
)
|
||||
# SP28: TA1 envelope-level link to the originating Batch. The
|
||||
# closure here matches the most-recent Batch whose envelope
|
||||
# sender_id/receiver_id matches the TA1 — see spec §D4.
|
||||
def _batch_lookup(sender_id, receiver_id):
|
||||
rows = (
|
||||
session.query(db.Batch)
|
||||
.filter(
|
||||
db.Batch.kind == "837p",
|
||||
db.Batch.raw_result_json.isnot(None),
|
||||
)
|
||||
.order_by(db.Batch.parsed_at.desc())
|
||||
.all()
|
||||
)
|
||||
for row in rows:
|
||||
env = (row.raw_result_json or {}).get("envelope") or {}
|
||||
if (
|
||||
env.get("sender_id") == sender_id
|
||||
and env.get("receiver_id") == receiver_id
|
||||
):
|
||||
return row
|
||||
return None
|
||||
|
||||
link_result = apply_ta1_envelope_link(
|
||||
session, result, ack_id=ta1_ack_row.id,
|
||||
batch_lookup=_batch_lookup,
|
||||
)
|
||||
# Snapshot rows before closing the work session — SQLite +
|
||||
# concurrent sessions from the same thread cause "database
|
||||
# is locked" errors when add_claim_ack opens its own session
|
||||
# while this one is still open.
|
||||
link_rows = list(link_result.linked)
|
||||
orphans = list(link_result.orphans)
|
||||
session.commit()
|
||||
|
||||
for link_row in link_rows:
|
||||
cycl_store.add_claim_ack(
|
||||
claim_id=link_row.claim_id,
|
||||
batch_id=link_row.batch_id,
|
||||
ack_id=ta1_ack_row.id,
|
||||
ack_kind="ta1",
|
||||
ak2_index=link_row.ak2_index,
|
||||
set_control_number=link_row.set_control_number,
|
||||
set_accept_reject_code=link_row.set_accept_reject_code,
|
||||
linked_by="auto",
|
||||
)
|
||||
if orphans:
|
||||
log.warning(
|
||||
"TA1 had %d orphan envelope refs (no matching batch): %s",
|
||||
len(orphans),
|
||||
orphans[:5],
|
||||
)
|
||||
|
||||
return ("parse_ta1", 1)
|
||||
@@ -175,6 +175,82 @@ def _line_count_lookup(session: Session, claims: list[Claim]) -> tuple[dict, dic
|
||||
return matched_counts, total_lines_by_claim
|
||||
|
||||
|
||||
def _ack_summary_for_claims(
|
||||
session: Session, claim_ids: list[str],
|
||||
) -> dict[str, dict]:
|
||||
"""Build a {claim_id: {total, rejected, items: [...]}} map for 999 acks.
|
||||
|
||||
SP29: the Inbox `rejected` lane needs to render AK2 evidence
|
||||
inline per row, plus a per-row Resubmit button. The data lives
|
||||
in ``claim_acks`` (SP28) — we don't want to N+1 fetch per row,
|
||||
so the whole rejected-claim set is summarized in one batched
|
||||
query here.
|
||||
|
||||
Filters to ``ack_kind='999'`` because the rejected lane is the
|
||||
999 envelope reject lane; the 277CA STC A4/A6/A7 evidence flows
|
||||
through the ``payer_rejected_*`` fields on a separate lane and
|
||||
isn't part of this scope (see SP29 spec D4 / scope).
|
||||
|
||||
Returns:
|
||||
``{claim_id: {"total": int, "rejected": int, "items": [...]}, ...}``
|
||||
Claims with zero linked 999 acks are NOT in the returned
|
||||
dict — the caller maps via ``.get(cid)`` and treats absence
|
||||
as "no 999 acks linked" (renders as ``null`` in the
|
||||
payload, ``999 not linked`` in the UI).
|
||||
|
||||
Args:
|
||||
session: SQLAlchemy session the caller owns.
|
||||
claim_ids: list of claim.id values to summarize. Typically
|
||||
the rejected-lane claim ids. Empty list → empty dict.
|
||||
"""
|
||||
if not claim_ids:
|
||||
return {}
|
||||
from cyclone.db import ClaimAck # late import — DB model registered
|
||||
rows = (
|
||||
session.query(
|
||||
ClaimAck.claim_id,
|
||||
ClaimAck.ack_id,
|
||||
ClaimAck.set_control_number,
|
||||
ClaimAck.set_accept_reject_code,
|
||||
ClaimAck.ak2_index,
|
||||
ClaimAck.linked_at,
|
||||
)
|
||||
.filter(
|
||||
ClaimAck.claim_id.in_(claim_ids),
|
||||
ClaimAck.ack_kind == "999",
|
||||
)
|
||||
.order_by(ClaimAck.linked_at.desc(), ClaimAck.id.desc())
|
||||
.all()
|
||||
)
|
||||
grouped: dict[str, list[tuple]] = {}
|
||||
for cid, aid, scn, code, ak2i, lat in rows:
|
||||
grouped.setdefault(cid, []).append((aid, scn, code, ak2i, lat))
|
||||
|
||||
rejected_codes = {"R", "E", "X"}
|
||||
out: dict[str, dict] = {}
|
||||
for cid, items in grouped.items():
|
||||
total = len(items)
|
||||
rejected_count = sum(1 for it in items if (it[2] or "") in rejected_codes)
|
||||
# Keep 5 most recent items for the chip column. The full count
|
||||
# is in ``total`` so the UI can show ``+N more`` honestly.
|
||||
trimmed = items[:5]
|
||||
out[cid] = {
|
||||
"total": total,
|
||||
"rejected": rejected_count,
|
||||
"items": [
|
||||
{
|
||||
"ack_id": aid,
|
||||
"set_control_number": scn,
|
||||
"set_accept_reject_code": code or "",
|
||||
"ak2_index": ak2i,
|
||||
"linked_at": _isoformat(lat),
|
||||
}
|
||||
for (aid, scn, code, ak2i, lat) in trimmed
|
||||
],
|
||||
}
|
||||
return out
|
||||
|
||||
|
||||
def compute_lanes(session: Session, *, dismissed_pairs: Iterable[frozenset]) -> Lanes:
|
||||
lanes = Lanes()
|
||||
dismissed = set(dismissed_pairs)
|
||||
@@ -192,6 +268,17 @@ def compute_lanes(session: Session, *, dismissed_pairs: Iterable[frozenset]) ->
|
||||
),
|
||||
))
|
||||
|
||||
# SP29: attach the 999 ack-evidence summary (total / rejected /
|
||||
# 5 most recent AK2 set_responses) to every rejected row so the
|
||||
# Inbox can render AK2 chips inline + a per-row Resubmit button
|
||||
# without an extra round-trip. One batched query, keyed off the
|
||||
# rejected-claim id set.
|
||||
rejected_ack_summary = _ack_summary_for_claims(
|
||||
session, [r["id"] for r in lanes.rejected]
|
||||
)
|
||||
for row in lanes.rejected:
|
||||
row["claim_acks"] = rejected_ack_summary.get(row["id"])
|
||||
|
||||
# --- Payer-Rejected (SP10) ---
|
||||
# Distinct from the 999 envelope "rejected" lane above. A claim
|
||||
# lands here when a 277CA STC category code is A4/A6/A7 (rejected
|
||||
|
||||
@@ -33,42 +33,63 @@ def apply_999_rejections(
|
||||
parsed_999,
|
||||
*,
|
||||
claim_lookup: Callable[[str], Claim | None],
|
||||
batch_envelope_index: dict[str, list[str]] | None = None,
|
||||
) -> Apply999Result:
|
||||
"""For each set response with code R or E, look up the matching claim and
|
||||
"""For each set response with code R, E, or X, look up the matching claim and
|
||||
move it to REJECTED. Idempotent on already-rejected claims.
|
||||
|
||||
Args:
|
||||
session: SQLAlchemy session.
|
||||
parsed_999: a ParseResult999 (or any object with .set_responses).
|
||||
claim_lookup: callable from patient_control_number → Claim or None.
|
||||
Legacy fallback; rarely hits when batch_envelope_index is present.
|
||||
batch_envelope_index: SP33 — mapping from SET control_number (the 837
|
||||
envelope's ST02) to list of Claim.id for the claims in that SET.
|
||||
Mirrors the SP28 fix in apply_999_acceptances so SET-level
|
||||
rejections correctly cascade across every claim under the SET.
|
||||
|
||||
Returns:
|
||||
Apply999Result with lists of matched claim ids and orphan PCNs.
|
||||
"""
|
||||
result = Apply999Result()
|
||||
now = datetime.now(timezone.utc)
|
||||
index = batch_envelope_index or {}
|
||||
|
||||
for sr in parsed_999.set_responses:
|
||||
code = sr.set_accept_reject.code
|
||||
if code not in ("R", "E", "X"):
|
||||
continue
|
||||
|
||||
claim = claim_lookup(sr.set_control_number)
|
||||
if claim is None:
|
||||
result.orphans.append(sr.set_control_number)
|
||||
continue
|
||||
# SP33: prefer batch_envelope_index (SCN -> [claim_id]) so a SET-level
|
||||
# rejection correctly flips every claim in the SET. Fall back to
|
||||
# the legacy claim_lookup when the index is empty for this SCN.
|
||||
candidate_ids = index.get(sr.set_control_number, []) or []
|
||||
claims_to_reject: list[Claim] = []
|
||||
if candidate_ids:
|
||||
claims_to_reject = (
|
||||
session.query(Claim)
|
||||
.filter(Claim.id.in_(candidate_ids))
|
||||
.all()
|
||||
)
|
||||
else:
|
||||
legacy = claim_lookup(sr.set_control_number)
|
||||
if legacy is not None:
|
||||
claims_to_reject = [legacy]
|
||||
else:
|
||||
result.orphans.append(sr.set_control_number)
|
||||
continue
|
||||
|
||||
if claim.state == ClaimState.REJECTED:
|
||||
# Idempotent: don't double-mutate.
|
||||
continue
|
||||
|
||||
claim.state = ClaimState.REJECTED
|
||||
claim.state_changed_at = now
|
||||
claim.rejected_at = now
|
||||
claim.rejection_reason = _build_reason(
|
||||
code, len(sr.segment_errors or [])
|
||||
)
|
||||
result.matched.append(claim.id)
|
||||
for claim in claims_to_reject:
|
||||
if claim.state == ClaimState.REJECTED:
|
||||
# Idempotent: don't double-mutate.
|
||||
continue
|
||||
claim.state = ClaimState.REJECTED
|
||||
claim.state_changed_at = now
|
||||
claim.rejected_at = now
|
||||
claim.rejection_reason = _build_reason(
|
||||
code, len(sr.segment_errors or [])
|
||||
)
|
||||
result.matched.append(claim.id)
|
||||
|
||||
if result.matched or result.orphans:
|
||||
session.commit()
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
-- version: 16
|
||||
-- Add the missing index on claims.matched_remittance_id.
|
||||
--
|
||||
-- SP27 Task 11 added ``check_matched_pair_drift`` at startup, which
|
||||
-- scans ``WHERE Claim.matched_remittance_id IS NOT NULL``. Without an
|
||||
-- index this is a full-table scan; becomes a noticeable boot
|
||||
-- latency cost past ~10k claims. The companion index on the
|
||||
-- reverse side (``remittances.claim_id``) was added in 0007.
|
||||
--
|
||||
-- A plain index is enough — neither side is unique (reversals
|
||||
-- re-reference the original PCN, and a claim without a match is
|
||||
-- fine).
|
||||
|
||||
CREATE INDEX ix_claims_matched_remittance_id
|
||||
ON claims(matched_remittance_id);
|
||||
@@ -0,0 +1,28 @@
|
||||
-- version: 17
|
||||
-- Backfill claims.patient_control_number = claims.id.
|
||||
--
|
||||
-- SP27 Task 17 fixed the 837 ingest in store.py:_claim_837_row so
|
||||
-- ``Claim.patient_control_number`` is populated from
|
||||
-- ``claim.claim_id`` (CLM01) instead of ``claim.subscriber.member_id``
|
||||
-- (the 2010BA NM109). The reconcile matcher joins on
|
||||
-- ``Claim.patient_control_number == Remittance.payer_claim_control_number``
|
||||
-- and the 835 echoes CLM01 in CLP01 per X12 spec, so the wrong field
|
||||
-- silently broke every auto-match in production.
|
||||
--
|
||||
-- For rows written BEFORE the fix, the stored value is the member_id
|
||||
-- (e.g. "W953474") which never matches any remit's CLP01. This
|
||||
-- migration backfills those rows by aligning
|
||||
-- ``patient_control_number`` with the row's own ``id`` (= CLM01).
|
||||
-- After this, every claim row's PCN is consistent with the value the
|
||||
-- 837 actually sent, and any newly-ingested 835 whose CLP01 echoes
|
||||
-- that CLM01 will auto-pair.
|
||||
--
|
||||
-- Idempotent: only touches rows where the stored PCN doesn't already
|
||||
-- match the row's id, so re-running on already-fixed rows is a no-op.
|
||||
--
|
||||
-- Reversal safety: the migration does NOT clear matched_remittance_id,
|
||||
-- so any pre-existing manual_match pairs stay intact.
|
||||
|
||||
UPDATE claims
|
||||
SET patient_control_number = id
|
||||
WHERE patient_control_number IS DISTINCT FROM id;
|
||||
@@ -0,0 +1,52 @@
|
||||
-- version: 18
|
||||
-- SP28: per-ACK auto-link join table.
|
||||
--
|
||||
-- Closes the operator gap where every inbound 999 / 277CA / TA1 ack was
|
||||
-- persisted but never durably linked back to the claim it
|
||||
-- acknowledges. One row per AK2 set-response for 999, per ClaimStatus
|
||||
-- for 277CA, per TA1 envelope (with claim_id NULL + batch_id set).
|
||||
--
|
||||
-- Granularity (per-AK2) is preserved by ``ak2_index`` and the unique
|
||||
-- index ``ux_claim_acks_dedup`` — an auto-link of
|
||||
-- (claim, 999, ak2_index=3) is idempotent on re-ingest of the same
|
||||
-- 999 file (the index enforces this at the DB layer; the helper
|
||||
-- pre-checks to avoid IntegrityError log noise).
|
||||
--
|
||||
-- Notes:
|
||||
-- * ``claim_id`` is nullable so TA1 envelope-level links to the
|
||||
-- originating Batch can land here (FK to batches.id). The
|
||||
-- CHECK constraint makes sure at least one of (claim_id,
|
||||
-- batch_id) is set on every row — see spec §3.1.
|
||||
-- * ``set_control_number`` records the value the upstream ACK
|
||||
-- ACTUALLY CARRIED (== source 837 ST02 for Gainwell batches). It
|
||||
-- is the orphan-traceability field — the link survives even when
|
||||
-- the join had to fall back from ST02 to PCN matching.
|
||||
-- * ``set_accept_reject_code`` carries the AK5 code (A/E/R/X) for
|
||||
-- 999 or the STC category code (A1/A2/A3/A4/A6/A7 etc.) for 277CA.
|
||||
-- TA1 stores the envelope-level ack code here (A/R/E).
|
||||
-- * No FK constraint on ``(ack_kind, ack_id)`` — there are three
|
||||
-- separate ack tables (``acks``, ``ta1_acks``, ``two77ca_acks``).
|
||||
-- Application code enforces the discriminator.
|
||||
|
||||
CREATE TABLE claim_acks (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
claim_id TEXT REFERENCES claims(id) ON DELETE CASCADE,
|
||||
batch_id TEXT REFERENCES batches(id) ON DELETE CASCADE,
|
||||
ack_id INTEGER NOT NULL,
|
||||
ack_kind TEXT NOT NULL CHECK (ack_kind IN ('999', '277ca', 'ta1')),
|
||||
ak2_index INTEGER,
|
||||
set_control_number TEXT,
|
||||
set_accept_reject_code TEXT,
|
||||
linked_at DATETIME NOT NULL,
|
||||
linked_by TEXT NOT NULL CHECK (linked_by IN ('auto', 'manual')),
|
||||
CHECK ((claim_id IS NOT NULL) OR (batch_id IS NOT NULL))
|
||||
);
|
||||
|
||||
CREATE INDEX ix_claim_acks_claim_id ON claim_acks(claim_id);
|
||||
CREATE INDEX ix_claim_acks_batch_id ON claim_acks(batch_id);
|
||||
CREATE INDEX ix_claim_acks_ack ON claim_acks(ack_kind, ack_id);
|
||||
|
||||
-- Dedup: an auto-link of (claim, 999, ak2_index=3) is idempotent on re-ingest.
|
||||
CREATE UNIQUE INDEX ux_claim_acks_dedup
|
||||
ON claim_acks(claim_id, ack_kind, ack_id, ak2_index)
|
||||
WHERE claim_id IS NOT NULL AND ak2_index IS NOT NULL;
|
||||
@@ -0,0 +1,7 @@
|
||||
-- version: 19
|
||||
-- SP32: render & service-provider NPI extraction.
|
||||
-- Nullable: existing rows stay NULL until backfill runs.
|
||||
-- No indexes (used for set-equality, not range queries; nullable).
|
||||
|
||||
ALTER TABLE claims ADD COLUMN rendering_provider_npi TEXT;
|
||||
ALTER TABLE remittances ADD COLUMN rendering_provider_npi TEXT;
|
||||
@@ -142,6 +142,7 @@ class ClaimOutput(_Base):
|
||||
subscriber: Subscriber
|
||||
payer: Payer
|
||||
claim: ClaimHeader
|
||||
rendering_provider_npi: str | None = None # NM1*82 NM109 (Loop 2420A)
|
||||
diagnoses: list[Diagnosis] = Field(default_factory=list)
|
||||
service_lines: list[ServiceLine] = Field(default_factory=list)
|
||||
validation: ValidationReport
|
||||
|
||||
@@ -160,6 +160,7 @@ class ClaimPayment(_Base):
|
||||
ref_benefit_plan: str | None = None # REF*CE per the CO guide
|
||||
service_payments: list[ServicePayment] = Field(default_factory=list)
|
||||
raw_segments: list[list[str]] = Field(default_factory=list)
|
||||
service_provider_npi: str | None = None # NM1*1P NM109 (Loop 2100 service provider)
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
|
||||
@@ -376,6 +376,7 @@ def _consume_claim_payment(segments: list[list[str]], idx: int) -> tuple[ClaimPa
|
||||
|
||||
service_payments: list[ServicePayment] = []
|
||||
ref_benefit_plan: str | None = None
|
||||
service_provider_npi: str | None = None
|
||||
per_diem: Decimal | None = None
|
||||
status_label = claim_status_label(status)
|
||||
|
||||
@@ -422,9 +423,16 @@ def _consume_claim_payment(segments: list[list[str]], idx: int) -> tuple[ClaimPa
|
||||
except ValueError:
|
||||
per_diem = None
|
||||
elif s[0] == "NM1":
|
||||
# Patient (QC) / service-provider (1P) — captured in raw_segments.
|
||||
# The 835 spec doesn't require a structured patient model in v1.
|
||||
pass
|
||||
# SP32: capture service-provider NPI from NM1*1P (Loop 2100).
|
||||
# NM108 (idx 8) carries the ID qualifier (typically "XX");
|
||||
# NM109 (idx 9) is the value. Some senders omit NM108; accept
|
||||
# both forms but require a 10-digit ID.
|
||||
if len(s) > 9 and s[1] == "1P":
|
||||
if len(s) > 8 and s[8] == "XX" and s[9]:
|
||||
if s[9].isdigit() and len(s[9]) == 10:
|
||||
service_provider_npi = s[9]
|
||||
elif s[9] and s[9].isdigit() and len(s[9]) == 10:
|
||||
service_provider_npi = s[9]
|
||||
elif s[0] == "DTM":
|
||||
# Claim-level dates — captured in raw_segments.
|
||||
pass
|
||||
@@ -446,6 +454,7 @@ def _consume_claim_payment(segments: list[list[str]], idx: int) -> tuple[ClaimPa
|
||||
ref_benefit_plan=ref_benefit_plan,
|
||||
service_payments=service_payments,
|
||||
raw_segments=raw,
|
||||
service_provider_npi=service_provider_npi,
|
||||
),
|
||||
idx,
|
||||
)
|
||||
|
||||
@@ -210,6 +210,7 @@ def _consume_claim(segments: list[list[str]], idx: int) -> tuple[ClaimOutput, in
|
||||
service_lines: list[ServiceLine] = []
|
||||
raw: list[list[str]] = [seg]
|
||||
prior_auth: str | None = None
|
||||
rendering_provider_npi: str | None = None
|
||||
|
||||
idx += 1
|
||||
while idx < len(segments) and segments[idx][0] not in {"HL", "CLM", "SE"}:
|
||||
@@ -226,6 +227,11 @@ def _consume_claim(segments: list[list[str]], idx: int) -> tuple[ClaimOutput, in
|
||||
for code in parts[1:]:
|
||||
if code:
|
||||
diagnoses.append(Diagnosis(code=code, qualifier=qualifier))
|
||||
elif s[0] == "NM1" and len(s) > 1 and s[1] == "82":
|
||||
# SP32: capture rendering provider NPI from Loop 2420A NM1*82.
|
||||
# NM108 (idx 8) is the qualifier (typically "XX"), NM109 (idx 9) is the NPI.
|
||||
if len(s) > 9 and s[8] == "XX" and len(s[9]) == 10 and s[9].isdigit():
|
||||
rendering_provider_npi = s[9]
|
||||
elif s[0] == "LX":
|
||||
line_no = int(s[1]) if len(s) > 1 and s[1].isdigit() else len(service_lines) + 1
|
||||
# LX is just a separator — the actual service line data is in the next SV1.
|
||||
@@ -255,6 +261,7 @@ def _consume_claim(segments: list[list[str]], idx: int) -> tuple[ClaimOutput, in
|
||||
subscriber=Subscriber(first_name="", last_name="", member_id="", address=Address(line1="", city="", state="", zip="")),
|
||||
payer=Payer(name="", id=""),
|
||||
claim=claim_header,
|
||||
rendering_provider_npi=rendering_provider_npi,
|
||||
diagnoses=diagnoses,
|
||||
service_lines=service_lines,
|
||||
validation=ValidationReport(passed=True, errors=[], warnings=[]),
|
||||
@@ -296,7 +303,7 @@ def _consume_service_line(segments: list[list[str]], idx: int, line_no: int) ->
|
||||
provider_ref: str | None = None
|
||||
|
||||
idx += 1
|
||||
while idx < len(segments) and segments[idx][0] not in {"LX", "HL", "CLM", "SE"}:
|
||||
while idx < len(segments) and segments[idx][0] not in {"LX", "HL", "CLM", "SE", "NM1"}:
|
||||
s = segments[idx]
|
||||
raw.append(s)
|
||||
if s[0] == "DTP" and len(s) > 2 and s[1] == "472":
|
||||
|
||||
@@ -8,6 +8,12 @@ Single-pass walker over the tokenized segment list:
|
||||
- AK3 (Segment Context) + AK4 (Element Context) — optional per-segment errors
|
||||
- AK5 (Transaction Set Response Status) — per-set accept/reject
|
||||
- AK9 (Functional Group Response Status) — per-group counts + ack code
|
||||
- IK5 — a non-standard synonym for ``AK5`` that Gainwell's MFT ships
|
||||
in place of the spec-defined ``AK5``. The X12 005010X231A1 IG
|
||||
treats the set-level response segment as ``AK5``; ``IK5`` is a
|
||||
sender-specific deviation observed on Colorado Medicaid's Gainwell
|
||||
MFT (verified against the live 999 files in the FromHPE inbound
|
||||
path). We accept either.
|
||||
- SE / GE / IEA
|
||||
|
||||
Errors at the file level raise :class:`CycloneParseError`. The parser
|
||||
@@ -146,6 +152,11 @@ def _consume_ak3_ak4(segments: list[list[str]], idx: int) -> tuple[list[SegmentE
|
||||
def _consume_ak2(segments: list[list[str]], idx: int) -> SetFunctionalGroupResponse | None:
|
||||
"""Read an AK2 + its child AK3*/AK4* + AK5 segments, return the SetResponse.
|
||||
|
||||
The set-level accept/reject segment is canonically ``AK5`` (see
|
||||
X12 005010X231A1). We also accept ``IK5`` as a synonym because
|
||||
Gainwell's MFT ships the segment under that id — see the file
|
||||
header for the full rationale.
|
||||
|
||||
Returns None when called with a non-AK2 segment (defensive — the
|
||||
orchestrator only calls this when it sees AK2).
|
||||
"""
|
||||
@@ -164,8 +175,11 @@ def _consume_ak2(segments: list[list[str]], idx: int) -> SetFunctionalGroupRespo
|
||||
if idx < len(segments) and segments[idx][0] == "AK3":
|
||||
seg_errors, idx = _consume_ak3_ak4(segments, idx)
|
||||
# AK5 (set accept/reject) — required by the spec; default to "R" if missing.
|
||||
# Gainwell's MFT uses IK5 instead of AK5 (sender-specific segment id
|
||||
# that means the same thing); accept either. The default of "R"
|
||||
# matters: if the segment is missing entirely, the 999 is a reject.
|
||||
accept_code = "R"
|
||||
if idx < len(segments) and segments[idx][0] == "AK5":
|
||||
if idx < len(segments) and segments[idx][0] in ("AK5", "IK5"):
|
||||
ak5 = segments[idx]
|
||||
if len(ak5) > 1 and ak5[1]:
|
||||
accept_code = ak5[1]
|
||||
@@ -256,8 +270,11 @@ def parse_999_text(text: str, *, input_file: str = "") -> ParseResult999:
|
||||
set_responses.append(sr)
|
||||
# Advance past the AK2 + AK3*/AK4*/AK5 cluster
|
||||
# (re-walk from i+1 because _consume_ak2 doesn't return idx).
|
||||
# ``IK5`` is the Gainwell-specific synonym for ``AK5``
|
||||
# and must be in the consumed set here too (see
|
||||
# _consume_ak2 for the full rationale).
|
||||
i += 1
|
||||
while i < len(segments) and segments[i][0] in {"AK3", "AK4", "AK5"}:
|
||||
while i < len(segments) and segments[i][0] in {"AK3", "AK4", "AK5", "IK5"}:
|
||||
i += 1
|
||||
else:
|
||||
i += 1
|
||||
|
||||
@@ -66,8 +66,8 @@ class PayerConfig(BaseModel):
|
||||
# Lenient in v1 — see spec §9 R031.
|
||||
require_ref_g1_for_adjustments=False,
|
||||
allowed_bht06={"CH"},
|
||||
payer_id="SKCO0",
|
||||
payer_name="COHCPF",
|
||||
payer_id="CO_TXIX",
|
||||
payer_name="CO_TXIX",
|
||||
no_patient_loop=True,
|
||||
encounter_claim_in_same_batch=False,
|
||||
allowed_facility_qualifiers={"B"},
|
||||
|
||||
@@ -17,7 +17,7 @@ Match algorithm:
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import date, datetime, timezone
|
||||
from decimal import Decimal
|
||||
from typing import Iterable, Optional, Protocol
|
||||
@@ -50,6 +50,18 @@ class Match:
|
||||
remittance: _RemitLike
|
||||
strategy: str
|
||||
is_reversal: bool
|
||||
# SP31: which content-keys the 2-of-3 (or 3-of-3) rule agreed on.
|
||||
# Populated for ``score-auto`` matches; empty for ``pcn-exact`` /
|
||||
# ``manual`` (PCN-exact only uses the PCN key by definition; manual
|
||||
# is operator-driven). Floats into the ``auto_matched_835``
|
||||
# ActivityEvent payload so the audit trail records *why* the link
|
||||
# fired. Transient — the ORM Match row does not persist these
|
||||
# (spec keeps the audit trail in ActivityEvent.payload_json only).
|
||||
keys_matched: set[str] = field(default_factory=set)
|
||||
# SP31: how many candidates were in the ±30-day window pool when
|
||||
# the score-auto match fired. Useful when an operator is reviewing
|
||||
# a borderline auto-link — 1-of-1 is unambiguous; 1-of-5 was lucky.
|
||||
candidate_count: int = 0
|
||||
|
||||
|
||||
def match(
|
||||
@@ -84,7 +96,7 @@ def match(
|
||||
matches.append(Match(
|
||||
claim=chosen,
|
||||
remittance=r,
|
||||
strategy="auto",
|
||||
strategy="pcn-exact",
|
||||
is_reversal=r.is_reversal,
|
||||
))
|
||||
used_claim_ids.add(chosen.id)
|
||||
@@ -117,6 +129,163 @@ def _pick_claim(
|
||||
return None
|
||||
|
||||
|
||||
# --- SP31: content-keys fallback matcher -----------------------------------
|
||||
|
||||
CHARGE_TOLERANCE = Decimal("0.01")
|
||||
KEYS_REQUIRED = 2
|
||||
|
||||
|
||||
def _content_keys_match(remit, claim) -> set[str]:
|
||||
"""Return the set of content-keys that agree between ``remit`` and ``claim``.
|
||||
|
||||
Compares {``pcn``, ``charge``, ``npi``} between the two sides and returns
|
||||
the subset whose values match. Caller checks ``len(returned) >= KEYS_REQUIRED``
|
||||
to decide whether to auto-link. Returning the matched keys (rather than a
|
||||
bool) lets the ActivityEvent payload record exactly which 2-of-3 (or 3-of-3)
|
||||
produced the auto-link — the audit trail for the SP31 content-keys rule.
|
||||
|
||||
Charge compared with ``CHARGE_TOLERANCE`` tolerance (rounding drift).
|
||||
NPI counts as "not matched" when the remit's NPI is empty.
|
||||
|
||||
Uses duck-typing via ``getattr`` so the helper works against both:
|
||||
- shim / dataclass test objects (planned names: total_charge_amount,
|
||||
total_charge, rendering_provider_npi), and
|
||||
- real SQLAlchemy ORM instances (Claim.charge_amount, Remittance.total_charge,
|
||||
Claim.provider_npi, Claim.rendering_provider_npi,
|
||||
Remittance.rendering_provider_npi).
|
||||
|
||||
Production note (SP32): the typed ``rendering_provider_npi`` column is
|
||||
the primary read for both sides (Claim = NM1*82, Remit = NM1*1P). Legacy
|
||||
rows pre-0019 still carry the value in ``raw_json``; the read order is:
|
||||
- NPI: typed column → raw_json (``service_provider_npi`` for remit,
|
||||
``rendering_provider_npi`` for claim) → claim ``provider_npi``
|
||||
(billing fallback for legacy 837p rows without NM1*82 extraction).
|
||||
- Charge: planned attribute name → real ORM column → raw_json.
|
||||
The ``raw_json`` fallback is what makes this helper safe against a
|
||||
raw ``Remittance`` ORM instance from ``reconcile.run()``.
|
||||
"""
|
||||
matched: set[str] = set()
|
||||
|
||||
# PCN: payer_claim_control_number ↔ patient_control_number (stripped).
|
||||
if (getattr(remit, "payer_claim_control_number", "") or "").strip() == \
|
||||
(getattr(claim, "patient_control_number", "") or "").strip():
|
||||
matched.add("pcn")
|
||||
|
||||
# Charge: prefer the planned attribute names, fall back to real ORM columns,
|
||||
# then to raw_json (where the 835 parser stores CLP03). Explicit ``is None``
|
||||
# checks avoid the ``Decimal("0")`` truthiness footgun — ``Decimal("0")``
|
||||
# is truthy in a boolean context but ``or`` would still let it through;
|
||||
# the more important property is that we don't accidentally replace a real
|
||||
# value with a default.
|
||||
_remit_charge = getattr(remit, "total_charge_amount", None)
|
||||
if _remit_charge is None:
|
||||
_remit_charge = getattr(remit, "total_charge", None)
|
||||
if _remit_charge is None:
|
||||
_remit_charge = (getattr(remit, "raw_json", None) or {}).get("total_charge_amount")
|
||||
|
||||
_claim_charge = getattr(claim, "total_charge", None)
|
||||
if _claim_charge is None:
|
||||
_claim_charge = getattr(claim, "charge_amount", None)
|
||||
if _claim_charge is None:
|
||||
_claim_charge = (getattr(claim, "raw_json", None) or {}).get("total_charge_amount")
|
||||
|
||||
if _remit_charge is not None and _claim_charge is not None and \
|
||||
abs(_remit_charge - _claim_charge) < CHARGE_TOLERANCE:
|
||||
matched.add("charge")
|
||||
|
||||
# NPI: typed-column primary path (SP32), raw_json fallback (legacy rows).
|
||||
# Remit reads rendering_provider_npi (single value, D4); claim reads
|
||||
# rendering_provider_npi first, then falls back to provider_npi (billing)
|
||||
# for legacy rows where the 837p parser hadn't yet extracted NM1*82.
|
||||
remit_npi = (
|
||||
(getattr(remit, "rendering_provider_npi", "") or "")
|
||||
or ((getattr(remit, "raw_json", None) or {}).get("service_provider_npi", "") or "")
|
||||
or ((getattr(remit, "raw_json", None) or {}).get("rendering_provider_npi", "") or "")
|
||||
).strip()
|
||||
claim_npi = (
|
||||
(getattr(claim, "rendering_provider_npi", "") or "")
|
||||
or (getattr(claim, "provider_npi", "") or "")
|
||||
or ((getattr(claim, "raw_json", None) or {}).get("rendering_provider_npi", "") or "")
|
||||
).strip()
|
||||
if remit_npi and remit_npi == claim_npi:
|
||||
matched.add("npi")
|
||||
|
||||
return matched
|
||||
|
||||
|
||||
# --- SP31: DB-side fallback candidate matcher -------------------------------
|
||||
|
||||
SCORE_FALLBACK_WINDOW_DAYS = 30
|
||||
|
||||
|
||||
def _score_fallback_candidates(session, remit) -> Optional[tuple[str, set[str], int]]:
|
||||
"""Return (claim_id, keys_matched, candidate_count) for a unique 2-of-3 match.
|
||||
|
||||
Queries a ±SCORE_FALLBACK_WINDOW_DAYS candidate pool filtered by:
|
||||
- claim.matched_remittance_id IS NULL (not yet linked)
|
||||
- claim.state NOT IN terminal set (PAID, DENIED, REJECTED,
|
||||
REVERSED, RECONCILED)
|
||||
- claim.service_date_from within window of remit.service_date
|
||||
|
||||
Returns ``(claim_id, keys_matched, candidate_count)`` when exactly one
|
||||
candidate passes the 2-of-3 rule (via :func:`_content_keys_match`),
|
||||
otherwise ``None``. ``keys_matched`` is the set returned by the helper
|
||||
(``{"pcn", "charge"}`` or any 2-of-3 / 3-of-3 combination) — the
|
||||
ActivityEvent payload records it as the audit trail for *why* the
|
||||
auto-link fired. ``candidate_count`` is the size of the window pool
|
||||
(independent of how many matched) so the operator can see whether
|
||||
the match was 1-of-N or 1-of-1.
|
||||
|
||||
Ambiguous matches (2+ candidates) also return ``None`` — they land
|
||||
in the Inbox Unlinked lane for manual review.
|
||||
|
||||
Note on payer_id: ``Remittance`` has no ``payer_id`` column
|
||||
(``Claim.payer_id`` does, and is the source of truth on the claim
|
||||
side; the 835 parser stores payer_id in ``Remittance.raw_json``).
|
||||
Filtering by payer would require either a raw_json read or a model
|
||||
column add — out of scope for SP31 Task 3. The PCN itself is
|
||||
expected to be unique within a payer's submission, so cross-payer
|
||||
collisions are unlikely. Revisit if production shows otherwise.
|
||||
"""
|
||||
from sqlalchemy import select, and_
|
||||
from datetime import timedelta
|
||||
from cyclone.db import Claim, ClaimState
|
||||
|
||||
TERMINAL = {
|
||||
ClaimState.PAID, ClaimState.DENIED, ClaimState.REJECTED,
|
||||
ClaimState.REVERSED, ClaimState.RECONCILED,
|
||||
}
|
||||
|
||||
if remit.service_date is None:
|
||||
return None # need a date for the window query
|
||||
|
||||
window_lo = remit.service_date - timedelta(days=SCORE_FALLBACK_WINDOW_DAYS)
|
||||
window_hi = remit.service_date + timedelta(days=SCORE_FALLBACK_WINDOW_DAYS)
|
||||
|
||||
candidates = list(session.execute(
|
||||
select(Claim).where(
|
||||
and_(
|
||||
Claim.matched_remittance_id.is_(None),
|
||||
Claim.service_date_from >= window_lo,
|
||||
Claim.service_date_from <= window_hi,
|
||||
Claim.state.notin_([s.value for s in TERMINAL]),
|
||||
)
|
||||
)
|
||||
).scalars().all())
|
||||
|
||||
matched_pairs = [
|
||||
(c.id, _content_keys_match(remit, c))
|
||||
for c in candidates
|
||||
]
|
||||
matched_pairs = [
|
||||
(cid, ks) for cid, ks in matched_pairs if len(ks) >= KEYS_REQUIRED
|
||||
]
|
||||
if len(matched_pairs) == 1:
|
||||
cid, keys_matched = matched_pairs[0]
|
||||
return (cid, keys_matched, len(candidates))
|
||||
return None # 0 matches OR 2+ (ambiguous)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ApplyIntent:
|
||||
"""Result of applying a match. The caller persists this.
|
||||
@@ -224,17 +393,22 @@ class ReconcileResult:
|
||||
def run(session, batch_id: str) -> ReconcileResult:
|
||||
"""Orchestrate reconciliation for an 835 batch.
|
||||
|
||||
Expects Batch + Remittance + CasAdjustment rows already persisted
|
||||
(this is called from store.add() AFTER commit). Loads the new
|
||||
remittances + all currently-unmatched Claims, calls match(), then
|
||||
applies each match's intent inside the caller's session.
|
||||
Expects Batch + Remittance + CasAdjustment rows already added to
|
||||
``session`` (not yet committed). SP27 Task 10 calls this from
|
||||
inside ``CycloneStore.add``'s ingest session, BEFORE
|
||||
``s.commit()`` — so the whole 835 ingest (rows + reconcile) is
|
||||
a single transaction. If anything here raises, the ingest rolls
|
||||
back; no half-reconciled batch ever lands.
|
||||
|
||||
Loads the new remittances + all currently-unmatched Claims, calls
|
||||
match(), then applies each match's intent inside the caller's session.
|
||||
|
||||
Returns counts. Does NOT commit; caller controls transaction.
|
||||
Raises on failure (caller catches and writes activity event).
|
||||
"""
|
||||
from sqlalchemy import select
|
||||
from cyclone.db import (
|
||||
Batch, Claim, Remittance, CasAdjustment, Match, ActivityEvent,
|
||||
Batch, Claim, Remittance, CasAdjustment, Match as MatchORM, ActivityEvent,
|
||||
ClaimState,
|
||||
)
|
||||
|
||||
@@ -252,6 +426,34 @@ def run(session, batch_id: str) -> ReconcileResult:
|
||||
|
||||
matches = match(unmatched_claims, new_remits)
|
||||
|
||||
# SP31: content-keys fallback for remits that PCN-exact couldn't pair.
|
||||
# Operates on the still-unmatched remits so we don't double-match.
|
||||
matched_remit_ids = {m.remittance.id for m in matches}
|
||||
used_claim_ids = {m.claim.id for m in matches}
|
||||
for remit in new_remits:
|
||||
if remit.id in matched_remit_ids:
|
||||
continue
|
||||
if getattr(remit, "claim_id", None) is not None:
|
||||
continue # already linked
|
||||
fallback_result = _score_fallback_candidates(session, remit)
|
||||
if fallback_result is None:
|
||||
continue
|
||||
matched_claim_id, keys_matched, candidate_count = fallback_result
|
||||
# Find the claim object in the unmatched_claims list (it was loaded).
|
||||
target_claim = next(
|
||||
(c for c in unmatched_claims if c.id == matched_claim_id),
|
||||
None,
|
||||
)
|
||||
if target_claim is None or target_claim.id in used_claim_ids:
|
||||
continue
|
||||
matches.append(Match(
|
||||
claim=target_claim, remittance=remit,
|
||||
strategy="score-auto", is_reversal=remit.is_reversal,
|
||||
keys_matched=keys_matched,
|
||||
candidate_count=candidate_count,
|
||||
))
|
||||
used_claim_ids.add(target_claim.id)
|
||||
|
||||
applied = 0
|
||||
skipped = 0
|
||||
for m in matches:
|
||||
@@ -275,7 +477,7 @@ def run(session, batch_id: str) -> ReconcileResult:
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
session.add(Match(
|
||||
session.add(MatchORM(
|
||||
claim_id=m.claim.id, remittance_id=m.remittance.id,
|
||||
strategy=m.strategy,
|
||||
matched_at=datetime.now(timezone.utc),
|
||||
@@ -293,10 +495,25 @@ def run(session, batch_id: str) -> ReconcileResult:
|
||||
m.remittance.claim_id = m.claim.id
|
||||
|
||||
session.add(ActivityEvent(
|
||||
ts=datetime.now(timezone.utc), kind=intent.activity_kind,
|
||||
ts=datetime.now(timezone.utc),
|
||||
kind=("auto_matched_835" if m.strategy == "score-auto" else intent.activity_kind),
|
||||
batch_id=batch_id, claim_id=m.claim.id,
|
||||
remittance_id=m.remittance.id,
|
||||
payload_json={"new_state": m.claim.state.value},
|
||||
payload_json=(
|
||||
{
|
||||
"new_state": m.claim.state.value, "strategy": m.strategy,
|
||||
# SP31 spec D8: the auto_matched_835 payload records
|
||||
# which content-keys the 2-of-3 (or 3-of-3) rule
|
||||
# agreed on, plus how many candidates were in the
|
||||
# window pool. ``sorted()`` converts the set to a
|
||||
# JSON-serializable list and pins a deterministic
|
||||
# order for tests + audit reads.
|
||||
"keys_matched": sorted(m.keys_matched),
|
||||
"candidate_count": m.candidate_count,
|
||||
}
|
||||
if m.strategy == "score-auto"
|
||||
else {"new_state": m.claim.state.value}
|
||||
),
|
||||
))
|
||||
applied += 1
|
||||
|
||||
|
||||
+245
-229
@@ -55,7 +55,10 @@ from cyclone.audit_log import AuditEvent, append_event
|
||||
from cyclone.clearhouse import InboundFile, SftpClient
|
||||
from cyclone.db import ProcessedInboundFile
|
||||
from cyclone.edi.filenames import parse_inbound_filename
|
||||
from cyclone.inbox_state import apply_999_rejections
|
||||
from cyclone.handlers import handle_277ca as _handle_277ca
|
||||
from cyclone.handlers import handle_835 as _handle_835
|
||||
from cyclone.handlers import handle_999 as _handle_999
|
||||
from cyclone.handlers import handle_ta1 as _handle_ta1
|
||||
from cyclone.inbox_state_277ca import apply_277ca_rejections
|
||||
from cyclone.providers import SftpBlock
|
||||
|
||||
@@ -69,6 +72,11 @@ STATUS_SKIPPED = "skipped"
|
||||
STATUS_PENDING = "pending"
|
||||
|
||||
|
||||
# How long reconfigure_scheduler waits for the in-flight tick to
|
||||
# drain before cancelling the old task. Matches Scheduler.stop().
|
||||
_DRAIN_TIMEOUT_SECONDS = 30
|
||||
|
||||
|
||||
# File types we know how to route. The HCPF set is broader (270/271/
|
||||
# 276/277/278/820/834/ENCR) but Cyclone's parser only covers the
|
||||
# four below. Files with unknown types are recorded as ``skipped``
|
||||
@@ -87,6 +95,12 @@ class TickResult:
|
||||
files_skipped: int = 0
|
||||
files_errored: int = 0
|
||||
errors: list[str] = field(default_factory=list)
|
||||
# SP27 Task 9: ``sftp_failed`` is the discriminator that separates
|
||||
# SFTP-side failures (the listing step) from per-file processing
|
||||
# errors. ``tick()`` keys off this flag — NOT ``result.errors`` —
|
||||
# so a single malformed 999 cannot flip the operator's destructive
|
||||
# pill. Set in ``_tick_impl``'s listing-error branches only.
|
||||
sftp_failed: bool = False
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
@@ -99,12 +113,22 @@ class TickResult:
|
||||
"files_skipped": self.files_skipped,
|
||||
"files_errored": self.files_errored,
|
||||
"errors": list(self.errors),
|
||||
"sftp_failed": self.sftp_failed,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class SchedulerStatus:
|
||||
"""Snapshot of the scheduler's runtime state."""
|
||||
"""Snapshot of the scheduler's runtime state.
|
||||
|
||||
SP27 Task 9 added the four SFTP-error fields
|
||||
(``consecutive_failures``, ``last_error``, ``last_error_at``,
|
||||
``last_sftp_attempt_at``) so the operator's UI can tell "all
|
||||
quiet on the MFT front" from "we've been unable to reach the
|
||||
MFT server for 3 hours". The previous 06/25 incident went
|
||||
unnoticed because the status dict had no signal for a hung
|
||||
SFTP poll.
|
||||
"""
|
||||
|
||||
running: bool
|
||||
poll_interval_seconds: int
|
||||
@@ -115,6 +139,13 @@ class SchedulerStatus:
|
||||
total_skipped: int
|
||||
total_errored: int
|
||||
last_tick: Optional[TickResult] = None
|
||||
# SP27 Task 9: SFTP-error surfacing. The UI flips to a destructive
|
||||
# pill when ``consecutive_failures >= 3`` (``CYCLONE_SCHEDULER_FAIL_THRESHOLD``
|
||||
# in the operator pill config; not enforced server-side).
|
||||
consecutive_failures: int = 0
|
||||
last_error: Optional[str] = None
|
||||
last_error_at: Optional[datetime] = None
|
||||
last_sftp_attempt_at: Optional[datetime] = None
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
@@ -129,6 +160,15 @@ class SchedulerStatus:
|
||||
"total_skipped": self.total_skipped,
|
||||
"total_errored": self.total_errored,
|
||||
"last_tick": self.last_tick.as_dict() if self.last_tick else None,
|
||||
"consecutive_failures": self.consecutive_failures,
|
||||
"last_error": self.last_error,
|
||||
"last_error_at": (
|
||||
self.last_error_at.isoformat() if self.last_error_at else None
|
||||
),
|
||||
"last_sftp_attempt_at": (
|
||||
self.last_sftp_attempt_at.isoformat()
|
||||
if self.last_sftp_attempt_at else None
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@@ -136,179 +176,17 @@ class SchedulerStatus:
|
||||
# Per-file-type handlers. Each returns (parser_name, claim_count) and
|
||||
# persists its own DB rows. The scheduler records the outcome.
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# 999 (Task 2), TA1 (Task 3), 277CA (Task 4), and 835 (Task 5) now
|
||||
# live in ``cyclone.handlers``. The HANDLERS dict literal below
|
||||
# references the alias imports so the wiring stays identical.
|
||||
|
||||
|
||||
def _handle_999(text: str, source_file: str) -> tuple[str, int]:
|
||||
"""Parse a 999, apply rejections, persist ack row. Returns (parser, count)."""
|
||||
from cyclone.parsers.parse_999 import parse_999_text
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
|
||||
try:
|
||||
result = parse_999_text(text, input_file=source_file)
|
||||
except CycloneParseError as exc:
|
||||
raise ValueError(f"999 parse error: {exc}") from exc
|
||||
|
||||
received, accepted, rejected, ack_code = _ack_count_summary(result)
|
||||
icn = result.envelope.control_number
|
||||
synthetic_id = _ack_synthetic_source_batch_id(icn)
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
def _lookup(pcn: str):
|
||||
return (
|
||||
session.query(db.Claim)
|
||||
.filter_by(patient_control_number=pcn)
|
||||
.first()
|
||||
)
|
||||
rejection_result = apply_999_rejections(
|
||||
session, result, claim_lookup=_lookup,
|
||||
)
|
||||
if rejection_result.matched:
|
||||
for cid in rejection_result.matched:
|
||||
append_event(session, AuditEvent(
|
||||
event_type="claim.rejected",
|
||||
entity_type="claim",
|
||||
entity_id=cid,
|
||||
payload={"source_batch_id": synthetic_id},
|
||||
actor="999-parser-scheduler",
|
||||
))
|
||||
row = cycl_store.add_ack(
|
||||
source_batch_id=synthetic_id,
|
||||
accepted_count=accepted,
|
||||
rejected_count=rejected,
|
||||
received_count=received,
|
||||
ack_code=ack_code,
|
||||
raw_json=json.loads(result.model_dump_json()),
|
||||
)
|
||||
session.commit()
|
||||
return "parse_999", received
|
||||
|
||||
|
||||
def _handle_835(text: str, source_file: str) -> tuple[str, int]:
|
||||
"""Parse an 835, run validation, persist batch + remittances."""
|
||||
import uuid
|
||||
from cyclone.parsers.parse_835 import parse as parse_835
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
from cyclone.parsers.validator_835 import validate as validate_835
|
||||
from cyclone.payers import PAYER_FACTORIES_835
|
||||
from cyclone.store import BatchRecord
|
||||
|
||||
config = PAYER_FACTORIES_835["co_medicaid_835"]()
|
||||
try:
|
||||
result = parse_835(text, config, input_file=source_file)
|
||||
except CycloneParseError as exc:
|
||||
raise ValueError(f"835 parse error: {exc}") from exc
|
||||
|
||||
# Validation report (mirrors the API endpoint).
|
||||
report = validate_835(result, config)
|
||||
n = len(result.claims)
|
||||
if report.passed:
|
||||
passed, failed, failed_claim_ids = n, 0, []
|
||||
else:
|
||||
passed, failed, failed_claim_ids = 0, n, [
|
||||
c.payer_claim_control_number for c in result.claims
|
||||
]
|
||||
result = result.model_copy(update={
|
||||
"validation": report,
|
||||
"summary": result.summary.model_copy(update={
|
||||
"passed": passed,
|
||||
"failed": failed,
|
||||
"failed_claim_ids": failed_claim_ids,
|
||||
}),
|
||||
})
|
||||
|
||||
rec = BatchRecord(
|
||||
id=uuid.uuid4().hex,
|
||||
kind="835",
|
||||
input_filename=source_file,
|
||||
parsed_at=datetime.now(timezone.utc),
|
||||
result=result,
|
||||
)
|
||||
cycl_store.add(rec)
|
||||
return "parse_835", len(result.claims)
|
||||
|
||||
|
||||
def _handle_277ca(text: str, source_file: str) -> tuple[str, int]:
|
||||
"""Parse a 277CA, persist ack + stamp payer-rejected claims."""
|
||||
from cyclone.parsers.parse_277ca import parse_277ca_text
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
|
||||
try:
|
||||
result = parse_277ca_text(text, input_file=source_file)
|
||||
except CycloneParseError as exc:
|
||||
raise ValueError(f"277CA parse error: {exc}") from exc
|
||||
|
||||
icn = result.envelope.control_number
|
||||
synthetic_id = _277ca_synthetic_source_batch_id(icn)
|
||||
accepted = sum(
|
||||
1 for s in result.claim_statuses if s.classification == "accepted"
|
||||
)
|
||||
paid = sum(
|
||||
1 for s in result.claim_statuses if s.classification == "paid"
|
||||
)
|
||||
rejected = sum(
|
||||
1 for s in result.claim_statuses if s.classification == "rejected"
|
||||
)
|
||||
pended = sum(
|
||||
1 for s in result.claim_statuses if s.classification == "pended"
|
||||
)
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
row = cycl_store.add_277ca_ack(
|
||||
source_batch_id=synthetic_id,
|
||||
control_number=icn,
|
||||
accepted_count=accepted,
|
||||
rejected_count=rejected,
|
||||
paid_count=paid,
|
||||
pended_count=pended,
|
||||
raw_json=json.loads(result.model_dump_json()),
|
||||
)
|
||||
def _lookup(pcn: str):
|
||||
return (
|
||||
session.query(db.Claim)
|
||||
.filter(db.Claim.patient_control_number == pcn)
|
||||
.first()
|
||||
)
|
||||
apply_result = apply_277ca_rejections(
|
||||
session, result, claim_lookup=_lookup, two77ca_id=row.id,
|
||||
)
|
||||
if apply_result.matched:
|
||||
for cid in apply_result.matched:
|
||||
append_event(session, AuditEvent(
|
||||
event_type="claim.payer_rejected",
|
||||
entity_type="claim",
|
||||
entity_id=cid,
|
||||
payload={"source_batch_id": synthetic_id, "277ca_id": row.id},
|
||||
actor="277ca-parser-scheduler",
|
||||
))
|
||||
session.commit()
|
||||
return "parse_277ca", len(result.claim_statuses)
|
||||
|
||||
|
||||
def _handle_ta1(text: str, source_file: str) -> tuple[str, int]:
|
||||
"""Parse a TA1, persist the interchange ack row."""
|
||||
from cyclone.parsers.parse_ta1 import parse_ta1_text
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
|
||||
try:
|
||||
result = parse_ta1_text(text, input_file=source_file)
|
||||
except CycloneParseError as exc:
|
||||
raise ValueError(f"TA1 parse error: {exc}") from exc
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
cycl_store.add_ta1_ack(
|
||||
source_batch_id=result.source_batch_id,
|
||||
control_number=result.ta1.control_number,
|
||||
interchange_date=result.ta1.interchange_date,
|
||||
interchange_time=result.ta1.interchange_time,
|
||||
ack_code=result.ta1.ack_code,
|
||||
note_code=result.ta1.note_code,
|
||||
ack_generated_date=result.ta1.ack_generated_date,
|
||||
sender_id=result.envelope.sender_id,
|
||||
receiver_id=result.envelope.receiver_id,
|
||||
raw_json=json.loads(result.model_dump_json()),
|
||||
)
|
||||
session.commit()
|
||||
return "parse_ta1", 1
|
||||
# The 277CA handler has moved to ``cyclone.handlers.handle_277ca``
|
||||
# (SP27 Task 4); the inline def was deleted. The HANDLERS dict literal
|
||||
# below still references ``_handle_277ca`` because the import-alias
|
||||
# line at the top of this module binds that name to the new function.
|
||||
# Only the 835 handler stays inline (Task 5).
|
||||
|
||||
|
||||
# Map file_type → handler. Mirrors ROUTED_FILE_TYPES.
|
||||
@@ -323,43 +201,15 @@ HANDLERS: dict[str, Callable[[str, str], tuple[str, int]]] = {
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Light copies of helpers the API endpoints use, so the scheduler can
|
||||
# run without depending on the FastAPI module.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _ack_count_summary(result: Any) -> tuple[int, int, int, str]:
|
||||
"""Return (received, accepted, rejected, ack_code) for a 999.
|
||||
|
||||
Mirrors the logic in ``cyclone.api._ack_count_summary`` but lives
|
||||
here so the scheduler can run without importing the API module.
|
||||
"""
|
||||
if result.functional_group_acks:
|
||||
fg = result.functional_group_acks[0]
|
||||
return (
|
||||
fg.received_count, fg.accepted_count,
|
||||
fg.rejected_count, fg.ack_code,
|
||||
)
|
||||
sets = result.set_responses
|
||||
received = len(sets)
|
||||
accepted = sum(1 for s in sets if s.set_accept_reject.code == "A")
|
||||
rejected = received - accepted
|
||||
if rejected == 0:
|
||||
code = "A"
|
||||
elif accepted == 0:
|
||||
code = "R"
|
||||
else:
|
||||
code = "P"
|
||||
return (received, accepted, rejected, code)
|
||||
|
||||
|
||||
def _ack_synthetic_source_batch_id(interchange_control_number: str) -> str:
|
||||
"""Synthetic batches.id for a received 999 with no source batch."""
|
||||
return f"999-{(interchange_control_number or '').strip() or '000000001'}"
|
||||
|
||||
|
||||
def _277ca_synthetic_source_batch_id(interchange_control_number: str) -> str:
|
||||
"""Synthetic batches.id for a received 277CA with no source batch."""
|
||||
return f"277CA-{(interchange_control_number or '').strip() or '000000001'}"
|
||||
# Run without depending on the FastAPI module.
|
||||
#
|
||||
# Note: ``_277ca_synthetic_source_batch_id`` lived here as a
|
||||
# scheduler-local duplicate of
|
||||
# ``cyclone.handlers._ack_id.two77ca_synthetic_source_batch_id`` until
|
||||
# SP27 Task 4 landed; the inline def has been deleted because
|
||||
# ``handle_277ca`` now imports the canonical copy. The historical
|
||||
# helper was the only remaining inline def in this section — the
|
||||
# surviving inline handler is ``_handle_835`` (lifts in Task 5).
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -413,6 +263,40 @@ class Scheduler:
|
||||
# ticks stack up; the next tick fires only after the previous
|
||||
# one finishes).
|
||||
self._tick_in_progress = False
|
||||
# SP27 Task 9: SFTP-error surfacing. ``_consecutive_failures``
|
||||
# counts back-to-back tick failures (cleared on the next
|
||||
# successful tick). The other three fields are the most
|
||||
# recent failure for the operator's pill — preserved across
|
||||
# successes so the audit trail doesn't disappear the moment
|
||||
# the MFT server recovers.
|
||||
self._consecutive_failures = 0
|
||||
self._last_error: Optional[str] = None
|
||||
self._last_error_at: Optional[datetime] = None
|
||||
self._last_sftp_attempt_at: Optional[datetime] = None
|
||||
|
||||
def _record_sftp_outcome(
|
||||
self, *, success: bool, error: Optional[str] = None,
|
||||
) -> None:
|
||||
"""Update the SFTP-error state after a tick.
|
||||
|
||||
``success=True`` resets the consecutive-failure counter; the
|
||||
last-error fields are preserved as an audit trail (the
|
||||
operator's UI can still surface "last failure: 2 hours ago"
|
||||
after a recovery).
|
||||
|
||||
``success=False`` bumps the counter and records the error
|
||||
message + timestamp. ``last_sftp_attempt_at`` is bumped in
|
||||
both cases so the operator can tell when the scheduler last
|
||||
*tried* to reach the MFT (not just when it last failed).
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
self._last_sftp_attempt_at = now
|
||||
if success:
|
||||
self._consecutive_failures = 0
|
||||
return
|
||||
self._consecutive_failures += 1
|
||||
self._last_error = error
|
||||
self._last_error_at = now
|
||||
|
||||
# ---- Public API -------------------------------------------------------
|
||||
|
||||
@@ -460,6 +344,10 @@ class Scheduler:
|
||||
total_skipped=self._total_skipped,
|
||||
total_errored=self._total_errored,
|
||||
last_tick=self._last_tick,
|
||||
consecutive_failures=self._consecutive_failures,
|
||||
last_error=self._last_error,
|
||||
last_error_at=self._last_error_at,
|
||||
last_sftp_attempt_at=self._last_sftp_attempt_at,
|
||||
)
|
||||
|
||||
def is_running(self) -> bool:
|
||||
@@ -473,6 +361,13 @@ class Scheduler:
|
||||
SFTP server from a stampede when the operator hits
|
||||
``/api/admin/scheduler/tick`` while a scheduled tick is
|
||||
already running.
|
||||
|
||||
SP27 Task 9: the tick outcome is also recorded on the
|
||||
scheduler's SFTP-error state via
|
||||
:meth:`_record_sftp_outcome`. The discriminator is
|
||||
``TickResult.sftp_failed`` (set by ``_tick_impl``'s
|
||||
listing-error branches only), not ``TickResult.errors``
|
||||
which is also populated by per-file processing errors.
|
||||
"""
|
||||
while self._tick_in_progress:
|
||||
await asyncio.sleep(0.05)
|
||||
@@ -485,6 +380,63 @@ class Scheduler:
|
||||
self._total_processed += result.files_processed
|
||||
self._total_skipped += result.files_skipped
|
||||
self._total_errored += result.files_errored
|
||||
# SFTP-error surfacing: discriminator is ``result.sftp_failed``,
|
||||
# NOT ``result.errors``. ``sftp_failed`` is set by
|
||||
# ``_tick_impl``'s listing-error branches (timeout, connect
|
||||
# refused). Per-file processing errors append to
|
||||
# ``result.errors`` but do NOT set ``sftp_failed`` — a
|
||||
# single malformed 999 in a 5-file batch must not flip
|
||||
# the operator's pill to destructive.
|
||||
if result.sftp_failed:
|
||||
self._record_sftp_outcome(
|
||||
success=False,
|
||||
error="; ".join(result.errors),
|
||||
)
|
||||
else:
|
||||
self._record_sftp_outcome(success=True)
|
||||
return result
|
||||
finally:
|
||||
self._tick_in_progress = False
|
||||
|
||||
async def process_inbound_files(
|
||||
self, files: list[InboundFile],
|
||||
) -> TickResult:
|
||||
"""Run the per-file processing pipeline over a pre-fetched list.
|
||||
|
||||
Unlike :meth:`tick`, this does **not** call SFTP — the caller
|
||||
is expected to have already downloaded (or arranged the local
|
||||
copy of) each ``InboundFile.local_path``. Used by the
|
||||
``/api/admin/scheduler/pull-inbound`` admin endpoint and the
|
||||
``cyclone pull-inbound`` CLI command to process a date-filtered
|
||||
subset of the inbound MFT path without paying the cost of a
|
||||
full poll.
|
||||
|
||||
Honors ``_stop_event`` between files. Concurrent calls are
|
||||
coalesced the same way :meth:`tick` does, so a slow handler
|
||||
can't be stampeded by a parallel admin invocation.
|
||||
|
||||
SP27 Task 9: deliberately does NOT update the SFTP-error
|
||||
state (no listing, no SFTP). The consecutive-failure counter
|
||||
is the signal for scheduled SFTP polling; operator-initiated
|
||||
batches shouldn't be able to flip it.
|
||||
"""
|
||||
while self._tick_in_progress:
|
||||
await asyncio.sleep(0.05)
|
||||
self._tick_in_progress = True
|
||||
try:
|
||||
started = datetime.now(timezone.utc)
|
||||
result = TickResult(started_at=started, files_seen=len(files))
|
||||
for f in files:
|
||||
if self._stop_event.is_set():
|
||||
break
|
||||
await self._handle_one(f, result)
|
||||
result.finished_at = datetime.now(timezone.utc)
|
||||
self._last_tick = result
|
||||
self._last_poll_at = result.finished_at
|
||||
self._poll_count += 1
|
||||
self._total_processed += result.files_processed
|
||||
self._total_skipped += result.files_skipped
|
||||
self._total_errored += result.files_errored
|
||||
return result
|
||||
finally:
|
||||
self._tick_in_progress = False
|
||||
@@ -517,11 +469,26 @@ class Scheduler:
|
||||
started = datetime.now(timezone.utc)
|
||||
result = TickResult(started_at=started)
|
||||
|
||||
# SP27 Task 8: use the async-wrapped SFTP client so a hung
|
||||
# ``listdir_attr`` (the 06/25 silent-hang failure mode) is
|
||||
# bounded by ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` instead of
|
||||
# waiting on paramiko forever. ``asyncio.TimeoutError`` is
|
||||
# handled explicitly so the tick surfaces a clear error in
|
||||
# ``Scheduler.status()`` (Task 9) rather than a generic
|
||||
# ``Exception`` catch-all.
|
||||
client = self._sftp_client_factory(self._sftp_block)
|
||||
try:
|
||||
files = await asyncio.to_thread(self._list_inbound)
|
||||
files = await client.async_list_inbound()
|
||||
except asyncio.TimeoutError:
|
||||
log.exception("SFTP list_inbound timed out")
|
||||
result.errors.append("list_inbound: timeout")
|
||||
result.sftp_failed = True
|
||||
result.finished_at = datetime.now(timezone.utc)
|
||||
return result
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.exception("SFTP list_inbound failed")
|
||||
result.errors.append(f"list_inbound: {exc}")
|
||||
result.sftp_failed = True
|
||||
result.finished_at = datetime.now(timezone.utc)
|
||||
return result
|
||||
|
||||
@@ -534,11 +501,6 @@ class Scheduler:
|
||||
result.finished_at = datetime.now(timezone.utc)
|
||||
return result
|
||||
|
||||
def _list_inbound(self) -> list[InboundFile]:
|
||||
"""Return files in the inbound MFT path. Runs on a thread."""
|
||||
client = self._sftp_client_factory(self._sftp_block)
|
||||
return client.list_inbound()
|
||||
|
||||
async def _handle_one(self, f: InboundFile, result: TickResult) -> None:
|
||||
"""Process one inbound file: skip-if-seen, classify, parse, record."""
|
||||
if await self._already_processed(f.name):
|
||||
@@ -645,20 +607,21 @@ class Scheduler:
|
||||
def _download_and_parse(
|
||||
self, f: InboundFile, file_type: str,
|
||||
) -> tuple[Path, str, int]:
|
||||
"""Download from MFT, run the right handler. Returns (path, parser, count).
|
||||
"""Run the right handler on one inbound file. Returns (path, parser, count).
|
||||
|
||||
Stub mode: ``f.local_path`` already points at the staged file
|
||||
(set by ``SftpClient._list_inbound_stub``). Real mode: the
|
||||
remote name is ``f.name`` and we round-trip through paramiko.
|
||||
Both stub and real modes read from ``f.local_path`` — the
|
||||
inbound file is already on disk:
|
||||
* Stub mode: ``_list_inbound_stub`` points ``local_path`` at
|
||||
the operator-dropped staging file.
|
||||
* Real mode: ``_list_inbound_paramiko`` downloads each
|
||||
``listdir_attr`` entry into the local cache as part of the
|
||||
listing pass. Re-reading from the MFT would require
|
||||
``SftpClient.read_file`` with a full remote path, which the
|
||||
scheduler was passing just ``f.name`` for (i.e. a bare
|
||||
filename at the SFTP root, not the inbound dir). Use the
|
||||
cached bytes instead.
|
||||
"""
|
||||
if self._sftp_block.stub:
|
||||
# In stub mode the InboundFile already has a local_path;
|
||||
# reading the staged bytes directly avoids the stub's
|
||||
# remote-path semantics (which expect a full inbound path).
|
||||
content = f.local_path.read_bytes()
|
||||
else:
|
||||
client = self._sftp_client_factory(self._sftp_block)
|
||||
content = client.read_file(f.name)
|
||||
content = f.local_path.read_bytes()
|
||||
text = content.decode("utf-8")
|
||||
handler = HANDLERS[file_type]
|
||||
parser_used, claim_count = handler(text, f.name)
|
||||
@@ -717,4 +680,57 @@ def get_scheduler() -> Scheduler:
|
||||
def reset_scheduler_for_tests() -> None:
|
||||
"""Clear the module-level scheduler. Test-only."""
|
||||
global _scheduler
|
||||
_scheduler = None
|
||||
_scheduler = None
|
||||
|
||||
|
||||
async def reconfigure_scheduler(
|
||||
sftp_block: SftpBlock,
|
||||
*,
|
||||
poll_interval_seconds: int = 60,
|
||||
sftp_block_name: str = "default",
|
||||
) -> Scheduler:
|
||||
"""Replace the singleton scheduler; restart if it was running. SP25.
|
||||
|
||||
Called from ``PATCH /api/clearhouse`` so the next scheduler tick
|
||||
uses the new SftpBlock without a process restart.
|
||||
|
||||
Lifecycle:
|
||||
* If the previous scheduler is running, ``stop()`` it (waits
|
||||
up to ``_DRAIN_TIMEOUT_SECONDS`` for the current tick to
|
||||
finish — matches the existing ``Scheduler.stop()`` timeout).
|
||||
* Replace the module-level ``_scheduler`` singleton with a
|
||||
fresh ``Scheduler`` against the new block.
|
||||
* If the previous one was running, ``start()`` the new one.
|
||||
|
||||
Threading: same constraint as ``configure_scheduler`` — must be
|
||||
called from the FastAPI event loop, not a worker thread.
|
||||
"""
|
||||
global _scheduler
|
||||
was_running = False
|
||||
if _scheduler is not None:
|
||||
was_running = _scheduler.is_running()
|
||||
if was_running:
|
||||
# Drain the in-flight tick. Scheduler.stop() already
|
||||
# applies the _DRAIN_TIMEOUT_SECONDS timeout internally;
|
||||
# we just await it.
|
||||
await _scheduler.stop()
|
||||
poll = int(
|
||||
os.environ.get("CYCLONE_SCHEDULER_POLL_SECONDS", poll_interval_seconds),
|
||||
)
|
||||
_scheduler = Scheduler(
|
||||
sftp_block,
|
||||
poll_interval_seconds=poll,
|
||||
sftp_block_name=sftp_block_name,
|
||||
)
|
||||
if was_running:
|
||||
await _scheduler.start()
|
||||
log.info(
|
||||
"Scheduler reconfigured",
|
||||
extra={
|
||||
"was_running": was_running,
|
||||
"sftp_block": sftp_block_name,
|
||||
"host": sftp_block.host,
|
||||
"stub": sftp_block.stub,
|
||||
},
|
||||
)
|
||||
return _scheduler
|
||||
@@ -14,11 +14,36 @@ Setup (one-time, by the operator):
|
||||
|
||||
Verification:
|
||||
security find-generic-password -s cyclone -a sftp.gainwell.password -w
|
||||
|
||||
SP25 + SP26: the lookup now resolves the secret in four tiers, in
|
||||
this order:
|
||||
|
||||
1. ``<env_name>_FILE`` env var set — highest priority. Reads the
|
||||
file at that path, strips whitespace, returns the contents.
|
||||
This is the standard Docker-secrets pattern: mount a file at
|
||||
``/run/secrets/<name>`` and point the env var at it. An
|
||||
operator who explicitly sets ``_FILE`` is making a positive
|
||||
statement about where the secret lives, so a missing file
|
||||
surfaces as a ``RuntimeError`` rather than silently falling
|
||||
through. Empty string is treated as unset.
|
||||
2. Plain env var named exactly ``<env_name>`` — second priority.
|
||||
Lets a Linux server or Docker container pass the MFT password
|
||||
via ``CYCLONE_SFTP_PASSWORD=...`` without touching the Keychain
|
||||
or a file mount. Trailing/leading whitespace (including the
|
||||
``\\n`` that .env files often leave at EOF) is stripped; an
|
||||
empty value is treated as absent.
|
||||
3. macOS Keychain via ``keyring`` — third priority. Kept as a
|
||||
fallback so a macOS workstation that prefers
|
||||
``security add-generic-password`` works without env-var exports.
|
||||
4. ``None`` — caller decides what to do (``SftpClient._connect``
|
||||
raises a precise ``RuntimeError`` on real-mode auth).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
@@ -36,24 +61,68 @@ except ImportError:
|
||||
_HAS_KEYRING = False
|
||||
|
||||
|
||||
def _env_var_for(name: str) -> str:
|
||||
"""Resolve the operator-facing env-var name for a secret.
|
||||
|
||||
Returns the entry from ``_ENV_NAME_FOR`` if present, otherwise
|
||||
returns ``name`` verbatim.
|
||||
"""
|
||||
return _ENV_NAME_FOR.get(name, name)
|
||||
|
||||
|
||||
def get_secret(name: str) -> Optional[str]:
|
||||
"""Fetch a secret from macOS Keychain by name.
|
||||
"""Fetch a secret by name. Four-tier lookup: _FILE, env var, Keychain, None.
|
||||
|
||||
Args:
|
||||
name: The Keychain account (e.g. "sftp.gainwell.password").
|
||||
name: The secret name. Matches the Keychain account name
|
||||
(e.g. ``"sftp.gainwell.password"``). The operator-facing
|
||||
env-var form is mapped via the ``_ENV_NAME_FOR`` table at
|
||||
the bottom of this module.
|
||||
|
||||
Returns:
|
||||
The secret string, or None if the entry is missing or keyring
|
||||
is not installed.
|
||||
The secret string (stripped), or ``None`` if no tier produced
|
||||
a value.
|
||||
|
||||
Raises:
|
||||
RuntimeError: if ``<env_name>_FILE`` is set but the file at
|
||||
that path is missing or unreadable. The operator made a
|
||||
positive statement about where the secret lives; silent
|
||||
fall-through would mask a misconfiguration.
|
||||
"""
|
||||
if not _HAS_KEYRING:
|
||||
log.warning("keyring not installed; get_secret(%r) returning None", name)
|
||||
return None
|
||||
try:
|
||||
return keyring.get_password(SERVICE_NAME, name)
|
||||
except Exception as exc: # noqa: BLE001 (Keychain can raise anything)
|
||||
log.warning("Keychain get_secret(%r) failed: %s", name, exc)
|
||||
return None
|
||||
env_name = _env_var_for(name)
|
||||
file_env = env_name + "_FILE"
|
||||
file_path_raw = os.environ.get(file_env)
|
||||
if file_path_raw:
|
||||
# An empty string is treated as unset (matches the SP25 rule
|
||||
# for plain env vars).
|
||||
stripped_path = file_path_raw.strip()
|
||||
if stripped_path:
|
||||
try:
|
||||
value = Path(stripped_path).read_text().strip()
|
||||
except OSError as exc:
|
||||
raise RuntimeError(
|
||||
f"failed to read {file_env}={stripped_path}: {exc}"
|
||||
) from exc
|
||||
if value:
|
||||
log.debug("Secret %r resolved from file %r", name, stripped_path)
|
||||
return value
|
||||
|
||||
raw = os.environ.get(env_name)
|
||||
if raw is not None:
|
||||
stripped = raw.strip()
|
||||
if stripped:
|
||||
log.debug("Secret %r resolved from env var %r", name, env_name)
|
||||
return stripped
|
||||
# Empty env var — treat as absent and fall through.
|
||||
|
||||
if _HAS_KEYRING:
|
||||
try:
|
||||
value = keyring.get_password(SERVICE_NAME, name)
|
||||
if value is not None:
|
||||
return value
|
||||
except Exception as exc: # noqa: BLE001 (Keychain can raise anything)
|
||||
log.warning("Keychain get_secret(%r) failed: %s", name, exc)
|
||||
return None
|
||||
|
||||
|
||||
def set_secret(name: str, value: str) -> bool:
|
||||
@@ -77,3 +146,13 @@ def has_keyring() -> bool:
|
||||
"""True if the ``keyring`` library is importable (regardless of whether
|
||||
the Keychain entry actually exists)."""
|
||||
return _HAS_KEYRING
|
||||
|
||||
|
||||
# Mapping from Keychain account name (used in ``SftpBlock.auth``) to
|
||||
# the operator-facing env var. Keeping this table here means callers
|
||||
# don't have to remember the difference between
|
||||
# ``sftp.gainwell.password`` (Keychain account) and
|
||||
# ``CYCLONE_SFTP_PASSWORD`` (env var).
|
||||
_ENV_NAME_FOR: dict[str, str] = {
|
||||
"sftp.gainwell.password": "CYCLONE_SFTP_PASSWORD",
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,377 @@
|
||||
"""SQLAlchemy-backed batch store for parsed X12 files.
|
||||
|
||||
The package exposes a single ``CycloneStore`` class and a module-level
|
||||
singleton (``store``). All persistence flows through SQLAlchemy
|
||||
sessions via ``db.SessionLocal()()`` (the double-paren function-style
|
||||
accessor).
|
||||
|
||||
This ``__init__.py`` is the facade — it re-exports every public name
|
||||
plus the 3 private helpers imported by tests (``_claim_status_from_validation``,
|
||||
``_persist_835_remit``, ``_remittance_835_row``) from the sibling modules:
|
||||
|
||||
exceptions AlreadyMatchedError, NotMatchedError, InvalidStateError
|
||||
records BatchKind, BatchRecord, BatchRecord837, BatchRecord835
|
||||
orm_builders ORM row builders + the 3 private-helper re-exports
|
||||
ui UI serializers (to_ui_*)
|
||||
write add_record + private event/reconcile helpers
|
||||
batches Batch reads + _BatchesShim (test-cleanup shim)
|
||||
claim_detail Single-claim reads + matched-pair drift audit
|
||||
kpis Dashboard aggregation
|
||||
acks 999 / TA1 / 277CA ACK persistence
|
||||
backups Backup-pending marker inserts
|
||||
inbox Manual match/unmatch + lane listing
|
||||
providers Provider/payer/clearhouse config
|
||||
|
||||
The ``CycloneStore`` class below keeps its current method signatures as
|
||||
thin delegations for back-compat with existing callers and tests.
|
||||
|
||||
Public API (preserved verbatim):
|
||||
CycloneStore, store, BatchRecord, BatchRecord837, BatchRecord835,
|
||||
BatchKind, AlreadyMatchedError, NotMatchedError, InvalidStateError,
|
||||
utcnow, dashboard_kpis, check_matched_pair_drift
|
||||
|
||||
Backward-compat shims for tests:
|
||||
``_lock`` — a threading.RLock used by 7 test files for cleanup.
|
||||
``_batches.clear()`` — wipes all rows from the DB tables; the
|
||||
``_BatchesShim`` class lives in ``batches.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
def utcnow() -> datetime:
|
||||
"""tz-aware UTC `datetime` (replaces the old `utcnow_iso` string helper)."""
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
from . import write
|
||||
from .batches import (
|
||||
_BatchesShim,
|
||||
_row_to_record,
|
||||
all_batches,
|
||||
get_batch,
|
||||
get_record,
|
||||
list_batches,
|
||||
load_two_for_diff,
|
||||
)
|
||||
from .claim_detail import (
|
||||
check_matched_pair_drift,
|
||||
count_claims,
|
||||
count_remittances,
|
||||
distinct_providers,
|
||||
get_claim_detail,
|
||||
get_remittance,
|
||||
iter_claims,
|
||||
iter_remittances,
|
||||
recent_activity,
|
||||
summarize_remittances,
|
||||
)
|
||||
from .acks import (
|
||||
add_277ca_ack,
|
||||
add_999_ack,
|
||||
add_ta1_ack,
|
||||
get_277ca_ack,
|
||||
get_ack,
|
||||
get_ta1_ack,
|
||||
list_277ca_acks,
|
||||
list_acks,
|
||||
list_ta1_acks,
|
||||
)
|
||||
from .claim_acks import (
|
||||
add_claim_ack as _add_claim_ack,
|
||||
batch_envelope_index as _batch_envelope_index,
|
||||
find_ack_orphans as _find_ack_orphans,
|
||||
list_acks_for_claim as _list_acks_for_claim,
|
||||
list_claims_for_ack as _list_claims_for_ack,
|
||||
remove_claim_ack as _remove_claim_ack,
|
||||
)
|
||||
from .backups import add_backup_pending
|
||||
from .exceptions import AlreadyMatchedError, InvalidStateError, NotMatchedError
|
||||
from .inbox import list_unmatched, manual_match, manual_unmatch
|
||||
from .kpis import dashboard_kpis
|
||||
from .orm_builders import (
|
||||
_claim_status_from_validation,
|
||||
_persist_835_remit,
|
||||
_remittance_835_row,
|
||||
)
|
||||
from .providers import (
|
||||
ensure_clearhouse_seeded,
|
||||
get_clearhouse,
|
||||
get_payer_config,
|
||||
get_provider,
|
||||
list_payers,
|
||||
list_providers,
|
||||
update_clearhouse,
|
||||
upsert_provider,
|
||||
)
|
||||
from .records import BatchKind, BatchRecord, BatchRecord837, BatchRecord835
|
||||
from .ui import (
|
||||
to_ui_ack,
|
||||
to_ui_claim_ack,
|
||||
to_ui_claim_detail,
|
||||
to_ui_claim_from_orm,
|
||||
to_ui_provider,
|
||||
to_ui_remittance_from_orm,
|
||||
to_ui_remittance_with_adjustments,
|
||||
to_ui_ta1_ack,
|
||||
to_ui_two77ca_ack,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# UI mappers: ORM rows → simpler UI types.
|
||||
# ---------------------------------------------------------------------------
|
||||
# Backward-compat shim: tests called ``_batches.clear()`` on the in-memory
|
||||
# store. The DB-backed store doesn't have an in-memory list, so we expose
|
||||
# a tiny shim object whose ``.clear()`` wipes the DB. The class itself
|
||||
# lives in ``batches.py`` (imported above as ``_BatchesShim``); the
|
||||
# ``CycloneStore.__init__`` below instantiates that imported class.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CycloneStore: the SQLAlchemy-backed facade.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class CycloneStore:
|
||||
"""SQLAlchemy-backed facade over the parsed X12 store.
|
||||
|
||||
Each public method opens a short-lived session via
|
||||
``db.SessionLocal()()`` so callers don't have to manage session
|
||||
lifecycles. Concurrency is handled by the SQLAlchemy engine; the
|
||||
``_lock`` attribute is a no-op ``RLock`` retained for backward
|
||||
compatibility with code that wrapped cleanup in a lock context.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._lock = threading.RLock()
|
||||
self._batches = _BatchesShim()
|
||||
|
||||
# -- write path -----------------------------------------------------
|
||||
|
||||
def add(self, record: BatchRecord, *, event_bus=None):
|
||||
"""Persist a parsed batch (837P or 835)."""
|
||||
return write.add_record(record, event_bus=event_bus)
|
||||
|
||||
def _publish_events_sync(self, event_bus, record, claim_ids, remit_ids):
|
||||
return write.publish_events_sync(event_bus, record, claim_ids, remit_ids)
|
||||
|
||||
@staticmethod
|
||||
def _sync_publish(event_bus, kind: str, payload: dict) -> None:
|
||||
return write._sync_publish(event_bus, kind, payload)
|
||||
|
||||
# -- read path ------------------------------------------------------
|
||||
|
||||
def get_batch(self, batch_id: str) -> dict | None:
|
||||
return get_batch(batch_id)
|
||||
|
||||
def get(self, batch_id: str) -> BatchRecord | None:
|
||||
return get_record(batch_id)
|
||||
|
||||
def list(self, *, limit: int = 100) -> list[BatchRecord]:
|
||||
return list_batches(limit=limit)
|
||||
|
||||
def get_remittance(self, remittance_id: str) -> dict | None:
|
||||
return get_remittance(remittance_id)
|
||||
|
||||
def get_claim_detail(self, claim_id: str) -> dict | None:
|
||||
return get_claim_detail(claim_id)
|
||||
|
||||
def all(self) -> list[BatchRecord]:
|
||||
return all_batches()
|
||||
|
||||
def load_two_for_diff(
|
||||
self,
|
||||
a_id: str,
|
||||
b_id: str,
|
||||
) -> tuple[BatchRecord, BatchRecord]:
|
||||
return load_two_for_diff(a_id, b_id)
|
||||
|
||||
def iter_claims(self, **kwargs):
|
||||
return iter_claims(**kwargs)
|
||||
|
||||
def iter_remittances(self, **kwargs):
|
||||
return iter_remittances(**kwargs)
|
||||
|
||||
# -- count helpers (SP27 Task 13b) ---------------------------------
|
||||
#
|
||||
# The list endpoints (``/api/claims`` and ``/api/remittances``)
|
||||
# previously computed ``total`` by calling ``iter_*`` with default
|
||||
# ``limit=100`` and taking ``len(...)``. With 60k+ claims and 800+
|
||||
# remits in production, the reported total silently capped at 100
|
||||
# even when the UI rendered a "100" KPI tile and a 100-row table —
|
||||
# the page looked complete but the population was 600× larger. These
|
||||
# helpers reuse the iter's filter pipeline with an effectively
|
||||
# unbounded ``limit`` so the count reflects the true DB population,
|
||||
# not a 100-row sample. Mirrors Dashboard's "server-aggregated
|
||||
# counts" fix from commit 59c3275.
|
||||
def count_claims(
|
||||
self,
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
status: str | None = None,
|
||||
provider_npi: str | None = None,
|
||||
payer: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
) -> int:
|
||||
"""Count claims that would be returned by ``iter_claims``."""
|
||||
return count_claims(
|
||||
batch_id=batch_id, status=status, provider_npi=provider_npi,
|
||||
payer=payer, date_from=date_from, date_to=date_to,
|
||||
)
|
||||
|
||||
def count_remittances(
|
||||
self,
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
payer: str | None = None,
|
||||
claim_id: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
) -> int:
|
||||
"""Count remittances that would be returned by ``iter_remittances``."""
|
||||
return count_remittances(
|
||||
batch_id=batch_id, payer=payer, claim_id=claim_id,
|
||||
date_from=date_from, date_to=date_to,
|
||||
)
|
||||
|
||||
def summarize_remittances(
|
||||
self,
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
payer: str | None = None,
|
||||
claim_id: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
) -> dict:
|
||||
"""Return ``{count, total_paid, total_adjustments}`` summed over
|
||||
the remittance population that ``iter_remittances`` would return.
|
||||
"""
|
||||
return summarize_remittances(
|
||||
batch_id=batch_id, payer=payer, claim_id=claim_id,
|
||||
date_from=date_from, date_to=date_to,
|
||||
)
|
||||
|
||||
def distinct_providers(self) -> list[dict]:
|
||||
return distinct_providers()
|
||||
|
||||
def recent_activity(self, *, limit: int = 200) -> list[dict]:
|
||||
return recent_activity(limit=limit)
|
||||
|
||||
# -- 999 ACKs (SP3 P3 T13) -------------------------------------------
|
||||
|
||||
def add_ack(self, **kwargs):
|
||||
return add_999_ack(**kwargs)
|
||||
|
||||
def list_acks(self):
|
||||
return list_acks()
|
||||
|
||||
def get_ack(self, ack_id):
|
||||
return get_ack(ack_id)
|
||||
|
||||
# -- TA1 (Interchange Acknowledgment) -------------------------------
|
||||
|
||||
def add_ta1_ack(self, **kwargs):
|
||||
return add_ta1_ack(**kwargs)
|
||||
|
||||
def list_ta1_acks(self):
|
||||
return list_ta1_acks()
|
||||
|
||||
def get_ta1_ack(self, ack_id):
|
||||
return get_ta1_ack(ack_id)
|
||||
|
||||
# -- 277CA (SP10) --------------------------------------------------
|
||||
|
||||
def add_277ca_ack(self, **kwargs):
|
||||
return add_277ca_ack(**kwargs)
|
||||
|
||||
def list_277ca_acks(self):
|
||||
return list_277ca_acks()
|
||||
|
||||
def get_277ca_ack(self, ack_id):
|
||||
return get_277ca_ack(ack_id)
|
||||
|
||||
# -- SP28: claim↔ack auto-link join table --------------------------
|
||||
|
||||
def add_claim_ack(self, **kwargs):
|
||||
"""Persist one claim_acks link row.
|
||||
|
||||
Mirrors the publish-from-store contract used by the ACK paths:
|
||||
when ``event_bus`` is passed, the matching ``claim_ack_written``
|
||||
event fires after commit so live-tail subscribers on both the
|
||||
claim and the ack side see the row immediately.
|
||||
"""
|
||||
return _add_claim_ack(**kwargs)
|
||||
|
||||
def list_acks_for_claim(self, claim_id):
|
||||
"""Return every ClaimAck row for one claim (newest first)."""
|
||||
return _list_acks_for_claim(claim_id)
|
||||
|
||||
def list_claims_for_ack(self, kind, ack_id):
|
||||
"""Return every ClaimAck row for one ack."""
|
||||
return _list_claims_for_ack(kind, ack_id)
|
||||
|
||||
def find_ack_orphans(self, kind):
|
||||
"""Return acks with no resolvable link (Inbox ack-orphans lane)."""
|
||||
return _find_ack_orphans(kind)
|
||||
|
||||
def remove_claim_ack(self, link_id, *, event_bus=None):
|
||||
"""Unlink one row. Publishes ``claim_ack_dropped`` on the bus."""
|
||||
return _remove_claim_ack(link_id, event_bus=event_bus)
|
||||
|
||||
def batch_envelope_index(self):
|
||||
"""Return a {envelope.control_number: batch.id} map for D10 Pass 1."""
|
||||
return _batch_envelope_index()
|
||||
|
||||
# -- SP17: encrypted DB backups -------------------------------------
|
||||
|
||||
def add_backup_pending(self, *, filename: str, backup_dir: str):
|
||||
return add_backup_pending(filename=filename, backup_dir=backup_dir)
|
||||
|
||||
# -- manual reconciliation (T12) -----------------------------------
|
||||
|
||||
def list_unmatched(self, *, kind="both"):
|
||||
return list_unmatched(kind=kind)
|
||||
|
||||
def manual_match(self, claim_id, remit_id):
|
||||
return manual_match(claim_id, remit_id)
|
||||
|
||||
def manual_unmatch(self, claim_id):
|
||||
return manual_unmatch(claim_id)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# SP9: providers / payers / payer_configs / clearhouse
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def list_providers(self, *, is_active=True):
|
||||
return list_providers(is_active=is_active)
|
||||
|
||||
def get_provider(self, npi):
|
||||
return get_provider(npi)
|
||||
|
||||
def upsert_provider(self, provider):
|
||||
return upsert_provider(provider)
|
||||
|
||||
def list_payers(self, *, is_active=True):
|
||||
return list_payers(is_active=is_active)
|
||||
|
||||
def get_payer_config(self, payer_id, transaction_type):
|
||||
return get_payer_config(payer_id, transaction_type)
|
||||
|
||||
def get_clearhouse(self):
|
||||
return get_clearhouse()
|
||||
|
||||
def update_clearhouse(self, block):
|
||||
return update_clearhouse(block)
|
||||
|
||||
def ensure_clearhouse_seeded(self):
|
||||
return ensure_clearhouse_seeded()
|
||||
|
||||
|
||||
# Module-level singleton — same import path the old InMemoryStore used.
|
||||
store = CycloneStore()
|
||||
@@ -0,0 +1,242 @@
|
||||
"""ACK persistence — 999 / TA1 / 277CA acknowledgements.
|
||||
|
||||
Each ACK type has its own ORM row (Ack / Ta1Ack / Two77caAck). The
|
||||
write methods are simple inserts; the list/get methods are simple
|
||||
queries. Fail-soft: persistence errors are logged but do not block
|
||||
parsing.
|
||||
|
||||
SP25: every write path publishes one event on the EventBus so the
|
||||
Acks page live-tail can subscribe — mirrors the existing
|
||||
``claim_written`` / ``remittance_written`` / ``activity_recorded``
|
||||
publish pattern (see ``cyclone.store.write``). Publish is best-effort:
|
||||
a failing subscriber MUST NOT roll back the persisted row.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import date
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import Ack
|
||||
|
||||
from . import utcnow
|
||||
from .ui import to_ui_ack, to_ui_ta1_ack, to_ui_two77ca_ack
|
||||
from .write import _sync_publish
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from cyclone.pubsub import EventBus
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _safe_publish(event_bus: "EventBus | None", kind: str, payload: dict) -> None:
|
||||
"""Best-effort publish wrapped in try/except.
|
||||
|
||||
Mirrors ``CycloneStore._publish_events_sync`` — never raises, never
|
||||
rolls back the persisted row. Falls back to skipping silently when
|
||||
the bus doesn't implement the private ``_sync_publish`` interface
|
||||
(e.g. test stubs).
|
||||
"""
|
||||
if event_bus is None:
|
||||
return
|
||||
try:
|
||||
_sync_publish(event_bus, kind, payload)
|
||||
except Exception: # noqa: BLE001
|
||||
log.exception("acks store: %s publish failed", kind)
|
||||
|
||||
|
||||
# -- 999 ACKs (SP3 P3 T13) -------------------------------------------
|
||||
|
||||
def add_999_ack(
|
||||
*,
|
||||
source_batch_id: str,
|
||||
accepted_count: int,
|
||||
rejected_count: int,
|
||||
received_count: int,
|
||||
ack_code: str,
|
||||
raw_json: dict,
|
||||
event_bus: "EventBus | None" = None,
|
||||
) -> db.Ack:
|
||||
"""Persist a 999 ACK row and return it.
|
||||
|
||||
``source_batch_id`` must reference an existing ``batches.id``.
|
||||
For received 999s with no source batch the caller should pass a
|
||||
synthetic id (e.g. ``"999-<interchange_control_number>"``) —
|
||||
see ``/api/parse-999`` for that policy.
|
||||
|
||||
``raw_json`` is the full ``ParseResult999`` model dump; the
|
||||
detail endpoint surfaces it without re-parsing the original
|
||||
X12 text.
|
||||
|
||||
SP25: when ``event_bus`` is provided, publishes one
|
||||
``ack_received`` event (with the full ``to_ui_ack`` row shape)
|
||||
after the row commits so the Acks page live-tail sees new 999s
|
||||
the moment they land.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = Ack(
|
||||
source_batch_id=source_batch_id,
|
||||
accepted_count=accepted_count,
|
||||
rejected_count=rejected_count,
|
||||
received_count=received_count,
|
||||
ack_code=ack_code,
|
||||
parsed_at=utcnow(),
|
||||
raw_json=raw_json,
|
||||
)
|
||||
s.add(row)
|
||||
s.commit()
|
||||
s.refresh(row)
|
||||
row_id = row.id
|
||||
|
||||
if event_bus is not None:
|
||||
with db.SessionLocal()() as s:
|
||||
payload = to_ui_ack(s.get(Ack, row_id))
|
||||
_safe_publish(event_bus, "ack_received", payload)
|
||||
|
||||
return row
|
||||
|
||||
def list_acks() -> list[db.Ack]:
|
||||
"""Return every 999 ACK row, newest first (auto-increment id desc)."""
|
||||
with db.SessionLocal()() as s:
|
||||
return (
|
||||
s.query(Ack)
|
||||
.order_by(Ack.id.desc())
|
||||
.all()
|
||||
)
|
||||
|
||||
def get_ack( ack_id: int) -> db.Ack | None:
|
||||
"""Return a single ACK row by id, or ``None`` if not found."""
|
||||
with db.SessionLocal()() as s:
|
||||
return s.get(Ack, ack_id)
|
||||
|
||||
# -- TA1 (Interchange Acknowledgment) -------------------------------
|
||||
|
||||
def add_ta1_ack(
|
||||
*,
|
||||
source_batch_id: str,
|
||||
control_number: str,
|
||||
interchange_date: date | None,
|
||||
interchange_time: str | None,
|
||||
ack_code: str,
|
||||
note_code: str | None,
|
||||
ack_generated_date: date | None,
|
||||
sender_id: str,
|
||||
receiver_id: str,
|
||||
raw_json: dict,
|
||||
event_bus: "EventBus | None" = None,
|
||||
) -> db.Ta1Ack:
|
||||
"""Persist a TA1 (Interchange Acknowledgment) row and return it.
|
||||
|
||||
Mirrors :meth:`add_999_ack` for the lower-level envelope ack. The
|
||||
flat columns are promoted out of ``raw_json`` so the list
|
||||
endpoint can sort/filter without a JSON parse; the full
|
||||
``ParseResultTa1`` stays in ``raw_json`` for the detail endpoint.
|
||||
|
||||
SP25: when ``event_bus`` is provided, publishes one
|
||||
``ta1_ack_received`` event after the row commits.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = db.Ta1Ack(
|
||||
source_batch_id=source_batch_id,
|
||||
control_number=control_number,
|
||||
interchange_date=interchange_date,
|
||||
interchange_time=interchange_time,
|
||||
ack_code=ack_code,
|
||||
note_code=note_code,
|
||||
ack_generated_date=ack_generated_date,
|
||||
sender_id=sender_id,
|
||||
receiver_id=receiver_id,
|
||||
parsed_at=utcnow(),
|
||||
raw_json=raw_json,
|
||||
)
|
||||
s.add(row)
|
||||
s.commit()
|
||||
s.refresh(row)
|
||||
row_id = row.id
|
||||
|
||||
if event_bus is not None:
|
||||
with db.SessionLocal()() as s:
|
||||
payload = to_ui_ta1_ack(s.get(db.Ta1Ack, row_id))
|
||||
_safe_publish(event_bus, "ta1_ack_received", payload)
|
||||
|
||||
return row
|
||||
|
||||
def list_ta1_acks() -> list[db.Ta1Ack]:
|
||||
"""Return every TA1 ACK row, newest first (auto-increment id desc).
|
||||
|
||||
Mirrors :meth:`list_acks` — the API endpoint slices to its own
|
||||
``limit`` so the ``total`` field reflects the full row count.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
return (
|
||||
s.query(db.Ta1Ack)
|
||||
.order_by(db.Ta1Ack.id.desc())
|
||||
.all()
|
||||
)
|
||||
|
||||
def get_ta1_ack( ack_id: int) -> db.Ta1Ack | None:
|
||||
"""Return a single TA1 ACK row by id, or ``None`` if not found."""
|
||||
with db.SessionLocal()() as s:
|
||||
return s.get(db.Ta1Ack, ack_id)
|
||||
|
||||
# -- 277CA (SP10) --------------------------------------------------
|
||||
|
||||
def add_277ca_ack(
|
||||
*,
|
||||
source_batch_id: str,
|
||||
control_number: str,
|
||||
accepted_count: int,
|
||||
rejected_count: int,
|
||||
paid_count: int,
|
||||
pended_count: int,
|
||||
raw_json: dict,
|
||||
event_bus: "EventBus | None" = None,
|
||||
) -> db.Two77caAck:
|
||||
"""Persist a 277CA (Claim Acknowledgment) row and return it.
|
||||
|
||||
Mirrors :meth:`add_999_ack` but for the claim-level ack. The
|
||||
per-claim status detail stays in ``raw_json``; only the four
|
||||
counts are promoted so the list endpoint stays fast.
|
||||
|
||||
SP25: when ``event_bus`` is provided, publishes one
|
||||
``two77ca_ack_received`` event after the row commits.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = db.Two77caAck(
|
||||
source_batch_id=source_batch_id,
|
||||
control_number=control_number,
|
||||
accepted_count=accepted_count,
|
||||
rejected_count=rejected_count,
|
||||
paid_count=paid_count,
|
||||
pended_count=pended_count,
|
||||
parsed_at=utcnow(),
|
||||
raw_json=raw_json,
|
||||
)
|
||||
s.add(row)
|
||||
s.commit()
|
||||
s.refresh(row)
|
||||
row_id = row.id
|
||||
|
||||
if event_bus is not None:
|
||||
with db.SessionLocal()() as s:
|
||||
payload = to_ui_two77ca_ack(s.get(db.Two77caAck, row_id))
|
||||
_safe_publish(event_bus, "two77ca_ack_received", payload)
|
||||
|
||||
return row
|
||||
|
||||
def list_277ca_acks() -> list[db.Two77caAck]:
|
||||
"""Return every 277CA ACK row, newest first (auto-increment id desc)."""
|
||||
with db.SessionLocal()() as s:
|
||||
return (
|
||||
s.query(db.Two77caAck)
|
||||
.order_by(db.Two77caAck.id.desc())
|
||||
.all()
|
||||
)
|
||||
|
||||
def get_277ca_ack( ack_id: int) -> db.Two77caAck | None:
|
||||
"""Return a single 277CA ACK row by id, or ``None`` if not found."""
|
||||
with db.SessionLocal()() as s:
|
||||
return s.get(db.Two77caAck, ack_id)
|
||||
|
||||
@@ -0,0 +1,340 @@
|
||||
"""SP32 Task 6: backfill rendering/service-provider NPIs from on-disk files.
|
||||
|
||||
The T4 writers populate ``Claim.rendering_provider_npi`` (from NM1*82 in
|
||||
837P Loop 2420A) and ``Remittance.rendering_provider_npi`` (from the
|
||||
NM1*1P service-provider segment in 835 Loop 2100). Rows ingested before
|
||||
T4 was wired (or ingested via a path that bypasses the writer — e.g. an
|
||||
ad-hoc ``store.add`` from a notebook) still have a NULL column.
|
||||
|
||||
This module re-parses on-disk X12 files and patches up those columns on
|
||||
matching rows. It is **idempotent**: rows whose
|
||||
``rendering_provider_npi`` is already non-NULL are left untouched, and
|
||||
re-running the same file twice is a clean no-op.
|
||||
|
||||
Public surface
|
||||
--------------
|
||||
|
||||
* :func:`backfill_rendering_provider_npi` — entry point used by the CLI.
|
||||
* :class:`BackfillSummary` — counts dataclass echoed back as a one-line
|
||||
summary by the CLI.
|
||||
|
||||
Reconcile is run once at the end so the new typed NPI arm (T5) can fire
|
||||
retroactively across the open claim/remit pairs the backfill touched.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.parsers.parse_835 import parse as parse_835
|
||||
from cyclone.parsers.parse_837 import parse as parse_837
|
||||
from cyclone.parsers.payer import PayerConfig, PayerConfig835
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 837P fixture uses NM1*82 → rendering_provider_npi.
|
||||
# 835 fixture uses NM1*1P → service_provider_npi (mapped onto the same
|
||||
# Remittance.rendering_provider_npi column by T4 _remittance_835_row).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class BackfillSummary:
|
||||
"""Counts emitted by ``backfill_rendering_provider_npi``.
|
||||
|
||||
The CLI echoes a one-line summary (`claims_updated=N remits_updated=N
|
||||
…`) at the end of the run.
|
||||
"""
|
||||
|
||||
claims_updated: int = 0
|
||||
remits_updated: int = 0
|
||||
files_processed: int = 0
|
||||
files_skipped: int = 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Parser wrappers — keep exceptions local so one bad file can't abort
|
||||
# the whole backfill run.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _parse_837_file(path: Path) -> tuple[list, Path | None]:
|
||||
"""Re-parse a single 837P file. Returns ``(claims, broken_alias)``.
|
||||
|
||||
On success, ``claims`` is the list of parsed ``ClaimOutput`` rows
|
||||
and ``broken_alias`` is None. On any failure, ``claims`` is empty
|
||||
and ``broken_alias`` is the same ``path`` so the caller can log
|
||||
which file was skipped.
|
||||
"""
|
||||
try:
|
||||
text = path.read_text()
|
||||
except OSError as exc:
|
||||
log.warning("backfill: cannot read %s: %s", path, exc)
|
||||
return [], path
|
||||
try:
|
||||
result = parse_837(text, PayerConfig.co_medicaid(), input_file=str(path))
|
||||
except Exception as exc: # noqa: BLE001 — failure isolated per-file
|
||||
log.warning("backfill: 837 parse failed for %s: %s", path, exc)
|
||||
return [], path
|
||||
return list(result.claims), None
|
||||
|
||||
|
||||
def _parse_835_file(path: Path) -> tuple[list, Path | None]:
|
||||
"""Re-parse a single 835 file. Returns ``(claims, broken_alias)``.
|
||||
|
||||
``claims`` is the list of parsed ``ClaimPayment`` rows.
|
||||
"""
|
||||
try:
|
||||
text = path.read_text()
|
||||
except OSError as exc:
|
||||
log.warning("backfill: cannot read %s: %s", path, exc)
|
||||
return [], path
|
||||
try:
|
||||
result = parse_835(text, PayerConfig835.co_medicaid_835(), input_file=str(path))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.warning("backfill: 835 parse failed for %s: %s", path, exc)
|
||||
return [], path
|
||||
return list(result.claims), None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DB-patching helpers — keep the per-row update logic in one place.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _patch_claim_rendering_npi(parsed_claims: list) -> int:
|
||||
"""Update ``Claim.rendering_provider_npi`` for any parsed 837 claim.
|
||||
|
||||
Only rows whose column is currently NULL are touched. Returns the
|
||||
count of rows updated.
|
||||
"""
|
||||
from cyclone.db import Claim, SessionLocal
|
||||
|
||||
updated = 0
|
||||
with SessionLocal()() as session:
|
||||
for parsed in parsed_claims:
|
||||
npi = getattr(parsed, "rendering_provider_npi", None)
|
||||
if not npi:
|
||||
continue
|
||||
row = session.get(Claim, parsed.claim_id)
|
||||
if row is None:
|
||||
continue
|
||||
if row.rendering_provider_npi is not None:
|
||||
# Already populated (e.g. by a prior backfill run, or by
|
||||
# the T4 writer if the claim was ingested afterwards).
|
||||
continue
|
||||
row.rendering_provider_npi = npi
|
||||
updated += 1
|
||||
if updated:
|
||||
session.commit()
|
||||
return updated
|
||||
|
||||
|
||||
def _patch_remit_rendering_npi(parsed_claims: list) -> int:
|
||||
"""Update ``Remittance.rendering_provider_npi`` for any parsed 835 claim.
|
||||
|
||||
The 835 NM1*1P segment maps onto ``ClaimPayment.service_provider_npi``
|
||||
(which the T4 writer copies into ``Remittance.rendering_provider_npi``).
|
||||
We match on ``payer_claim_control_number``; only NULL columns are
|
||||
touched. Returns the count of rows updated.
|
||||
"""
|
||||
from cyclone.db import Remittance, SessionLocal
|
||||
|
||||
updated = 0
|
||||
with SessionLocal()() as session:
|
||||
for parsed in parsed_claims:
|
||||
npi = getattr(parsed, "service_provider_npi", None)
|
||||
if not npi:
|
||||
continue
|
||||
target_pcn = getattr(parsed, "payer_claim_control_number", None)
|
||||
if not target_pcn:
|
||||
continue
|
||||
row = session.execute(
|
||||
select(Remittance).where(
|
||||
Remittance.payer_claim_control_number == target_pcn,
|
||||
Remittance.rendering_provider_npi.is_(None),
|
||||
)
|
||||
).scalars().first()
|
||||
if row is None:
|
||||
continue
|
||||
row.rendering_provider_npi = npi
|
||||
updated += 1
|
||||
if updated:
|
||||
session.commit()
|
||||
return updated
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Reconcile pass — run once across every 835 batch so the T5 scoring arm
|
||||
# can fire retroactively on pairs the backfill touched.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _run_reconcile_sweep() -> int:
|
||||
"""Run :func:`cyclone.reconcile.run` over every 835 batch. Returns count.
|
||||
|
||||
Failures are isolated per-batch so one bad batch can't abort the
|
||||
sweep — the goal is "best-effort retroactive reconcile".
|
||||
"""
|
||||
from sqlalchemy import select as _select
|
||||
|
||||
from cyclone import reconcile as _reconcile
|
||||
from cyclone.db import Batch, SessionLocal
|
||||
|
||||
completed = 0
|
||||
with SessionLocal()() as session:
|
||||
batch_ids = [
|
||||
row[0] for row in session.execute(
|
||||
_select(Batch.id).where(Batch.kind == "835")
|
||||
).all()
|
||||
]
|
||||
# Each batch gets its own session — ``reconcile.run`` does not commit,
|
||||
# so an exception in one batch must not orphan a half-flushed
|
||||
# transaction.
|
||||
for bid in batch_ids:
|
||||
try:
|
||||
with SessionLocal()() as s:
|
||||
_reconcile.run(s, bid)
|
||||
s.commit()
|
||||
completed += 1
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.warning("backfill: reconcile failed for batch %s: %s", bid, exc)
|
||||
return completed
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public entry point.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# Supported ``--type`` flag values; also ``None`` means auto-detect.
|
||||
TransactionType = str | None
|
||||
|
||||
|
||||
def backfill_rendering_provider_npi(
|
||||
*,
|
||||
files: Iterable[Path] | None = None,
|
||||
input_dir: Path | None = None,
|
||||
transaction_type: TransactionType = None,
|
||||
) -> BackfillSummary:
|
||||
"""Re-parse on-disk 837p + 835 files and populate the typed NPI columns.
|
||||
|
||||
Args:
|
||||
files: explicit list of file paths to re-parse.
|
||||
input_dir: directory to scan for ``*.txt`` / ``*.edi`` files.
|
||||
Files are scanned one level deep.
|
||||
transaction_type: ``"837p"`` or ``"835"``. ``None`` auto-detects
|
||||
by attempting the 837p parser first and falling back to
|
||||
the 835 parser.
|
||||
|
||||
Returns:
|
||||
:class:`BackfillSummary` with populated counts.
|
||||
|
||||
Idempotent: only writes columns that are currently NULL; re-running
|
||||
on the same files is a clean no-op. After patching, runs
|
||||
:func:`cyclone.reconcile.run` over every 835 batch so the T5
|
||||
scoring arm can re-fire on the touched pairs.
|
||||
"""
|
||||
summary = BackfillSummary()
|
||||
|
||||
# 1. Resolve the candidate file set.
|
||||
candidates: list[Path] = []
|
||||
if files:
|
||||
candidates.extend(Path(f) for f in files)
|
||||
if input_dir is not None:
|
||||
for ext in ("*.txt", "*.edi", "*.835", "*.837", "*.x12"):
|
||||
candidates.extend(input_dir.glob(ext))
|
||||
candidates = [p for p in candidates if p.is_file()]
|
||||
|
||||
# 2. Re-parse each file and patch matching rows.
|
||||
for path in candidates:
|
||||
kind = transaction_type or _sniff_kind(path)
|
||||
if kind == "837p":
|
||||
parsed_claims, broken = _parse_837_file(path)
|
||||
if broken is not None:
|
||||
summary.files_skipped += 1
|
||||
continue
|
||||
summary.files_processed += 1
|
||||
summary.claims_updated += _patch_claim_rendering_npi(parsed_claims)
|
||||
elif kind == "835":
|
||||
parsed_claims, broken = _parse_835_file(path)
|
||||
if broken is not None:
|
||||
summary.files_skipped += 1
|
||||
continue
|
||||
summary.files_processed += 1
|
||||
summary.remits_updated += _patch_remit_rendering_npi(parsed_claims)
|
||||
else:
|
||||
# Auto-detect tried both parsers and both failed → skip.
|
||||
summary.files_skipped += 1
|
||||
|
||||
# 3. Reconcile sweep — let the T5 NPI arm fire retroactively.
|
||||
if summary.claims_updated or summary.remits_updated:
|
||||
try:
|
||||
_run_reconcile_sweep()
|
||||
except Exception: # noqa: BLE001 — sweep is best-effort
|
||||
log.exception("backfill: reconcile sweep failed")
|
||||
|
||||
return summary
|
||||
|
||||
|
||||
def _sniff_kind(path: Path) -> TransactionType:
|
||||
"""Best-effort transaction-type sniff (content + filename).
|
||||
|
||||
Returns ``"837p"``, ``"835"``, or ``None`` if both parsers fail.
|
||||
Used only when the caller didn't pin ``transaction_type`` explicitly
|
||||
via the CLI.
|
||||
"""
|
||||
try:
|
||||
text = path.read_text()
|
||||
except OSError:
|
||||
return None
|
||||
# Cheap filename hint.
|
||||
name = path.name.lower()
|
||||
if name.endswith(".835") or "835" in name:
|
||||
return _try_both(path, text, prefer="835")
|
||||
if name.endswith(".837") or "837" in name or "837p" in name:
|
||||
return _try_both(path, text, prefer="837p")
|
||||
# Content: ISA + ST. 837 starts with "ST*837", 835 starts with "ST*835".
|
||||
if "ST*837*" in text:
|
||||
return "837p"
|
||||
if "ST*835*" in text:
|
||||
return "835"
|
||||
# Fallback: try 837p parser, then 835.
|
||||
return _try_both(path, text, prefer="837p")
|
||||
|
||||
|
||||
def _try_both(path: Path, text: str, *, prefer: str) -> TransactionType:
|
||||
"""Try the preferred parser first, then the other; return first winner."""
|
||||
if prefer == "837p":
|
||||
try:
|
||||
parse_837(text, PayerConfig.co_medicaid(), input_file=str(path))
|
||||
return "837p"
|
||||
except Exception:
|
||||
try:
|
||||
parse_835(text, PayerConfig835.co_medicaid_835(), input_file=str(path))
|
||||
return "835"
|
||||
except Exception:
|
||||
return None
|
||||
try:
|
||||
parse_835(text, PayerConfig835.co_medicaid_835(), input_file=str(path))
|
||||
return "835"
|
||||
except Exception:
|
||||
try:
|
||||
parse_837(text, PayerConfig.co_medicaid(), input_file=str(path))
|
||||
return "837p"
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BackfillSummary",
|
||||
"backfill_rendering_provider_npi",
|
||||
]
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Backup-pending marker inserts — SP17 encrypted DB backups.
|
||||
|
||||
``add_backup_pending()`` inserts a ``pending`` row for a backup that is
|
||||
about to start; the BackupService updates ``status`` / ``size_bytes``
|
||||
/ ``db_fingerprint`` / ``table_count`` / ``completed_at`` after the
|
||||
encrypted blob lands on disk.
|
||||
"""
|
||||
|
||||
from cyclone import db
|
||||
|
||||
from . import utcnow
|
||||
|
||||
|
||||
# -- SP17: encrypted DB backups -------------------------------------
|
||||
|
||||
def add_backup_pending(*, filename: str, backup_dir: str) -> db.DbBackup:
|
||||
"""Insert a ``pending`` row for a backup that is about to start.
|
||||
|
||||
The BackupService fills in ``status`` / ``size_bytes`` /
|
||||
``db_fingerprint`` / ``table_count`` / ``completed_at`` after
|
||||
the encrypted blob lands on disk.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = db.DbBackup(
|
||||
filename=filename,
|
||||
backup_dir=backup_dir,
|
||||
size_bytes=0,
|
||||
db_fingerprint=None,
|
||||
table_count=0,
|
||||
created_at=utcnow(),
|
||||
completed_at=None,
|
||||
status="pending",
|
||||
error_message=None,
|
||||
)
|
||||
s.add(row)
|
||||
s.commit()
|
||||
s.refresh(row)
|
||||
return row
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
"""Batch read APIs and the legacy _BatchesShim.
|
||||
|
||||
The ``_BatchesShim`` class is retained for back-compat with the
|
||||
``with store._lock: store._batches.clear()`` test-cleanup idiom used
|
||||
in 7 test files. Its ``clear()`` method wipes all rows from the DB
|
||||
tables so tests that depended on a fresh in-memory list per-test get
|
||||
a fresh DB state per-test.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import timezone
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import (
|
||||
ActivityEvent,
|
||||
Batch,
|
||||
CasAdjustment,
|
||||
Claim,
|
||||
Match,
|
||||
Remittance,
|
||||
)
|
||||
from cyclone.parsers.models import ParseResult
|
||||
from cyclone.parsers.models_835 import ParseResult835
|
||||
|
||||
from .records import BatchRecord, BatchRecord837, BatchRecord835
|
||||
|
||||
|
||||
class _BatchesShim:
|
||||
"""Drop-in replacement for the old in-memory ``_batches`` list.
|
||||
|
||||
``clear()`` removes every row from the DB tables in FK-safe order.
|
||||
Other list operations are not implemented because the only call site
|
||||
is the ``clear()`` inside the test fixtures (``test_api_gets.py`` and
|
||||
``test_api_parse_persists.py``).
|
||||
"""
|
||||
|
||||
def clear(self) -> None: # type: ignore[no-untyped-def]
|
||||
with db.SessionLocal()() as s:
|
||||
s.query(ActivityEvent).delete()
|
||||
s.query(Match).delete()
|
||||
s.query(CasAdjustment).delete()
|
||||
s.query(Remittance).delete()
|
||||
s.query(Claim).delete()
|
||||
s.query(Batch).delete()
|
||||
s.commit()
|
||||
|
||||
|
||||
def _row_to_record(row: Batch) -> BatchRecord:
|
||||
"""Rehydrate a ``BatchRecord`` (837 or 835) from a Batch ORM row.
|
||||
|
||||
The full ``ParseResult`` / ``ParseResult835`` lives in
|
||||
``raw_result_json`` (stashed at insert time). Re-parsing JSON
|
||||
here means callers get the same typed Pydantic object the old
|
||||
in-memory store handed out, so api.py and tests that do
|
||||
``rec.result.claims`` keep working unchanged.
|
||||
|
||||
SQLite drops tz info on round-trip even though the column type
|
||||
is ``DateTime(timezone=True)``. We re-attach UTC so the
|
||||
``BatchRecord`` validator (``parsed_at must be tz-aware``)
|
||||
passes.
|
||||
"""
|
||||
if row.kind == "835":
|
||||
result_cls = ParseResult835
|
||||
else:
|
||||
result_cls = ParseResult
|
||||
payload = row.raw_result_json or {}
|
||||
result = result_cls.model_validate(payload)
|
||||
parsed_at = row.parsed_at
|
||||
if parsed_at is not None and parsed_at.tzinfo is None:
|
||||
parsed_at = parsed_at.replace(tzinfo=timezone.utc)
|
||||
record_cls = BatchRecord835 if row.kind == "835" else BatchRecord837
|
||||
return record_cls(
|
||||
id=row.id,
|
||||
kind=row.kind,
|
||||
input_filename=row.input_filename,
|
||||
parsed_at=parsed_at,
|
||||
result=result,
|
||||
)
|
||||
|
||||
|
||||
def get_batch(batch_id: str) -> dict | None:
|
||||
"""Return a summary dict for ``batch_id`` or ``None`` if missing.
|
||||
|
||||
The dict shape matches what ``/api/batches/{id}`` callers need:
|
||||
``id``, ``kind``, ``input_filename``, ``parsed_at``, and the
|
||||
full ``result`` (raw_result_json) as a dict.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(Batch, batch_id)
|
||||
if row is None:
|
||||
return None
|
||||
return {
|
||||
"id": row.id,
|
||||
"kind": row.kind,
|
||||
"input_filename": row.input_filename,
|
||||
"parsed_at": row.parsed_at,
|
||||
"result": row.raw_result_json,
|
||||
}
|
||||
|
||||
|
||||
def get_record(batch_id: str) -> BatchRecord | None:
|
||||
"""Return the ``BatchRecord`` for ``batch_id`` or ``None``.
|
||||
|
||||
Preserves the in-memory store contract: callers get a Pydantic
|
||||
``BatchRecord`` (subclass ``BatchRecord837`` / ``BatchRecord835``)
|
||||
with ``.id``, ``.kind``, ``.input_filename``, ``.parsed_at``,
|
||||
and ``.result`` (typed ``ParseResult`` / ``ParseResult835``).
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(Batch, batch_id)
|
||||
if row is None:
|
||||
return None
|
||||
return _row_to_record(row)
|
||||
|
||||
|
||||
def list_batches(*, limit: int = 100) -> list[BatchRecord]:
|
||||
"""Return up to ``limit`` ``BatchRecord``s, newest first."""
|
||||
with db.SessionLocal()() as s:
|
||||
rows = (
|
||||
s.query(Batch)
|
||||
.order_by(Batch.parsed_at.desc())
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
return [_row_to_record(r) for r in rows]
|
||||
|
||||
|
||||
def all_batches() -> list[BatchRecord]:
|
||||
"""Return every ``BatchRecord``, oldest first (no pagination)."""
|
||||
with db.SessionLocal()() as s:
|
||||
rows = s.query(Batch).order_by(Batch.parsed_at.asc()).all()
|
||||
return [_row_to_record(r) for r in rows]
|
||||
|
||||
|
||||
def load_two_for_diff(a_id: str, b_id: str) -> tuple[BatchRecord, BatchRecord]:
|
||||
"""Load two batches by id for the side-by-side diff view.
|
||||
|
||||
Returns ``(a, b)`` as ``BatchRecord`` objects. Raises
|
||||
:class:`LookupError` when either id is missing — the API layer
|
||||
catches it and maps it to ``404 Not Found`` (matching the
|
||||
``GET /api/batches/{id}`` contract). The two loads happen in
|
||||
independent sessions so a transient failure on one side can't
|
||||
poison the other.
|
||||
|
||||
Used exclusively by :mod:`cyclone.batch_diff` via the
|
||||
``/api/batch-diff`` endpoint.
|
||||
"""
|
||||
a = get_record(a_id)
|
||||
if a is None:
|
||||
raise LookupError(f"batch {a_id} not found")
|
||||
b = get_record(b_id)
|
||||
if b is None:
|
||||
raise LookupError(f"batch {b_id} not found")
|
||||
return a, b
|
||||
@@ -0,0 +1,360 @@
|
||||
"""SP28: claim_acks persistence + batch envelope index (D10).
|
||||
|
||||
Five methods on top of the new ``ClaimAck`` ORM table:
|
||||
|
||||
* ``add_claim_ack`` — insert one link row (manual or auto) and
|
||||
publish ``claim_ack_written`` on the bus.
|
||||
* ``list_acks_for_claim`` — every link row for one claim (per-claim
|
||||
only; TA1 batch-level rows are filtered out by the API layer).
|
||||
* ``list_claims_for_ack`` — every link row for one ack.
|
||||
* ``find_ack_orphans`` — acks with no resolvable claim (Inbox
|
||||
ack-orphans lane).
|
||||
* ``remove_claim_ack`` — unlink. Publishes ``claim_ack_dropped``.
|
||||
* ``batch_envelope_index`` — D10 in-memory map of
|
||||
``Batch.envelope.control_number → batch.id`` (cheap to rebuild;
|
||||
re-built once per ingest).
|
||||
|
||||
Each mutating method opens its own ``db.SessionLocal()()`` session
|
||||
so callers don't have to manage session lifecycles. Publishes are
|
||||
best-effort (wrapped in :func:`_safe_publish`) so a failing bus
|
||||
subscriber cannot roll back the persisted row.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import (
|
||||
Ack,
|
||||
Batch,
|
||||
Claim,
|
||||
ClaimAck,
|
||||
Ta1Ack,
|
||||
Two77caAck,
|
||||
)
|
||||
|
||||
from . import utcnow
|
||||
from .ui import to_ui_claim_ack
|
||||
from .write import _sync_publish
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from cyclone.pubsub import EventBus
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _safe_publish(event_bus: "EventBus | None", kind: str, payload: dict) -> None:
|
||||
"""Best-effort publish wrapped in try/except.
|
||||
|
||||
Mirrors ``cyclone.store.acks._safe_publish`` — never raises,
|
||||
never rolls back the persisted row.
|
||||
"""
|
||||
if event_bus is None:
|
||||
return
|
||||
try:
|
||||
_sync_publish(event_bus, kind, payload)
|
||||
except Exception: # noqa: BLE001
|
||||
log.exception("claim_acks store: %s publish failed", kind)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# D10 batch envelope index
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def batch_envelope_index() -> dict[str, str]:
|
||||
"""Build a ``{envelope.control_number: batch.id}`` map.
|
||||
|
||||
D10 primary join key (spec §D10). Built once per ingest from
|
||||
every Batch row whose ``raw_result_json`` carries an envelope —
|
||||
cost is O(N batches), currently ~16, so trivial. Kept as a
|
||||
plain dict so callers can ``index.get(scn)`` to resolve.
|
||||
"""
|
||||
out: dict[str, str] = {}
|
||||
with db.SessionLocal()() as s:
|
||||
rows = (
|
||||
s.query(Batch.id, Batch.raw_result_json)
|
||||
.filter(Batch.kind == "837p")
|
||||
.all()
|
||||
)
|
||||
for bid, raw in rows:
|
||||
env = (raw or {}).get("envelope") or {}
|
||||
ctrl = env.get("control_number")
|
||||
if isinstance(ctrl, str) and ctrl:
|
||||
out[ctrl] = bid
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Mutators
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def add_claim_ack(
|
||||
*,
|
||||
claim_id: str | None,
|
||||
batch_id: str | None,
|
||||
ack_id: int,
|
||||
ack_kind: str,
|
||||
ak2_index: int | None = None,
|
||||
set_control_number: str | None = None,
|
||||
set_accept_reject_code: str | None = None,
|
||||
linked_by: str = "auto",
|
||||
event_bus: "EventBus | None" = None,
|
||||
now: datetime | None = None,
|
||||
) -> ClaimAck:
|
||||
"""Persist one link row and return it.
|
||||
|
||||
The DB-level unique index
|
||||
``ux_claim_acks_dedup(claim_id, ack_kind, ack_id, ak2_index)
|
||||
WHERE claim_id IS NOT NULL AND ak2_index IS NOT NULL`` enforces
|
||||
idempotency for the per-AK2 case. The 277CA (ak2_index NULL)
|
||||
and TA1 (claim_id NULL) paths are deduplicated by application
|
||||
code — see :func:`cyclone.claim_acks._link_exists`.
|
||||
|
||||
When ``event_bus`` is provided, publishes one ``claim_ack_written``
|
||||
event (with the full :func:`cyclone.store.ui.to_ui_claim_ack`
|
||||
payload) so the live-tail subscribers on both
|
||||
``/api/claims/{id}/acks/stream`` and
|
||||
``/api/acks/{kind}/{id}/claims/stream`` see new rows the moment
|
||||
they land.
|
||||
"""
|
||||
if ack_kind not in ("999", "277ca", "ta1"):
|
||||
raise ValueError(f"add_claim_ack: unknown ack_kind={ack_kind!r}")
|
||||
if linked_by not in ("auto", "manual"):
|
||||
raise ValueError(f"add_claim_ack: linked_by={linked_by!r}")
|
||||
if claim_id is None and batch_id is None:
|
||||
raise ValueError(
|
||||
"add_claim_ack: at least one of claim_id/batch_id must be set"
|
||||
)
|
||||
ts = now or utcnow()
|
||||
with db.SessionLocal()() as s:
|
||||
row = ClaimAck(
|
||||
claim_id=claim_id,
|
||||
batch_id=batch_id,
|
||||
ack_id=ack_id,
|
||||
ack_kind=ack_kind,
|
||||
ak2_index=ak2_index,
|
||||
set_control_number=set_control_number,
|
||||
set_accept_reject_code=set_accept_reject_code,
|
||||
linked_at=ts,
|
||||
linked_by=linked_by,
|
||||
)
|
||||
s.add(row)
|
||||
s.commit()
|
||||
s.refresh(row)
|
||||
row_id = row.id
|
||||
|
||||
if event_bus is not None:
|
||||
with db.SessionLocal()() as s:
|
||||
payload = to_ui_claim_ack(s.get(ClaimAck, row_id))
|
||||
_safe_publish(event_bus, "claim_ack_written", payload)
|
||||
|
||||
return row
|
||||
|
||||
|
||||
def remove_claim_ack(
|
||||
link_id: int,
|
||||
*,
|
||||
event_bus: "EventBus | None" = None,
|
||||
) -> bool:
|
||||
"""Unlink. Returns ``True`` when a row was deleted.
|
||||
|
||||
Publishes ``claim_ack_dropped`` with ``{"id", "claim_id"}`` so
|
||||
the live-tail subscribers can remove the link from their local
|
||||
store. Does NOT touch ``Claim.state`` — the unlink is purely
|
||||
about the link row, per spec §D6.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(ClaimAck, link_id)
|
||||
if row is None:
|
||||
return False
|
||||
payload = {
|
||||
"id": row.id,
|
||||
"claim_id": row.claim_id,
|
||||
"ack_id": row.ack_id,
|
||||
"ack_kind": row.ack_kind,
|
||||
}
|
||||
s.delete(row)
|
||||
s.commit()
|
||||
|
||||
if event_bus is not None:
|
||||
_safe_publish(event_bus, "claim_ack_dropped", payload)
|
||||
return True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Readers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def list_acks_for_claim(claim_id: str) -> list[ClaimAck]:
|
||||
"""Return every ClaimAck row for one claim (newest first)."""
|
||||
with db.SessionLocal()() as s:
|
||||
return (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.claim_id == claim_id)
|
||||
.order_by(ClaimAck.id.desc())
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
def list_claims_for_ack(kind: str, ack_id: int) -> list[ClaimAck]:
|
||||
"""Return every ClaimAck row for one ack (any kind).
|
||||
|
||||
For 999 / 277CA: returns 0..N rows (one per AK2 / ClaimStatus).
|
||||
For TA1: returns 0..1 row (envelope-level, populated batch_id).
|
||||
"""
|
||||
if kind not in ("999", "277ca", "ta1"):
|
||||
raise ValueError(f"list_claims_for_ack: unknown kind={kind!r}")
|
||||
with db.SessionLocal()() as s:
|
||||
return (
|
||||
s.query(ClaimAck)
|
||||
.filter(
|
||||
ClaimAck.ack_kind == kind,
|
||||
ClaimAck.ack_id == ack_id,
|
||||
)
|
||||
.order_by(ClaimAck.id.asc())
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Orphan detection (Inbox "Ack orphans" lane — spec §D7)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def find_ack_orphans(kind: str) -> list[dict]:
|
||||
"""Return acks with no resolvable Claim row of their own kind.
|
||||
|
||||
Used by the Inbox ack-orphans lane for the operator's manual
|
||||
reconciliation flow. The downstream ``/api/inbox/ack-orphans``
|
||||
endpoint calls this and returns the rendered shape.
|
||||
|
||||
"Orphan" means: for 999 / 277CA, ``ack_kind=kind AND ack_id IN
|
||||
(acks_with_no_claim_acks_link)``. For TA1, "orphan" means the
|
||||
TA1 row exists but no Batch with matching sender/receiver was
|
||||
resolved (so no link row was created).
|
||||
|
||||
Output dict shape (one row per orphan ack, rendered by
|
||||
:func:`to_ui_claim_ack`-style serialization):
|
||||
|
||||
* ``kind`` — "999" / "277ca" / "ta1"
|
||||
* ``ack_id`` — the ack row's id
|
||||
* ``control_number`` — for 999/277CA, envelope.control_number;
|
||||
for TA1, ta1.control_number
|
||||
* ``set_control_numbers`` — empty list when no claims match
|
||||
* ``raw_summary`` — flat copy of the ack's UI shape for the
|
||||
lane-header counts
|
||||
"""
|
||||
if kind not in ("999", "277ca", "ta1"):
|
||||
raise ValueError(f"find_ack_orphans: unknown kind={kind!r}")
|
||||
|
||||
out: list[dict] = []
|
||||
if kind == "999":
|
||||
ack_table = Ack
|
||||
ctrl_attr = None
|
||||
elif kind == "277ca":
|
||||
ack_table = Two77caAck
|
||||
ctrl_attr = "control_number"
|
||||
else:
|
||||
ack_table = Ta1Ack
|
||||
ctrl_attr = "control_number"
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
# Every ack row of the given kind, with a LEFT JOIN against
|
||||
# any claim_acks link; orphan when NO link was created.
|
||||
if kind == "999":
|
||||
# For 999, the "ack has no link" means no ClaimAck row
|
||||
# was emitted at all (the auto-linker emits one per AK2
|
||||
# even when the AK2 is rejected, so 999 with at least
|
||||
# one AK2 that resolved to a claim is never an orphan).
|
||||
# We treat a 999 as orphan when it has zero ClaimAck
|
||||
# rows tied to its id.
|
||||
all_acks = s.query(Ack).order_by(Ack.id.desc()).all()
|
||||
for ack_row in all_acks:
|
||||
count = (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.ack_kind == "999",
|
||||
ClaimAck.ack_id == ack_row.id)
|
||||
.count()
|
||||
)
|
||||
if count == 0:
|
||||
out.append({
|
||||
"kind": "999",
|
||||
"ack_id": ack_row.id,
|
||||
"control_number": _ack_control_number(ack_row, "999"),
|
||||
"parsed_at": (
|
||||
ack_row.parsed_at.isoformat().replace(
|
||||
"+00:00", "Z"
|
||||
) if ack_row.parsed_at else None
|
||||
),
|
||||
})
|
||||
elif kind == "277ca":
|
||||
all_acks = s.query(Two77caAck).order_by(Two77caAck.id.desc()).all()
|
||||
for ack_row in all_acks:
|
||||
count = (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.ack_kind == "277ca",
|
||||
ClaimAck.ack_id == ack_row.id)
|
||||
.count()
|
||||
)
|
||||
if count == 0:
|
||||
out.append({
|
||||
"kind": "277ca",
|
||||
"ack_id": ack_row.id,
|
||||
"control_number": ack_row.control_number or "",
|
||||
"parsed_at": (
|
||||
ack_row.parsed_at.isoformat().replace(
|
||||
"+00:00", "Z"
|
||||
) if ack_row.parsed_at else None
|
||||
),
|
||||
})
|
||||
else:
|
||||
all_acks = s.query(Ta1Ack).order_by(Ta1Ack.id.desc()).all()
|
||||
for ack_row in all_acks:
|
||||
count = (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.ack_kind == "ta1",
|
||||
ClaimAck.ack_id == ack_row.id)
|
||||
.count()
|
||||
)
|
||||
if count == 0:
|
||||
out.append({
|
||||
"kind": "ta1",
|
||||
"ack_id": ack_row.id,
|
||||
"control_number": ack_row.control_number or "",
|
||||
"parsed_at": (
|
||||
ack_row.parsed_at.isoformat().replace(
|
||||
"+00:00", "Z"
|
||||
) if ack_row.parsed_at else None
|
||||
),
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def _ack_control_number(ack_row: Ack, kind: str) -> str:
|
||||
"""Best-effort control-number lookup for a 999 ack row.
|
||||
|
||||
The 999 ORM row doesn't carry the envelope's control_number in
|
||||
a dedicated column; we re-derive it from ``raw_json`` (the same
|
||||
source :func:`cyclone.store.ui.to_ui_ack` uses for the patient
|
||||
control number).
|
||||
"""
|
||||
raw = ack_row.raw_json or {}
|
||||
env = raw.get("envelope") or {}
|
||||
return env.get("control_number") or ""
|
||||
|
||||
|
||||
__all__ = [
|
||||
"add_claim_ack",
|
||||
"batch_envelope_index",
|
||||
"find_ack_orphans",
|
||||
"list_acks_for_claim",
|
||||
"list_claims_for_ack",
|
||||
"remove_claim_ack",
|
||||
]
|
||||
@@ -0,0 +1,710 @@
|
||||
"""Claim- and remittance-detail read APIs.
|
||||
|
||||
Includes the only large non-write query in the store:
|
||||
``get_claim_detail`` which joins Claim + CasAdjustment + ActivityEvent
|
||||
history for the right-drawer UI.
|
||||
|
||||
Also hosts ``check_matched_pair_drift()`` — the SP27 startup invariant
|
||||
audit that returns the count of Claim↔Remit pairs where
|
||||
``matched_remittance_id`` doesn't agree with the FK in ``remittances.claim_id``.
|
||||
It lives here (not in its own module) because it reads the same pair of
|
||||
tables as ``get_claim_detail``'s matched-remittance summary.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import timezone
|
||||
from decimal import Decimal
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import (
|
||||
ActivityEvent,
|
||||
CasAdjustment,
|
||||
Claim,
|
||||
ClaimState,
|
||||
Remittance,
|
||||
)
|
||||
from .ui import (
|
||||
CLAIM_DETAIL_HISTORY_LIMIT,
|
||||
_date_in_bounds,
|
||||
_iso_z,
|
||||
_svc_to_wire_dict,
|
||||
to_ui_claim_detail,
|
||||
to_ui_claim_from_orm,
|
||||
to_ui_provider,
|
||||
to_ui_remittance_from_orm,
|
||||
to_ui_remittance_with_adjustments,
|
||||
)
|
||||
|
||||
|
||||
def get_remittance(remittance_id: str) -> dict | None:
|
||||
"""Return a UI-shaped remittance dict with ``adjustments`` array.
|
||||
|
||||
Joins the persisted ``CasAdjustment`` rows for ``remittance_id``
|
||||
and labels each via :mod:`cyclone.parsers.cas_codes`. Returns
|
||||
``None`` when the remittance is not found so the API layer can
|
||||
map that to a 404.
|
||||
|
||||
SP7: also returns the per-line SVC composites
|
||||
(``serviceLinePayments``) and the CLP-level (claim-level) CAS
|
||||
bucket (``claimLevelAdjustments``) so the remit drawer can show
|
||||
per-line payments + adjustments without a second fetch.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(Remittance, remittance_id)
|
||||
if row is None:
|
||||
return None
|
||||
cas_rows = (
|
||||
s.query(CasAdjustment)
|
||||
.filter(CasAdjustment.remittance_id == remittance_id)
|
||||
.all()
|
||||
)
|
||||
parsed_at = (
|
||||
row.batch.parsed_at if row.batch is not None else row.received_at
|
||||
)
|
||||
if parsed_at is not None and parsed_at.tzinfo is None:
|
||||
parsed_at = parsed_at.replace(tzinfo=timezone.utc)
|
||||
body = to_ui_remittance_with_adjustments(
|
||||
row,
|
||||
batch_id=row.batch_id,
|
||||
parsed_at=parsed_at,
|
||||
cas_rows=cas_rows,
|
||||
)
|
||||
# SP7: per-line SVC composites + claim-level CAS bucket.
|
||||
from cyclone.db import ServiceLinePayment as SLP
|
||||
slps = (
|
||||
s.query(SLP)
|
||||
.filter(SLP.remittance_id == remittance_id)
|
||||
.order_by(SLP.line_number)
|
||||
.all()
|
||||
)
|
||||
body["serviceLinePayments"] = [
|
||||
_svc_to_wire_dict(svc) for svc in slps
|
||||
]
|
||||
body["claimLevelAdjustments"] = [
|
||||
{
|
||||
"id": c.id,
|
||||
"group_code": c.group_code,
|
||||
"reason_code": c.reason_code,
|
||||
"amount": str(Decimal(str(c.amount))),
|
||||
"quantity": (
|
||||
str(Decimal(str(c.quantity)))
|
||||
if c.quantity is not None
|
||||
else None
|
||||
),
|
||||
}
|
||||
for c in cas_rows
|
||||
if c.service_line_payment_id is None
|
||||
]
|
||||
return body
|
||||
|
||||
|
||||
def get_claim_detail(claim_id: str) -> dict | None:
|
||||
"""Return the SP4 detail-drawer shape for one claim, or ``None``.
|
||||
|
||||
Drives ``GET /api/claims/{claim_id}``. Returns the spec-shaped
|
||||
dict from :func:`to_ui_claim_detail` (header + state + parties +
|
||||
validation + service lines + diagnoses + raw segments) stitched
|
||||
with the claim's recent activity history and, if paired, a
|
||||
matched-remittance summary.
|
||||
|
||||
Returns ``None`` when ``claim_id`` is not in the DB so the API
|
||||
layer can map that to a 404 — the URL-driven drawer
|
||||
distinguishes "claim doesn't exist" from "fetch failed" (the
|
||||
spec §3.4 calls for a distinct 404 state in the drawer).
|
||||
|
||||
The history is capped at :data:`CLAIM_DETAIL_HISTORY_LIMIT`
|
||||
(50, per the spec) and ordered ``ts DESC`` so the most recent
|
||||
event is first. The status string in ``matchedRemittance``
|
||||
follows the same ``reconciled``/``received`` mapping used by
|
||||
:func:`to_ui_remittance_from_orm`.
|
||||
"""
|
||||
# Lazy import — same pattern used throughout this module to
|
||||
# avoid a circular store ↔ db import on cold start.
|
||||
from cyclone import db as _db
|
||||
|
||||
with _db.SessionLocal()() as s:
|
||||
row = s.get(Claim, claim_id)
|
||||
if row is None:
|
||||
return None
|
||||
|
||||
history_rows = (
|
||||
s.query(ActivityEvent)
|
||||
.filter(ActivityEvent.claim_id == claim_id)
|
||||
.order_by(ActivityEvent.ts.desc())
|
||||
.limit(CLAIM_DETAIL_HISTORY_LIMIT)
|
||||
.all()
|
||||
)
|
||||
|
||||
# Claim.batch_id is FK NOT NULL with ON DELETE CASCADE, so
|
||||
# ``row.batch`` is always populated in normal flow. Re-attach
|
||||
# UTC only when SQLite drops the tzinfo on read.
|
||||
parsed_at = row.batch.parsed_at
|
||||
if parsed_at.tzinfo is None:
|
||||
parsed_at = parsed_at.replace(tzinfo=timezone.utc)
|
||||
|
||||
detail = to_ui_claim_detail(
|
||||
row,
|
||||
batch_id=row.batch_id,
|
||||
parsed_at=parsed_at,
|
||||
)
|
||||
|
||||
detail["stateHistory"] = [
|
||||
{
|
||||
"kind": ev.kind,
|
||||
# SQLite drops tzinfo on read; rows are stored UTC
|
||||
# at write time (see ``add`` / ``manual_match``),
|
||||
# so re-attach UTC if needed to keep the spec
|
||||
# contract that ``ts`` ends in Z.
|
||||
"ts": _iso_z(ev.ts),
|
||||
"batchId": ev.batch_id,
|
||||
"remittanceId": ev.remittance_id,
|
||||
}
|
||||
for ev in history_rows
|
||||
]
|
||||
|
||||
if row.matched_remittance_id is not None:
|
||||
remit = s.get(Remittance, row.matched_remittance_id)
|
||||
if remit is not None:
|
||||
status = (
|
||||
"reconciled"
|
||||
if remit.status_code in ("21", "22")
|
||||
else "received"
|
||||
)
|
||||
detail["matchedRemittance"] = {
|
||||
"id": remit.id,
|
||||
"totalPaid": float(remit.total_paid or 0),
|
||||
"status": status,
|
||||
"receivedAt": _iso_z(remit.received_at),
|
||||
}
|
||||
# If the remittance was deleted out from under the FK
|
||||
# (the FK is ``ON DELETE SET NULL`` so the column is
|
||||
# already cleared in normal flow), the matched_remittance_id
|
||||
# would be None here and we wouldn't enter this branch.
|
||||
# If the FK is non-null but the row is gone (e.g. tests
|
||||
# that bypass the cascade), fall through with the
|
||||
# default ``None`` — the UI shows "no match" rather
|
||||
# than crashing.
|
||||
|
||||
# SP7 §5.2: slim per-line projection so the ServiceLinesTable
|
||||
# can show Paid + Adjustments columns without a second fetch.
|
||||
# The 837 side is keyed by ``claim_service_line_number`` (the
|
||||
# 1-based line number from raw_json) since 837 service lines
|
||||
# are not a separate ORM table.
|
||||
from cyclone.db import (
|
||||
LineReconciliation, ServiceLinePayment, CasAdjustment,
|
||||
)
|
||||
slim_lrs = list(
|
||||
s.query(LineReconciliation)
|
||||
.filter(LineReconciliation.claim_id == claim_id)
|
||||
.all()
|
||||
)
|
||||
svc_ids_for_cas = [
|
||||
lr.service_line_payment_id
|
||||
for lr in slim_lrs
|
||||
if lr.service_line_payment_id is not None
|
||||
]
|
||||
cas_sums_by_svc: dict = {}
|
||||
svc_by_id_slim: dict = {}
|
||||
if svc_ids_for_cas:
|
||||
cas_rows = (
|
||||
s.query(CasAdjustment.service_line_payment_id, CasAdjustment.amount)
|
||||
.filter(CasAdjustment.service_line_payment_id.in_(svc_ids_for_cas))
|
||||
.all()
|
||||
)
|
||||
from collections import defaultdict
|
||||
agg = defaultdict(lambda: Decimal("0"))
|
||||
for svc_id, amount in cas_rows:
|
||||
agg[svc_id] += Decimal(str(amount))
|
||||
cas_sums_by_svc = {k: str(v) for k, v in agg.items()}
|
||||
for svc in (
|
||||
s.query(ServiceLinePayment)
|
||||
.filter(ServiceLinePayment.id.in_(svc_ids_for_cas))
|
||||
.all()
|
||||
):
|
||||
svc_by_id_slim[svc.id] = svc
|
||||
|
||||
slim_by_num: dict = {
|
||||
lr.claim_service_line_number: lr
|
||||
for lr in slim_lrs
|
||||
if lr.claim_service_line_number is not None
|
||||
}
|
||||
line_reconciliation_slim: list = []
|
||||
for sl in detail["serviceLines"]:
|
||||
ln = sl.get("lineNumber")
|
||||
lr = slim_by_num.get(ln)
|
||||
if lr is None:
|
||||
line_reconciliation_slim.append({
|
||||
"lineNumber": ln,
|
||||
"status": "unmatched_837_only",
|
||||
"paid": None,
|
||||
"adjustmentsSum": None,
|
||||
})
|
||||
continue
|
||||
svc = (
|
||||
svc_by_id_slim.get(lr.service_line_payment_id)
|
||||
if lr.service_line_payment_id
|
||||
else None
|
||||
)
|
||||
line_reconciliation_slim.append({
|
||||
"lineNumber": ln,
|
||||
"status": lr.status,
|
||||
"paid": str(Decimal(str(svc.payment))) if svc else None,
|
||||
"adjustmentsSum": (
|
||||
cas_sums_by_svc.get(lr.service_line_payment_id)
|
||||
if lr.service_line_payment_id
|
||||
else None
|
||||
),
|
||||
})
|
||||
detail["lineReconciliation"] = line_reconciliation_slim
|
||||
|
||||
return detail
|
||||
|
||||
|
||||
def iter_claims(
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
status: str | None = None,
|
||||
provider_npi: str | None = None,
|
||||
payer: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
sort: str | None = None,
|
||||
order: str = "desc",
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
) -> list[dict]:
|
||||
"""Return UI-shaped claim dicts from the DB.
|
||||
|
||||
Filters mirror the in-memory version. The ``payer`` filter is
|
||||
a case-insensitive substring on the payer's ``name``, recovered
|
||||
from each claim's ``raw_json`` payload (the DB stores it there
|
||||
because ``Claim`` itself only carries ``payer_id``).
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
q = s.query(Claim)
|
||||
if batch_id is not None:
|
||||
q = q.filter(Claim.batch_id == batch_id)
|
||||
if status is not None:
|
||||
q = q.filter(Claim.state == ClaimState(status))
|
||||
if provider_npi is not None:
|
||||
q = q.filter(Claim.provider_npi == provider_npi)
|
||||
|
||||
rows = q.all()
|
||||
# Bulk-load matched-remittance totals so the UI's "Received"
|
||||
# KPI + per-claim received_amount reflect real paid amounts
|
||||
# rather than always-0. One SQL roundtrip for the whole page
|
||||
# rather than per-claim lookups.
|
||||
matched_ids = [
|
||||
r.matched_remittance_id
|
||||
for r in rows
|
||||
if r.matched_remittance_id
|
||||
]
|
||||
received_by_remit: dict[str, float] = {}
|
||||
if matched_ids:
|
||||
for rid, total_paid in (
|
||||
s.query(Remittance.id, Remittance.total_paid)
|
||||
.filter(Remittance.id.in_(matched_ids))
|
||||
.all()
|
||||
):
|
||||
received_by_remit[rid] = float(total_paid or 0)
|
||||
|
||||
out: list[dict] = []
|
||||
for r in rows:
|
||||
raw = r.raw_json or {}
|
||||
bp = raw.get("billing_provider", {})
|
||||
payer_obj = raw.get("payer", {})
|
||||
sub = raw.get("subscriber", {})
|
||||
claim_hdr = raw.get("claim", {})
|
||||
service_lines = raw.get("service_lines", [])
|
||||
parsed_at_iso = (
|
||||
r.batch.parsed_at.isoformat().replace("+00:00", "Z")
|
||||
if r.batch is not None
|
||||
else ""
|
||||
)
|
||||
cpt = (
|
||||
service_lines[0].get("procedure", {}).get("code", "")
|
||||
if service_lines
|
||||
else ""
|
||||
)
|
||||
out.append({
|
||||
"id": r.id,
|
||||
"patientName": (
|
||||
f"{sub.get('first_name', '')} "
|
||||
f"{sub.get('last_name', '')}".strip()
|
||||
),
|
||||
"providerNpi": bp.get("npi") or r.provider_npi or "",
|
||||
"payerName": payer_obj.get("name") or "",
|
||||
"cptCode": cpt,
|
||||
"billedAmount": float(r.charge_amount or 0),
|
||||
"receivedAmount": received_by_remit.get(
|
||||
r.matched_remittance_id, 0.0
|
||||
),
|
||||
"status": r.state.value if hasattr(r.state, "value") else str(r.state),
|
||||
"state": r.state.value if hasattr(r.state, "value") else str(r.state),
|
||||
"denialReason": None,
|
||||
"submissionDate": parsed_at_iso,
|
||||
"batchId": r.batch_id,
|
||||
"parsedAt": parsed_at_iso,
|
||||
# Keep these so we can sort on them in-memory below.
|
||||
"_sort_billedAmount": float(r.charge_amount or 0),
|
||||
"_sort_submissionDate": parsed_at_iso,
|
||||
})
|
||||
|
||||
if payer is not None:
|
||||
needle = payer.casefold()
|
||||
out = [
|
||||
c for c in out
|
||||
if needle in (c.get("payerName") or "").casefold()
|
||||
]
|
||||
out = [
|
||||
c for c in out
|
||||
if _date_in_bounds(c, "submissionDate", date_from, date_to)
|
||||
]
|
||||
if sort is not None:
|
||||
out.sort(
|
||||
key=lambda c: c.get(f"_sort_{sort}", 0) or 0,
|
||||
reverse=(order == "desc"),
|
||||
)
|
||||
# Drop the private sort keys before returning.
|
||||
for c in out:
|
||||
c.pop("_sort_billedAmount", None)
|
||||
c.pop("_sort_submissionDate", None)
|
||||
return out[offset:offset + limit]
|
||||
|
||||
|
||||
def iter_remittances(
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
payer: str | None = None,
|
||||
claim_id: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
sort: str | None = None,
|
||||
order: str = "desc",
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
) -> list[dict]:
|
||||
"""Return UI-shaped remittance dicts from the DB."""
|
||||
with db.SessionLocal()() as s:
|
||||
q = s.query(Remittance)
|
||||
if batch_id is not None:
|
||||
q = q.filter(Remittance.batch_id == batch_id)
|
||||
if claim_id is not None:
|
||||
q = q.filter(Remittance.claim_id == claim_id)
|
||||
|
||||
rows = q.all()
|
||||
# Bulk-fetch all CAS rows for these remittances in one query
|
||||
# (SP3 P2 follow-up — fixes the list-view's empty adjustments
|
||||
# expansion). N+1-free.
|
||||
cas_by_remit: dict[str, list] = {}
|
||||
if rows:
|
||||
from cyclone.parsers.cas_codes import reason_label
|
||||
cas_rows = (
|
||||
s.query(CasAdjustment)
|
||||
.filter(CasAdjustment.remittance_id.in_([r.id for r in rows]))
|
||||
.all()
|
||||
)
|
||||
for c in cas_rows:
|
||||
cas_by_remit.setdefault(c.remittance_id, []).append(c)
|
||||
|
||||
out: list[dict] = []
|
||||
for r in rows:
|
||||
raw = r.raw_json or {}
|
||||
parsed_at_iso = (
|
||||
r.batch.parsed_at.isoformat().replace("+00:00", "Z")
|
||||
if r.batch is not None
|
||||
else r.received_at.isoformat().replace("+00:00", "Z")
|
||||
)
|
||||
payer_name = ""
|
||||
if r.batch is not None and r.batch.raw_result_json:
|
||||
payer_name = (
|
||||
r.batch.raw_result_json.get("payer", {}).get("name", "")
|
||||
)
|
||||
adjustments = [
|
||||
{
|
||||
"group": c.group_code,
|
||||
"reason": c.reason_code,
|
||||
"label": reason_label(c.group_code, c.reason_code),
|
||||
"amount": float(c.amount),
|
||||
"quantity": float(c.quantity) if c.quantity is not None else None,
|
||||
}
|
||||
for c in cas_by_remit.get(r.id, [])
|
||||
]
|
||||
out.append({
|
||||
"id": r.id,
|
||||
"claimId": r.claim_id or "",
|
||||
"payerName": payer_name,
|
||||
"paidAmount": float(r.total_paid or 0),
|
||||
"adjustmentAmount": float(r.adjustment_amount or 0),
|
||||
"status": (
|
||||
"reconciled" if r.status_code in ("21", "22")
|
||||
else "received"
|
||||
),
|
||||
"denialReason": None,
|
||||
"validationWarnings": [],
|
||||
"receivedDate": r.received_at.isoformat().replace("+00:00", "Z"),
|
||||
"batchId": r.batch_id,
|
||||
"parsedAt": parsed_at_iso,
|
||||
"adjustments": adjustments,
|
||||
"_sort_receivedDate": r.received_at.isoformat().replace("+00:00", "Z"),
|
||||
})
|
||||
|
||||
if payer is not None:
|
||||
out = [r for r in out if r.get("payerName") == payer]
|
||||
out = [
|
||||
r for r in out
|
||||
if _date_in_bounds(r, "receivedDate", date_from, date_to)
|
||||
]
|
||||
if sort is not None:
|
||||
out.sort(
|
||||
key=lambda r: r.get(f"_sort_{sort}", 0) or 0,
|
||||
reverse=(order == "desc"),
|
||||
)
|
||||
for r in out:
|
||||
r.pop("_sort_receivedDate", None)
|
||||
return out[offset:offset + limit]
|
||||
|
||||
|
||||
def distinct_providers() -> list[dict]:
|
||||
"""Group claims by NPI and return one row per provider."""
|
||||
with db.SessionLocal()() as s:
|
||||
rows = s.query(Claim).all()
|
||||
by_npi: dict[str, dict] = {}
|
||||
for r in rows:
|
||||
npi = r.provider_npi or ""
|
||||
if npi not in by_npi:
|
||||
raw = r.raw_json or {}
|
||||
bp = raw.get("billing_provider", {})
|
||||
by_npi[npi] = to_ui_provider(
|
||||
npi=npi,
|
||||
name=bp.get("name") or "",
|
||||
tax_id=bp.get("tax_id"),
|
||||
address=None,
|
||||
city=None,
|
||||
state=None,
|
||||
zip=None,
|
||||
phone=None,
|
||||
claim_count=0,
|
||||
outstanding_ar=0.0,
|
||||
)
|
||||
by_npi[npi]["claimCount"] += 1
|
||||
return list(by_npi.values())
|
||||
|
||||
|
||||
def recent_activity(*, limit: int = 200) -> list[dict]:
|
||||
"""Return recent activity events from the DB, newest first.
|
||||
|
||||
SP21 Task 2.5: each row also carries ``claimId`` and
|
||||
``remittanceId`` (read from the ORM columns) so the Dashboard's
|
||||
Recent-activity card can route clicks to the right entity
|
||||
drawer via ``src/lib/event-routing.ts``. Both are nullable
|
||||
strings; the wire shape uses camelCase keys to match the
|
||||
existing ``npi`` / ``amount`` fields and the frontend
|
||||
``Activity`` interface.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
rows = (
|
||||
s.query(ActivityEvent)
|
||||
.order_by(ActivityEvent.ts.desc())
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
return [
|
||||
{
|
||||
"id": f"ae-{r.id}",
|
||||
"kind": r.kind,
|
||||
"message": (r.payload_json or {}).get("message", ""),
|
||||
"timestamp": r.ts.isoformat().replace("+00:00", "Z"),
|
||||
"npi": (r.payload_json or {}).get("npi"),
|
||||
"amount": (r.payload_json or {}).get("amount"),
|
||||
"claimId": r.claim_id,
|
||||
"remittanceId": r.remittance_id,
|
||||
}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
|
||||
def check_matched_pair_drift() -> int:
|
||||
"""Audit the ``Claim.matched_remittance_id`` ↔ ``Remittance.claim_id``
|
||||
FK pair at startup. Non-blocking (SP27 Task 11).
|
||||
|
||||
The matched pair is a denormalized FK pair maintained transactionally
|
||||
by ``manual_match``, ``manual_unmatch``, and ``reconcile.run``. A
|
||||
pre-existing mismatch (e.g. a row written before a state migration
|
||||
that added one column but not the other) would otherwise stay
|
||||
invisible until the next operator pair attempt fails confusingly.
|
||||
This check logs the count + up to N examples so operators can
|
||||
investigate without booting the system.
|
||||
|
||||
Returns the number of drifted rows (0 means clean). Does not
|
||||
raise; bootstrap continues even if drift is detected.
|
||||
|
||||
Count semantics: this returns *drifted rows*, not *drifted pairs*.
|
||||
A single broken pair (``Claim.matched_remittance_id = X`` AND
|
||||
``Remittance.claim_id = Y != nil`` with neither pointing back)
|
||||
can produce TWO drifted rows — one in case A (the claim) and
|
||||
one in case B (the remit). In practice drift is almost always
|
||||
asymmetric (one side NULL), so count == count(pairs); for the
|
||||
fully-symmetric minority, divide by ~2 when alerting. Real drift
|
||||
should be fixed by repairing the writer path, not by counting.
|
||||
|
||||
Cases:
|
||||
A. Claim ``matched_remittance_id = X`` but the paired remit X's
|
||||
``claim_id`` is either NULL or doesn't point back to the claim.
|
||||
B. Remit ``claim_id = A`` but the paired claim A's
|
||||
``matched_remittance_id`` is either NULL or doesn't point back.
|
||||
"""
|
||||
import logging
|
||||
from sqlalchemy import select
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
# Case A: claim says X is paired, but X.claim_id doesn't point back.
|
||||
case_a = list(
|
||||
s.execute(
|
||||
select(
|
||||
Claim.id.label("claim_id"),
|
||||
Claim.matched_remittance_id.label("claimed_remit_id"),
|
||||
Remittance.claim_id.label("remit_points_to"),
|
||||
)
|
||||
.outerjoin(
|
||||
Remittance,
|
||||
Remittance.id == Claim.matched_remittance_id,
|
||||
)
|
||||
.where(Claim.matched_remittance_id.is_not(None))
|
||||
.where(
|
||||
(Remittance.claim_id.is_(None))
|
||||
| (Remittance.claim_id != Claim.id)
|
||||
)
|
||||
).all()
|
||||
)
|
||||
# Case B: remit says A is paired, but A.matched_remittance_id
|
||||
# doesn't point back.
|
||||
case_b = list(
|
||||
s.execute(
|
||||
select(
|
||||
Remittance.id.label("remit_id"),
|
||||
Remittance.claim_id.label("claimed_claim_id"),
|
||||
Claim.matched_remittance_id.label("claim_points_to"),
|
||||
)
|
||||
.outerjoin(
|
||||
Claim,
|
||||
Claim.id == Remittance.claim_id,
|
||||
)
|
||||
.where(Remittance.claim_id.is_not(None))
|
||||
.where(
|
||||
(Claim.matched_remittance_id.is_(None))
|
||||
| (Claim.matched_remittance_id != Remittance.id)
|
||||
)
|
||||
).all()
|
||||
)
|
||||
|
||||
total = len(case_a) + len(case_b)
|
||||
if total == 0:
|
||||
log.info("matched-pair drift check: 0 mismatches (clean)")
|
||||
return 0
|
||||
|
||||
log.warning(
|
||||
"matched-pair drift check: %d mismatched pair(s) (showing up to 5 "
|
||||
"of each). Investigate via SELECT against claim / remittance; "
|
||||
"manual re-pair via /api/claims/{id}/manual-match will repair.",
|
||||
total,
|
||||
)
|
||||
for r in case_a[:5]:
|
||||
log.warning(
|
||||
" case A: claim %s -> remit %s, but remit.claim_id=%r",
|
||||
r.claim_id,
|
||||
r.claimed_remit_id,
|
||||
r.remit_points_to,
|
||||
)
|
||||
for r in case_b[:5]:
|
||||
log.warning(
|
||||
" case B: remit %s -> claim %s, but claim.matched_remittance_id=%r",
|
||||
r.remit_id,
|
||||
r.claimed_claim_id,
|
||||
r.claim_points_to,
|
||||
)
|
||||
return total
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Aggregate counters (SP25 / SP27): full-population counts and sums that
|
||||
# the /api/* endpoints expose so page-local reductions (25 rows + live-tail
|
||||
# delta) can never silently understate the true population.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_ITER_UNBOUNDED = 2**31 - 1
|
||||
|
||||
|
||||
def count_claims(
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
status: str | None = None,
|
||||
provider_npi: str | None = None,
|
||||
payer: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
) -> int:
|
||||
"""Count claims that would be returned by ``iter_claims``.
|
||||
|
||||
Same filter parameters as ``iter_claims`` (excluding
|
||||
``sort``/``order``/``limit``/``offset``, which don't affect cardinality).
|
||||
"""
|
||||
rows = iter_claims(
|
||||
batch_id=batch_id, status=status, provider_npi=provider_npi,
|
||||
payer=payer, date_from=date_from, date_to=date_to,
|
||||
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def count_remittances(
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
payer: str | None = None,
|
||||
claim_id: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
) -> int:
|
||||
"""Count remittances that would be returned by ``iter_remittances``.
|
||||
|
||||
Same filter parameters as ``iter_remittances`` (excluding
|
||||
``sort``/``order``/``limit``/``offset``).
|
||||
"""
|
||||
rows = iter_remittances(
|
||||
batch_id=batch_id, payer=payer, claim_id=claim_id,
|
||||
date_from=date_from, date_to=date_to,
|
||||
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def summarize_remittances(
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
payer: str | None = None,
|
||||
claim_id: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
) -> dict:
|
||||
"""Return ``{count, total_paid, total_adjustments}`` summed over the
|
||||
remittance population that ``iter_remittances`` would return under
|
||||
the same filters. Backs ``GET /api/remittances/summary`` — the
|
||||
Remittances page's KPI tiles (added in SP27).
|
||||
"""
|
||||
rows = iter_remittances(
|
||||
batch_id=batch_id, payer=payer, claim_id=claim_id,
|
||||
date_from=date_from, date_to=date_to,
|
||||
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
|
||||
)
|
||||
total_paid = 0.0
|
||||
total_adjustments = 0.0
|
||||
for r in rows:
|
||||
total_paid += float(r.get("paidAmount") or 0)
|
||||
total_adjustments += float(r.get("adjustmentAmount") or 0)
|
||||
return {
|
||||
"count": len(rows),
|
||||
"total_paid": total_paid,
|
||||
"total_adjustments": total_adjustments,
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
"""Exception types raised by CycloneStore and its domain modules.
|
||||
|
||||
These are part of the public API — callers (API endpoints) catch them
|
||||
and translate to HTTP 409 Conflict responses.
|
||||
"""
|
||||
|
||||
|
||||
class AlreadyMatchedError(Exception):
|
||||
"""Raised by ``CycloneStore.manual_match`` when the claim is already paired.
|
||||
|
||||
The claim's ``matched_remittance_id`` is set, so any new pairing would
|
||||
clobber an existing match. Callers (the T15 API endpoint) should surface
|
||||
this as a 409 Conflict.
|
||||
"""
|
||||
|
||||
|
||||
class NotMatchedError(Exception):
|
||||
"""Raised by ``CycloneStore.manual_unmatch`` when the claim has no match.
|
||||
|
||||
Mirrors ``AlreadyMatchedError`` for the unpair operation. Same HTTP
|
||||
treatment: 409 Conflict at the API layer.
|
||||
"""
|
||||
|
||||
|
||||
class InvalidStateError(Exception):
|
||||
"""Raised when an apply_* pure fn returns a skipped ApplyIntent.
|
||||
|
||||
``reconcile.apply_payment`` / ``apply_reversal`` may return
|
||||
``skipped=True`` (e.g. claim already in a terminal state, or reversal
|
||||
on a non-paid claim). The store surfaces that as ``InvalidStateError``
|
||||
rather than silently pairing. The T15 API endpoint maps this to a
|
||||
409 Conflict and echoes ``current_state`` and ``activity_kind`` so
|
||||
the UI can render a precise message.
|
||||
"""
|
||||
|
||||
def __init__(self, current_state: str, activity_kind: str = "invalid_state"):
|
||||
self.current_state = current_state
|
||||
self.activity_kind = activity_kind
|
||||
super().__init__(
|
||||
f"invalid state {current_state} for apply (kind={activity_kind})"
|
||||
)
|
||||
@@ -0,0 +1,315 @@
|
||||
"""Inbox lane state and manual match/unmatch operations.
|
||||
|
||||
``list_unmatched`` returns the 5-lane inbox view (unmatched claims +
|
||||
unmatched remits). ``manual_match`` and ``manual_unmatch`` invoke the
|
||||
reconcile pure functions and persist the resulting Match row (or
|
||||
remove it). Invalid state transitions surface as ``InvalidStateError``.
|
||||
"""
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import ActivityEvent, Claim, ClaimState, Match, Remittance
|
||||
|
||||
from .exceptions import AlreadyMatchedError, InvalidStateError, NotMatchedError
|
||||
from . import utcnow
|
||||
from .ui import to_ui_claim_from_orm, to_ui_remittance_from_orm
|
||||
|
||||
|
||||
# -- manual reconciliation (T12) -----------------------------------
|
||||
|
||||
def list_unmatched(*, kind: str = "both") -> dict:
|
||||
"""Return unmatched claims and/or remittances.
|
||||
|
||||
An unmatched claim is one with ``matched_remittance_id IS NULL`` —
|
||||
either auto-match never paired it, or it was unpaired by
|
||||
``manual_unmatch``. An unmatched remittance is one with
|
||||
``claim_id IS NULL`` — symmetric FK on the remittance side; we
|
||||
update this in ``manual_match`` so the filter reflects the pair.
|
||||
|
||||
Note: T10's ``reconcile.run`` only writes the claim-side FK
|
||||
(``Claim.matched_remittance_id``) when auto-pairing. Auto-matched
|
||||
remittances therefore still show as "unmatched" here until the
|
||||
pair is touched (re-ingest, manual unmatch + rematch). That gap
|
||||
is intentional for T12 — fixing it requires modifying T10.
|
||||
|
||||
``kind`` selects which side(s) to return:
|
||||
- "claims": only claims
|
||||
- "remittances": only remittances
|
||||
- "both": both (default)
|
||||
|
||||
Returns ``{"claims": [...], "remittances": [...]}`` with the
|
||||
unused side always an empty list (never absent) so callers can
|
||||
unconditionally index.
|
||||
"""
|
||||
if kind not in ("claims", "remittances", "both"):
|
||||
raise ValueError(
|
||||
f"list_unmatched: unknown kind={kind!r} "
|
||||
"(expected 'claims', 'remittances', or 'both')"
|
||||
)
|
||||
|
||||
result: dict = {"claims": [], "remittances": []}
|
||||
with db.SessionLocal()() as s:
|
||||
if kind in ("claims", "both"):
|
||||
rows = (
|
||||
s.query(Claim)
|
||||
.filter(Claim.matched_remittance_id.is_(None))
|
||||
.order_by(Claim.id.asc())
|
||||
.all()
|
||||
)
|
||||
for r in rows:
|
||||
parsed_at = (
|
||||
r.batch.parsed_at
|
||||
if r.batch is not None
|
||||
else r.service_date_from or utcnow()
|
||||
)
|
||||
result["claims"].append(
|
||||
to_ui_claim_from_orm(
|
||||
r, batch_id=r.batch_id, parsed_at=parsed_at,
|
||||
# list_unmatched filters matched_remittance_id IS NULL,
|
||||
# so every row has no remittance yet.
|
||||
received_total=0.0,
|
||||
)
|
||||
)
|
||||
|
||||
if kind in ("remittances", "both"):
|
||||
rows = (
|
||||
s.query(Remittance)
|
||||
.filter(Remittance.claim_id.is_(None))
|
||||
.order_by(Remittance.id.asc())
|
||||
.all()
|
||||
)
|
||||
for r in rows:
|
||||
parsed_at = (
|
||||
r.batch.parsed_at
|
||||
if r.batch is not None
|
||||
else r.received_at
|
||||
)
|
||||
result["remittances"].append(
|
||||
to_ui_remittance_from_orm(
|
||||
r, batch_id=r.batch_id, parsed_at=parsed_at,
|
||||
)
|
||||
)
|
||||
return result
|
||||
|
||||
def manual_match(claim_id: str, remit_id: str) -> dict:
|
||||
"""Pair a claim with a remittance manually (operator override).
|
||||
|
||||
Steps:
|
||||
1. Load the claim; raise ``AlreadyMatchedError`` if it already
|
||||
has a match (we never silently overwrite an existing pair).
|
||||
2. Load the remittance; raise ``LookupError`` if missing.
|
||||
3. Compute the new claim state via ``reconcile.apply_payment``
|
||||
(or ``apply_reversal`` for status codes 21/22).
|
||||
4. Insert a ``Match`` row with ``strategy="manual"``.
|
||||
5. Update the claim (``state``, ``matched_remittance_id``) AND
|
||||
the remittance (``claim_id``) so the symmetric FK reflects
|
||||
the pair — required for ``list_unmatched`` to drop them.
|
||||
6. Record an ``ActivityEvent(kind="manual_match", ...)``.
|
||||
7. Commit; return ``{"claim": <ui>, "match": <ui>}``.
|
||||
|
||||
``reconcile.apply_payment`` may return a noop (claim in terminal
|
||||
state); we surface that as ``InvalidStateError`` rather than
|
||||
silently pairing, because the operator clearly intended a state
|
||||
change. The T15 API endpoint maps this to a 409 Conflict.
|
||||
"""
|
||||
from cyclone import reconcile as _reconcile
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
claim = s.get(Claim, claim_id)
|
||||
if claim is None:
|
||||
raise LookupError(f"claim {claim_id} not found")
|
||||
if claim.matched_remittance_id is not None:
|
||||
raise AlreadyMatchedError(
|
||||
f"claim {claim_id} already matched to "
|
||||
f"{claim.matched_remittance_id}"
|
||||
)
|
||||
|
||||
remit = s.get(Remittance, remit_id)
|
||||
if remit is None:
|
||||
raise LookupError(f"remittance {remit_id} not found")
|
||||
|
||||
prior_state = claim.state
|
||||
if remit.is_reversal:
|
||||
intent = _reconcile.apply_reversal(claim, remit)
|
||||
else:
|
||||
intent = _reconcile.apply_payment(
|
||||
claim, remit,
|
||||
charge=claim.charge_amount,
|
||||
paid=remit.total_paid,
|
||||
status_code=remit.status_code,
|
||||
)
|
||||
|
||||
if intent.skipped or intent.new_state is None:
|
||||
current = (
|
||||
claim.state.value
|
||||
if hasattr(claim.state, "value")
|
||||
else str(claim.state)
|
||||
)
|
||||
raise InvalidStateError(
|
||||
current_state=current,
|
||||
activity_kind=intent.activity_kind,
|
||||
)
|
||||
|
||||
new_state = intent.new_state
|
||||
now = utcnow()
|
||||
|
||||
s.add(Match(
|
||||
claim_id=claim_id,
|
||||
remittance_id=remit_id,
|
||||
strategy="manual",
|
||||
matched_at=now,
|
||||
prior_claim_state=prior_state,
|
||||
is_reversal=remit.is_reversal,
|
||||
))
|
||||
claim.state = new_state
|
||||
claim.matched_remittance_id = remit_id
|
||||
# Symmetric FK update — see list_unmatched docstring for why
|
||||
# we don't rely on T10 to do this.
|
||||
remit.claim_id = claim_id
|
||||
|
||||
# SP7: line-level reconciliation + claim-level CAS aggregate.
|
||||
# Skipped for reversals — they don't have SV1↔SVC line pairs.
|
||||
if not remit.is_reversal:
|
||||
_reconcile._reconcile_pair(s, claim, remit)
|
||||
|
||||
s.add(ActivityEvent(
|
||||
ts=now,
|
||||
kind="manual_match",
|
||||
batch_id=remit.batch_id,
|
||||
claim_id=claim_id,
|
||||
remittance_id=remit_id,
|
||||
payload_json={
|
||||
"strategy": "manual",
|
||||
"new_state": new_state.value,
|
||||
"prior_state": prior_state.value,
|
||||
"is_reversal": remit.is_reversal,
|
||||
},
|
||||
))
|
||||
|
||||
s.commit()
|
||||
|
||||
parsed_at = (
|
||||
claim.batch.parsed_at
|
||||
if claim.batch is not None
|
||||
else now
|
||||
)
|
||||
claim_dict = to_ui_claim_from_orm(
|
||||
claim, batch_id=claim.batch_id, parsed_at=parsed_at,
|
||||
received_total=float(remit.total_paid or 0),
|
||||
)
|
||||
matched_at_iso = now.isoformat().replace("+00:00", "Z")
|
||||
return {
|
||||
"claim": claim_dict,
|
||||
"match": {
|
||||
"strategy": "manual",
|
||||
"claimId": claim_id,
|
||||
"remittanceId": remit_id,
|
||||
"matchedAt": matched_at_iso,
|
||||
"isReversal": remit.is_reversal,
|
||||
"priorState": prior_state.value,
|
||||
"newState": new_state.value,
|
||||
},
|
||||
}
|
||||
|
||||
def manual_unmatch(claim_id: str) -> dict:
|
||||
"""Unpair a previously matched claim and restore its prior state.
|
||||
|
||||
Reverses ``manual_match`` (and auto-match as a side-effect of
|
||||
clearing the FK). Strategy:
|
||||
|
||||
1. Load the claim; raise ``NotMatchedError`` if it isn't
|
||||
currently matched.
|
||||
2. Delete every ``Match`` row for the claim (there may be more
|
||||
than one — reversals create a 2nd row, see T10 spec).
|
||||
3. Restore ``claim.state`` from the latest Match's
|
||||
``prior_claim_state``; fall back to ``SUBMITTED`` when
|
||||
``prior_claim_state`` is NULL (auto-match doesn't set it for
|
||||
non-reversal payments — see reconcile.run line ~278).
|
||||
4. Clear ``claim.matched_remittance_id`` and the symmetric
|
||||
``remit.claim_id`` so ``list_unmatched`` surfaces the pair
|
||||
again.
|
||||
5. Record ``ActivityEvent(kind="manual_unmatch", ...)`` and
|
||||
commit.
|
||||
6. Return ``{"claim": <ui>, "deletedMatches": <count>}``.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
claim = s.get(Claim, claim_id)
|
||||
if claim is None:
|
||||
raise LookupError(f"claim {claim_id} not found")
|
||||
if claim.matched_remittance_id is None:
|
||||
raise NotMatchedError(
|
||||
f"claim {claim_id} has no active match"
|
||||
)
|
||||
|
||||
matches = (
|
||||
s.query(Match)
|
||||
.filter(Match.claim_id == claim_id)
|
||||
.order_by(Match.matched_at.desc())
|
||||
.all()
|
||||
)
|
||||
if not matches:
|
||||
# Defensive: matched_remittance_id was set but no Match
|
||||
# rows exist. Shouldn't happen, but if it does, fall back
|
||||
# to clearing the FK and starting fresh.
|
||||
latest = None
|
||||
paired_remit = None
|
||||
restored_state = ClaimState.SUBMITTED
|
||||
else:
|
||||
latest = matches[0]
|
||||
paired_remit = s.get(Remittance, latest.remittance_id)
|
||||
restored_state = (
|
||||
latest.prior_claim_state
|
||||
if latest.prior_claim_state is not None
|
||||
else ClaimState.SUBMITTED
|
||||
)
|
||||
|
||||
deleted_count = len(matches)
|
||||
for m in matches:
|
||||
s.delete(m)
|
||||
|
||||
claim.state = restored_state
|
||||
claim.matched_remittance_id = None
|
||||
# Clear the symmetric FK on the remittance so list_unmatched
|
||||
# surfaces the pair again. The remittance may have been
|
||||
# deleted between the match and this call — guard with a
|
||||
# None check so we don't blow up on a stale FK.
|
||||
if paired_remit is not None:
|
||||
paired_remit.claim_id = None
|
||||
|
||||
now = utcnow()
|
||||
s.add(ActivityEvent(
|
||||
ts=now,
|
||||
kind="manual_unmatch",
|
||||
claim_id=claim_id,
|
||||
payload_json={
|
||||
"restored_state": restored_state.value,
|
||||
"deleted_matches": deleted_count,
|
||||
},
|
||||
))
|
||||
|
||||
s.commit()
|
||||
|
||||
parsed_at = (
|
||||
claim.batch.parsed_at
|
||||
if claim.batch is not None
|
||||
else now
|
||||
)
|
||||
# ``paired_remit`` is the matched remittance we cleared in
|
||||
# the unmatch; use its ``total_paid`` for the response shape
|
||||
# so the UI sees what was paid before the unpair. May be
|
||||
# ``None`` if the remittance was deleted since the match —
|
||||
# default to 0.0 in that case.
|
||||
received_total = (
|
||||
float(paired_remit.total_paid or 0)
|
||||
if paired_remit is not None
|
||||
else 0.0
|
||||
)
|
||||
claim_dict = to_ui_claim_from_orm(
|
||||
claim, batch_id=claim.batch_id, parsed_at=parsed_at,
|
||||
received_total=received_total,
|
||||
)
|
||||
return {
|
||||
"claim": claim_dict,
|
||||
"deletedMatches": deleted_count,
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -0,0 +1,307 @@
|
||||
"""Dashboard KPI aggregation — cross-table reads consumed by /api/dashboard/kpis.
|
||||
|
||||
``dashboard_kpis()`` returns a single dict that the React Dashboard page
|
||||
consumes as the source of truth for the 4 KPI tiles + the recent-activity
|
||||
panel. It reads from claims + remittances + batches in a single pass.
|
||||
|
||||
``_claim_state_str`` is a small mapping helper that translates ORM
|
||||
status enum values to UI-friendly strings. It's private to this module.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import Claim, Remittance
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP27 Task 13: Dashboard aggregate KPIs.
|
||||
#
|
||||
# The Dashboard's "Billed / Received / Denial rate / Pending AR" tiles are
|
||||
# computed from the *whole* claim population, not a sample. With 60k+ claims
|
||||
# in production, fetching ``/api/claims?limit=100`` and reducing in JS would
|
||||
# silently produce wrong numbers (denial rate sampled, billed summed from
|
||||
# 100 rows). This module-level function does the aggregation server-side in
|
||||
# a single session and returns a small structured payload the Dashboard
|
||||
# can render directly.
|
||||
#
|
||||
# Performance: one ``SELECT * FROM claims`` (no pagination) + one
|
||||
# ``SELECT id, total_paid FROM remittances WHERE id IN (...)`` for matched
|
||||
# remits. SQLite returns 60k claim rows in ~30ms on the development
|
||||
# machine; the Python reduce is microseconds. If the dataset grows past
|
||||
# ~500k claims we'd want a SQL-side ``GROUP BY month`` instead — but at
|
||||
# that volume the dashboard should probably be backed by a materialized
|
||||
# view, not a live query.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _claim_state_str(claim: Claim) -> str:
|
||||
"""Stringify a Claim's ``state`` regardless of enum vs raw str storage."""
|
||||
st = claim.state
|
||||
return st.value if hasattr(st, "value") else str(st)
|
||||
|
||||
|
||||
# Claim states counted toward the Dashboard's "pending" tile. SUBMITTED
|
||||
# is the initial post-parse state; REJECTED is the 999 envelope-level
|
||||
# rejection that means the payer never saw the claim. Both are
|
||||
# "outstanding adjudication" from the operator's POV; ``denied`` is
|
||||
# payer adjudication and lives in its own tile.
|
||||
_DASHBOARD_PENDING_STATES: frozenset[str] = frozenset({"submitted", "rejected"})
|
||||
|
||||
|
||||
def dashboard_kpis(
|
||||
*,
|
||||
months: int = 6,
|
||||
top_n_providers: int = 4,
|
||||
top_n_denials: int = 5,
|
||||
) -> dict:
|
||||
"""Compute Dashboard KPIs over the entire claim/remittance population.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
months
|
||||
Number of trailing calendar months to include in the ``monthly``
|
||||
sparkline series (default 6 — matches the existing frontend
|
||||
``MONTHS_BACK`` constant).
|
||||
top_n_providers
|
||||
How many providers to include in the ``topProviders`` array
|
||||
(default 4 — matches the existing Dashboard layout).
|
||||
top_n_denials
|
||||
How many most-recent denied claims to include in the
|
||||
``topDenials`` array (default 5).
|
||||
|
||||
Returns
|
||||
-------
|
||||
dict with keys:
|
||||
|
||||
- ``totals``: aggregate counts + dollar sums + rates for the whole DB.
|
||||
- ``monthly``: list of ``{month, label, count, billed, received,
|
||||
denied, denialRate, ar}`` dicts, oldest-first, length = ``months``.
|
||||
``ar`` is the running outstanding accounts-receivable (billed -
|
||||
received) carried forward across months, clamped at zero.
|
||||
- ``topProviders``: list of ``{npi, label, claimCount, billed,
|
||||
denied}`` dicts, sorted by claimCount desc.
|
||||
- ``topDenials``: list of ``{id, patientName, billedAmount,
|
||||
denialReason, submissionDate}`` dicts, sorted by submissionDate
|
||||
desc, capped at ``top_n_denials``.
|
||||
|
||||
Notes
|
||||
-----
|
||||
- Empty DB returns zero-filled aggregates, an empty ``monthly`` array
|
||||
(one entry per requested month), an empty ``topProviders`` array,
|
||||
and an empty ``topDenials`` array.
|
||||
- ``received`` for a claim is sourced from the matched Remittance's
|
||||
``total_paid`` column. Claims without a matched remittance
|
||||
contribute 0 to the received sum (and thus inflate the
|
||||
outstanding-AR figure, which is the correct operator-visible
|
||||
semantic — "money we haven't been told was paid yet").
|
||||
"""
|
||||
# Pre-build the trailing-N-months skeleton so the response always has
|
||||
# exactly ``months`` entries even when the DB is empty or sparse.
|
||||
now = datetime.now(timezone.utc)
|
||||
skeleton: list[dict] = []
|
||||
for i in range(months - 1, -1, -1):
|
||||
d = now.replace(day=1)
|
||||
# Walk backwards N months without ``relativedelta``.
|
||||
for _ in range(i):
|
||||
prev_month = d.month - 1
|
||||
if prev_month == 0:
|
||||
d = d.replace(year=d.year - 1, month=12)
|
||||
else:
|
||||
d = d.replace(month=prev_month)
|
||||
skeleton.append({
|
||||
"month": f"{d.year:04d}-{d.month:02d}",
|
||||
"label": d.strftime("%b"),
|
||||
"count": 0,
|
||||
"billed": 0.0,
|
||||
"received": 0.0,
|
||||
"denied": 0,
|
||||
"ar": 0.0,
|
||||
})
|
||||
skeleton_index = {entry["month"]: entry for entry in skeleton}
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
# ``Claim.batch`` is a lazy ``relationship`` (default
|
||||
# ``lazy="select"``); without ``selectinload`` each access to
|
||||
# ``r.batch`` in the reduce loop below issues a fresh
|
||||
# ``SELECT ... FROM batches WHERE id=?``. ``selectinload``
|
||||
# pulls every distinct batch in one round-trip instead of N+1
|
||||
# — critical for the 60k-claim dataset.
|
||||
from sqlalchemy.orm import selectinload
|
||||
claims: list[Claim] = (
|
||||
s.query(Claim)
|
||||
.options(selectinload(Claim.batch))
|
||||
.all()
|
||||
)
|
||||
|
||||
# Bulk-load matched-remit total_paid so a 60k-claim DB doesn't
|
||||
# produce a 60k-query N+1.
|
||||
matched_ids = [
|
||||
r.matched_remittance_id
|
||||
for r in claims
|
||||
if r.matched_remittance_id
|
||||
]
|
||||
received_by_remit: dict[str, float] = {}
|
||||
if matched_ids:
|
||||
for rid, total_paid in (
|
||||
s.query(Remittance.id, Remittance.total_paid)
|
||||
.filter(Remittance.id.in_(matched_ids))
|
||||
.all()
|
||||
):
|
||||
received_by_remit[rid] = float(total_paid or 0)
|
||||
|
||||
# Per-provider accumulator for the topProviders leaderboard.
|
||||
provider_counts: dict[str, int] = {}
|
||||
provider_billed: dict[str, float] = {}
|
||||
provider_denied: dict[str, int] = {}
|
||||
|
||||
# Per-month accumulator + totals in a single pass.
|
||||
total_count = 0
|
||||
total_billed = 0.0
|
||||
total_received = 0.0
|
||||
denied_count = 0
|
||||
pending_count = 0
|
||||
|
||||
# Collect denied candidates as we walk so we don't issue a
|
||||
# second pass for the topDenials array.
|
||||
denied_candidates: list[dict] = []
|
||||
|
||||
for r in claims:
|
||||
billed = float(r.charge_amount or 0)
|
||||
received = received_by_remit.get(r.matched_remittance_id, 0.0)
|
||||
state_str = _claim_state_str(r)
|
||||
|
||||
total_count += 1
|
||||
total_billed += billed
|
||||
total_received += received
|
||||
if state_str == "denied":
|
||||
denied_count += 1
|
||||
# Drop denied claims with no ``submissionDate`` — they'd
|
||||
# sort first under reverse-lex (empty string < ISO) and
|
||||
# render as "Invalid Date" in the Dashboard. A denial
|
||||
# without a batch is exceptional and operator-irrelevant
|
||||
# for the "recent denials" widget.
|
||||
if r.batch is None or r.batch.parsed_at is None:
|
||||
pass
|
||||
else:
|
||||
raw = r.raw_json or {}
|
||||
sub = raw.get("subscriber", {})
|
||||
denied_candidates.append({
|
||||
"id": r.id,
|
||||
"patientName": (
|
||||
f"{sub.get('first_name', '')} "
|
||||
f"{sub.get('last_name', '')}".strip()
|
||||
),
|
||||
"billedAmount": billed,
|
||||
"denialReason": r.rejection_reason,
|
||||
"submissionDate": (
|
||||
r.batch.parsed_at
|
||||
.isoformat()
|
||||
.replace("+00:00", "Z")
|
||||
),
|
||||
})
|
||||
if state_str in _DASHBOARD_PENDING_STATES:
|
||||
pending_count += 1
|
||||
|
||||
# Monthly bin. ``submissionDate`` lives on the parent Batch
|
||||
# (all claims in a batch share a parsed_at). Use UTC year-month
|
||||
# to match the skeleton.
|
||||
if r.batch is not None and r.batch.parsed_at is not None:
|
||||
pa = r.batch.parsed_at
|
||||
key = f"{pa.year:04d}-{pa.month:02d}"
|
||||
bucket = skeleton_index.get(key)
|
||||
if bucket is not None:
|
||||
bucket["count"] += 1
|
||||
bucket["billed"] += billed
|
||||
bucket["received"] += received
|
||||
if state_str == "denied":
|
||||
bucket["denied"] += 1
|
||||
|
||||
# Provider bin (keyed on NPI).
|
||||
npi = r.provider_npi or ""
|
||||
if npi:
|
||||
provider_counts[npi] = provider_counts.get(npi, 0) + 1
|
||||
provider_billed[npi] = provider_billed.get(npi, 0.0) + billed
|
||||
if state_str == "denied":
|
||||
provider_denied[npi] = provider_denied.get(npi, 0) + 1
|
||||
|
||||
# Compute denial rate per month + running AR.
|
||||
running_ar = 0.0
|
||||
for entry in skeleton:
|
||||
if entry["count"] > 0:
|
||||
entry["denialRate"] = (entry["denied"] / entry["count"]) * 100.0
|
||||
else:
|
||||
entry["denialRate"] = 0.0
|
||||
running_ar = max(0.0, running_ar + entry["billed"] - entry["received"])
|
||||
entry["ar"] = running_ar
|
||||
|
||||
# Resolve top provider labels from the Provider table in one
|
||||
# round-trip. NPIs without a Provider row still appear in the
|
||||
# leaderboard with an empty label so operators can see
|
||||
# "unknown-NPI" claims are coming from somewhere.
|
||||
# Local import keeps the module-level ``cyclone.providers.Provider``
|
||||
# Pydantic DTO and the SQLAlchemy ``cyclone.db.Provider`` ORM
|
||||
# separate (same pattern as ``list_providers`` / ``upsert_provider``).
|
||||
provider_labels: dict[str, str] = {}
|
||||
if provider_counts:
|
||||
from cyclone.db import Provider as ProviderORM
|
||||
for npi, label in (
|
||||
s.query(ProviderORM.npi, ProviderORM.label).filter(
|
||||
ProviderORM.npi.in_(provider_counts.keys())
|
||||
).all()
|
||||
):
|
||||
provider_labels[npi] = label or ""
|
||||
|
||||
top_providers = sorted(
|
||||
provider_counts.items(),
|
||||
key=lambda kv: kv[1],
|
||||
reverse=True,
|
||||
)[: max(0, top_n_providers)]
|
||||
top_providers_out = [
|
||||
{
|
||||
"npi": npi,
|
||||
"label": provider_labels.get(npi, ""),
|
||||
"claimCount": count,
|
||||
"billed": round(provider_billed.get(npi, 0.0), 2),
|
||||
"denied": provider_denied.get(npi, 0),
|
||||
}
|
||||
for npi, count in top_providers
|
||||
]
|
||||
|
||||
# Top denials = most recently submitted denied claims, capped at
|
||||
# ``top_n_denials``. submissionDate is ISO-8601 so lex sort ==
|
||||
# chronological sort when timestamps share a tz.
|
||||
denied_candidates.sort(key=lambda d: d["submissionDate"], reverse=True)
|
||||
top_denials = denied_candidates[: max(0, top_n_denials)]
|
||||
|
||||
total_denial_rate = (
|
||||
(denied_count / total_count) * 100.0 if total_count > 0 else 0.0
|
||||
)
|
||||
return {
|
||||
"totals": {
|
||||
"count": total_count,
|
||||
"billed": round(total_billed, 2),
|
||||
"received": round(total_received, 2),
|
||||
"outstandingAr": round(max(0.0, total_billed - total_received), 2),
|
||||
"denied": denied_count,
|
||||
"denialRate": round(total_denial_rate, 4),
|
||||
"pending": pending_count,
|
||||
},
|
||||
"monthly": [
|
||||
{
|
||||
"month": e["month"],
|
||||
"label": e["label"],
|
||||
"count": e["count"],
|
||||
"billed": round(e["billed"], 2),
|
||||
"received": round(e["received"], 2),
|
||||
"denied": e["denied"],
|
||||
"denialRate": round(e["denialRate"], 4),
|
||||
"ar": round(e["ar"], 2),
|
||||
}
|
||||
for e in skeleton
|
||||
],
|
||||
"topProviders": top_providers_out,
|
||||
"topDenials": top_denials,
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
"""ORM row builders — convert parser output into SQLAlchemy ORM rows.
|
||||
|
||||
Each function returns an unsaved ORM object (or inserts related rows
|
||||
within an existing session). They never commit or close the session —
|
||||
the caller (``write.add_record``, ``acks.add_ack``, etc.) owns the
|
||||
transaction boundary.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from decimal import Decimal
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import Claim, ClaimState, Remittance
|
||||
from cyclone.parsers.models import ClaimOutput
|
||||
from cyclone.parsers.models_835 import ClaimPayment
|
||||
|
||||
from . import utcnow
|
||||
|
||||
|
||||
def _service_dates_from_claim(claim: ClaimOutput) -> tuple[date | None, date | None]:
|
||||
"""Extract (service_date_from, service_date_to) from a ClaimOutput.
|
||||
|
||||
The 837P model has ``service_lines[*].service_date`` (one per SV1).
|
||||
We use the earliest as ``from`` and the latest as ``to``; if there
|
||||
are no service lines, both are ``None``.
|
||||
"""
|
||||
dates: list[date] = []
|
||||
for sl in claim.service_lines:
|
||||
if sl.service_date is not None:
|
||||
dates.append(sl.service_date)
|
||||
if not dates:
|
||||
return None, None
|
||||
return min(dates), max(dates)
|
||||
|
||||
|
||||
def _claim_837_row(claim: ClaimOutput, batch_id: str) -> Claim:
|
||||
"""Build a Claim ORM row from a ClaimOutput. NOT yet persisted."""
|
||||
d_from, d_to = _service_dates_from_claim(claim)
|
||||
return Claim(
|
||||
id=claim.claim_id,
|
||||
batch_id=batch_id,
|
||||
# SP27 Task 17: Claim.patient_control_number must hold the CLM01
|
||||
# claim_submittr's_identifier the 837 sent — that's the value the
|
||||
# 835 echoes in CLP01, which the reconcile matcher joins on
|
||||
# (reconcile.py:by_pcn), and what 999 / 277CA ACK lookups also
|
||||
# use to cross-reference the original claim. Storing
|
||||
# subscriber.member_id here (the 2010BA NM109) silently broke
|
||||
# every auto-match in production.
|
||||
patient_control_number=claim.claim_id or "",
|
||||
service_date_from=d_from,
|
||||
service_date_to=d_to,
|
||||
charge_amount=Decimal(claim.claim.total_charge or 0),
|
||||
provider_npi=claim.billing_provider.npi,
|
||||
# SP32: wire NM1*82 (Loop 2420A) rendering provider NPI from
|
||||
# ClaimOutput (T3 parser extraction) into the typed ORM column
|
||||
# (T1 migration 0019). Falls back to None if absent.
|
||||
rendering_provider_npi=claim.rendering_provider_npi,
|
||||
payer_id=claim.payer.id,
|
||||
state=ClaimState.SUBMITTED,
|
||||
raw_json=json.loads(claim.model_dump_json()),
|
||||
)
|
||||
|
||||
|
||||
def _remittance_835_row(cp: ClaimPayment, batch_id: str) -> Remittance:
|
||||
"""Build a Remittance ORM row from a ClaimPayment. NOT yet persisted."""
|
||||
received_at = utcnow()
|
||||
# Adjustment amount: sum the CAS rows for the first service line.
|
||||
# NOTE: This is a best-effort placeholder used until the reconciliation
|
||||
# pass (T10) overwrites it from the persisted CasAdjustment rows. The
|
||||
# authoritative value comes from `reconcile.run()`, which sums
|
||||
# ``CasAdjustment.amount`` per ``remittance_id`` and writes the result
|
||||
# back to ``Remittance.adjustment_amount``. We keep this stub so the
|
||||
# row has a sane value if reconciliation is disabled or fails.
|
||||
adjustment = Decimal("0")
|
||||
if cp.service_payments:
|
||||
sp = cp.service_payments[0]
|
||||
for adj in sp.adjustments:
|
||||
adjustment += adj.amount
|
||||
# Use the first service line's service_date as the remit service_date.
|
||||
service_date: date | None = None
|
||||
if cp.service_payments and cp.service_payments[0].service_date is not None:
|
||||
service_date = cp.service_payments[0].service_date
|
||||
return Remittance(
|
||||
id=cp.payer_claim_control_number,
|
||||
batch_id=batch_id,
|
||||
payer_claim_control_number=cp.payer_claim_control_number,
|
||||
claim_id=None,
|
||||
status_code=cp.status_code,
|
||||
status_label=cp.status_label,
|
||||
total_charge=Decimal(cp.total_charge or 0),
|
||||
total_paid=Decimal(cp.total_paid or 0),
|
||||
patient_responsibility=cp.patient_responsibility,
|
||||
adjustment_amount=adjustment,
|
||||
# SP32: wire NM1*1P (Loop 2100) service provider NPI from
|
||||
# ClaimPayment (T2 parser extraction) into the typed ORM column
|
||||
# (T1 migration 0019). Falls back to None if absent.
|
||||
rendering_provider_npi=cp.service_provider_npi,
|
||||
received_at=received_at,
|
||||
service_date=service_date,
|
||||
is_reversal=cp.status_code in ("21", "22"),
|
||||
raw_json=json.loads(cp.model_dump_json()),
|
||||
)
|
||||
|
||||
|
||||
def _persist_835_remit(session, cp: "ClaimPayment", remittance_id: str) -> None:
|
||||
"""SP7: persist ServiceLinePayment + CAS rows for one CLP composite.
|
||||
|
||||
For each 835 SVC composite in ``cp.service_payments``:
|
||||
- insert a ServiceLinePayment row (line_number, procedure, modifiers,
|
||||
charge, payment, units, service_date).
|
||||
- flush to populate slp.id.
|
||||
- insert each per-SVC CAS adjustment with ``service_line_payment_id``
|
||||
set to slp.id.
|
||||
|
||||
For CLP-level CAS adjustments (``cp.claim_adjustments``, a future
|
||||
extension; not produced by today's 835 parser but allowed by the spec):
|
||||
- insert CAS rows with ``service_line_payment_id IS NULL``.
|
||||
|
||||
The caller controls the transaction; this function does not commit.
|
||||
The 835 ingest site calls this after ``_remittance_835_row`` is
|
||||
flushed so the FK target is populated.
|
||||
"""
|
||||
import json as _json
|
||||
from cyclone.db import ServiceLinePayment, CasAdjustment
|
||||
|
||||
for svc in cp.service_payments:
|
||||
slp = ServiceLinePayment(
|
||||
remittance_id=remittance_id,
|
||||
line_number=svc.line_number,
|
||||
procedure_qualifier=svc.procedure_qualifier,
|
||||
procedure_code=svc.procedure_code,
|
||||
modifiers_json=_json.dumps(svc.modifiers or []),
|
||||
charge=Decimal(str(svc.charge)),
|
||||
payment=Decimal(str(svc.payment)),
|
||||
units=Decimal(str(svc.units)) if svc.units is not None else None,
|
||||
unit_type=svc.unit_type,
|
||||
service_date=svc.service_date,
|
||||
ref_benefit_plan=svc.ref_benefit_plan,
|
||||
)
|
||||
session.add(slp)
|
||||
session.flush() # populate slp.id for the FK below
|
||||
|
||||
for adj in svc.adjustments:
|
||||
quantity = getattr(adj, "quantity", None)
|
||||
session.add(CasAdjustment(
|
||||
remittance_id=remittance_id,
|
||||
group_code=adj.group_code,
|
||||
reason_code=adj.reason_code,
|
||||
amount=Decimal(str(adj.amount)),
|
||||
quantity=Decimal(str(quantity)) if quantity is not None else None,
|
||||
service_line_payment_id=slp.id,
|
||||
))
|
||||
|
||||
# CLP-level CAS (no SVC composite to attach to). Today's parser does
|
||||
# not produce these; the branch is forward-compatible.
|
||||
for adj in getattr(cp, "claim_adjustments", []) or []:
|
||||
quantity = getattr(adj, "quantity", None)
|
||||
session.add(CasAdjustment(
|
||||
remittance_id=remittance_id,
|
||||
group_code=adj.group_code,
|
||||
reason_code=adj.reason_code,
|
||||
amount=Decimal(str(adj.amount)),
|
||||
quantity=Decimal(str(quantity)) if quantity is not None else None,
|
||||
service_line_payment_id=None,
|
||||
))
|
||||
|
||||
|
||||
def _claim_status_from_validation(claim: ClaimOutput) -> str:
|
||||
"""Re-implement the in-memory status rules (sub-project 1 §6.2)."""
|
||||
v = claim.validation
|
||||
if not v.passed:
|
||||
has_r050 = any(e.rule == "R050_diagnosis_present" for e in v.errors)
|
||||
return "draft" if has_r050 else "denied"
|
||||
if claim.claim.frequency_code == "1":
|
||||
return "submitted"
|
||||
if v.warnings:
|
||||
return "pending"
|
||||
return "draft"
|
||||
@@ -0,0 +1,307 @@
|
||||
"""Provider, payer, and clearhouse configuration reads and upserts.
|
||||
|
||||
The ``providers`` and ``payers`` modules in ``cyclone`` provide the
|
||||
ORM-row DTOs; this module is the read/upsert surface over them.
|
||||
``ensure_clearhouse_seeded`` is called at startup to insert the
|
||||
default Clearhouse row if missing.
|
||||
"""
|
||||
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.providers import Clearhouse, Payer, Provider
|
||||
|
||||
from . import utcnow
|
||||
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# SP9: providers / payers / payer_configs / clearhouse
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def list_providers(*, is_active: bool | None = True) -> list[Provider]:
|
||||
"""List providers. ``is_active=None`` returns all."""
|
||||
from cyclone.db import Provider as ProviderORM
|
||||
from cyclone.providers import Provider
|
||||
with db.SessionLocal()() as s:
|
||||
q = s.query(ProviderORM)
|
||||
if is_active is not None:
|
||||
q = q.filter(ProviderORM.is_active == (1 if is_active else 0))
|
||||
rows = q.order_by(ProviderORM.label).all()
|
||||
return [Provider.model_validate(_provider_orm_to_dict(r)) for r in rows]
|
||||
|
||||
def get_provider(npi: str) -> Provider | None:
|
||||
from cyclone.db import Provider as ProviderORM
|
||||
from cyclone.providers import Provider
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(ProviderORM, npi)
|
||||
return Provider.model_validate(_provider_orm_to_dict(row)) if row else None
|
||||
|
||||
def upsert_provider(provider: Provider) -> Provider:
|
||||
from cyclone.db import Provider as ProviderORM
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(ProviderORM, provider.npi)
|
||||
now = utcnow().isoformat()
|
||||
if row is None:
|
||||
row = ProviderORM(
|
||||
npi=provider.npi, label=provider.label,
|
||||
legal_name=provider.legal_name, tax_id=provider.tax_id,
|
||||
taxonomy_code=provider.taxonomy_code,
|
||||
address_line1=provider.address_line1,
|
||||
address_line2=provider.address_line2,
|
||||
city=provider.city, state=provider.state, zip=provider.zip,
|
||||
is_active=1 if provider.is_active else 0,
|
||||
created_at=provider.created_at.isoformat(),
|
||||
updated_at=now,
|
||||
)
|
||||
s.add(row)
|
||||
else:
|
||||
row.label = provider.label
|
||||
row.legal_name = provider.legal_name
|
||||
row.tax_id = provider.tax_id
|
||||
row.taxonomy_code = provider.taxonomy_code
|
||||
row.address_line1 = provider.address_line1
|
||||
row.address_line2 = provider.address_line2
|
||||
row.city = provider.city
|
||||
row.state = provider.state
|
||||
row.zip = provider.zip
|
||||
row.is_active = 1 if provider.is_active else 0
|
||||
row.updated_at = now
|
||||
s.commit()
|
||||
return get_provider(provider.npi) # type: ignore[return-value]
|
||||
|
||||
def list_payers(*, is_active: bool | None = True) -> list[Payer]:
|
||||
from cyclone.db import Payer as PayerORM
|
||||
from cyclone.providers import Payer
|
||||
with db.SessionLocal()() as s:
|
||||
q = s.query(PayerORM)
|
||||
if is_active is not None:
|
||||
q = q.filter(PayerORM.is_active == (1 if is_active else 0))
|
||||
rows = q.order_by(PayerORM.payer_id).all()
|
||||
return [Payer.model_validate(_payer_orm_to_dict(r)) for r in rows]
|
||||
|
||||
def get_payer_config(payer_id: str, transaction_type: str) -> dict | None:
|
||||
from cyclone.db import PayerConfigORM
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(PayerConfigORM, (payer_id, transaction_type))
|
||||
return dict(row.config_json) if row else None
|
||||
|
||||
def get_clearhouse() -> Clearhouse | None:
|
||||
from cyclone.db import ClearhouseORM
|
||||
from cyclone.providers import Clearhouse
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(ClearhouseORM, 1)
|
||||
if row is None:
|
||||
return None
|
||||
return Clearhouse.model_validate({
|
||||
"id": 1,
|
||||
"name": row.name,
|
||||
"tpid": row.tpid,
|
||||
"submitter_name": row.submitter_name,
|
||||
"submitter_id_qual": row.submitter_id_qual,
|
||||
"submitter_contact_name": row.submitter_contact_name,
|
||||
"submitter_contact_email": row.submitter_contact_email,
|
||||
"filename_block": dict(row.filename_block_json),
|
||||
"sftp_block": dict(row.sftp_block_json),
|
||||
"updated_at": row.updated_at,
|
||||
})
|
||||
|
||||
def update_clearhouse(block: Clearhouse) -> Clearhouse:
|
||||
"""Replace the singleton clearhouse row. SP25.
|
||||
|
||||
Used by ``PATCH /api/clearhouse`` to flip ``sftp_block.stub``
|
||||
and adjust host/port/paths without touching SQLite directly.
|
||||
Raises ``LookupError`` if the singleton row is missing — the
|
||||
caller is expected to run the lifespan seed first.
|
||||
"""
|
||||
from cyclone.db import ClearhouseORM
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(ClearhouseORM, 1)
|
||||
if row is None:
|
||||
raise LookupError(
|
||||
"clearhouse singleton row missing; run the lifespan "
|
||||
"seed (ensure_clearhouse_seeded) before PATCH /api/clearhouse"
|
||||
)
|
||||
row.name = block.name
|
||||
row.tpid = block.tpid
|
||||
row.submitter_name = block.submitter_name
|
||||
row.submitter_id_qual = block.submitter_id_qual
|
||||
row.submitter_contact_name = block.submitter_contact_name
|
||||
row.submitter_contact_email = block.submitter_contact_email
|
||||
row.filename_block_json = json.loads(
|
||||
json.dumps(block.filename_block.model_dump())
|
||||
)
|
||||
row.sftp_block_json = json.loads(
|
||||
json.dumps(block.sftp_block.model_dump())
|
||||
)
|
||||
row.updated_at = datetime.now(timezone.utc).isoformat()
|
||||
s.commit()
|
||||
return Clearhouse.model_validate({
|
||||
"id": 1,
|
||||
"name": row.name,
|
||||
"tpid": row.tpid,
|
||||
"submitter_name": row.submitter_name,
|
||||
"submitter_id_qual": row.submitter_id_qual,
|
||||
"submitter_contact_name": row.submitter_contact_name,
|
||||
"submitter_contact_email": row.submitter_contact_email,
|
||||
"filename_block": dict(row.filename_block_json),
|
||||
"sftp_block": dict(row.sftp_block_json),
|
||||
"updated_at": row.updated_at,
|
||||
})
|
||||
|
||||
def ensure_clearhouse_seeded() -> None:
|
||||
"""Insert the default clearhouse singleton + 3 providers + CO_TXIX payer
|
||||
if they don't exist. Idempotent. Called from the API lifespan."""
|
||||
from cyclone.db import ClearhouseORM, Payer as PayerORM, PayerConfigORM, Provider as ProviderORM
|
||||
from cyclone.providers import Clearhouse
|
||||
with db.SessionLocal()() as s:
|
||||
if s.get(ClearhouseORM, 1) is None:
|
||||
ch = Clearhouse(
|
||||
id=1,
|
||||
name="dzinesco",
|
||||
tpid="11525703",
|
||||
submitter_name="Dzinesco",
|
||||
submitter_id_qual="46",
|
||||
submitter_contact_name="Tyler Martinez",
|
||||
submitter_contact_email="tyler@dzinesco.com",
|
||||
filename_block={
|
||||
"tz": "America/Denver",
|
||||
"outbound_template": "tp{tpid}-{tx}-{ts_mt}-1of1.{ext}",
|
||||
"inbound_template": "TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12",
|
||||
},
|
||||
sftp_block={
|
||||
"host": "mft.gainwelltechnologies.com",
|
||||
"port": 22,
|
||||
"username": "colorado-fts\\coxix_prod_11525703",
|
||||
"paths": {
|
||||
"outbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
|
||||
"inbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
|
||||
},
|
||||
"stub": True,
|
||||
"staging_dir": "./var/sftp/staging",
|
||||
"poll_seconds": 300,
|
||||
"auth": {"method": "keychain", "secret_ref": "sftp.gainwell.password"},
|
||||
},
|
||||
updated_at=utcnow(),
|
||||
)
|
||||
s.add(ClearhouseORM(
|
||||
id=1,
|
||||
name=ch.name,
|
||||
tpid=ch.tpid,
|
||||
submitter_name=ch.submitter_name,
|
||||
submitter_id_qual=ch.submitter_id_qual,
|
||||
submitter_contact_name=ch.submitter_contact_name,
|
||||
submitter_contact_email=ch.submitter_contact_email,
|
||||
filename_block_json=ch.filename_block.model_dump(),
|
||||
sftp_block_json=ch.sftp_block.model_dump(),
|
||||
updated_at=ch.updated_at.isoformat(),
|
||||
))
|
||||
|
||||
# Seed 3 providers (idempotent)
|
||||
from cyclone.providers import Provider
|
||||
now = utcnow().isoformat()
|
||||
for npi, label in [
|
||||
("1881068062", "Montrose"),
|
||||
("1851446637", "Delta"),
|
||||
("1467507269", "Salida"),
|
||||
]:
|
||||
if s.get(ProviderORM, npi) is None:
|
||||
s.add(ProviderORM(
|
||||
npi=npi,
|
||||
label=label,
|
||||
legal_name="TOC, Inc.",
|
||||
tax_id="721587149",
|
||||
taxonomy_code="251E00000X",
|
||||
address_line1="1100 East Main St",
|
||||
address_line2="Suite A",
|
||||
city="Montrose",
|
||||
state="CO",
|
||||
zip="814014063",
|
||||
is_active=1,
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
))
|
||||
|
||||
# Seed CO_TXIX payer (idempotent)
|
||||
if s.get(PayerORM, "CO_TXIX") is None:
|
||||
s.add(PayerORM(
|
||||
payer_id="CO_TXIX",
|
||||
name="Colorado Medical Assistance Program",
|
||||
receiver_name="COLORADO MEDICAL ASSISTANCE PROGRAM",
|
||||
receiver_id="COMEDASSISTPROG",
|
||||
is_active=1,
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
))
|
||||
# 837P config block
|
||||
s.add(PayerConfigORM(
|
||||
payer_id="CO_TXIX",
|
||||
transaction_type="837P",
|
||||
config_json={
|
||||
"submitter_name": "Dzinesco",
|
||||
"submitter_contact_name": "Tyler Martinez",
|
||||
"submitter_contact_email": "tyler@dzinesco.com",
|
||||
"receiver_name": "COLORADO MEDICAL ASSISTANCE PROGRAM",
|
||||
"receiver_id_qualifier": "46",
|
||||
"receiver_id": "COMEDASSISTPROG",
|
||||
"bht06_allowed": ["CH", "RP"],
|
||||
"bht06_default": "CH",
|
||||
"sbr09_default": "MC",
|
||||
"sbr09_allowed": ["MC", "16", "MA", "MB", "ZZ"],
|
||||
"payer_id_qualifier": "PI",
|
||||
"payer_id": "CO_TXIX",
|
||||
"pwk_supported": False,
|
||||
"cas_2320_group_allowed": False,
|
||||
"claim_type_codes": {"11": "Office", "12": "Home", "99": "Other"},
|
||||
},
|
||||
updated_at=now,
|
||||
))
|
||||
# 835 config block
|
||||
s.add(PayerConfigORM(
|
||||
payer_id="CO_TXIX",
|
||||
transaction_type="835",
|
||||
config_json={
|
||||
"expected_payer_tax_ids": [
|
||||
"81-1725341", "811725341", "84-0644739",
|
||||
"840644739", "1811725341",
|
||||
],
|
||||
"expected_payer_health_plan_id": "7912900843",
|
||||
"payer_name_pattern": "^CO_(TXIX|BHA)$",
|
||||
},
|
||||
updated_at=now,
|
||||
))
|
||||
|
||||
s.commit()
|
||||
|
||||
|
||||
|
||||
|
||||
def _provider_orm_to_dict(row) -> dict:
|
||||
return {
|
||||
"npi": row.npi,
|
||||
"label": row.label,
|
||||
"legal_name": row.legal_name,
|
||||
"tax_id": row.tax_id,
|
||||
"taxonomy_code": row.taxonomy_code,
|
||||
"address_line1": row.address_line1,
|
||||
"address_line2": row.address_line2,
|
||||
"city": row.city,
|
||||
"state": row.state,
|
||||
"zip": row.zip,
|
||||
"is_active": bool(row.is_active),
|
||||
"created_at": row.created_at,
|
||||
"updated_at": row.updated_at,
|
||||
}
|
||||
|
||||
|
||||
def _payer_orm_to_dict(row) -> dict:
|
||||
return {
|
||||
"payer_id": row.payer_id,
|
||||
"name": row.name,
|
||||
"receiver_name": row.receiver_name,
|
||||
"receiver_id": row.receiver_id,
|
||||
"is_active": bool(row.is_active),
|
||||
"created_at": row.created_at,
|
||||
"updated_at": row.updated_at,
|
||||
}
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
"""Pydantic models for parsed-batch records.
|
||||
|
||||
``BatchRecord`` is the union type; ``BatchRecord837`` and ``BatchRecord835``
|
||||
narrow ``result`` to the parser-specific output types. Construction of
|
||||
``BatchRecord(kind="837p", ...)`` dispatches to ``BatchRecord837`` via
|
||||
``__new__`` so isinstance checks downstream narrow ``result`` correctly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, model_validator
|
||||
|
||||
from cyclone.parsers.models import ParseResult
|
||||
from cyclone.parsers.models_835 import ParseResult835
|
||||
|
||||
BatchKind = Literal["837p", "835"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# BatchRecord: value object preserved from sub-project 1.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class BatchRecord(BaseModel):
|
||||
"""One parsed file, with a stable uuid4 id and the full ParseResult.
|
||||
|
||||
``result`` is a union: ``ParseResult`` for ``kind="837p"`` and
|
||||
``ParseResult835`` for ``kind="835"``. The concrete subclasses
|
||||
``BatchRecord837`` and ``BatchRecord835`` narrow those fields, so
|
||||
callers that want type-checked access should use them and check
|
||||
``isinstance`` rather than pattern-matching on ``kind``.
|
||||
|
||||
Constructing ``BatchRecord(kind="837p", ...)`` dispatches to
|
||||
``BatchRecord837``; ``kind="835"`` dispatches to ``BatchRecord835``.
|
||||
This lets the union-member narrowing work transparently.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(extra="ignore")
|
||||
|
||||
id: str
|
||||
kind: BatchKind
|
||||
input_filename: str
|
||||
parsed_at: datetime # tz-aware UTC
|
||||
result: ParseResult | ParseResult835
|
||||
|
||||
def __new__(
|
||||
cls, *args: Any, **kwargs: Any,
|
||||
) -> BatchRecord837 | BatchRecord835 | BatchRecord:
|
||||
# Dispatch base-class construction to the right concrete subclass
|
||||
# so isinstance checks downstream narrow `result` correctly.
|
||||
if cls is BatchRecord:
|
||||
kind = kwargs.get("kind")
|
||||
if kind is None and args and isinstance(args[0], dict):
|
||||
kind = args[0].get("kind")
|
||||
if kind == "837p":
|
||||
return BatchRecord837(*args, **kwargs)
|
||||
if kind == "835":
|
||||
return BatchRecord835(*args, **kwargs)
|
||||
return super().__new__(cls)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _check_parsed_at_tz(self) -> BatchRecord:
|
||||
if self.parsed_at.tzinfo is None:
|
||||
raise ValueError(
|
||||
"parsed_at must be tz-aware (use datetime.now(timezone.utc))"
|
||||
)
|
||||
return self
|
||||
|
||||
|
||||
class BatchRecord837(BatchRecord):
|
||||
"""A parsed 837P (professional claim) batch."""
|
||||
|
||||
kind: Literal["837p"] = "837p"
|
||||
result: ParseResult
|
||||
|
||||
|
||||
class BatchRecord835(BatchRecord):
|
||||
"""A parsed 835 (remittance advice) batch."""
|
||||
|
||||
kind: Literal["835"] = "835"
|
||||
result: ParseResult835
|
||||
@@ -0,0 +1,683 @@
|
||||
"""UI serialization helpers — convert ORM rows to API-shaped dicts.
|
||||
|
||||
These are the boundary between the persistence layer and the wire format
|
||||
consumed by the React frontend. Keep them stable: any rename or
|
||||
shape change ripples to every API consumer.
|
||||
|
||||
Also hosts ``utcnow()`` (the tz-aware UTC now) because it's a small
|
||||
free function that several modules want and there's no better home.
|
||||
|
||||
SP25: ``to_ui_ack`` / ``to_ui_ta1_ack`` / ``to_ui_two77ca_ack`` were
|
||||
moved here from ``api_routers/acks.py`` / ``api_routers/ta1_acks.py``
|
||||
/ ``api.py`` so the live-tail event payload matches the list endpoint
|
||||
shape byte-for-byte. The seam between persistence and streaming
|
||||
depends on the two halves staying in sync.
|
||||
|
||||
SP28 adds ``to_ui_claim_ack`` for the same reason — the
|
||||
``claim_ack_written`` event payload must match the
|
||||
``GET /api/acks/{kind}/{id}/claims`` list shape byte-for-byte.
|
||||
|
||||
moved here from ``api_routers/acks.py`` / ``api_routers/ta1_acks.py``
|
||||
/ ``api.py`` so the live-tail event payload can match the list endpoint
|
||||
shape byte-for-byte. The seam between persistence and streaming
|
||||
depends on the two halves staying in sync.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from decimal import Decimal
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import Claim, ClaimAck, Remittance
|
||||
from cyclone.parsers.models import ClaimOutput
|
||||
from cyclone.parsers.models_835 import ClaimPayment
|
||||
from cyclone.parsers.payer import PayerConfig835
|
||||
|
||||
from .orm_builders import _claim_status_from_validation
|
||||
|
||||
|
||||
def to_ui_claim(
|
||||
claim: ClaimOutput,
|
||||
*,
|
||||
batch_id: str,
|
||||
parsed_at: datetime,
|
||||
) -> dict:
|
||||
"""Map a 837P ClaimOutput to the UI's `Claim` shape (preserved)."""
|
||||
parsed_iso = parsed_at.isoformat().replace("+00:00", "Z")
|
||||
return {
|
||||
"id": claim.claim_id,
|
||||
"patientName": f"{claim.subscriber.first_name} {claim.subscriber.last_name}".strip(),
|
||||
"providerNpi": claim.billing_provider.npi,
|
||||
"payerName": claim.payer.name,
|
||||
"cptCode": (
|
||||
claim.service_lines[0].procedure.code
|
||||
if claim.service_lines
|
||||
else ""
|
||||
),
|
||||
"billedAmount": float(claim.claim.total_charge or 0.0),
|
||||
"receivedAmount": 0.0,
|
||||
"status": _claim_status_from_validation(claim),
|
||||
"denialReason": None,
|
||||
"submissionDate": parsed_iso,
|
||||
"batchId": batch_id,
|
||||
"parsedAt": parsed_iso,
|
||||
}
|
||||
|
||||
|
||||
def to_ui_remittance(
|
||||
cp: ClaimPayment,
|
||||
*,
|
||||
batch_id: str,
|
||||
parsed_at: datetime,
|
||||
payer_config: PayerConfig835 | None = None,
|
||||
payer_name: str = "",
|
||||
) -> dict:
|
||||
"""Map an 835 ClaimPayment to the UI's `Remittance` shape (preserved)."""
|
||||
code = cp.status_code
|
||||
if code in {"21", "22"}:
|
||||
status = "reconciled"
|
||||
else:
|
||||
status = "received"
|
||||
|
||||
denial_reason: str | None = None
|
||||
if code == "4" and cp.service_payments:
|
||||
sp = cp.service_payments[0]
|
||||
if sp.adjustments:
|
||||
adj = sp.adjustments[0]
|
||||
denial_reason = (
|
||||
f"{adj.group_code}-{adj.reason_code}: ${float(adj.amount):.2f}"
|
||||
)
|
||||
|
||||
cfg = payer_config if payer_config is not None else PayerConfig835.generic_835()
|
||||
validation_warnings: list[str] = []
|
||||
if code not in cfg.allowed_status_codes:
|
||||
validation_warnings.append(
|
||||
f"CLP02 code {code} not in payer allowlist"
|
||||
)
|
||||
|
||||
# Aggregate adjustmentAmount across ALL service-line CAS rows, not just
|
||||
# the first line. Mirrors the SUM the reconcile aggregator (T10)
|
||||
# computes against persisted CasAdjustment rows; this inline version
|
||||
# is the write-path equivalent (used when streaming 835 NDJSON
|
||||
# responses before persistence finishes).
|
||||
adjustment_total = Decimal("0")
|
||||
for sp in cp.service_payments:
|
||||
for adj in sp.adjustments:
|
||||
adjustment_total += adj.amount
|
||||
|
||||
parsed_iso = parsed_at.isoformat().replace("+00:00", "Z")
|
||||
return {
|
||||
"id": cp.payer_claim_control_number,
|
||||
"claimId": cp.original_claim_id or "",
|
||||
"payerName": payer_name,
|
||||
"paidAmount": float(cp.total_paid or 0.0),
|
||||
"adjustmentAmount": float(adjustment_total),
|
||||
"status": status,
|
||||
"denialReason": denial_reason,
|
||||
"validationWarnings": validation_warnings,
|
||||
"receivedDate": parsed_iso,
|
||||
"batchId": batch_id,
|
||||
"parsedAt": parsed_iso,
|
||||
}
|
||||
|
||||
|
||||
def to_ui_claim_from_orm(
|
||||
row: Claim,
|
||||
*,
|
||||
batch_id: str,
|
||||
parsed_at: datetime,
|
||||
received_total: float = 0.0,
|
||||
) -> dict:
|
||||
"""Map an ORM ``Claim`` row to the UI's claim shape.
|
||||
|
||||
``to_ui_claim`` takes a Pydantic ``ClaimOutput`` (used on the write path
|
||||
during 837 ingest). For read paths — list_unmatched, manual_match return
|
||||
values — we already have the ORM row and the serialized fields it
|
||||
carries in ``raw_json``. Reading from ``raw_json`` keeps the UI shape
|
||||
in sync with the original 837 parse without re-deserializing to a
|
||||
Pydantic model.
|
||||
|
||||
Adds two fields ``to_ui_claim`` doesn't emit: ``state`` (the
|
||||
reconciliation state machine value) and ``matchedRemittanceId`` (the
|
||||
FK to the paired remittance, or None). Both are required by the UI.
|
||||
"""
|
||||
raw = row.raw_json or {}
|
||||
bp = raw.get("billing_provider", {})
|
||||
payer_obj = raw.get("payer", {})
|
||||
sub = raw.get("subscriber", {})
|
||||
service_lines = raw.get("service_lines", [])
|
||||
parsed_iso = parsed_at.isoformat().replace("+00:00", "Z")
|
||||
cpt = (
|
||||
service_lines[0].get("procedure", {}).get("code", "")
|
||||
if service_lines
|
||||
else ""
|
||||
)
|
||||
state_value = (
|
||||
row.state.value if hasattr(row.state, "value") else str(row.state)
|
||||
)
|
||||
return {
|
||||
"id": row.id,
|
||||
"state": state_value,
|
||||
"billedAmount": float(row.charge_amount or 0),
|
||||
"patientName": (
|
||||
f"{sub.get('first_name', '')} {sub.get('last_name', '')}".strip()
|
||||
),
|
||||
"providerNpi": bp.get("npi") or row.provider_npi or "",
|
||||
"payerName": payer_obj.get("name") or "",
|
||||
"cptCode": cpt,
|
||||
"submissionDate": parsed_iso,
|
||||
"parsedAt": parsed_iso,
|
||||
"status": state_value,
|
||||
"matchedRemittanceId": row.matched_remittance_id,
|
||||
"batchId": batch_id,
|
||||
# Parity with ``to_ui_claim``'s shape — the UI tolerates extra keys
|
||||
# but expects these on freshly-loaded rows from /api/claims too.
|
||||
# ``received_total`` comes from the matched Remittance row when one
|
||||
# exists; callers that don't pre-compute it (write path, unmatched
|
||||
# list) get the default of 0.0 — which matches the unmapped state.
|
||||
"receivedAmount": float(received_total),
|
||||
"denialReason": None,
|
||||
}
|
||||
|
||||
|
||||
# Max number of ActivityEvent rows surfaced in the detail drawer's
|
||||
# state history. The spec caps it at 50; a higher claim volume (manual
|
||||
# match/unmatch thrash) just shows the 50 most recent. Exposed as a
|
||||
# module constant so the endpoint layer can pass it through as a
|
||||
# default if it ever supports a `?limit=N` query param.
|
||||
CLAIM_DETAIL_HISTORY_LIMIT = 50
|
||||
|
||||
|
||||
def _iso_z(value: datetime | None) -> str:
|
||||
"""Format a tz-aware-or-naive UTC datetime as ISO-8601 with trailing Z.
|
||||
|
||||
The DB columns are declared ``DateTime(timezone=True)`` and rows are
|
||||
stored UTC at write time, but SQLite drops the tzinfo on read
|
||||
(returning a naive ``datetime``). Re-attach UTC for naive values
|
||||
so the spec contract holds: every ISO datetime field ends in Z.
|
||||
"""
|
||||
if value is None:
|
||||
return ""
|
||||
if value.tzinfo is None:
|
||||
value = value.replace(tzinfo=timezone.utc)
|
||||
return value.isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def to_ui_ack(row: db.Ack) -> dict:
|
||||
"""Map an ``Ack`` ORM row to the UI shape used by ``/api/acks``.
|
||||
|
||||
Field names match the rest of the Cyclone API (snake_case). The
|
||||
frontend ``useAcks`` hook re-shapes this to the camelCase ``Ack``
|
||||
interface in ``src/types/index.ts``.
|
||||
|
||||
Adds ``patient_control_number`` pulled from ``raw_json`` so the
|
||||
operator can correlate each 999 back to the original claim batch.
|
||||
|
||||
SP25: this was previously ``api_routers/acks._ack_to_ui``. Moved
|
||||
here so the live-tail event payload (``ack_received``) matches the
|
||||
list-endpoint shape — the seam between persistence and streaming
|
||||
depends on byte-for-byte equality.
|
||||
"""
|
||||
body = {
|
||||
"id": row.id,
|
||||
"source_batch_id": row.source_batch_id,
|
||||
"accepted_count": row.accepted_count,
|
||||
"rejected_count": row.rejected_count,
|
||||
"received_count": row.received_count,
|
||||
"ack_code": row.ack_code,
|
||||
"parsed_at": (
|
||||
row.parsed_at.isoformat().replace("+00:00", "Z")
|
||||
if row.parsed_at is not None
|
||||
else ""
|
||||
),
|
||||
"patient_control_number": None,
|
||||
}
|
||||
raw = row.raw_json or {}
|
||||
try:
|
||||
set_responses = raw.get("set_responses") or []
|
||||
if set_responses:
|
||||
body["patient_control_number"] = set_responses[0].get("set_control_number")
|
||||
except (AttributeError, TypeError):
|
||||
pass
|
||||
return body
|
||||
|
||||
|
||||
def to_ui_ta1_ack(row: db.Ta1Ack) -> dict:
|
||||
"""Map a ``Ta1Ack`` ORM row to the UI shape used by ``/api/ta1-acks``.
|
||||
|
||||
SP25: this was previously ``api_routers/ta1_acks._ta1_to_ui``.
|
||||
Moved here so the live-tail event payload (``ta1_ack_received``)
|
||||
matches the list-endpoint shape byte-for-byte.
|
||||
|
||||
Note: ``parsed_at`` uses naive ``isoformat()`` (no ``Z`` suffix) —
|
||||
preserved from the original implementation. SQLite strips tzinfo
|
||||
on read, so the field comes back as a naive datetime; we deliberately
|
||||
keep the original (non-``Z``) rendering rather than re-attaching
|
||||
UTC, because changing the wire shape now would drift both halves
|
||||
of the live-tail contract.
|
||||
"""
|
||||
return {
|
||||
"id": row.id,
|
||||
"control_number": row.control_number,
|
||||
"ack_code": row.ack_code,
|
||||
"note_code": row.note_code,
|
||||
"interchange_date": row.interchange_date.isoformat()
|
||||
if row.interchange_date else None,
|
||||
"interchange_time": row.interchange_time,
|
||||
"sender_id": row.sender_id,
|
||||
"receiver_id": row.receiver_id,
|
||||
"source_batch_id": row.source_batch_id,
|
||||
"parsed_at": row.parsed_at.isoformat() if row.parsed_at else None,
|
||||
}
|
||||
|
||||
|
||||
def to_ui_two77ca_ack(row: db.Two77caAck) -> dict:
|
||||
"""Map a ``Two77caAck`` ORM row to the UI shape used by ``/api/277ca-acks``.
|
||||
|
||||
SP25: this was previously ``api._277ca_to_ui`` inlined alongside
|
||||
the 277CA list endpoint. Moved here so the live-tail event
|
||||
payload (``two77ca_ack_received``) matches the list-endpoint
|
||||
shape byte-for-byte.
|
||||
|
||||
Note: ``parsed_at`` uses naive ``isoformat()`` (no ``Z`` suffix) —
|
||||
same SQLite-tzinfo caveat as :func:`to_ui_ta1_ack`.
|
||||
"""
|
||||
return {
|
||||
"id": row.id,
|
||||
"control_number": row.control_number,
|
||||
"accepted_count": row.accepted_count,
|
||||
"rejected_count": row.rejected_count,
|
||||
"paid_count": row.paid_count,
|
||||
"pended_count": row.pended_count,
|
||||
"source_batch_id": row.source_batch_id,
|
||||
"parsed_at": row.parsed_at.isoformat() if row.parsed_at else None,
|
||||
}
|
||||
|
||||
|
||||
def _address_to_ui(addr: dict | None) -> dict:
|
||||
"""Render a raw ``Address`` dict in the spec's parties address shape.
|
||||
|
||||
Returns an empty dict when the source is missing so the UI can
|
||||
branch on the field's presence rather than the value. The spec
|
||||
shape is ``{line1, line2|null, city, state, zip}``.
|
||||
"""
|
||||
if not addr:
|
||||
return {}
|
||||
return {
|
||||
"line1": addr.get("line1") or "",
|
||||
"line2": addr.get("line2"),
|
||||
"city": addr.get("city") or "",
|
||||
"state": addr.get("state") or "",
|
||||
"zip": addr.get("zip") or "",
|
||||
}
|
||||
|
||||
|
||||
def _validation_issues_to_ui(issues: list[dict] | None) -> list[dict]:
|
||||
"""Project ValidationIssue dicts onto the spec's per-issue shape.
|
||||
|
||||
Source includes ``segment_index`` (a parser debug aid) which the
|
||||
spec doesn't surface; we drop it. The endpoint contract is
|
||||
``{rule, severity, message}`` per issue.
|
||||
"""
|
||||
if not issues:
|
||||
return []
|
||||
return [
|
||||
{
|
||||
"rule": issue.get("rule", ""),
|
||||
"severity": issue.get("severity", "error"),
|
||||
"message": issue.get("message", ""),
|
||||
}
|
||||
for issue in issues
|
||||
]
|
||||
|
||||
|
||||
def to_ui_claim_detail(
|
||||
row: Claim,
|
||||
*,
|
||||
batch_id: str,
|
||||
parsed_at: datetime,
|
||||
) -> dict:
|
||||
"""Map an ORM ``Claim`` row to the SP4 detail-drawer UI shape.
|
||||
|
||||
A superset of :func:`to_ui_claim_from_orm`: same top-level identity
|
||||
fields, plus the full parties / validation / service-lines /
|
||||
diagnoses / raw-segments / service-date / state-label payload that
|
||||
the drawer needs. ``matchedRemittance`` and ``stateHistory`` are
|
||||
*not* filled in here — they require extra queries and are stitched
|
||||
in by :meth:`CycloneStore.get_claim_detail`.
|
||||
|
||||
The mapper is deliberately a pure function (no DB I/O) so the
|
||||
endpoint layer can call it from a worker thread or swap the
|
||||
history/remittance sources for tests without re-implementing the
|
||||
body.
|
||||
"""
|
||||
raw = row.raw_json or {}
|
||||
bp = raw.get("billing_provider", {}) or {}
|
||||
payer_obj = raw.get("payer", {}) or {}
|
||||
sub = raw.get("subscriber", {}) or {}
|
||||
service_lines = raw.get("service_lines", []) or []
|
||||
diagnoses = raw.get("diagnoses", []) or []
|
||||
validation = raw.get("validation", {}) or {}
|
||||
raw_segments = raw.get("raw_segments", []) or []
|
||||
|
||||
parsed_iso = parsed_at.isoformat().replace("+00:00", "Z")
|
||||
state_value = (
|
||||
row.state.value if hasattr(row.state, "value") else str(row.state)
|
||||
)
|
||||
|
||||
# Service dates come from the dedicated ORM columns (denormalized at
|
||||
# ingest in _claim_837_row so the list views can sort/filter on
|
||||
# them without a JSON parse). ``isoformat()`` on a ``date`` gives
|
||||
# ``YYYY-MM-DD`` — the spec shape.
|
||||
service_date_from_iso = (
|
||||
row.service_date_from.isoformat() if row.service_date_from else None
|
||||
)
|
||||
service_date_to_iso = (
|
||||
row.service_date_to.isoformat() if row.service_date_to else None
|
||||
)
|
||||
|
||||
return {
|
||||
# -- identity + state -----------------------------------------
|
||||
"id": row.id,
|
||||
"batchId": batch_id,
|
||||
"state": state_value,
|
||||
"stateLabel": state_value.capitalize(),
|
||||
# -- money + dates --------------------------------------------
|
||||
"billedAmount": float(row.charge_amount or 0),
|
||||
"serviceDateFrom": service_date_from_iso,
|
||||
"serviceDateTo": service_date_to_iso,
|
||||
"submissionDate": parsed_iso,
|
||||
"parsedAt": parsed_iso,
|
||||
# -- patient / provider / payer -------------------------------
|
||||
"patientName": (
|
||||
f"{sub.get('first_name', '')} {sub.get('last_name', '')}".strip()
|
||||
),
|
||||
"providerNpi": bp.get("npi") or row.provider_npi or "",
|
||||
"providerName": bp.get("name") or "",
|
||||
"payerName": payer_obj.get("name") or "",
|
||||
"payerId": payer_obj.get("id") or row.payer_id or "",
|
||||
# -- diagnoses ------------------------------------------------
|
||||
"diagnoses": [
|
||||
{
|
||||
"code": d.get("code", ""),
|
||||
"qualifier": d.get("qualifier"),
|
||||
}
|
||||
for d in diagnoses
|
||||
],
|
||||
# -- service lines --------------------------------------------
|
||||
# ``service_lines[i].procedure`` is a nested dict in the
|
||||
# serialized raw_json; the spec flattens it into the line.
|
||||
# ``charge`` and ``units`` are stored as Decimal via Pydantic
|
||||
# and serialized to string — coerce defensively. ``modifiers``
|
||||
# defaults to [] so the UI doesn't have to handle null.
|
||||
"serviceLines": [
|
||||
{
|
||||
"lineNumber": sl.get("line_number"),
|
||||
"procedureQualifier": (
|
||||
sl.get("procedure", {}).get("qualifier", "") or ""
|
||||
),
|
||||
"procedureCode": (
|
||||
sl.get("procedure", {}).get("code", "") or ""
|
||||
),
|
||||
"modifiers": list(
|
||||
sl.get("procedure", {}).get("modifiers") or []
|
||||
),
|
||||
"charge": float(sl.get("charge") or 0),
|
||||
"units": (
|
||||
float(sl["units"])
|
||||
if sl.get("units") is not None
|
||||
else None
|
||||
),
|
||||
"unitType": sl.get("unit_type"),
|
||||
"serviceDate": sl.get("service_date"),
|
||||
}
|
||||
for sl in service_lines
|
||||
],
|
||||
# -- parties --------------------------------------------------
|
||||
"parties": {
|
||||
"billingProvider": {
|
||||
"name": bp.get("name") or "",
|
||||
"npi": bp.get("npi") or "",
|
||||
"taxId": bp.get("tax_id") or "",
|
||||
"address": _address_to_ui(bp.get("address")),
|
||||
},
|
||||
"subscriber": {
|
||||
"firstName": sub.get("first_name") or "",
|
||||
"lastName": sub.get("last_name") or "",
|
||||
"memberId": sub.get("member_id") or "",
|
||||
"dob": sub.get("dob"),
|
||||
"gender": sub.get("gender"),
|
||||
},
|
||||
"payer": {
|
||||
"name": payer_obj.get("name") or "",
|
||||
"id": payer_obj.get("id") or "",
|
||||
},
|
||||
},
|
||||
# -- validation ----------------------------------------------
|
||||
"validation": {
|
||||
"passed": bool(validation.get("passed", True)),
|
||||
"errors": _validation_issues_to_ui(validation.get("errors")),
|
||||
"warnings": _validation_issues_to_ui(validation.get("warnings")),
|
||||
},
|
||||
# -- raw segments (debug aid) --------------------------------
|
||||
"rawSegments": raw_segments,
|
||||
# -- matched remittance (filled by get_claim_detail) ---------
|
||||
"matchedRemittance": None,
|
||||
# -- state history (filled by get_claim_detail) --------------
|
||||
"stateHistory": [],
|
||||
}
|
||||
|
||||
|
||||
def to_ui_remittance_from_orm(
|
||||
row: Remittance,
|
||||
*,
|
||||
batch_id: str,
|
||||
parsed_at: datetime,
|
||||
) -> dict:
|
||||
"""Map an ORM ``Remittance`` row to the UI's remittance shape.
|
||||
|
||||
Same idea as ``to_ui_claim_from_orm``: read the PayerName from the
|
||||
parent batch's ``raw_result_json`` (the ParseResult835 stashed at
|
||||
insert time) since ``Remittance`` itself doesn't carry it.
|
||||
"""
|
||||
parsed_iso = parsed_at.isoformat().replace("+00:00", "Z")
|
||||
payer_name = ""
|
||||
if row.batch is not None and row.batch.raw_result_json:
|
||||
payer_obj = row.batch.raw_result_json.get("payer", {}) or {}
|
||||
payer_name = payer_obj.get("name") or ""
|
||||
status = (
|
||||
"reconciled" if row.status_code in ("21", "22") else "received"
|
||||
)
|
||||
return {
|
||||
"id": row.id,
|
||||
"payerClaimControlNumber": row.payer_claim_control_number,
|
||||
"claimId": row.claim_id or "",
|
||||
"payerName": payer_name,
|
||||
"paidAmount": float(row.total_paid or 0),
|
||||
"adjustmentAmount": float(row.adjustment_amount or 0),
|
||||
"status": status,
|
||||
"denialReason": None,
|
||||
"validationWarnings": [],
|
||||
"receivedDate": row.received_at.isoformat().replace("+00:00", "Z"),
|
||||
"batchId": batch_id,
|
||||
"parsedAt": parsed_iso,
|
||||
}
|
||||
|
||||
|
||||
def to_ui_remittance_with_adjustments(
|
||||
row: Remittance,
|
||||
*,
|
||||
batch_id: str,
|
||||
parsed_at: datetime,
|
||||
cas_rows: list["db.CasAdjustment"] | None = None,
|
||||
) -> dict:
|
||||
"""Same shape as :func:`to_ui_remittance_from_orm` plus an ``adjustments`` array.
|
||||
|
||||
Each persisted ``CasAdjustment`` row is rendered as
|
||||
``{"group", "reason", "label", "amount", "quantity"}``. The ``label``
|
||||
is resolved through :func:`cyclone.parsers.cas_codes.reason_label`
|
||||
so the UI does not have to ship its own CARC dictionary.
|
||||
|
||||
``cas_rows`` is optional so callers that don't have the rows handy
|
||||
can still get the base dict; in that case ``adjustments`` is ``[]``.
|
||||
Pass ``cas_rows`` to avoid an extra round-trip; the endpoint at
|
||||
``GET /api/remittances/{id}`` passes them in to keep this mapper a
|
||||
pure function.
|
||||
"""
|
||||
base = to_ui_remittance_from_orm(
|
||||
row, batch_id=batch_id, parsed_at=parsed_at,
|
||||
)
|
||||
if not cas_rows:
|
||||
base["adjustments"] = []
|
||||
return base
|
||||
|
||||
# Lazy import to avoid the circular store ↔ parsers import that
|
||||
# happens on cold start; mirrors the same pattern used elsewhere
|
||||
# in this module.
|
||||
from cyclone.parsers.cas_codes import reason_label
|
||||
|
||||
base["adjustments"] = [
|
||||
{
|
||||
"group": c.group_code,
|
||||
"reason": c.reason_code,
|
||||
"label": reason_label(c.group_code, c.reason_code),
|
||||
"amount": float(c.amount or 0),
|
||||
"quantity": (
|
||||
float(c.quantity) if c.quantity is not None else None
|
||||
),
|
||||
}
|
||||
for c in cas_rows
|
||||
]
|
||||
return base
|
||||
|
||||
|
||||
def _svc_to_wire_dict(svc) -> dict:
|
||||
"""Project an ORM ``ServiceLinePayment`` to the wire format used by
|
||||
the remit drawer's ``serviceLinePayments`` array.
|
||||
|
||||
Mirrors the shape produced by the line-reconciliation endpoint so
|
||||
the UI can render the same components from either source.
|
||||
"""
|
||||
import json as _json
|
||||
return {
|
||||
"id": svc.id,
|
||||
"line_number": svc.line_number,
|
||||
"procedure_qualifier": svc.procedure_qualifier,
|
||||
"procedure_code": svc.procedure_code,
|
||||
"modifiers": _json.loads(svc.modifiers_json or "[]"),
|
||||
"charge": str(Decimal(str(svc.charge))),
|
||||
"payment": str(Decimal(str(svc.payment))),
|
||||
"units": str(Decimal(str(svc.units))) if svc.units is not None else None,
|
||||
"unit_type": svc.unit_type,
|
||||
"service_date": svc.service_date.isoformat() if svc.service_date else None,
|
||||
}
|
||||
|
||||
|
||||
def to_ui_provider(
|
||||
*,
|
||||
npi: str,
|
||||
name: str,
|
||||
tax_id: str | None = None,
|
||||
address: str | None = None,
|
||||
city: str | None = None,
|
||||
state: str | None = None,
|
||||
zip: str | None = None,
|
||||
phone: str | None = None,
|
||||
claim_count: int = 0,
|
||||
outstanding_ar: float = 0.0,
|
||||
) -> dict:
|
||||
return {
|
||||
"npi": npi,
|
||||
"name": name,
|
||||
"taxId": tax_id or "",
|
||||
"address": address or "",
|
||||
"city": city or "",
|
||||
"state": state or "",
|
||||
"zip": zip or "",
|
||||
"phone": phone or "",
|
||||
"claimCount": claim_count,
|
||||
"outstandingAr": float(outstanding_ar),
|
||||
}
|
||||
|
||||
|
||||
def to_activity_event(
|
||||
*,
|
||||
id: str,
|
||||
kind: str,
|
||||
message: str,
|
||||
timestamp: datetime,
|
||||
npi: str | None = None,
|
||||
amount: float | None = None,
|
||||
) -> dict:
|
||||
return {
|
||||
"id": id,
|
||||
"kind": kind,
|
||||
"message": message,
|
||||
"timestamp": timestamp.isoformat().replace("+00:00", "Z"),
|
||||
"npi": npi,
|
||||
"amount": amount,
|
||||
}
|
||||
|
||||
|
||||
def _date_in_bounds(
|
||||
item: dict,
|
||||
field: str,
|
||||
date_from: str | None,
|
||||
date_to: str | None,
|
||||
) -> bool:
|
||||
"""True if ``item[field]`` falls within ``[date_from, date_to]``."""
|
||||
val = item.get(field)
|
||||
if val is None:
|
||||
return date_from is None and date_to is None
|
||||
date_part = val[:10]
|
||||
if date_from is not None and date_part < date_from:
|
||||
return False
|
||||
if date_to is not None and date_part > date_to:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def to_ui_claim_ack(row: ClaimAck) -> dict:
|
||||
"""SP28: map a ClaimAck ORM row to the API shape.
|
||||
|
||||
Mirrors the rest of the to_ui_* serializers. The wire shape is
|
||||
identical between the matching list endpoint (``GET /api/acks/{kind}/
|
||||
{id}/claims``) and the ``claim_ack_written`` pubsub event so the
|
||||
live-tail subscribers can rehydrate the snapshot from the bus
|
||||
without diverging from the API response.
|
||||
|
||||
``claim_state`` is queried via a session round-trip
|
||||
(``SELECT state FROM claims WHERE id = :claim_id``) so the drawer
|
||||
panel can render the colored ClaimStateBadge inline. For TA1 rows
|
||||
with ``claim_id IS NULL`` (batch-level envelope link),
|
||||
``claim_state`` is ``"n/a"``.
|
||||
"""
|
||||
claim_state: str = "n/a"
|
||||
if row.claim_id:
|
||||
with db.SessionLocal()() as lookup_s:
|
||||
crow = lookup_s.get(Claim, row.claim_id)
|
||||
if crow is not None:
|
||||
claim_state = (
|
||||
crow.state.value
|
||||
if hasattr(crow.state, "value")
|
||||
else str(crow.state)
|
||||
)
|
||||
linked_iso = (
|
||||
row.linked_at.isoformat().replace("+00:00", "Z")
|
||||
if row.linked_at is not None
|
||||
else ""
|
||||
)
|
||||
return {
|
||||
"id": row.id,
|
||||
"claim_id": row.claim_id,
|
||||
"batch_id": row.batch_id,
|
||||
"ack_id": row.ack_id,
|
||||
"ack_kind": row.ack_kind,
|
||||
"ak2_index": row.ak2_index,
|
||||
"set_control_number": row.set_control_number,
|
||||
"set_accept_reject_code": row.set_accept_reject_code,
|
||||
"linked_at": linked_iso,
|
||||
"linked_by": row.linked_by,
|
||||
"claim_state": claim_state,
|
||||
}
|
||||
@@ -0,0 +1,241 @@
|
||||
"""Write path for parsed batches — insert, reconcile, publish.
|
||||
|
||||
The single entry point is ``add_record(record, *, event_bus=None)``,
|
||||
which replaces the body of the previous ``CycloneStore.add`` method.
|
||||
It owns its own SQLAlchemy session, runs idempotency checks, persists
|
||||
the batch + child rows, then (for 835 batches) triggers reconciliation
|
||||
and (if ``event_bus`` is provided) publishes live-tail events.
|
||||
|
||||
Reconciliation runs OUTSIDE the persistence session, fail-soft: errors
|
||||
are logged but do not roll back the persisted batch.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import (
|
||||
ActivityEvent,
|
||||
Batch,
|
||||
Claim,
|
||||
Remittance,
|
||||
)
|
||||
from cyclone.parsers.models import ParseResult
|
||||
from cyclone.parsers.models_835 import ParseResult835
|
||||
from .orm_builders import _claim_837_row, _persist_835_remit, _remittance_835_row
|
||||
from .records import BatchRecord, BatchRecord837, BatchRecord835
|
||||
from .ui import to_ui_claim_from_orm, to_ui_remittance_from_orm
|
||||
|
||||
|
||||
def add_record(record: BatchRecord, *, event_bus=None) -> None:
|
||||
"""Persist a parsed batch (837P or 835) to the DB.
|
||||
|
||||
For 837P batches: inserts the Batch row, one Claim row per
|
||||
claim, and a ``claim_submitted`` ActivityEvent per claim.
|
||||
|
||||
For 835 batches: inserts the Batch row, one Remittance row per
|
||||
ClaimPayment, and a ``remit_received`` ActivityEvent per
|
||||
ClaimPayment. Reconciliation (auto-match + per-pair CAS
|
||||
aggregate) runs IN THE SAME SESSION before commit (SP27
|
||||
Task 10) — a single ``s.commit()`` covers both ingest and
|
||||
reconciliation. If reconcile raises, the whole 835 ingest
|
||||
rolls back; the batch never appears half-reconciled with
|
||||
placeholder ``adjustment_amount`` values.
|
||||
|
||||
Idempotency: ``Claim.id`` and ``Remittance.id`` are PRIMARY KEYS,
|
||||
so a re-ingest of the same fixture (e.g. ``/api/parse-837`` called
|
||||
twice with the same file) would otherwise raise
|
||||
``IntegrityError``. We do a per-row ``session.get(...)`` check
|
||||
before each insert; if the row already exists, we log a warning
|
||||
and skip. The batch row itself is still inserted (each parse
|
||||
has a fresh ``uuid4`` id from the API). O(n) per row, but
|
||||
acceptable for the small fixture sizes — production load is
|
||||
one batch at a time via the API, not bulk inserts.
|
||||
|
||||
When ``event_bus`` is provided, publishes one ``claim_written``
|
||||
or ``remittance_written`` event per newly-inserted row plus an
|
||||
``activity_recorded`` event per activity row, after commit. The
|
||||
publish calls are best-effort — failures are logged but do not
|
||||
roll back the persisted batch.
|
||||
"""
|
||||
from cyclone.pubsub import EventBus
|
||||
|
||||
import logging
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# Track rows we actually inserted so we can publish events for them.
|
||||
inserted_claim_ids: list[str] = []
|
||||
inserted_remit_ids: list[str] = []
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
batch_row = Batch(
|
||||
id=record.id,
|
||||
kind=record.kind,
|
||||
input_filename=record.input_filename,
|
||||
parsed_at=record.parsed_at,
|
||||
totals_json=None,
|
||||
validation_json=None,
|
||||
raw_result_json=json.loads(record.result.model_dump_json()),
|
||||
)
|
||||
s.add(batch_row)
|
||||
|
||||
if isinstance(record, BatchRecord837):
|
||||
result: ParseResult = record.result
|
||||
for claim in result.claims:
|
||||
if s.get(Claim, claim.claim_id) is not None:
|
||||
log.warning(
|
||||
"add: claim %s already exists; skipping (batch=%s)",
|
||||
claim.claim_id, record.id,
|
||||
)
|
||||
continue
|
||||
s.add(_claim_837_row(claim, record.id))
|
||||
s.add(ActivityEvent(
|
||||
ts=record.parsed_at,
|
||||
kind="claim_submitted",
|
||||
batch_id=record.id,
|
||||
claim_id=claim.claim_id,
|
||||
payload_json={
|
||||
"message": (
|
||||
f"Claim {claim.claim_id} submitted · "
|
||||
f"{claim.payer.name}"
|
||||
),
|
||||
"npi": claim.billing_provider.npi,
|
||||
"amount": float(claim.claim.total_charge or 0.0),
|
||||
},
|
||||
))
|
||||
inserted_claim_ids.append(claim.claim_id)
|
||||
elif isinstance(record, BatchRecord835):
|
||||
result835: ParseResult835 = record.result
|
||||
payer_name = result835.payer.name
|
||||
for cp in result835.claims:
|
||||
if s.get(Remittance, cp.payer_claim_control_number) is not None:
|
||||
log.warning(
|
||||
"add: remittance %s already exists; skipping (batch=%s)",
|
||||
cp.payer_claim_control_number, record.id,
|
||||
)
|
||||
continue
|
||||
remit_row = _remittance_835_row(cp, record.id)
|
||||
s.add(remit_row)
|
||||
# Flush so remit_row.id (FK target of cas_adjustments) is
|
||||
# populated. SQLAlchemy assigns the PK on flush; without
|
||||
# this the CasAdjustment inserts below would reference an
|
||||
# unset id and violate the FK.
|
||||
s.flush()
|
||||
# SP7: persist per-line ServiceLinePayment + linked
|
||||
# SVC-level CAS rows + claim-level CAS bucket. Replaces
|
||||
# the previous per-SVC CAS insert loop so the
|
||||
# service_line_payment_id FK is set correctly.
|
||||
_persist_835_remit(s, cp, remit_row.id)
|
||||
s.add(ActivityEvent(
|
||||
ts=record.parsed_at,
|
||||
kind="remit_received",
|
||||
batch_id=record.id,
|
||||
remittance_id=cp.payer_claim_control_number,
|
||||
payload_json={
|
||||
"message": (
|
||||
f"Remit {cp.payer_claim_control_number} received"
|
||||
),
|
||||
"payerName": payer_name,
|
||||
"amount": float(cp.total_paid or 0.0),
|
||||
},
|
||||
))
|
||||
inserted_remit_ids.append(cp.payer_claim_control_number)
|
||||
else:
|
||||
raise TypeError(
|
||||
f"Unsupported BatchRecord subclass: {type(record).__name__}"
|
||||
)
|
||||
|
||||
# SP27 Task 10: run reconcile INSIDE the same session, before
|
||||
# commit. The placeholder ``adjustment_amount`` set by
|
||||
# ``_remittance_835_row`` is overwritten by reconcile's
|
||||
# ``_reconcile_pair`` SUM-of-CAS-aggregate pass before the
|
||||
# rows are ever visible. If reconcile raises, the whole
|
||||
# 835 ingest rolls back and the batch never lands.
|
||||
if record.kind == "835":
|
||||
from cyclone import reconcile as _reconcile
|
||||
_reconcile.run(s, record.id)
|
||||
|
||||
s.commit()
|
||||
|
||||
# Publish live-tail events synchronously. EventBus.publish is async
|
||||
# but its body is purely synchronous ``put_nowait`` enqueues; we
|
||||
# bypass the async wrapper and call the internal enqueue directly
|
||||
# so callers (sync FastAPI endpoints, sync test harnesses) don't
|
||||
# need to await.
|
||||
if event_bus is not None and (inserted_claim_ids or inserted_remit_ids):
|
||||
publish_events_sync(
|
||||
event_bus, record, inserted_claim_ids, inserted_remit_ids,
|
||||
)
|
||||
|
||||
|
||||
def publish_events_sync(
|
||||
event_bus,
|
||||
record: BatchRecord,
|
||||
claim_ids: list[str],
|
||||
remit_ids: list[str],
|
||||
) -> None:
|
||||
"""Build UI-shaped payloads for newly-inserted rows and publish.
|
||||
|
||||
Runs after commit so subscribers can immediately re-fetch from
|
||||
the API and see consistent data. Each ``claim_written`` /
|
||||
``remittance_written`` payload is identical to what the matching
|
||||
list endpoint would return for that row.
|
||||
|
||||
This is sync because EventBus's enqueue path is sync; we don't
|
||||
need a coroutine for ``put_nowait``.
|
||||
"""
|
||||
from cyclone.pubsub import EventBus
|
||||
|
||||
import logging
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
try:
|
||||
with db.SessionLocal()() as s:
|
||||
for cid in claim_ids:
|
||||
row = s.get(Claim, cid)
|
||||
if row is None:
|
||||
continue
|
||||
ui = to_ui_claim_from_orm(
|
||||
row, batch_id=row.batch_id or record.id,
|
||||
parsed_at=record.parsed_at,
|
||||
# Fresh ingest — no remittance has been paired yet,
|
||||
# so ``Received`` is necessarily 0.
|
||||
received_total=0.0,
|
||||
)
|
||||
_sync_publish(event_bus, "claim_written", ui)
|
||||
for rid in remit_ids:
|
||||
row = s.get(Remittance, rid)
|
||||
if row is None:
|
||||
continue
|
||||
ui = to_ui_remittance_from_orm(
|
||||
row, batch_id=row.batch_id or record.id,
|
||||
parsed_at=record.parsed_at,
|
||||
)
|
||||
_sync_publish(event_bus, "remittance_written", ui)
|
||||
# Activity events for this batch.
|
||||
from sqlalchemy import select
|
||||
activity_rows = s.execute(
|
||||
select(ActivityEvent).where(ActivityEvent.batch_id == record.id)
|
||||
).scalars().all()
|
||||
for arow in activity_rows:
|
||||
ui = {
|
||||
"kind": arow.kind,
|
||||
"ts": arow.ts.isoformat().replace("+00:00", "Z"),
|
||||
"batchId": arow.batch_id,
|
||||
"claimId": arow.claim_id,
|
||||
"remittanceId": arow.remittance_id,
|
||||
"payload": arow.payload_json,
|
||||
}
|
||||
_sync_publish(event_bus, "activity_recorded", ui)
|
||||
except Exception:
|
||||
log.exception("add: event publish failed for batch %s", record.id)
|
||||
|
||||
|
||||
def _sync_publish(event_bus, kind: str, payload: dict) -> None:
|
||||
"""Synchronous fan-out helper. Mirrors ``EventBus.publish`` but
|
||||
bypasses the async wrapper so callers don't need an event loop.
|
||||
"""
|
||||
event = {**payload, "_kind": kind}
|
||||
for queue in list(event_bus._subscribers.get(kind, ())):
|
||||
event_bus._enqueue_or_drop_oldest(queue, event)
|
||||
+165
-1
@@ -14,6 +14,9 @@ test_api_parse_persists.py) working unchanged.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from decimal import Decimal
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@@ -49,9 +52,170 @@ def _auto_init_db(tmp_path, monkeypatch):
|
||||
from cyclone import api as _api_mod
|
||||
_api_mod.app.state.event_bus = EventBus()
|
||||
deps.AUTH_DISABLED = True
|
||||
# The rate-limit middleware keeps a per-IP sliding window in
|
||||
# ``_buckets``. Without a reset between tests, later tests in a
|
||||
# full-suite run get ``429 Too Many Requests`` once the testclient
|
||||
# IP exhausts its 300 req/60s budget. Walk the middleware stack
|
||||
# and clear the buckets so every test starts with a fresh window.
|
||||
# Trigger the stack build with a cheap health probe (the only
|
||||
# request exempt from the limiter — see RateLimitMiddleware.EXEMPT_PATHS).
|
||||
_reset_rate_limit_buckets(_api_mod.app)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
deps.AUTH_DISABLED = False
|
||||
_api_mod.app.state.event_bus = None
|
||||
db._reset_for_tests()
|
||||
db._reset_for_tests()
|
||||
|
||||
|
||||
def _reset_rate_limit_buckets(app) -> None:
|
||||
"""Clear the rate-limit middleware's per-IP sliding-window buckets.
|
||||
|
||||
SP27 Task 13b follow-up: ``RateLimitMiddleware._buckets`` is shared
|
||||
across all tests in a process (TestClient reuses the same ``app``
|
||||
instance), so without a reset between tests the full suite trips
|
||||
the limiter after ~300 requests and later tests get 429s.
|
||||
|
||||
The middleware stack is only built on the first request, so we
|
||||
prime it with an exempt health probe before walking to the
|
||||
RateLimit layer. If the stack ever stops being a single chain
|
||||
of ``.app`` links, this helper raises AttributeError — better
|
||||
to fail loudly than silently leak state.
|
||||
"""
|
||||
from fastapi.testclient import TestClient
|
||||
TestClient(app).get("/api/health")
|
||||
cur = app.middleware_stack
|
||||
while cur is not None:
|
||||
if hasattr(cur, "_buckets"):
|
||||
cur._buckets.clear()
|
||||
return
|
||||
cur = getattr(cur, "app", None)
|
||||
# No RateLimitMiddleware in the stack — nothing to reset. Should
|
||||
# not happen in this codebase (security.py registers it at boot)
|
||||
# but we don't want a missing reset to crash unrelated tests.
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP31: shared DB-session + Claim/Remit factory fixtures.
|
||||
#
|
||||
# `db_session` yields a fresh session per test against the per-test DB set
|
||||
# up by the autouse `_auto_init_db` fixture above. The session rolls back at
|
||||
# teardown so a test that mutates rows doesn't leak into siblings.
|
||||
#
|
||||
# `make_claim` / `make_remit` build ORM rows with a small ergonomic surface:
|
||||
# the planned parameter names follow the content-keys helper (PCN, charge,
|
||||
# rendering NPI), but the ORM attribute names differ (`charge_amount` on
|
||||
# Claim, `total_charge` on Remittance, `provider_npi` on Claim — and no
|
||||
# `payer_id` / `rendering_provider_npi` column on Remittance at all). The
|
||||
# factories map the planned names onto the real ORM attributes and stash
|
||||
# `rendering_provider_npi` as a transient attribute so
|
||||
# ``reconcile._content_keys_match`` can still read it via ``getattr``.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def db_session():
|
||||
"""Yield a fresh SQLAlchemy session, rolling back at teardown."""
|
||||
from cyclone import db as _db
|
||||
session = _db.SessionLocal()()
|
||||
try:
|
||||
yield session
|
||||
session.rollback()
|
||||
finally:
|
||||
session.close()
|
||||
|
||||
|
||||
def _ensure_batch(session, batch_id: str = "test-batch") -> None:
|
||||
"""Create the Batch row a Claim/Remittance FKs to (idempotent)."""
|
||||
from cyclone.db import Batch
|
||||
if session.get(Batch, batch_id) is None:
|
||||
session.add(Batch(
|
||||
id=batch_id, kind="837p", input_filename="test.txt",
|
||||
parsed_at=datetime.now(timezone.utc),
|
||||
))
|
||||
session.flush()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def make_claim(db_session):
|
||||
"""Factory: build & flush a Claim with the planned content-keys params.
|
||||
|
||||
Planned params (match the SP31 spec / content-keys test surface):
|
||||
patient_control_number, total_charge, rendering_provider_npi,
|
||||
service_date_from, matched_remittance_id=None, state=None,
|
||||
claim_id=None
|
||||
|
||||
`state=None` defaults to ``ClaimState.SUBMITTED`` so existing tests
|
||||
that omit it keep behaving. `rendering_provider_npi` is wired through
|
||||
the real ``Claim.rendering_provider_npi`` column (SP32 migration 0019).
|
||||
"""
|
||||
def _make(
|
||||
patient_control_number: str,
|
||||
total_charge,
|
||||
rendering_provider_npi: str,
|
||||
service_date_from,
|
||||
matched_remittance_id=None,
|
||||
state=None,
|
||||
claim_id: str | None = None,
|
||||
):
|
||||
from cyclone.db import Claim, ClaimState as _CS
|
||||
|
||||
_ensure_batch(db_session)
|
||||
cid = claim_id or f"clm-{patient_control_number}-{service_date_from.isoformat()}"
|
||||
c = Claim(
|
||||
id=cid,
|
||||
batch_id="test-batch",
|
||||
patient_control_number=patient_control_number,
|
||||
service_date_from=service_date_from,
|
||||
charge_amount=total_charge,
|
||||
state=state if state is not None else _CS.SUBMITTED,
|
||||
matched_remittance_id=matched_remittance_id,
|
||||
rendering_provider_npi=rendering_provider_npi,
|
||||
)
|
||||
db_session.add(c)
|
||||
db_session.flush()
|
||||
return c
|
||||
|
||||
return _make
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def make_remit(db_session):
|
||||
"""Factory: build & flush a Remittance with the planned content-keys params.
|
||||
|
||||
Planned params:
|
||||
payer_claim_control_number, total_charge_amount, rendering_provider_npi,
|
||||
service_date, remit_id=None
|
||||
|
||||
`total_charge_amount` is mapped to ``Remittance.total_charge`` (real ORM
|
||||
field). `rendering_provider_npi` is wired through the real
|
||||
``Remittance.rendering_provider_npi`` column (SP32 migration 0019).
|
||||
"""
|
||||
def _make(
|
||||
payer_claim_control_number: str,
|
||||
total_charge_amount,
|
||||
rendering_provider_npi: str,
|
||||
service_date,
|
||||
remit_id: str | None = None,
|
||||
):
|
||||
from cyclone.db import Remittance
|
||||
|
||||
_ensure_batch(db_session)
|
||||
rid = remit_id or f"remit-{payer_claim_control_number}-{service_date.isoformat()}"
|
||||
r = Remittance(
|
||||
id=rid,
|
||||
batch_id="test-batch",
|
||||
payer_claim_control_number=payer_claim_control_number,
|
||||
status_code="1",
|
||||
total_charge=total_charge_amount,
|
||||
total_paid=Decimal("0"),
|
||||
received_at=datetime.now(timezone.utc),
|
||||
service_date=service_date,
|
||||
is_reversal=False,
|
||||
rendering_provider_npi=rendering_provider_npi,
|
||||
)
|
||||
db_session.add(r)
|
||||
db_session.flush()
|
||||
return r
|
||||
|
||||
return _make
|
||||
+2
-2
@@ -17,7 +17,7 @@ NM1*IL*1*Balliache*Marianela****MI*P060946~
|
||||
N3*1811 PAVILION DR APT 303~
|
||||
N4*Montrose*CO*814016072~
|
||||
DMG*D8*19590223*F~
|
||||
NM1*PR*2*COHCPF*****PI*SKCO0~
|
||||
NM1*PR*2*CO_TXIX*****PI*CO_TXIX~
|
||||
CLM*t991102984o1c1d*85.40***12:B:1*Y*A*Y*Y~
|
||||
REF*G1*3173~
|
||||
HI*ABK:R69~
|
||||
@@ -35,7 +35,7 @@ NM1*IL*1*Barella*Victoria****MI*H582447~
|
||||
N3*1900 Kellie DR~
|
||||
N4*Montrose*CO*814019524~
|
||||
DMG*D8*19570727*F~
|
||||
NM1*PR*2*COHCPF*****PI*SKCO0~
|
||||
NM1*PR*2*CO_TXIX*****PI*CO_TXIX~
|
||||
CLM*t991102984o1c2d*155.76***12:B:1*Y*A*Y*Y~
|
||||
REF*G1*3173~
|
||||
HI*ABK:R69~
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
ISA*00* *00* *ZZ*11525703 *ZZ*COMEDASSISTPROG*260617*1937*^*00501*991102984*1*P*:~
|
||||
GS*HC*11525703*COMEDASSISTPROG*20260617*193715*991102984*X*005010X222A1~
|
||||
ST*837*991102984*005010X222A1~
|
||||
BHT*0019*00*co-fixture-001*20260617*193715*CH~
|
||||
NM1*41*2*Dzinesco*****46*11525703~
|
||||
PER*IC*Tester*EM*tester@example.com~
|
||||
NM1*40*2*COLORADO MEDICAL ASSISTANCE PROGRAM*****46*COMEDASSISTPROG~
|
||||
HL*1**20*1~
|
||||
PRV*BI*PXC*251E00000X~
|
||||
NM1*85*2*TOC, Inc.*****XX*1881068062~
|
||||
N3*1100 East Main St*Suite A~
|
||||
N4*Montrose*CO*814014063~
|
||||
REF*EI*721587149~
|
||||
HL*2*1*22*0~
|
||||
SBR*P*18*******MC~
|
||||
NM1*IL*1*Balliache*Marianela****MI*P060946~
|
||||
N3*1811 PAVILION DR APT 303~
|
||||
N4*Montrose*CO*814016072~
|
||||
DMG*D8*19590223*F~
|
||||
NM1*PR*2*CO_TXIX*****PI*CO_TXIX~
|
||||
CLM*t991102984o1c1d*85.40***12:B:1*Y*A*Y*Y~
|
||||
REF*G1*3173~
|
||||
HI*ABK:R69~
|
||||
LX*1~
|
||||
SV1*HC:T1019:U2*42.70*UN*6.00***1~
|
||||
DTP*472*D8*20260602~
|
||||
REF*6R*t991102984v769804d~
|
||||
LX*2~
|
||||
SV1*HC:T1019:U2*42.70*UN*6.00***1~
|
||||
DTP*472*D8*20260603~
|
||||
REF*6R*t991102984v770058d~
|
||||
NM1*82*2*RENDERING PROVIDER*****XX*1234567893~
|
||||
HL*3*1*22*0~
|
||||
SBR*P*18*******MC~
|
||||
NM1*IL*1*Barella*Victoria****MI*H582447~
|
||||
N3*1900 Kellie DR~
|
||||
N4*Montrose*CO*814019524~
|
||||
DMG*D8*19570727*F~
|
||||
NM1*PR*2*CO_TXIX*****PI*CO_TXIX~
|
||||
CLM*t991102984o1c2d*155.76***12:B:1*Y*A*Y*Y~
|
||||
REF*G1*3173~
|
||||
HI*ABK:R69~
|
||||
LX*1~
|
||||
SV1*HC:S5130:U2*38.94*UN*6.00***1~
|
||||
DTP*472*D8*20260602~
|
||||
REF*6R*t991102984v769805d~
|
||||
LX*2~
|
||||
SV1*HC:S5130:U2*38.94*UN*6.00***1~
|
||||
DTP*472*D8*20260603~
|
||||
REF*6R*t991102984v770059d~
|
||||
LX*3~
|
||||
SV1*HC:S5130:U2*38.94*UN*6.00***1~
|
||||
DTP*472*D8*20260604~
|
||||
REF*6R*t991102984v770308d~
|
||||
LX*4~
|
||||
SV1*HC:S5130:U2*38.94*UN*6.00***1~
|
||||
DTP*472*D8*20260605~
|
||||
REF*6R*t991102984v770668d~
|
||||
SE*45*991102984~
|
||||
GE*1*991102984~
|
||||
IEA*1*991102984~
|
||||
+1
-1
@@ -17,7 +17,7 @@ NM1*IL*1*Doe*John****MI*ABC123~
|
||||
N3*456 Member St~
|
||||
N4*Denver*CO*80203~
|
||||
DMG*D8*19800101*M~
|
||||
NM1*PR*2*COHCPF*****PI*SKCO0~
|
||||
NM1*PR*2*CO_TXIX*****PI*CO_TXIX~
|
||||
CLM*CLM001*100.00***12:B:1*Y*A*Y*Y~
|
||||
REF*G1*PA123~
|
||||
HI*ABK:Z00~
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
ISA*00* *00* *ZZ*COMEDASSISTPROG*ZZ*11525703 *260527*2303*^*00501*000000001*0*P*:~
|
||||
GS*FA*COMEDASSISTPROG*11525703*20260527*2303*1*X*005010X231A1~
|
||||
ST*999*0001*005010X231A1~
|
||||
AK1*HC*1*005010X222A1~
|
||||
AK2*837*991102989*005010X222A1~
|
||||
IK5*A~
|
||||
AK9*A*1*1*1~
|
||||
SE*6*0001~
|
||||
GE*1*1~
|
||||
IEA*1*000000001~
|
||||
@@ -0,0 +1,71 @@
|
||||
"""Regression: GET /api/acks, /api/ta1-acks, /api/277ca-acks must be reachable
|
||||
for an authenticated admin. The PERMISSIONS matrix only listed the
|
||||
``POST /api/acks`` (parse-999) entry, so the GET list/detail endpoints
|
||||
returned 403 to admins and broke the Inbox / 999 ACKs pages on the UI.
|
||||
|
||||
This test exercises the matrix via the public login route — not by
|
||||
flipping ``AUTH_DISABLED`` — so a missing ``("GET", "/api/<kind>-acks"):
|
||||
ALL_ROLES`` entry would surface as 403.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
from sqlalchemy import delete
|
||||
|
||||
from cyclone.api import app
|
||||
from cyclone.auth import users
|
||||
from cyclone.db import Session as DbSession
|
||||
from cyclone.db import SessionLocal, User
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clear(monkeypatch):
|
||||
monkeypatch.setattr("cyclone.auth.deps.AUTH_DISABLED", False)
|
||||
with SessionLocal()() as db:
|
||||
db.execute(delete(DbSession))
|
||||
db.execute(delete(User))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal()() as db:
|
||||
db.execute(delete(DbSession))
|
||||
db.execute(delete(User))
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def admin_client(client):
|
||||
with SessionLocal()() as db:
|
||||
users.create(db, username="admin", password="adminpassword1", role="admin")
|
||||
resp = client.post(
|
||||
"/api/auth/login",
|
||||
json={"username": "admin", "password": "adminpassword1"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
return client
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", [
|
||||
"/api/acks",
|
||||
"/api/acks/0",
|
||||
"/api/ta1-acks",
|
||||
"/api/ta1-acks/0",
|
||||
"/api/277ca-acks",
|
||||
"/api/277ca-acks/0",
|
||||
])
|
||||
def test_admin_can_list_and_detail_all_ack_kinds(admin_client, path):
|
||||
resp = admin_client.get(path, headers={"Accept": "application/json"})
|
||||
# 200 for the list endpoint (empty rows is fine), 404 for the detail
|
||||
# of a non-existent id (the gate must let the request through).
|
||||
assert resp.status_code in (200, 404), (
|
||||
f"{path} returned {resp.status_code}: {resp.text}"
|
||||
)
|
||||
assert resp.status_code != 403, (
|
||||
f"{path} returned 403 — PERMISSIONS matrix is missing a GET entry"
|
||||
)
|
||||
@@ -0,0 +1,142 @@
|
||||
"""Tests for the dedup-ed ack ID helpers (SP27 Task 1).
|
||||
|
||||
Locks the contract for the helpers that ``scheduler.py`` + ``api.py``
|
||||
will both import from one place in ``cyclone.handlers._ack_id``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
|
||||
from cyclone.handlers._ack_id import (
|
||||
ack_count_summary,
|
||||
ack_synthetic_source_batch_id,
|
||||
two77ca_synthetic_source_batch_id,
|
||||
)
|
||||
from cyclone.parsers.parse_999 import parse_999_text
|
||||
|
||||
|
||||
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
|
||||
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
|
||||
|
||||
|
||||
def _parse(path: Path):
|
||||
return parse_999_text(path.read_text(), input_file=path.name)
|
||||
|
||||
|
||||
def test_ack_count_summary_all_accepted():
|
||||
result = _parse(ACCEPTED)
|
||||
recv, acc, rej, code = ack_count_summary(result)
|
||||
assert recv == 1
|
||||
assert acc == 1
|
||||
assert rej == 0
|
||||
assert code == "A"
|
||||
|
||||
|
||||
def test_ack_count_summary_all_rejected():
|
||||
result = _parse(REJECTED)
|
||||
recv, acc, rej, code = ack_count_summary(result)
|
||||
assert recv == 1
|
||||
assert acc == 0
|
||||
assert rej == 1
|
||||
assert code == "R"
|
||||
|
||||
|
||||
def test_ack_count_summary_partial_uses_p_code():
|
||||
# Synthesize a 2-set 999 inline with one AK5=A and one AK5=R.
|
||||
# ISA is exactly 106 chars (positions 0-105); the parser slices
|
||||
# positionally then splits the body on ``~``.
|
||||
# SE count = ST + AK1 + AK2_1 + AK5_1 + AK2_2 + AK5_2 + AK9 + SE = 8.
|
||||
partial_999 = (
|
||||
"ISA*00* *00* *ZZ*SUBMITTERID *ZZ*RECEIVERID "
|
||||
"*240101*1200*^*00501*000000001*0*P*:~"
|
||||
"GS*FA*SUBMITTERID*RECEIVERID*20240101*1200*1*X*005010X231A1~"
|
||||
"ST*999*0001*005010X231A1~"
|
||||
"AK1*HC*0001*1*2~"
|
||||
"AK2*837*1~"
|
||||
"AK5*A~"
|
||||
"AK2*837*2~"
|
||||
"AK5*R~"
|
||||
"AK9*P*2*1*1~"
|
||||
"SE*8*0001~"
|
||||
"GE*1*1~"
|
||||
"IEA*1*000000001~"
|
||||
)
|
||||
result = parse_999_text(partial_999, input_file="partial.999")
|
||||
recv, acc, rej, code = ack_count_summary(result)
|
||||
assert recv == 2
|
||||
assert acc == 1
|
||||
assert rej == 1
|
||||
assert code == "P"
|
||||
|
||||
|
||||
def test_ack_synthetic_with_pcn_and_filename_is_deterministic_and_prefix():
|
||||
bsid1 = ack_synthetic_source_batch_id(
|
||||
"000000001", pcn="PCN-12345", source_filename="tp_999.x12",
|
||||
)
|
||||
bsid2 = ack_synthetic_source_batch_id(
|
||||
"000000001", pcn="PCN-12345", source_filename="tp_999.x12",
|
||||
)
|
||||
assert bsid1 == bsid2, "same inputs must produce same id"
|
||||
assert bsid1.startswith("999-PCN-12345-")
|
||||
# Hash is 8 hex chars
|
||||
suffix = bsid1.split("-")[-1]
|
||||
assert len(suffix) == 8 and all(
|
||||
c in "0123456789abcdef" for c in suffix
|
||||
), suffix
|
||||
|
||||
|
||||
def test_ack_synthetic_with_pcn_strips_whitespace():
|
||||
bsid = ack_synthetic_source_batch_id(
|
||||
"000000001", pcn=" PCN ", source_filename="f.x12",
|
||||
)
|
||||
assert bsid.startswith("999-PCN-")
|
||||
|
||||
|
||||
def test_ack_synthetic_without_pcn_uses_icn():
|
||||
bsid = ack_synthetic_source_batch_id(
|
||||
"777777777", pcn=None, source_filename="f.x12",
|
||||
)
|
||||
assert bsid.startswith("999-777777777-")
|
||||
# <= 32 chars total (VARCHAR(32) constraint)
|
||||
assert len(bsid) <= 32
|
||||
|
||||
|
||||
def test_ack_synthetic_default_icn_when_empty():
|
||||
bsid = ack_synthetic_source_batch_id("", pcn=None, source_filename=None)
|
||||
# No PCN, no filename → falls through to default ICN ``000000001``
|
||||
# (no hash suffix without a filename).
|
||||
assert bsid == "999-000000001"
|
||||
|
||||
|
||||
def test_two77ca_synthetic_uses_icn():
|
||||
assert two77ca_synthetic_source_batch_id("000012345") == "277CA-000012345"
|
||||
|
||||
|
||||
def test_two77ca_synthetic_empty_falls_back_to_default():
|
||||
assert two77ca_synthetic_source_batch_id("") == "277CA-000000001"
|
||||
assert two77ca_synthetic_source_batch_id(None) == "277CA-000000001"
|
||||
|
||||
|
||||
def test_ack_count_summary_empty_set_responses_returns_zeros():
|
||||
"""A 999 envelope without any AK2/IK5 sets is malformed per the spec,
|
||||
but the helper must still return a sane (0, 0, 0, 'A') tuple — the
|
||||
scheduler adds it to result.errors and the caller decides."""
|
||||
class _StubResult:
|
||||
set_responses = []
|
||||
recv, acc, rej, code = ack_count_summary(_StubResult())
|
||||
assert recv == 0
|
||||
assert acc == 0
|
||||
assert rej == 0
|
||||
# rejected == 0 → "A" (no rejection signal in this degenerate case)
|
||||
assert code == "A"
|
||||
|
||||
|
||||
def test_ack_synthetic_uses_real_sha1_truncation():
|
||||
"""The hash should match hashlib.sha1(filename).hexdigest()[:8] so
|
||||
a future migration to a different hash is intentional, not drift."""
|
||||
expected = hashlib.sha1(b"my-file.x12").hexdigest()[:8]
|
||||
bsid = ack_synthetic_source_batch_id(
|
||||
"000000001", pcn=None, source_filename="my-file.x12",
|
||||
)
|
||||
assert bsid.endswith(f"-{expected}")
|
||||
+159
-4
@@ -51,21 +51,25 @@ def test_migration_0002_creates_acks_table():
|
||||
|
||||
def test_migration_latest_idempotent_on_fresh_db():
|
||||
"""Re-running the migration on the same DB must be a no-op (PRAGMA
|
||||
user_version already at the latest version — currently 15 after
|
||||
user_version already at the latest version — currently 19 after
|
||||
0004-0006 line_reconciliation, 0005 ta1_acks, SP9's 0007
|
||||
providers/payers/clearhouse, SP10's 0008 payer_rejected,
|
||||
SP11's 0009 audit_log, SP14's 0010 payer_rejected_acknowledged,
|
||||
SP16's 0011 processed_inbound_files, SP17's 0012 db_backups,
|
||||
SP-auth's 0013 users + sessions, SP-audit's 0014 audit_log.user_id,
|
||||
SP22's 0015 drop_claims_unique_constraint)."""
|
||||
SP22's 0015 drop_claims_unique_constraint, SP27-Task 11's 0016
|
||||
claims.matched_remittance_id index, SP27-Task 17's 0017
|
||||
claim.patient_control_number backfill UPDATE, SP28's 0018
|
||||
claim_acks join table, SP32's 0019
|
||||
rendering_provider_npi + service_provider_npi)."""
|
||||
with db.engine().begin() as c:
|
||||
v1 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
|
||||
assert v1 == 15
|
||||
assert v1 == 19
|
||||
# A second run should not raise and should not bump the version.
|
||||
db_migrate.run(db.engine())
|
||||
with db.engine().begin() as c:
|
||||
v2 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
|
||||
assert v2 == 15
|
||||
assert v2 == 19
|
||||
|
||||
|
||||
def test_add_ack_persists_row():
|
||||
@@ -128,3 +132,154 @@ def test_get_ack_returns_row_when_present():
|
||||
assert fetched.id == row.id
|
||||
assert fetched.source_batch_id == "b-1"
|
||||
assert fetched.raw_json == {"hello": "world"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Hotfix 2026-06-29: acks list endpoint silently capped at limit=100
|
||||
# without exposing the true total or full-set aggregates, so the Acks page
|
||||
# rendered "100 on file" and KPI totals summed from the page only when the
|
||||
# row count exceeded 100. These tests pin the new offset param + the
|
||||
# server-side `aggregates` field so the silent-failure mode can't regress.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed_acks(n: int) -> list[int]:
|
||||
"""Insert n ack rows with deterministic per-row counts. Returns the ids."""
|
||||
_make_batch("b-hotfix")
|
||||
ids: list[int] = []
|
||||
for i in range(n):
|
||||
row = store.add_ack(
|
||||
source_batch_id="b-hotfix",
|
||||
accepted_count=i + 1,
|
||||
rejected_count=i,
|
||||
received_count=i + 1,
|
||||
ack_code="A",
|
||||
raw_json={"order": i},
|
||||
)
|
||||
ids.append(row.id)
|
||||
return ids
|
||||
|
||||
|
||||
def test_list_acks_endpoint_pagination_offsets_correctly():
|
||||
"""`offset` walks the full set, `has_more` flips at the boundary."""
|
||||
from fastapi.testclient import TestClient
|
||||
from cyclone.api import app
|
||||
from cyclone.auth.users import create
|
||||
from cyclone.db import SessionLocal
|
||||
|
||||
with SessionLocal()() as s:
|
||||
create(s, username="u", password="p", role="admin")
|
||||
s.commit()
|
||||
client = TestClient(app)
|
||||
client.post("/api/auth/login", json={"username": "u", "password": "p"})
|
||||
|
||||
_seed_acks(5)
|
||||
|
||||
r = client.get("/api/acks?limit=2&offset=0")
|
||||
d = r.json()
|
||||
assert d["total"] == 5
|
||||
assert d["returned"] == 2
|
||||
assert d["has_more"] is True
|
||||
assert len(d["items"]) == 2
|
||||
|
||||
r = client.get("/api/acks?limit=2&offset=4")
|
||||
d = r.json()
|
||||
assert d["total"] == 5
|
||||
assert d["returned"] == 1
|
||||
assert d["has_more"] is False
|
||||
assert len(d["items"]) == 1
|
||||
|
||||
|
||||
def test_list_acks_endpoint_aggregates_reflect_full_set():
|
||||
"""`aggregates` sums over the full row set, not the page.
|
||||
|
||||
Without this the Acks KPI strip silently under-reports once the
|
||||
row count exceeds the page size — the silent-failure the operator
|
||||
flagged on 2026-06-29.
|
||||
"""
|
||||
from fastapi.testclient import TestClient
|
||||
from cyclone.api import app
|
||||
from cyclone.auth.users import create
|
||||
from cyclone.db import SessionLocal
|
||||
|
||||
with SessionLocal()() as s:
|
||||
create(s, username="u", password="p", role="admin")
|
||||
s.commit()
|
||||
client = TestClient(app)
|
||||
client.post("/api/auth/login", json={"username": "u", "password": "p"})
|
||||
|
||||
# 5 rows: accepted = 1+2+3+4+5 = 15, rejected = 0+1+2+3+4 = 10,
|
||||
# received = same as accepted.
|
||||
_seed_acks(5)
|
||||
|
||||
# Page 1 of 2 — full set is 5 rows, page only shows 2, but aggregates
|
||||
# must reflect all 5.
|
||||
r = client.get("/api/acks?limit=2&offset=0")
|
||||
d = r.json()
|
||||
assert d["total"] == 5
|
||||
assert d["returned"] == 2
|
||||
assert d["aggregates"]["accepted_count"] == 15
|
||||
assert d["aggregates"]["rejected_count"] == 10
|
||||
assert d["aggregates"]["received_count"] == 15
|
||||
|
||||
# Page 2 must return identical aggregates — page slice mustn't shift them.
|
||||
r = client.get("/api/acks?limit=2&offset=2")
|
||||
d = r.json()
|
||||
assert d["aggregates"]["accepted_count"] == 15
|
||||
assert d["aggregates"]["rejected_count"] == 10
|
||||
assert d["aggregates"]["received_count"] == 15
|
||||
|
||||
|
||||
def test_list_acks_endpoint_limit_cap_is_5000():
|
||||
"""The validator still enforces an upper bound so a client can't
|
||||
request 1,000,000 rows and OOM the SQLite-backed list call."""
|
||||
from fastapi.testclient import TestClient
|
||||
from cyclone.api import app
|
||||
from cyclone.auth.users import create
|
||||
from cyclone.db import SessionLocal
|
||||
|
||||
with SessionLocal()() as s:
|
||||
create(s, username="u", password="p", role="admin")
|
||||
s.commit()
|
||||
client = TestClient(app)
|
||||
client.post("/api/auth/login", json={"username": "u", "password": "p"})
|
||||
|
||||
r = client.get("/api/acks?limit=10000")
|
||||
assert r.status_code == 422 # FastAPI validation error
|
||||
|
||||
r = client.get("/api/acks?limit=5000")
|
||||
assert r.status_code == 200
|
||||
|
||||
|
||||
def test_list_acks_endpoint_offset_past_end_returns_empty_page():
|
||||
"""`offset` past the row count must yield an empty page, not 500.
|
||||
|
||||
Pinning this so a future refactor that introduces streaming or
|
||||
cursor-based pagination can't accidentally error or 500 when the
|
||||
UI holds stale page state across a row count change.
|
||||
"""
|
||||
from fastapi.testclient import TestClient
|
||||
from cyclone.api import app
|
||||
from cyclone.auth.users import create
|
||||
from cyclone.db import SessionLocal
|
||||
|
||||
with SessionLocal()() as s:
|
||||
create(s, username="u", password="p", role="admin")
|
||||
s.commit()
|
||||
client = TestClient(app)
|
||||
client.post("/api/auth/login", json={"username": "u", "password": "p"})
|
||||
|
||||
_seed_acks(3)
|
||||
|
||||
r = client.get("/api/acks?limit=2&offset=99")
|
||||
assert r.status_code == 200
|
||||
d = r.json()
|
||||
assert d["total"] == 3
|
||||
assert d["returned"] == 0
|
||||
assert d["has_more"] is False
|
||||
assert d["items"] == []
|
||||
# Aggregates must still reflect the full 3-row set on the empty page —
|
||||
# otherwise a stale UI page state would silently zero out the KPI strip.
|
||||
assert d["aggregates"]["accepted_count"] == 1 + 2 + 3
|
||||
assert d["aggregates"]["rejected_count"] == 0 + 1 + 2
|
||||
assert d["aggregates"]["received_count"] == 1 + 2 + 3
|
||||
|
||||
@@ -18,6 +18,7 @@ from cyclone import __version__
|
||||
|
||||
|
||||
FIXTURE = Path(__file__).parent / "fixtures" / "co_medicaid_837p.txt"
|
||||
FIXTURE_835 = Path(__file__).parent / "fixtures" / "co_medicaid_835.txt"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -216,3 +217,124 @@ def test_cors_extra_origins_via_env(client: TestClient, monkeypatch):
|
||||
# Reload once more so the module-level allow-list returns to its
|
||||
# default for any test that imports `cyclone.api` after this one.
|
||||
importlib.reload(api_module)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# SP35: parse-837 input guards (defense in depth against misroute ingest)
|
||||
# --------------------------------------------------------------------------- #
|
||||
#
|
||||
# Before SP35, /api/parse-837 silently accepted any file that had a parseable
|
||||
# ISA envelope. An 835 file dropped on the Upload page while the dropdown
|
||||
# still said "837p" would land in the DB as an empty batch (claims=[]) and a
|
||||
# bogus row on the History tab. SP35 fixes that at three layers:
|
||||
#
|
||||
# 1. Server envelope check: ST*837 (or ST*837P) required, else 400 with
|
||||
# error="Mismatched file kind".
|
||||
# 2. Server empty-claims check: even with the right envelope, if zero CLM
|
||||
# segments were parsed, return 400 with error="No claims parsed"
|
||||
# and DO NOT persist the batch.
|
||||
# 3. UI auto-detect (separate file: src/pages/Upload.test.tsx).
|
||||
#
|
||||
# These tests are the server-layer regression locks. They run against the
|
||||
# TestClient and use the existing fixtures. The 835 fixture has an ST*835
|
||||
# envelope; posting it to /api/parse-837 must surface a 400 and must not
|
||||
# create a BatchRecord.
|
||||
|
||||
|
||||
def test_parse_837_endpoint_rejects_835_input(client: TestClient):
|
||||
"""Uploading an 835 file to /api/parse-837 must fail loudly, not persist.
|
||||
|
||||
Repro for the original bug: user drops an 835 file on the Upload page
|
||||
while the kind dropdown still says "837p" (Upload.tsx default). Before
|
||||
SP35 the endpoint accepted it, ran it through the 837 parser (which
|
||||
found zero CLM segments because the file has none), and persisted a
|
||||
claims=[] batch — a bogus row on the History tab and on
|
||||
/api/batches. SP35 closes the door at the server so the UI bug becomes
|
||||
cosmetic instead of data-corrupting.
|
||||
"""
|
||||
text_835 = FIXTURE_835.read_text()
|
||||
# Sanity check: the fixture really is an 835 file. If this ever flips,
|
||||
# the test would still pass for the wrong reason.
|
||||
assert "ST*835" in text_835, "fixture is no longer ST*835 — update SP35 tests"
|
||||
|
||||
# Snapshot the batch count BEFORE the bad upload so we can assert the
|
||||
# request did NOT persist anything. Using the public /api/batches JSON
|
||||
# endpoint (already exercised by the Dashboard).
|
||||
before = client.get("/api/batches", headers={"Accept": "application/json"}).json()
|
||||
total_before = before.get("total", len(before.get("items", [])))
|
||||
|
||||
resp = client.post(
|
||||
"/api/parse-837",
|
||||
files={"file": ("co_medicaid_835.txt", text_835, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
|
||||
assert resp.status_code == 400, resp.text
|
||||
body = resp.json()
|
||||
assert body["error"] == "Mismatched file kind"
|
||||
assert body["expected"] == "837p"
|
||||
assert body["detected_st"].startswith("835")
|
||||
|
||||
after = client.get("/api/batches", headers={"Accept": "application/json"}).json()
|
||||
total_after = after.get("total", len(after.get("items", [])))
|
||||
assert total_after == total_before, "Server persisted a batch from a 835 file"
|
||||
|
||||
|
||||
def test_parse_837_endpoint_rejects_empty_envelope(client: TestClient):
|
||||
"""Right envelope (ST*837), zero CLM segments → 400 No claims parsed.
|
||||
|
||||
Synthetic input: a complete ISA/GS/ST envelope with a BHT, a closing
|
||||
SE/GE/IEA, and no CLM loops. The 837 parser will tokenize and build
|
||||
the envelope cleanly, then return claims=[]. SP35 must surface this as
|
||||
a 400 with error="No claims parsed" and must not persist a batch.
|
||||
"""
|
||||
# Bare 837 envelope — no HL/CLM loops. A real X12 file with ST*837
|
||||
# but no claims is unusual but possible (e.g. a header-only test file
|
||||
# or a truncated/cancelled run). The right behavior is to reject,
|
||||
# not to silently persist an empty batch.
|
||||
synthetic = (
|
||||
"ISA*00* *00* *ZZ*SENDER *ZZ*RECEIVER "
|
||||
"*260617*1937*^*00501*000000001*0*P*:~"
|
||||
"GS*HC*SENDER*RECEIVER*20260706*1937*1*X*005010X222A1~"
|
||||
"ST*837*0001*005010X222A1~"
|
||||
"BHT*0019*00*0001*20260706*1937*CH~"
|
||||
"SE*2*0001~"
|
||||
"GE*1*1~"
|
||||
"IEA*1*000000001~"
|
||||
)
|
||||
assert "ST*837" in synthetic
|
||||
|
||||
before = client.get("/api/batches", headers={"Accept": "application/json"}).json()
|
||||
total_before = before.get("total", len(before.get("items", [])))
|
||||
|
||||
resp = client.post(
|
||||
"/api/parse-837",
|
||||
files={"file": ("empty_837.txt", synthetic, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
|
||||
assert resp.status_code == 400, resp.text
|
||||
body = resp.json()
|
||||
assert body["error"] == "No claims parsed"
|
||||
|
||||
after = client.get("/api/batches", headers={"Accept": "application/json"}).json()
|
||||
total_after = after.get("total", len(after.get("items", [])))
|
||||
assert total_after == total_before, "Server persisted an empty-claims batch"
|
||||
|
||||
|
||||
def test_parse_837_endpoint_happy_path_still_works(client: TestClient):
|
||||
"""Regression guard: real 837 fixture must still parse → 200 with claims.
|
||||
|
||||
Sits next to the new SP35 rejection tests so any future tightening of
|
||||
the guards that accidentally blocks the happy path fails here loudly.
|
||||
"""
|
||||
text = FIXTURE.read_text()
|
||||
resp = client.post(
|
||||
"/api/parse-837",
|
||||
files={"file": ("co_medicaid_837p.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body.get("summary", {}).get("total_claims", 0) >= 1
|
||||
assert "batch_id" in body and len(body["batch_id"]) == 32
|
||||
|
||||
@@ -167,3 +167,48 @@ class TestInboxPayerRejectedLane:
|
||||
# The rejected lane (999 envelope) must be empty — we haven't
|
||||
# uploaded a 999, so this claim isn't there.
|
||||
assert "c1" not in [c["id"] for c in lanes["rejected"]]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# SP35: parse-277ca envelope regression lock
|
||||
# --------------------------------------------------------------------------- #
|
||||
#
|
||||
# The 277CA parser already raises CycloneParseError("Expected ST*277 or
|
||||
# ST*277CA, got ST*<other>") when fed a file with the wrong ST envelope
|
||||
# (parse_277ca.py line 298). This regression lock confirms the HTTP
|
||||
# surface converts that error into a 400 (never 200, never 500).
|
||||
|
||||
|
||||
def test_parse_277ca_endpoint_rejects_835_input(client: TestClient):
|
||||
"""Posting an 835 file to /api/parse-277ca must surface 400, not 200."""
|
||||
wrong_kind = Path(__file__).parent / "fixtures" / "co_medicaid_835.txt"
|
||||
text = wrong_kind.read_text()
|
||||
assert "ST*835" in text # sanity check on the fixture
|
||||
|
||||
resp = client.post(
|
||||
"/api/parse-277ca",
|
||||
files={"file": ("co_medicaid_835.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 400, resp.text
|
||||
body = resp.json()
|
||||
assert "error" in body
|
||||
# The parser-level message must survive (mentions "ST" and the
|
||||
# expected vs actual set id).
|
||||
detail = body.get("detail", "")
|
||||
assert "ST" in detail and ("277" in detail or body["error"] == "Parse error"), body
|
||||
|
||||
|
||||
def test_parse_277ca_endpoint_rejects_837_input(client: TestClient):
|
||||
"""Posting an 837P file to /api/parse-277ca must surface 400, not 200."""
|
||||
wrong_kind = Path(__file__).parent / "fixtures" / "co_medicaid_837p.txt"
|
||||
text = wrong_kind.read_text()
|
||||
assert "ST*837" in text # sanity check
|
||||
|
||||
resp = client.post(
|
||||
"/api/parse-277ca",
|
||||
files={"file": ("co_medicaid_837p.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 400, resp.text
|
||||
assert "error" in resp.json()
|
||||
|
||||
@@ -13,6 +13,7 @@ from cyclone.store import store as global_store
|
||||
|
||||
FIXTURE = Path(__file__).parent / "fixtures" / "co_medicaid_835.txt"
|
||||
UNBALANCED = Path(__file__).parent / "fixtures" / "unbalanced_835.txt"
|
||||
FIXTURE_837P = Path(__file__).parent / "fixtures" / "co_medicaid_837p.txt"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -337,3 +338,117 @@ def test_prodfile_round_trip_persists_separately(client: TestClient):
|
||||
# No duplicate PCNs survived the dedup; sanity check on persistence.
|
||||
pcns = [r["claimId"] for r in all_remits if r["claimId"]]
|
||||
assert len(pcns) == len(set(pcns)), "duplicate PCN across batches would be a persistence bug"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# SP35: parse-835 input guards (mirror of the parse-837 guards)
|
||||
# --------------------------------------------------------------------------- #
|
||||
#
|
||||
# Before SP35, /api/parse-835 had the same silent-corruption shape as
|
||||
# /api/parse-837: any file with a parseable ISA envelope was accepted, and
|
||||
# the 835 parser returned claims=[] when the file had no CLP segments.
|
||||
# This produced empty 835 batches and bogus rows on the History tab.
|
||||
# These tests are the server-layer regression locks for the 835 endpoint.
|
||||
|
||||
|
||||
def test_parse_835_endpoint_rejects_837_input(client: TestClient):
|
||||
"""Uploading an 837P file to /api/parse-835 must fail loudly, not persist.
|
||||
|
||||
Repro for the symmetric bug: user drops an 837P file on the Upload page
|
||||
while the dropdown still says "835" (Upload.tsx default). Before SP35 the
|
||||
endpoint accepted it, ran it through the 835 parser (which found zero
|
||||
CLP segments), and persisted an empty claims=[] batch. SP35 closes the
|
||||
door at the server so the UI bug becomes cosmetic instead of
|
||||
data-corrupting.
|
||||
"""
|
||||
text_837p = FIXTURE_837P.read_text()
|
||||
# Sanity check: the fixture really is an 837P file. If this ever flips,
|
||||
# the test would still pass for the wrong reason.
|
||||
assert "ST*837" in text_837p, "fixture is no longer ST*837 — update SP35 tests"
|
||||
|
||||
before = client.get("/api/batches", headers={"Accept": "application/json"}).json()
|
||||
total_before = before.get("total", len(before.get("items", [])))
|
||||
|
||||
resp = client.post(
|
||||
"/api/parse-835",
|
||||
files={"file": ("co_medicaid_837p.txt", text_837p, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
|
||||
assert resp.status_code == 400, resp.text
|
||||
body = resp.json()
|
||||
assert body["error"] == "Mismatched file kind"
|
||||
assert body["expected"] == "835"
|
||||
assert body["detected_st"].startswith("837")
|
||||
|
||||
after = client.get("/api/batches", headers={"Accept": "application/json"}).json()
|
||||
total_after = after.get("total", len(after.get("items", [])))
|
||||
assert total_after == total_before, "Server persisted a batch from a 837P file"
|
||||
|
||||
|
||||
def test_parse_835_endpoint_rejects_empty_envelope(client: TestClient):
|
||||
"""Right envelope (ST*835), zero CLP segments → 400 No claims parsed.
|
||||
|
||||
Synthetic input: a complete ISA/GS/ST envelope with a BPR + TRN, a
|
||||
closing SE/GE/IEA, and no CLP loops. The 835 parser will tokenize
|
||||
and build the envelope cleanly, then return claims=[]. SP35 must
|
||||
surface this as a 400 with error="No claims parsed" and must not
|
||||
persist a batch.
|
||||
"""
|
||||
# Bare 835 envelope — no LX/CLP loops. A real X12 835 with no claims
|
||||
# is unusual but possible (header-only test file or a cancelled run).
|
||||
# The right behavior is to reject, not to silently persist.
|
||||
synthetic = (
|
||||
"ISA*00* *00* *ZZ*SENDER *ZZ*RECEIVER "
|
||||
"*260617*1937*^*00501*000000001*0*P*:~"
|
||||
"GS*HP*SENDER*RECEIVER*20260706*1937*1*X*005010X221A1~"
|
||||
"ST*835*0001~"
|
||||
"BPR*I*100.00*C*ACH*CCP*01*021000021*DA*123456*1512345678**01*021000021*DA*123456*20260706~"
|
||||
"TRN*1*0001*1512345678~"
|
||||
"N1*PR*PAYER NAME~"
|
||||
"N3*123 PAYER ST~"
|
||||
"N4*DENVER*CO*80202~"
|
||||
"PER*BL*MEMBER SERVICES*TE*8005551212~"
|
||||
"N1*PE*PAYEE NAME~"
|
||||
"N3*456 PAYEE ST~"
|
||||
"N4*DENVER*CO*80202~"
|
||||
"REF*TJ*123456789~"
|
||||
"SE*9*0001~"
|
||||
"GE*1*1~"
|
||||
"IEA*1*000000001~"
|
||||
)
|
||||
assert "ST*835" in synthetic
|
||||
|
||||
before = client.get("/api/batches", headers={"Accept": "application/json"}).json()
|
||||
total_before = before.get("total", len(before.get("items", [])))
|
||||
|
||||
resp = client.post(
|
||||
"/api/parse-835",
|
||||
files={"file": ("empty_835.txt", synthetic, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
|
||||
assert resp.status_code == 400, resp.text
|
||||
body = resp.json()
|
||||
assert body["error"] == "No claims parsed"
|
||||
|
||||
after = client.get("/api/batches", headers={"Accept": "application/json"}).json()
|
||||
total_after = after.get("total", len(after.get("items", [])))
|
||||
assert total_after == total_before, "Server persisted an empty-claims 835 batch"
|
||||
|
||||
|
||||
def test_parse_835_endpoint_happy_path_still_works(client: TestClient):
|
||||
"""Regression guard: real 835 fixture must still parse → 200 with claims.
|
||||
|
||||
Sits next to the new SP35 rejection tests so any future tightening of
|
||||
the guards that accidentally blocks the happy path fails here loudly.
|
||||
"""
|
||||
text = FIXTURE.read_text()
|
||||
resp = client.post(
|
||||
"/api/parse-835",
|
||||
files={"file": ("co_medicaid_835.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body.get("summary", {}).get("total_claims", 0) >= 1
|
||||
|
||||
@@ -140,3 +140,54 @@ def test_get_ack_404_for_missing(client: TestClient) -> None:
|
||||
"""GET /api/acks/{id} returns 404 for a missing id (not 500)."""
|
||||
resp = client.get("/api/acks/9999", headers={"Accept": "application/json"})
|
||||
assert resp.status_code == 404
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# SP35: parse-999 envelope regression lock
|
||||
# --------------------------------------------------------------------------- #
|
||||
#
|
||||
# The 999 parser already raises CycloneParseError("No AK9 (Functional Group
|
||||
# Response Status) segment found") when fed a non-999 file (parse_999.py
|
||||
# line 290). This regression lock confirms the HTTP surface converts that
|
||||
# error into a 400 (never 200, never 500) so a misroute upload fails loudly
|
||||
# instead of silently creating a corrupt ack row.
|
||||
|
||||
|
||||
def test_parse_999_endpoint_rejects_837_input(client: TestClient):
|
||||
"""Posting an 837P file to /api/parse-999 must surface 400, not 200.
|
||||
|
||||
Before SP35, the 999 parser's envelope guard (no AK9) was already
|
||||
strict at the parser level. This test makes the HTTP contract
|
||||
explicit: a wrong-kind file POSTed to the 999 endpoint MUST come
|
||||
back as 400, not as 200 with an empty ack.
|
||||
"""
|
||||
wrong_kind = Path(__file__).parent / "fixtures" / "co_medicaid_837p.txt"
|
||||
text = wrong_kind.read_text()
|
||||
assert "ST*837" in text # sanity check on the fixture
|
||||
|
||||
resp = client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("co_medicaid_837p.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 400, resp.text
|
||||
body = resp.json()
|
||||
assert "error" in body
|
||||
# The parser-level message must survive (not be replaced by a generic
|
||||
# "Internal server error" or similar).
|
||||
assert "AK9" in body.get("detail", "") or "Parse" in body["error"], body
|
||||
|
||||
|
||||
def test_parse_999_endpoint_rejects_835_input(client: TestClient):
|
||||
"""Posting an 835 file to /api/parse-999 must surface 400, not 200."""
|
||||
wrong_kind = Path(__file__).parent / "fixtures" / "co_medicaid_835.txt"
|
||||
text = wrong_kind.read_text()
|
||||
assert "ST*835" in text # sanity check
|
||||
|
||||
resp = client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("co_medicaid_835.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 400, resp.text
|
||||
assert "error" in resp.json()
|
||||
|
||||
@@ -0,0 +1,303 @@
|
||||
"""Integration tests for /api/acks/stream and /api/ta1-acks/stream.
|
||||
|
||||
SP25: both endpoints follow the live-tail wire format used by
|
||||
/api/claims/stream, /api/remittances/stream, /api/activity/stream:
|
||||
|
||||
* Snapshot: one ``item`` line per existing row (newest first).
|
||||
* Closing: one ``snapshot_end`` line.
|
||||
* Live: forwarded ``item`` lines for each ``ack_received`` /
|
||||
``ta1_ack_received`` event published on the bus.
|
||||
|
||||
These tests follow the same direct-coroutine pattern as
|
||||
``test_api_stream_live.py`` because ``httpx.ASGITransport`` buffers
|
||||
the response and never delivers a disconnect message — calling the
|
||||
endpoint coroutine with a synthetic ``Request`` lets us iterate the
|
||||
body iterator byte-by-byte and use ``body_iterator.aclose()`` to
|
||||
simulate a client disconnect.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from datetime import date
|
||||
|
||||
import pytest
|
||||
from starlette.requests import Request
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.api import app
|
||||
from cyclone.api_routers.acks import acks_stream
|
||||
from cyclone.api_routers.ta1_acks import ta1_acks_stream
|
||||
from cyclone.pubsub import EventBus
|
||||
from cyclone.store import store
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared helpers — mirror the patterns in test_api_stream_live.py
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _make_request(path: str = "/api/acks/stream") -> Request:
|
||||
"""Synthetic Starlette ``Request`` with a no-op receive channel."""
|
||||
|
||||
async def receive():
|
||||
# Block forever; ``is_disconnected``'s cancel scope will cancel
|
||||
# this await and we'll return None (treated as "no message").
|
||||
await asyncio.Event().wait()
|
||||
|
||||
scope = {
|
||||
"type": "http",
|
||||
"method": "GET",
|
||||
"headers": [],
|
||||
"query_string": b"",
|
||||
"path": path,
|
||||
"app": app,
|
||||
"scheme": "http",
|
||||
"server": ("testserver", 80),
|
||||
"client": ("testclient", 50000),
|
||||
}
|
||||
return Request(scope, receive=receive)
|
||||
|
||||
|
||||
def _resolve_query_defaults(endpoint) -> dict:
|
||||
"""Resolve FastAPI Query(...) defaults to their inner ``.default``."""
|
||||
import inspect
|
||||
sig = inspect.signature(endpoint)
|
||||
resolved: dict = {}
|
||||
for name, param in sig.parameters.items():
|
||||
default = param.default
|
||||
if hasattr(default, "default"):
|
||||
resolved[name] = default.default
|
||||
else:
|
||||
resolved[name] = default
|
||||
return resolved
|
||||
|
||||
|
||||
async def _call_endpoint(endpoint, path: str):
|
||||
request = _make_request(path)
|
||||
defaults = _resolve_query_defaults(endpoint)
|
||||
defaults.pop("request", None)
|
||||
return await endpoint(request, **defaults)
|
||||
|
||||
|
||||
async def _read_lines(body_iter, n: int, timeout: float = 5.0) -> list[str]:
|
||||
buffer = b""
|
||||
lines: list[str] = []
|
||||
async with asyncio.timeout(timeout):
|
||||
while len(lines) < n:
|
||||
try:
|
||||
chunk = await body_iter.__anext__()
|
||||
except StopAsyncIteration:
|
||||
break
|
||||
buffer += chunk
|
||||
while b"\n" in buffer and len(lines) < n:
|
||||
raw, buffer = buffer.split(b"\n", 1)
|
||||
if raw:
|
||||
lines.append(raw.decode("utf-8"))
|
||||
return lines
|
||||
|
||||
|
||||
async def _read_one_line(body_iter, timeout: float = 5.0) -> str | None:
|
||||
buffer = b""
|
||||
async with asyncio.timeout(timeout):
|
||||
while True:
|
||||
try:
|
||||
chunk = await body_iter.__anext__()
|
||||
except StopAsyncIteration:
|
||||
return None
|
||||
buffer += chunk
|
||||
if b"\n" in buffer:
|
||||
raw, _ = buffer.split(b"\n", 1)
|
||||
if raw:
|
||||
return raw.decode("utf-8")
|
||||
|
||||
|
||||
async def _read_until_type(
|
||||
body_iter, target: str, *, timeout: float = 5.0, max_lines: int = 50
|
||||
) -> dict | None:
|
||||
buffer = b""
|
||||
seen = 0
|
||||
async with asyncio.timeout(timeout):
|
||||
while seen < max_lines:
|
||||
try:
|
||||
chunk = await body_iter.__anext__()
|
||||
except StopAsyncIteration:
|
||||
return None
|
||||
buffer += chunk
|
||||
while b"\n" in buffer:
|
||||
raw, buffer = buffer.split(b"\n", 1)
|
||||
if not raw:
|
||||
continue
|
||||
seen += 1
|
||||
obj = json.loads(raw.decode("utf-8"))
|
||||
if obj.get("type") == target:
|
||||
return obj
|
||||
return None
|
||||
|
||||
|
||||
async def _drain_until_disconnect(body_iter, max_chunks: int = 20) -> None:
|
||||
try:
|
||||
await body_iter.aclose()
|
||||
except (asyncio.CancelledError, GeneratorExit):
|
||||
pass
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-test heartbeat patch — 0.2s so idle generators exit promptly.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _short_heartbeat(monkeypatch):
|
||||
monkeypatch.setenv("CYCLONE_TAIL_HEARTBEAT_S", "0.2")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def bus() -> EventBus:
|
||||
"""Fresh EventBus attached to app.state. Tests publish via ``bus.publish``.
|
||||
|
||||
``add_ack`` / ``add_ta1_ack`` already publish to whichever bus is
|
||||
passed in, but the live-event tests want to publish synchronously
|
||||
against ``app.state.event_bus`` so the existing-tail subscription
|
||||
picks them up.
|
||||
"""
|
||||
new_bus = EventBus(max_queue_size=64)
|
||||
app.state.event_bus = new_bus
|
||||
return new_bus
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# /api/acks/stream
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_acks_stream_snapshots_existing_rows(bus: EventBus):
|
||||
"""An existing 999 row surfaces as the first ``item`` line."""
|
||||
with db.SessionLocal()() as s:
|
||||
store.add_ack(
|
||||
source_batch_id="999-SEED-1",
|
||||
accepted_count=1,
|
||||
rejected_count=0,
|
||||
received_count=1,
|
||||
ack_code="A",
|
||||
raw_json={
|
||||
"envelope": {"control_number": "000000001"},
|
||||
"set_responses": [{"set_control_number": "PCN-1"}],
|
||||
},
|
||||
event_bus=bus,
|
||||
)
|
||||
|
||||
response = await _call_endpoint(acks_stream, "/api/acks/stream")
|
||||
assert response.media_type.startswith("application/x-ndjson")
|
||||
|
||||
lines = await _read_lines(response.body_iterator, n=2)
|
||||
items = [json.loads(line) for line in lines]
|
||||
assert items[0]["type"] == "item"
|
||||
assert items[0]["data"]["source_batch_id"] == "999-SEED-1"
|
||||
assert items[0]["data"]["patient_control_number"] == "PCN-1"
|
||||
assert items[1] == {"type": "snapshot_end", "data": {"count": 1}}
|
||||
await _drain_until_disconnect(response.body_iterator)
|
||||
|
||||
|
||||
async def test_acks_stream_emits_live_event(bus: EventBus):
|
||||
"""A 999 written while the stream is open shows up as a live item."""
|
||||
response = await _call_endpoint(acks_stream, "/api/acks/stream")
|
||||
|
||||
# Drain snapshot (1 snapshot_end line, no prior rows).
|
||||
snap = await _read_until_type(
|
||||
response.body_iterator, "snapshot_end", timeout=2.0,
|
||||
)
|
||||
assert snap == {"type": "snapshot_end", "data": {"count": 0}}
|
||||
|
||||
# Advance one step so ``subscribe_raw`` registers. Next line may be
|
||||
# a heartbeat — that's fine, we just need the subscription live.
|
||||
_ = await _read_one_line(response.body_iterator, timeout=2.0)
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
store.add_ack(
|
||||
source_batch_id="999-LIVE-1",
|
||||
accepted_count=1,
|
||||
rejected_count=0,
|
||||
received_count=1,
|
||||
ack_code="A",
|
||||
raw_json={
|
||||
"envelope": {"control_number": "000000002"},
|
||||
"set_responses": [],
|
||||
},
|
||||
event_bus=bus,
|
||||
)
|
||||
|
||||
obj = await _read_until_type(
|
||||
response.body_iterator, "item", timeout=2.0,
|
||||
)
|
||||
assert obj is not None, "no item line arrived after publish"
|
||||
assert obj["data"]["source_batch_id"] == "999-LIVE-1"
|
||||
await _drain_until_disconnect(response.body_iterator)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# /api/ta1-acks/stream
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def test_ta1_acks_stream_snapshots_existing_rows(bus: EventBus):
|
||||
"""An existing TA1 row surfaces as the first ``item`` line."""
|
||||
with db.SessionLocal()() as s:
|
||||
store.add_ta1_ack(
|
||||
source_batch_id="TA1-SEED-1",
|
||||
control_number="000000001",
|
||||
interchange_date=date(2026, 7, 2),
|
||||
interchange_time="1200",
|
||||
ack_code="A",
|
||||
note_code="000",
|
||||
ack_generated_date=None,
|
||||
sender_id="S",
|
||||
receiver_id="R",
|
||||
raw_json={"envelope": {"control_number": "000000001"}},
|
||||
event_bus=bus,
|
||||
)
|
||||
|
||||
response = await _call_endpoint(ta1_acks_stream, "/api/ta1-acks/stream")
|
||||
assert response.media_type.startswith("application/x-ndjson")
|
||||
|
||||
lines = await _read_lines(response.body_iterator, n=2)
|
||||
items = [json.loads(line) for line in lines]
|
||||
assert items[0]["type"] == "item"
|
||||
assert items[0]["data"]["control_number"] == "000000001"
|
||||
assert items[0]["data"]["interchange_date"] == "2026-07-02"
|
||||
assert items[1] == {"type": "snapshot_end", "data": {"count": 1}}
|
||||
await _drain_until_disconnect(response.body_iterator)
|
||||
|
||||
|
||||
async def test_ta1_acks_stream_emits_live_event(bus: EventBus):
|
||||
"""A TA1 written while the stream is open shows up as a live item."""
|
||||
response = await _call_endpoint(ta1_acks_stream, "/api/ta1-acks/stream")
|
||||
|
||||
snap = await _read_until_type(
|
||||
response.body_iterator, "snapshot_end", timeout=2.0,
|
||||
)
|
||||
assert snap == {"type": "snapshot_end", "data": {"count": 0}}
|
||||
_ = await _read_one_line(response.body_iterator, timeout=2.0)
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
store.add_ta1_ack(
|
||||
source_batch_id="TA1-LIVE-1",
|
||||
control_number="000000099",
|
||||
interchange_date=date(2026, 7, 2),
|
||||
interchange_time="1330",
|
||||
ack_code="A",
|
||||
note_code="000",
|
||||
ack_generated_date=None,
|
||||
sender_id="SX",
|
||||
receiver_id="RX",
|
||||
raw_json={"envelope": {"control_number": "000000099"}},
|
||||
event_bus=bus,
|
||||
)
|
||||
|
||||
obj = await _read_until_type(
|
||||
response.body_iterator, "item", timeout=2.0,
|
||||
)
|
||||
assert obj is not None, "no item line arrived after publish"
|
||||
assert obj["data"]["control_number"] == "000000099"
|
||||
assert obj["data"]["sender_id"] == "SX"
|
||||
await _drain_until_disconnect(response.body_iterator)
|
||||
@@ -0,0 +1,359 @@
|
||||
"""API tests for SP28 ack-claim surface.
|
||||
|
||||
Spec §6 mandates these named tests:
|
||||
|
||||
* ``test_claim_detail_includes_ack_links`` — GET ``/api/claims/{id}``
|
||||
after ingesting a 999, assert ``ack_links`` is populated.
|
||||
* ``test_acks_list_includes_linked_claim_ids`` — GET ``/api/acks``
|
||||
after ingesting, assert ``linked_claim_ids`` is populated per row.
|
||||
* ``test_claim_acks_stream_emits_claim_ack_written`` — the live-tail
|
||||
endpoint emits claim_ack_written on the bus.
|
||||
|
||||
Plus manual-match (D5/D9):
|
||||
|
||||
* ``test_manual_match_creates_link_via_api``
|
||||
* ``test_manual_match_idempotent_returns_existing``
|
||||
* ``test_manual_match_terminal_claim_returns_409``
|
||||
* ``test_manual_unlink_via_api``
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.api import app
|
||||
from cyclone.db import Batch, Claim, ClaimAck, ClaimState
|
||||
from cyclone.store import store
|
||||
|
||||
|
||||
GAINWELL_999 = (
|
||||
Path(__file__).parent / "fixtures" / "minimal_999_ik5_gainwell.txt"
|
||||
)
|
||||
ACCEPTED_999 = Path(__file__).parent / "fixtures" / "minimal_999.txt"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def clear_store():
|
||||
with store._lock:
|
||||
store._batches.clear()
|
||||
yield
|
||||
with store._lock:
|
||||
store._batches.clear()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client() -> TestClient:
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _seed_batch_envelope(*, batch_id: str, envelope_control: str):
|
||||
with db.SessionLocal()() as s:
|
||||
b = Batch(
|
||||
id=batch_id,
|
||||
kind="837p",
|
||||
input_filename=f"{batch_id}.txt",
|
||||
parsed_at=datetime(2026, 7, 2, 12, 0, tzinfo=timezone.utc),
|
||||
raw_result_json={
|
||||
"envelope": {
|
||||
"sender_id": "SUBMITTER",
|
||||
"receiver_id": "RECEIVER",
|
||||
"control_number": envelope_control,
|
||||
"transaction_date": "2026-07-02",
|
||||
},
|
||||
},
|
||||
)
|
||||
s.add(b)
|
||||
s.commit()
|
||||
|
||||
|
||||
def _seed_claim(*, claim_id: str, batch_id: str, patient_control_number: str,
|
||||
state: ClaimState = ClaimState.SUBMITTED):
|
||||
with db.SessionLocal()() as s:
|
||||
c = Claim(
|
||||
id=claim_id,
|
||||
batch_id=batch_id,
|
||||
patient_control_number=patient_control_number,
|
||||
charge_amount=Decimal("100.00"),
|
||||
state=state,
|
||||
)
|
||||
s.add(c)
|
||||
s.commit()
|
||||
|
||||
|
||||
def test_claim_detail_includes_ack_links(client: TestClient):
|
||||
"""GET /api/claims/{id} after ingesting a 999, assert ack_links
|
||||
is populated."""
|
||||
_seed_batch_envelope(batch_id="B-A", envelope_control="991102989")
|
||||
_seed_claim(claim_id="CLM-DETAIL", batch_id="B-A",
|
||||
patient_control_number="t991102989o1cA")
|
||||
# Ingest a 999 that resolves via Pass 1.
|
||||
text = GAINWELL_999.read_text()
|
||||
resp = client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("minimal_999_ik5_gainwell.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
|
||||
# GET /api/claims/{claim_id} — the SP4 detail endpoint must
|
||||
# surface ack_links so the ClaimDrawer panel can render.
|
||||
detail_resp = client.get("/api/claims/CLM-DETAIL")
|
||||
assert detail_resp.status_code == 200, detail_resp.text
|
||||
body = detail_resp.json()
|
||||
assert "ack_links" in body
|
||||
assert len(body["ack_links"]) == 1
|
||||
link = body["ack_links"][0]
|
||||
assert link["ack_kind"] == "999"
|
||||
assert link["set_accept_reject_code"] == "A"
|
||||
assert link["linked_by"] == "auto"
|
||||
|
||||
|
||||
def test_acks_list_includes_linked_claim_ids(client: TestClient):
|
||||
"""GET /api/acks after ingesting, assert linked_claim_ids is
|
||||
populated per row."""
|
||||
_seed_batch_envelope(batch_id="B-LIST", envelope_control="991102989")
|
||||
_seed_claim(claim_id="CLM-LIST-1", batch_id="B-LIST",
|
||||
patient_control_number="t991102989o1cA")
|
||||
text = GAINWELL_999.read_text()
|
||||
client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("minimal_999_ik5_gainwell.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
|
||||
resp = client.get("/api/acks")
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["total"] == 1
|
||||
assert "linked_claim_ids" in body["items"][0]
|
||||
assert body["items"][0]["linked_claim_ids"] == ["CLM-LIST-1"]
|
||||
|
||||
|
||||
def test_claim_acks_stream_emits_claim_ack_written(client: TestClient):
|
||||
"""The per-claim live-tail endpoint streams claim_ack_written
|
||||
events the moment a link is created.
|
||||
|
||||
Uses the direct-endpoint-invocation pattern from
|
||||
``test_api_stream_live.py`` so we can iterate the body iterator
|
||||
with byte-level streaming + asyncio cancellation cleanup.
|
||||
"""
|
||||
import asyncio
|
||||
from starlette.requests import Request as StarletteRequest
|
||||
from cyclone.api_routers.claim_acks import (
|
||||
claim_acks_stream,
|
||||
)
|
||||
|
||||
_seed_batch_envelope(batch_id="B-STREAM", envelope_control="991102989")
|
||||
_seed_claim(claim_id="CLM-STREAM", batch_id="B-STREAM",
|
||||
patient_control_number="t991102989o1cA")
|
||||
|
||||
async def _drain_until_disconnect(body_iter):
|
||||
try:
|
||||
await body_iter.aclose()
|
||||
except (asyncio.CancelledError, GeneratorExit):
|
||||
pass
|
||||
|
||||
async def _read_until_type(body_iter, target, *, timeout=5.0):
|
||||
buffer = b""
|
||||
async with asyncio.timeout(timeout):
|
||||
while True:
|
||||
chunk = await body_iter.__anext__()
|
||||
buffer += chunk
|
||||
while b"\n" in buffer:
|
||||
raw, buffer = buffer.split(b"\n", 1)
|
||||
if not raw:
|
||||
continue
|
||||
obj = json.loads(raw.decode("utf-8"))
|
||||
if obj.get("type") == target:
|
||||
return obj
|
||||
return None
|
||||
|
||||
async def _scenario():
|
||||
# Ingest a 999 that resolves via Pass 1 so we have a link
|
||||
# row to stream.
|
||||
text = GAINWELL_999.read_text()
|
||||
resp = client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("minimal_999_ik5_gainwell.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
|
||||
# Build a synthetic request and call the endpoint coroutine
|
||||
# directly so we get true byte-streaming.
|
||||
async def receive():
|
||||
await asyncio.Event().wait() # never delivered
|
||||
|
||||
scope = {
|
||||
"type": "http",
|
||||
"method": "GET",
|
||||
"headers": [],
|
||||
"query_string": b"",
|
||||
"path": "/api/claims/CLM-STREAM/acks/stream",
|
||||
"app": app,
|
||||
"scheme": "http",
|
||||
"server": ("testserver", 80),
|
||||
"client": ("testclient", 50000),
|
||||
}
|
||||
request = StarletteRequest(scope, receive=receive)
|
||||
response = await claim_acks_stream(request, claim_id="CLM-STREAM")
|
||||
assert response.media_type.startswith("application/x-ndjson")
|
||||
|
||||
# The snapshot has 1 item + 1 snapshot_end. Read until
|
||||
# snapshot_end.
|
||||
snap_end = await _read_until_type(
|
||||
response.body_iterator, "snapshot_end", timeout=2.0,
|
||||
)
|
||||
assert snap_end is not None
|
||||
assert snap_end["data"]["count"] == 1
|
||||
# The first emitted item was the snapshot itself; check it
|
||||
# carries the right claim_id.
|
||||
await _drain_until_disconnect(response.body_iterator)
|
||||
|
||||
asyncio.run(_scenario())
|
||||
|
||||
|
||||
def test_manual_match_creates_link_via_api(client: TestClient):
|
||||
"""POST /api/acks/{kind}/{ack_id}/match-claim creates a link row."""
|
||||
# Pre-seed an ack via the API so we have an ack_id to reference.
|
||||
_seed_batch_envelope(batch_id="B-MM", envelope_control="991102989")
|
||||
_seed_claim(claim_id="CLM-MM", batch_id="B-MM",
|
||||
patient_control_number="t991102989o1cA")
|
||||
text = GAINWELL_999.read_text()
|
||||
resp = client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("minimal_999_ik5_gainwell.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
body = resp.json()
|
||||
ack_id = body["ack"]["id"]
|
||||
|
||||
# Now manually match a second claim to the same ack.
|
||||
_seed_claim(claim_id="CLM-MM-MANUAL", batch_id="B-MM",
|
||||
patient_control_number="manual-pcn-001")
|
||||
resp2 = client.post(
|
||||
f"/api/acks/999/{ack_id}/match-claim",
|
||||
json={"claim_id": "CLM-MM-MANUAL"},
|
||||
)
|
||||
assert resp2.status_code == 200, resp2.text
|
||||
body2 = resp2.json()
|
||||
assert body2["created"] is True
|
||||
assert body2["link"]["claim_id"] == "CLM-MM-MANUAL"
|
||||
assert body2["link"]["linked_by"] == "manual"
|
||||
|
||||
|
||||
def test_manual_match_idempotent_returns_existing(client: TestClient):
|
||||
"""Re-calling match-claim with the same claim_id returns the
|
||||
existing row (200), not a duplicate.
|
||||
|
||||
Uses a 999 that the auto-linker can't resolve (no matching
|
||||
Batch.envelope.control_number, no matching PCN) so the manual
|
||||
match is the FIRST link to land.
|
||||
"""
|
||||
# Seed a Batch + Claim whose envelope control and PCN both do
|
||||
# NOT match the 999's ST02 / set_control_number.
|
||||
_seed_batch_envelope(batch_id="B-MM-IDEMP", envelope_control="UNRELATED")
|
||||
_seed_claim(claim_id="CLM-MM-IDEMP", batch_id="B-MM-IDEMP",
|
||||
patient_control_number="manual-test-pcn")
|
||||
text = GAINWELL_999.read_text()
|
||||
resp = client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("minimal_999_ik5_gainwell.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
ack_id = resp.json()["ack"]["id"]
|
||||
|
||||
# First call — created.
|
||||
r1 = client.post(
|
||||
f"/api/acks/999/{ack_id}/match-claim",
|
||||
json={"claim_id": "CLM-MM-IDEMP"},
|
||||
)
|
||||
assert r1.status_code == 200, r1.text
|
||||
body1 = r1.json()
|
||||
assert body1["created"] is True
|
||||
|
||||
# Second call — same row, idempotent.
|
||||
r2 = client.post(
|
||||
f"/api/acks/999/{ack_id}/match-claim",
|
||||
json={"claim_id": "CLM-MM-IDEMP"},
|
||||
)
|
||||
assert r2.status_code == 200, r2.text
|
||||
body2 = r2.json()
|
||||
assert body2["created"] is False
|
||||
assert body2["link"]["id"] == body1["link"]["id"]
|
||||
|
||||
|
||||
def test_manual_match_terminal_claim_returns_409(client: TestClient):
|
||||
"""A REVERSED claim cannot be matched — 409."""
|
||||
_seed_batch_envelope(batch_id="B-MM-TERM", envelope_control="991102989")
|
||||
_seed_claim(claim_id="CLM-MM-TERM", batch_id="B-MM-TERM",
|
||||
patient_control_number="t991102989o1cA",
|
||||
state=ClaimState.REVERSED)
|
||||
text = GAINWELL_999.read_text()
|
||||
resp = client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("minimal_999_ik5_gainwell.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
ack_id = resp.json()["ack"]["id"]
|
||||
|
||||
resp2 = client.post(
|
||||
f"/api/acks/999/{ack_id}/match-claim",
|
||||
json={"claim_id": "CLM-MM-TERM"},
|
||||
)
|
||||
assert resp2.status_code == 409, resp2.text
|
||||
|
||||
|
||||
def test_manual_unlink_via_api(client: TestClient):
|
||||
"""DELETE /api/acks/{kind}/{ack_id}/match-claim/{claim_id} removes
|
||||
the link row and publishes claim_ack_dropped."""
|
||||
_seed_batch_envelope(batch_id="B-UNL", envelope_control="991102989")
|
||||
_seed_claim(claim_id="CLM-UNL", batch_id="B-UNL",
|
||||
patient_control_number="t991102989o1cA")
|
||||
text = GAINWELL_999.read_text()
|
||||
resp = client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("minimal_999_ik5_gainwell.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
ack_id = resp.json()["ack"]["id"]
|
||||
|
||||
resp_del = client.delete(f"/api/acks/999/{ack_id}/match-claim/CLM-UNL")
|
||||
assert resp_del.status_code == 200, resp_del.text
|
||||
assert resp_del.json()["removed"] is True
|
||||
|
||||
# Verify the row is gone.
|
||||
with db.SessionLocal()() as s:
|
||||
n = (
|
||||
s.query(ClaimAck)
|
||||
.filter_by(claim_id="CLM-UNL", ack_kind="999", ack_id=ack_id)
|
||||
.count()
|
||||
)
|
||||
assert n == 0
|
||||
|
||||
|
||||
def test_inbox_ack_orphans_returns_unresolved_acks(client: TestClient):
|
||||
"""An ack that resolves to no claim surfaces in the inbox
|
||||
ack-orphans lane."""
|
||||
# Ingest a 999 with no matching batch + no matching PCN — orphan.
|
||||
text = ACCEPTED_999.read_text()
|
||||
resp = client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("minimal_999.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
|
||||
orphans_resp = client.get("/api/inbox/ack-orphans")
|
||||
assert orphans_resp.status_code == 200, orphans_resp.text
|
||||
body = orphans_resp.json()
|
||||
assert body["total"] >= 1
|
||||
# The 999 we just ingested is in the orphan list.
|
||||
kinds = [item["kind"] for item in body["items"]]
|
||||
assert "999" in kinds
|
||||
@@ -0,0 +1,174 @@
|
||||
"""SP25 — PATCH /api/clearhouse endpoint.
|
||||
|
||||
Lets the operator flip sftp_block.stub and adjust host/port/paths
|
||||
without raw SQL. The endpoint validates via the Clearhouse Pydantic
|
||||
model, writes via store.update_clearhouse(), then hot-reloads the
|
||||
running scheduler via scheduler.reconfigure_scheduler().
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from cyclone import scheduler as sched_mod
|
||||
from cyclone.api import app
|
||||
from cyclone.auth import deps
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _seed_clearhouse():
|
||||
"""Insert the default clearhouse row used by SP9's lifespan seed."""
|
||||
cycl_store.ensure_clearhouse_seeded()
|
||||
|
||||
|
||||
def _clearhouse_body_from_get(client, **overrides) -> dict:
|
||||
"""GET the current clearhouse row and apply ``overrides`` to the
|
||||
sftp_block. The PATCH endpoint requires a full Clearhouse body,
|
||||
so we always round-trip through GET first to get the right
|
||||
updated_at + filename_block shape."""
|
||||
r = client.get("/api/clearhouse")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
sb = dict(body["sftp_block"])
|
||||
sb.update(overrides)
|
||||
body["sftp_block"] = sb
|
||||
return body
|
||||
|
||||
|
||||
def _real_block_overrides() -> dict:
|
||||
"""Default overrides for stub=False PATCHes — the auth shape must
|
||||
match what ``SftpClient._connect`` reads (the seed uses a different
|
||||
legacy shape with ``secret_ref``, which PATCH rejects on real
|
||||
blocks)."""
|
||||
return {
|
||||
"stub": False,
|
||||
"host": "mft.example.com",
|
||||
"auth": {"password_keychain_account": "sftp.gainwell.password"},
|
||||
}
|
||||
|
||||
|
||||
def test_patch_flips_stub_and_get_reflects(client):
|
||||
_seed_clearhouse()
|
||||
overrides = _real_block_overrides()
|
||||
body = _clearhouse_body_from_get(client, **overrides)
|
||||
r = client.patch("/api/clearhouse", json=body)
|
||||
assert r.status_code == 200, r.text
|
||||
payload = r.json()
|
||||
assert payload["sftp_block"]["stub"] is False
|
||||
assert payload["sftp_block"]["host"] == "mft.example.com"
|
||||
|
||||
r2 = client.get("/api/clearhouse")
|
||||
assert r2.status_code == 200
|
||||
assert r2.json()["sftp_block"]["stub"] is False
|
||||
|
||||
|
||||
def test_patch_with_string_stub_returns_422(client):
|
||||
_seed_clearhouse()
|
||||
body = _clearhouse_body_from_get(client)
|
||||
body["sftp_block"]["stub"] = "yes" # string instead of bool
|
||||
r = client.patch("/api/clearhouse", json=body)
|
||||
assert r.status_code == 422
|
||||
|
||||
|
||||
def test_patch_real_block_requires_host(client):
|
||||
_seed_clearhouse()
|
||||
overrides = _real_block_overrides()
|
||||
overrides["host"] = "" # override the real-block default
|
||||
body = _clearhouse_body_from_get(client, **overrides)
|
||||
r = client.patch("/api/clearhouse", json=body)
|
||||
assert r.status_code == 422
|
||||
|
||||
|
||||
def test_patch_real_block_requires_password_keychain_account(client):
|
||||
_seed_clearhouse()
|
||||
overrides = _real_block_overrides()
|
||||
overrides["auth"] = {"password_keychain_account": ""}
|
||||
body = _clearhouse_body_from_get(client, **overrides)
|
||||
r = client.patch("/api/clearhouse", json=body)
|
||||
assert r.status_code == 422
|
||||
|
||||
|
||||
def test_patch_without_session_returns_401(client):
|
||||
"""With AUTH_DISABLED off (the production gate), a request with no
|
||||
session cookie must be rejected at the gate before reaching the
|
||||
endpoint body. Build the body manually since the GET used by
|
||||
``_clearhouse_body_from_get`` is itself gated."""
|
||||
_seed_clearhouse()
|
||||
# Build a valid body without going through the gated GET.
|
||||
overrides = _real_block_overrides()
|
||||
body = {
|
||||
"id": 1,
|
||||
"name": "dzinesco",
|
||||
"tpid": "11525703",
|
||||
"submitter_name": "Dzinesco",
|
||||
"submitter_id_qual": "46",
|
||||
"submitter_contact_name": "Tyler Martinez",
|
||||
"submitter_contact_email": "tyler@dzinesco.com",
|
||||
"filename_block": {
|
||||
"tz": "America/Denver",
|
||||
"outbound_template": "{tpid}-{tx}-{ts}-1of1.{ext}",
|
||||
"inbound_template": "TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12",
|
||||
},
|
||||
"sftp_block": {
|
||||
"host": overrides.get("host", "mft.example.com"),
|
||||
"port": 22,
|
||||
"username": "colorado-fts\\coxix_prod_11525703",
|
||||
"paths": {
|
||||
"outbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
|
||||
"inbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
|
||||
},
|
||||
"stub": overrides.get("stub", False),
|
||||
"staging_dir": "./var/sftp/staging",
|
||||
"auth": overrides.get("auth", {"password_keychain_account": "sftp.gainwell.password"}),
|
||||
},
|
||||
"updated_at": "2026-06-24T00:00:00+00:00",
|
||||
}
|
||||
|
||||
deps.AUTH_DISABLED = False
|
||||
try:
|
||||
r = client.patch("/api/clearhouse", json=body)
|
||||
assert r.status_code == 401
|
||||
finally:
|
||||
deps.AUTH_DISABLED = True
|
||||
|
||||
|
||||
def test_patch_triggers_scheduler_reconfigure(client):
|
||||
"""The PATCH must call scheduler.reconfigure_scheduler() with the new
|
||||
SftpBlock so the next tick uses real MFT instead of the stub."""
|
||||
_seed_clearhouse()
|
||||
|
||||
with patch.object(
|
||||
sched_mod, "reconfigure_scheduler",
|
||||
AsyncMock(return_value=AsyncMock()),
|
||||
) as mock_reconf:
|
||||
overrides = _real_block_overrides()
|
||||
overrides["host"] = "mft.gainwelltechnologies.com"
|
||||
body = _clearhouse_body_from_get(client, **overrides)
|
||||
r = client.patch("/api/clearhouse", json=body)
|
||||
assert r.status_code == 200, r.text
|
||||
assert mock_reconf.await_count == 1
|
||||
# The new sftp_block is passed positionally; sftp_block_name
|
||||
# comes from the clearhouse row's ``name`` field.
|
||||
call_args = mock_reconf.await_args
|
||||
assert call_args.args[0].stub is False
|
||||
assert call_args.args[0].host == "mft.gainwelltechnologies.com"
|
||||
assert call_args.kwargs.get("sftp_block_name") == "dzinesco"
|
||||
|
||||
|
||||
def test_patch_returns_post_update_row(client):
|
||||
_seed_clearhouse()
|
||||
body = _clearhouse_body_from_get(client, **_real_block_overrides())
|
||||
r = client.patch("/api/clearhouse", json=body)
|
||||
assert r.status_code == 200
|
||||
payload = r.json()
|
||||
assert payload["name"] == "dzinesco"
|
||||
assert payload["tpid"] == "11525703"
|
||||
assert payload["sftp_block"]["stub"] is False
|
||||
assert payload["sftp_block"]["host"] == "mft.example.com"
|
||||
@@ -0,0 +1,112 @@
|
||||
"""Regression tests: api.py + scheduler.py delegate to handlers/_ack_id (SP27 Task 6).
|
||||
|
||||
After the dedup, the inline `_ack_count_summary`,
|
||||
`_ack_synthetic_source_batch_id`, and `_277ca_synthetic_source_batch_id`
|
||||
helpers should be gone from both `api.py` and `scheduler.py`. Their
|
||||
callers should reach the canonical copies under
|
||||
``cyclone.handlers._ack_id``.
|
||||
|
||||
This test pins two things:
|
||||
|
||||
1. The module attributes exist where expected (import paths intact).
|
||||
2. The endpoints that depend on the helpers still respond 200 (or
|
||||
401 if unauthenticated) — i.e. the dedup didn't break the import
|
||||
chain.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from cyclone import api, scheduler
|
||||
from cyclone.api import app
|
||||
from cyclone.handlers._ack_id import (
|
||||
ack_count_summary,
|
||||
ack_synthetic_source_batch_id,
|
||||
two77ca_synthetic_source_batch_id,
|
||||
)
|
||||
|
||||
|
||||
# ---- 1. The dedup actually happened ---------------------------------------
|
||||
|
||||
|
||||
def test_api_no_longer_defines_inline_ack_helpers():
|
||||
"""api.py must not re-define the helpers as local symbols.
|
||||
|
||||
The alias imports at the top of api.py (``from cyclone.handlers._ack_id
|
||||
import ack_count_summary as _ack_count_summary``) leave the
|
||||
underscore-prefixed names bound in ``api.__dict__`` — that's the
|
||||
whole point of an alias import and is required so the inline
|
||||
call sites keep working. What we want to check is that the
|
||||
*implementation source* is no longer the api module itself.
|
||||
|
||||
Concretely: the function's ``__module__`` attribute should be
|
||||
``cyclone.handlers._ack_id``, not ``cyclone.api``.
|
||||
"""
|
||||
for name in (
|
||||
"_ack_count_summary",
|
||||
"_ack_synthetic_source_batch_id",
|
||||
"_277ca_synthetic_source_batch_id",
|
||||
):
|
||||
assert hasattr(api, name), (
|
||||
f"api.py doesn't even have {name} — alias import missing"
|
||||
)
|
||||
fn = getattr(api, name)
|
||||
assert fn.__module__ == "cyclone.handlers._ack_id", (
|
||||
f"api.{name} resolves to {fn.__module__}, expected "
|
||||
f"cyclone.handlers._ack_id — inline copy still exists"
|
||||
)
|
||||
|
||||
|
||||
def test_scheduler_no_inline_ack_helpers():
|
||||
"""scheduler.py must not re-define the ack helpers as local symbols.
|
||||
|
||||
In SP27 Task 2 the inline copies of ``_ack_count_summary`` and
|
||||
``_ack_synthetic_source_batch_id`` were deleted from
|
||||
scheduler.py (they had become dead code after Task 1 lifted the
|
||||
canonical copies into ``cyclone.handlers._ack_id``). Task 4
|
||||
removed ``_277ca_synthetic_source_batch_id`` the same way. So
|
||||
none of the three names should be present in the scheduler
|
||||
module at all.
|
||||
"""
|
||||
for name in (
|
||||
"_ack_count_summary",
|
||||
"_ack_synthetic_source_batch_id",
|
||||
"_277ca_synthetic_source_batch_id",
|
||||
):
|
||||
assert not hasattr(scheduler, name), (
|
||||
f"scheduler.py still defines {name} inline; dedup incomplete"
|
||||
)
|
||||
|
||||
|
||||
def test_canonical_helpers_resolve_to_handlers_ack_id():
|
||||
"""The names exported by ``cyclone.handlers._ack_id`` must be
|
||||
reachable from the canonical import path."""
|
||||
assert callable(ack_count_summary)
|
||||
assert callable(ack_synthetic_source_batch_id)
|
||||
assert callable(two77ca_synthetic_source_batch_id)
|
||||
|
||||
|
||||
# ---- 2. Endpoints still respond -------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client() -> TestClient:
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def test_ack_endpoints_respond_after_dedup(client):
|
||||
"""Hitting each ack endpoint must not raise ImportError. Either
|
||||
200 (auth OK) or 401/403 (auth required) is acceptable; 500 means
|
||||
the import chain broke."""
|
||||
# Acquire an authenticated session by hitting login (idempotent —
|
||||
# the same admin user already exists in the test DB).
|
||||
for path in (
|
||||
"/api/acks",
|
||||
"/api/ta1-acks",
|
||||
"/api/277ca-acks",
|
||||
):
|
||||
resp = client.get(path)
|
||||
assert resp.status_code in (200, 401, 403), (
|
||||
f"{path} broke after dedup: {resp.status_code} {resp.text}"
|
||||
)
|
||||
@@ -101,6 +101,122 @@ def test_batches_claim_ids_empty_for_835(client: TestClient, tmp_path):
|
||||
assert item["claimIds"] == []
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# SP30: /api/batches billing-outcome fields
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def _sp30_seed_billing_outcome_batch() -> None:
|
||||
"""Insert one batch with 3 claims (paid / rejected / submitted)
|
||||
via the ORM directly. Mirrors the helper pattern in
|
||||
``test_dashboard_kpis.py`` — bypasses the parse pipeline so
|
||||
the test is independent of parser fixture drift.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
from decimal import Decimal
|
||||
|
||||
from cyclone.db import Batch, Claim, ClaimState
|
||||
from cyclone import db
|
||||
|
||||
raw_claim = {
|
||||
"subscriber": {"first_name": "Jane", "last_name": "Doe"},
|
||||
"payer": {"name": "CO_TXIX"},
|
||||
"billing_provider": {"npi": "1234567893"},
|
||||
"service_lines": [],
|
||||
}
|
||||
# ParseResult requires a `summary` (BatchSummary). The list endpoint
|
||||
# rehydrates Batch.raw_result_json through ParseResult.model_validate,
|
||||
# so the stub has to be shape-valid even though we don't read it.
|
||||
valid_raw_result = {
|
||||
"claims": [],
|
||||
"summary": {
|
||||
"input_file": "out.edi",
|
||||
"total_claims": 3,
|
||||
"passed": 3,
|
||||
"failed": 0,
|
||||
},
|
||||
}
|
||||
with db.SessionLocal()() as s:
|
||||
s.add(Batch(
|
||||
id="b-sp30-out",
|
||||
kind="837p",
|
||||
input_filename="out.edi",
|
||||
parsed_at=datetime(2026, 7, 1, 12, 0, tzinfo=timezone.utc),
|
||||
totals_json={"total_claims": 3},
|
||||
validation_json={"passed": True, "warnings": [], "errors": []},
|
||||
raw_result_json=valid_raw_result,
|
||||
))
|
||||
s.add(Claim(
|
||||
id="C-paid",
|
||||
batch_id="b-sp30-out",
|
||||
patient_control_number="PCN-paid",
|
||||
charge_amount=Decimal("100.00"),
|
||||
state=ClaimState.PAID,
|
||||
raw_json=raw_claim,
|
||||
))
|
||||
s.add(Claim(
|
||||
id="C-rej",
|
||||
batch_id="b-sp30-out",
|
||||
patient_control_number="PCN-rej",
|
||||
charge_amount=Decimal("50.00"),
|
||||
state=ClaimState.REJECTED,
|
||||
rejection_reason="999 AK5 R",
|
||||
raw_json=raw_claim,
|
||||
))
|
||||
s.add(Claim(
|
||||
id="C-sub",
|
||||
batch_id="b-sp30-out",
|
||||
patient_control_number="PCN-sub",
|
||||
charge_amount=Decimal("75.00"),
|
||||
state=ClaimState.SUBMITTED,
|
||||
raw_json=raw_claim,
|
||||
))
|
||||
s.commit()
|
||||
|
||||
|
||||
def test_batches_includes_billing_outcome(client: TestClient):
|
||||
"""SP30: the Dashboard widget reads accepted/rejected/pending counts,
|
||||
billed total, top rejection reason, and a has-problem flag off the
|
||||
list endpoint. One batch, three claims covering all three buckets."""
|
||||
_sp30_seed_billing_outcome_batch()
|
||||
resp = client.get("/api/batches", headers=JSON)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
# We may have other batches from earlier tests (the `seeded_store`
|
||||
# autouse isn't applied here); find the SP30 batch by id.
|
||||
item = next(i for i in body["items"] if i["id"] == "b-sp30-out")
|
||||
assert item["acceptedCount"] == 1
|
||||
assert item["rejectedCount"] == 1
|
||||
assert item["pendingCount"] == 1
|
||||
assert item["billedTotal"] == 225.0
|
||||
assert item["topRejectionReason"] == "999 AK5 R"
|
||||
assert item["hasProblem"] is True
|
||||
|
||||
|
||||
def test_batches_835_kind_returns_zero_billed_and_no_rejection(client: TestClient):
|
||||
"""SP30: 835 (ERA) batches have no Claim rows — the new fields are
|
||||
all zeroed out so the frontend widget doesn't render garbage (the
|
||||
widget shows "N payments" instead of a $ figure for ERAs).
|
||||
"""
|
||||
src = Path(__file__).parent / "fixtures" / "minimal_835.txt"
|
||||
files = {"file": ("minimal_835.txt", src.read_bytes(), "text/plain")}
|
||||
r = client.post(
|
||||
"/api/parse-835",
|
||||
params={"payer": "co_medicaid_835"},
|
||||
files=files,
|
||||
headers=JSON,
|
||||
)
|
||||
assert r.status_code == 200, r.text
|
||||
body = client.get("/api/batches", headers=JSON).json()
|
||||
item = next(i for i in body["items"] if i["kind"] == "835")
|
||||
assert item["acceptedCount"] == 0
|
||||
assert item["rejectedCount"] == 0
|
||||
assert item["pendingCount"] == 0
|
||||
assert item["billedTotal"] == 0.0
|
||||
assert item["topRejectionReason"] is None
|
||||
assert item["hasProblem"] is False
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# /api/batches/{id}
|
||||
# --------------------------------------------------------------------------- #
|
||||
@@ -509,14 +625,17 @@ def test_post_match_happy_path(client: TestClient):
|
||||
)
|
||||
from cyclone.store import BatchRecord837, BatchRecord835
|
||||
|
||||
# 837 Claim. member_id="M1" so the 835 PCN-based auto-match can't pair them.
|
||||
# 837 Claim. pcn deliberately differs from the 835's PCN so the
|
||||
# auto-matcher (which now joins on claim_id == pcn after the SP27
|
||||
# Task 17 PCN fix) doesn't pair them — we want an orphan so the
|
||||
# manual_match call has work to do.
|
||||
co = ClaimOutput(
|
||||
claim_id="CLM-1",
|
||||
control_number="0001",
|
||||
transaction_date=date(2026, 6, 19),
|
||||
billing_provider=BillingProvider(name="Test", npi="1234567890"),
|
||||
subscriber=Subscriber(
|
||||
first_name="Jane", last_name="Doe", member_id="M1",
|
||||
first_name="Jane", last_name="Doe", member_id="ORPHAN-MEMBER",
|
||||
),
|
||||
payer=Payer(name="Test Payer", id="P1"),
|
||||
claim=ClaimHeader(
|
||||
@@ -541,12 +660,12 @@ def test_post_match_happy_path(client: TestClient):
|
||||
),
|
||||
))
|
||||
|
||||
# 835 Remittance. PCN="CLM-1" but member_id on the claim side is "M1",
|
||||
# so the auto-matcher in reconcile.run does NOT pair them; we want
|
||||
# an orphan so manual_match has work to do. charge == paid == 100
|
||||
# so apply_payment picks ClaimState.PAID.
|
||||
# 835 Remittance. PCN="CLM-1-ORPHAN" deliberately differs from the
|
||||
# claim's claim_id="CLM-1" so the auto-matcher in reconcile.run does
|
||||
# NOT pair them; we want an orphan so manual_match has work to do.
|
||||
# charge == paid == 100 so apply_payment picks ClaimState.PAID.
|
||||
cp = ClaimPayment(
|
||||
payer_claim_control_number="CLM-1",
|
||||
payer_claim_control_number="CLM-1-ORPHAN",
|
||||
status_code="1",
|
||||
total_charge=Decimal("100"),
|
||||
total_paid=Decimal("100"),
|
||||
|
||||
@@ -0,0 +1,297 @@
|
||||
"""Tests for ``GET /api/remittances/summary`` (server-aggregated KPI
|
||||
backend for the Remittances page tiles).
|
||||
|
||||
Anomaly recap (see ``docs/superpowers/specs/2026-06-29-cyclone-...
|
||||
-design.md`` if it lands later):
|
||||
|
||||
* The Remittances page's "Total paid" + "Adjustments" KPI tiles
|
||||
previously summed ``items.reduce(...)`` over the *current page* (25
|
||||
rows) plus whatever live-tail events had arrived in the session.
|
||||
* For an empty-looking page with most-paid=0 rows, the UI rendered
|
||||
e.g. ``$16,934`` paid / ``$147`` adjustments — neither the page
|
||||
total ($1,334 / $143) nor the true DB total ($227,181 / $13,792).
|
||||
Same silent-incompleteness pattern the count bug retired in
|
||||
``d81b6ed``: the page looked complete but the numbers were wrong.
|
||||
|
||||
The fix adds a server-aggregated summary endpoint — ``count + paid
|
||||
+ adjustments`` summed over the full persisted row set with the same
|
||||
filter pipeline as ``/api/remittances``. ``Remittances.tsx`` swaps
|
||||
``items.reduce(...)`` for the new hook so the tiles reflect the true
|
||||
DB population regardless of dataset size.
|
||||
|
||||
The covered cases match ``test_list_endpoint_counts.py`` /
|
||||
``test_acks_aggregates.py`` for shape consistency.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from decimal import Decimal
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.api import app
|
||||
from cyclone.db import Batch, Remittance
|
||||
from cyclone.store import store as global_store
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client() -> TestClient:
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _seed_remit(
|
||||
s,
|
||||
remit_id: str,
|
||||
batch_id: str,
|
||||
*,
|
||||
received_at: datetime,
|
||||
total_paid: Decimal = Decimal("100.00"),
|
||||
adjustment_amount: Decimal = Decimal("0"),
|
||||
claim_id: str | None = None,
|
||||
payer_name: str = "Colorado Medicaid",
|
||||
status_code: str = "1",
|
||||
) -> None:
|
||||
# The remittance's batch row carries ``raw_result_json`` with the
|
||||
# payer name because ``iter_remittances`` reads payer name from
|
||||
# there. We stage a fresh ``Batch`` row per call so unique
|
||||
# ``payer_name`` values for tests like the per-payer filter case
|
||||
# don't collide — the existing pattern from
|
||||
# ``test_list_endpoint_counts.py:88``.
|
||||
s.add(Batch(
|
||||
id=batch_id,
|
||||
kind="835",
|
||||
input_filename=f"{remit_id}.edi",
|
||||
parsed_at=received_at,
|
||||
totals_json={"total_claims": 1},
|
||||
validation_json={"passed": True, "warnings": [], "errors": []},
|
||||
raw_result_json={"payer": {"name": payer_name}},
|
||||
))
|
||||
s.add(Remittance(
|
||||
id=remit_id,
|
||||
batch_id=batch_id,
|
||||
payer_claim_control_number=remit_id,
|
||||
claim_id=claim_id,
|
||||
status_code=status_code,
|
||||
total_charge=Decimal("100.00"),
|
||||
total_paid=total_paid,
|
||||
adjustment_amount=adjustment_amount,
|
||||
received_at=received_at,
|
||||
))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Module-level: summarize_remittances
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_summarize_remittances_zero_when_empty():
|
||||
assert global_store.summarize_remittances() == {
|
||||
"count": 0,
|
||||
"total_paid": 0.0,
|
||||
"total_adjustments": 0.0,
|
||||
}
|
||||
|
||||
|
||||
def test_summarize_remittances_unfiltered_returns_full_population():
|
||||
"""Regression: with 5 remits seeded, the OLD ``items.reduce(...)``
|
||||
page-local sum was per-page (25 rows); the new helper must
|
||||
aggregate over all 5."""
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
# 5 remits, paid=10/20/30/40/50 → sum=150, adj all 0.
|
||||
for i, amt in enumerate([10, 20, 30, 40, 50]):
|
||||
_seed_remit(
|
||||
s, f"RMT-SUM-{i:04d}", f"b-sum-pop-{i}",
|
||||
received_at=base + timedelta(minutes=i),
|
||||
total_paid=Decimal(amt),
|
||||
adjustment_amount=Decimal("0"),
|
||||
)
|
||||
s.commit()
|
||||
|
||||
assert global_store.summarize_remittances() == {
|
||||
"count": 5,
|
||||
"total_paid": 150.0,
|
||||
"total_adjustments": 0.0,
|
||||
}
|
||||
|
||||
|
||||
def test_summarize_remittances_sums_adjustments_across_rows():
|
||||
"""Adjustments accumulate independently from paid."""
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_remit(
|
||||
s, "RMT-A", "b-sum-adj-A",
|
||||
received_at=base, total_paid=Decimal("25.96"),
|
||||
adjustment_amount=Decimal("16.87"),
|
||||
)
|
||||
_seed_remit(
|
||||
s, "RMT-B", "b-sum-adj-B",
|
||||
received_at=base + timedelta(minutes=1),
|
||||
total_paid=Decimal("0"), adjustment_amount=Decimal("50.19"),
|
||||
)
|
||||
_seed_remit(
|
||||
s, "RMT-C", "b-sum-adj-C",
|
||||
received_at=base + timedelta(minutes=2),
|
||||
total_paid=Decimal("0"), adjustment_amount=Decimal("9.95"),
|
||||
)
|
||||
s.commit()
|
||||
|
||||
assert global_store.summarize_remittances() == {
|
||||
"count": 3,
|
||||
"total_paid": 25.96,
|
||||
"total_adjustments": 77.01, # 16.87 + 50.19 + 9.95
|
||||
}
|
||||
|
||||
|
||||
def test_summarize_remittances_filters_by_claim_id():
|
||||
"""claim_id filter narrows the summary the same way it narrows
|
||||
iter_remittances (DB-side WHERE remittance.claim_id = :claim_id).
|
||||
"""
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_remit(
|
||||
s, "RMT-X", "b-sum-cid-X",
|
||||
received_at=base, claim_id="CLM-1",
|
||||
total_paid=Decimal("10"), adjustment_amount=Decimal("0"),
|
||||
)
|
||||
_seed_remit(
|
||||
s, "RMT-Y", "b-sum-cid-Y",
|
||||
received_at=base, claim_id="CLM-2",
|
||||
total_paid=Decimal("20"), adjustment_amount=Decimal("1"),
|
||||
)
|
||||
_seed_remit(
|
||||
s, "RMT-Z", "b-sum-cid-Z",
|
||||
received_at=base, claim_id=None,
|
||||
total_paid=Decimal("30"), adjustment_amount=Decimal("2"),
|
||||
)
|
||||
s.commit()
|
||||
|
||||
assert global_store.summarize_remittances(claim_id="CLM-1") == {
|
||||
"count": 1,
|
||||
"total_paid": 10.0,
|
||||
"total_adjustments": 0.0,
|
||||
}
|
||||
assert global_store.summarize_remittances(claim_id="CLM-2") == {
|
||||
"count": 1,
|
||||
"total_paid": 20.0,
|
||||
"total_adjustments": 1.0,
|
||||
}
|
||||
assert global_store.summarize_remittances(claim_id="CLM-missing") == {
|
||||
"count": 0,
|
||||
"total_paid": 0.0,
|
||||
"total_adjustments": 0.0,
|
||||
}
|
||||
|
||||
|
||||
def test_summarize_remittances_filters_by_payer_exact_match():
|
||||
"""``payer`` filter is in-memory + case-sensitive exact match —
|
||||
the helper must apply it the same way ``iter_remittances`` does
|
||||
so summary and list views agree under the same chip."""
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_remit(
|
||||
s, "RMT-CO-1", "b-sum-payer-co-1",
|
||||
received_at=base, payer_name="Colorado Medicaid",
|
||||
total_paid=Decimal("7"), adjustment_amount=Decimal("0.50"),
|
||||
)
|
||||
_seed_remit(
|
||||
s, "RMT-CO-2", "b-sum-payer-co-2",
|
||||
received_at=base + timedelta(minutes=1),
|
||||
payer_name="Colorado Medicaid",
|
||||
total_paid=Decimal("13"), adjustment_amount=Decimal("0.25"),
|
||||
)
|
||||
_seed_remit(
|
||||
s, "RMT-AZ-1", "b-sum-payer-az-1",
|
||||
received_at=base + timedelta(minutes=2),
|
||||
payer_name="Arizona Medicaid",
|
||||
total_paid=Decimal("99"), adjustment_amount=Decimal("0"),
|
||||
)
|
||||
s.commit()
|
||||
|
||||
assert global_store.summarize_remittances(payer="Colorado Medicaid") == {
|
||||
"count": 2,
|
||||
"total_paid": 20.0,
|
||||
"total_adjustments": 0.75,
|
||||
}
|
||||
assert global_store.summarize_remittances(payer="Arizona Medicaid") == {
|
||||
"count": 1,
|
||||
"total_paid": 99.0,
|
||||
"total_adjustments": 0.0,
|
||||
}
|
||||
assert global_store.summarize_remittances(payer="colorado medicaid") == {
|
||||
"count": 0,
|
||||
"total_paid": 0.0,
|
||||
"total_adjustments": 0.0,
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# HTTP: /api/remittances/summary
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_api_remittances_summary_empty(client: TestClient):
|
||||
"""Empty DB → zero totals, well-formed response."""
|
||||
resp = client.get("/api/remittances/summary")
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert resp.json() == {
|
||||
"count": 0,
|
||||
"total_paid": 0,
|
||||
"total_adjustments": 0,
|
||||
}
|
||||
|
||||
|
||||
def test_api_remittances_summary_full_population(client: TestClient):
|
||||
"""Bug repro: seed 5 remits with known amounts, assert the
|
||||
endpoint reports the FULL sum, not a 25-row sample.
|
||||
"""
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
for i, amt in enumerate([100, 200, 300, 400, 500]):
|
||||
_seed_remit(
|
||||
s, f"RMT-HTTP-SUM-{i}", f"b-http-sum-{i}",
|
||||
received_at=base + timedelta(minutes=i),
|
||||
total_paid=Decimal(amt),
|
||||
adjustment_amount=Decimal("5"),
|
||||
)
|
||||
s.commit()
|
||||
|
||||
resp = client.get("/api/remittances/summary")
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body == {
|
||||
"count": 5,
|
||||
"total_paid": 1500, # 100+200+300+400+500
|
||||
"total_adjustments": 25, # 5*5
|
||||
}
|
||||
|
||||
|
||||
def test_api_remittances_summary_respects_claim_id_filter(client: TestClient):
|
||||
"""Filter narrows both count and sums."""
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_remit(
|
||||
s, "RMT-F-1", "b-http-cid-1",
|
||||
received_at=base, claim_id="CLM-F1",
|
||||
total_paid=Decimal("11"), adjustment_amount=Decimal("1"),
|
||||
)
|
||||
_seed_remit(
|
||||
s, "RMT-F-2", "b-http-cid-2",
|
||||
received_at=base, claim_id="CLM-F2",
|
||||
total_paid=Decimal("22"), adjustment_amount=Decimal("2"),
|
||||
)
|
||||
s.commit()
|
||||
|
||||
resp = client.get(
|
||||
"/api/remittances/summary",
|
||||
params={"claim_id": "CLM-F1"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {
|
||||
"count": 1,
|
||||
"total_paid": 11,
|
||||
"total_adjustments": 1,
|
||||
}
|
||||
@@ -34,13 +34,13 @@ def _stub_scheduler_env(tmp_path, monkeypatch):
|
||||
db._reset_for_tests()
|
||||
|
||||
staging = tmp_path / "staging"
|
||||
inbound = staging / "ToHPE"
|
||||
inbound = staging / "FromHPE"
|
||||
inbound.mkdir(parents=True)
|
||||
sftp_block = SftpBlock(
|
||||
host="mft.example.com",
|
||||
port=22,
|
||||
username="test",
|
||||
paths={"outbound": "/FromHPE", "inbound": "/ToHPE"},
|
||||
paths={"outbound": "/ToHPE", "inbound": "/FromHPE"},
|
||||
stub=True,
|
||||
staging_dir=str(staging),
|
||||
poll_seconds=60,
|
||||
@@ -54,7 +54,7 @@ def _stub_scheduler_env(tmp_path, monkeypatch):
|
||||
|
||||
|
||||
def _drop_file(staging: Path, name: str, body: bytes) -> Path:
|
||||
p = staging / "ToHPE" / name
|
||||
p = staging / "FromHPE" / name
|
||||
p.write_bytes(body)
|
||||
return p
|
||||
|
||||
|
||||
@@ -163,4 +163,54 @@ def test_list_ta1_acks_newest_first(client: TestClient):
|
||||
assert len(items) == 2
|
||||
# The REJECTED (uploaded second) is first.
|
||||
assert items[0]["ack_code"] == "R"
|
||||
assert items[1]["ack_code"] == "A"
|
||||
assert items[1]["ack_code"] == "A"
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# SP35: parse-ta1 envelope regression lock
|
||||
# --------------------------------------------------------------------------- #
|
||||
#
|
||||
# The TA1 parser already raises CycloneParseError("Expected TA1, got <other>")
|
||||
# when fed a file that doesn't have a TA1 segment as its first payload
|
||||
# segment (parse_ta1.py line 111). This regression lock confirms the HTTP
|
||||
# surface converts that error into a 400 (never 200, never 500). TA1 has
|
||||
# no ST envelope, so the test uses an 837 fixture (which has ISA + GS +
|
||||
# ST*837 but no TA1 segment) to exercise the parser-level guard.
|
||||
|
||||
|
||||
def test_parse_ta1_endpoint_rejects_837_input(client: TestClient):
|
||||
"""Posting an 837P file to /api/parse-ta1 must surface 400, not 200.
|
||||
|
||||
The TA1 envelope has no ST (it's the bare interchange-ack segment),
|
||||
so the wrong-kind check is structural — the parser looks for the TA1
|
||||
segment and raises when it doesn't find one. An 837 file has ISA +
|
||||
GS + ST*837 but no TA1, which triggers that branch.
|
||||
"""
|
||||
wrong_kind = Path(__file__).parent / "fixtures" / "co_medicaid_837p.txt"
|
||||
text = wrong_kind.read_text()
|
||||
assert "ST*837" in text # sanity check on the fixture
|
||||
|
||||
resp = client.post(
|
||||
"/api/parse-ta1",
|
||||
files={"file": ("co_medicaid_837p.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 400, resp.text
|
||||
body = resp.json()
|
||||
assert "error" in body
|
||||
detail = body.get("detail", "")
|
||||
assert "TA1" in detail or body["error"] == "Parse error", body
|
||||
|
||||
|
||||
def test_parse_ta1_endpoint_rejects_835_input(client: TestClient):
|
||||
"""Posting an 835 file to /api/parse-ta1 must surface 400, not 200."""
|
||||
wrong_kind = Path(__file__).parent / "fixtures" / "co_medicaid_835.txt"
|
||||
text = wrong_kind.read_text()
|
||||
assert "ST*835" in text # sanity check
|
||||
|
||||
resp = client.post(
|
||||
"/api/parse-ta1",
|
||||
files={"file": ("co_medicaid_835.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 400, resp.text
|
||||
assert "error" in resp.json()
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
"""SP33: apply_999_rejections must use batch_envelope_index (mirrors
|
||||
SP28's apply_999_acceptances fix) so SET-level 999 rejections cascade
|
||||
to claim state transitions.
|
||||
|
||||
The pre-SP33 bug: ``claim_lookup(sr.set_control_number)`` passed the
|
||||
SET control_number to a PCN-based lookup, so no claim was ever matched
|
||||
and rejections silently died. SP33 threads the same
|
||||
``batch_envelope_index`` that ``apply_999_acceptances`` already has.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from cyclone.db import ClaimState
|
||||
from cyclone.inbox_state import apply_999_rejections
|
||||
|
||||
|
||||
def _mk_set_response(code: str = "R", scn: str = "991102994"):
|
||||
"""Build a stand-in for ``ParseResult999.set_responses[0]``."""
|
||||
return SimpleNamespace(
|
||||
set_accept_reject=SimpleNamespace(code=code),
|
||||
set_control_number=scn,
|
||||
segment_errors=[],
|
||||
)
|
||||
|
||||
|
||||
def _mk_claim(cid: str = "claim-x", state=ClaimState.SUBMITTED):
|
||||
c = MagicMock()
|
||||
c.id = cid
|
||||
c.state = state
|
||||
return c
|
||||
|
||||
|
||||
def test_set_level_rejection_cascades_via_envelope_index():
|
||||
"""When the SET-level 999 has set_control_number=991102994 and the
|
||||
batch_envelope_index maps it to a list of 3 claim_ids, all 3 should
|
||||
transition to REJECTED.
|
||||
"""
|
||||
session = MagicMock()
|
||||
claims = [_mk_claim(f"c-{i}") for i in range(3)]
|
||||
session.query.return_value.filter.return_value.all.return_value = claims
|
||||
|
||||
parsed = SimpleNamespace(set_responses=[_mk_set_response(code="R", scn="991102994")])
|
||||
index = {"991102994": [c.id for c in claims]}
|
||||
|
||||
result = apply_999_rejections(
|
||||
session, parsed,
|
||||
claim_lookup=lambda _pcn: None,
|
||||
batch_envelope_index=index,
|
||||
)
|
||||
|
||||
assert sorted(result.matched) == sorted(c.id for c in claims)
|
||||
assert result.orphans == []
|
||||
session.commit.assert_called_once()
|
||||
for c in claims:
|
||||
# ClaimState.REJECTED is the lowercase 'rejected' string (the enum
|
||||
# value stored in the DB column via SQLAlchemy Enum type).
|
||||
assert c.state == ClaimState.REJECTED
|
||||
|
||||
|
||||
def test_set_level_rejection_without_index_still_uses_legacy_lookup():
|
||||
"""Backwards compat: when batch_envelope_index is None, the function
|
||||
must fall back to claim_lookup (existing behavior preserved).
|
||||
"""
|
||||
session = MagicMock()
|
||||
parsed = SimpleNamespace(set_responses=[_mk_set_response(code="R", scn="991102994")])
|
||||
legacy_claim = _mk_claim("legacy-claim")
|
||||
|
||||
result = apply_999_rejections(
|
||||
session, parsed,
|
||||
claim_lookup=lambda scn: legacy_claim if scn == "991102994" else None,
|
||||
)
|
||||
|
||||
assert result.matched == ["legacy-claim"]
|
||||
|
||||
|
||||
def test_no_envelope_index_no_match_becomes_orphan():
|
||||
"""Empty index + empty legacy lookup => the SET becomes an orphan."""
|
||||
session = MagicMock()
|
||||
parsed = SimpleNamespace(set_responses=[_mk_set_response(code="R", scn="991102999")])
|
||||
|
||||
result = apply_999_rejections(
|
||||
session, parsed,
|
||||
claim_lookup=lambda _pcn: None,
|
||||
batch_envelope_index={}, # empty index
|
||||
)
|
||||
assert result.matched == []
|
||||
assert "991102999" in result.orphans
|
||||
|
||||
|
||||
def test_accepted_999_does_not_reject():
|
||||
"""An 'A' code must not transition any claim to REJECTED."""
|
||||
session = MagicMock()
|
||||
parsed = SimpleNamespace(set_responses=[_mk_set_response(code="A", scn="991102994")])
|
||||
|
||||
result = apply_999_rejections(
|
||||
session, parsed,
|
||||
claim_lookup=lambda _pcn: None,
|
||||
batch_envelope_index={"991102994": ["claim-x"]},
|
||||
)
|
||||
assert result.matched == []
|
||||
session.query.assert_not_called()
|
||||
session.commit.assert_not_called()
|
||||
|
||||
|
||||
def test_already_rejected_is_idempotent():
|
||||
"""Pre-REJECTED claims in the index are left alone (no second commit)."""
|
||||
session = MagicMock()
|
||||
already = _mk_claim("already-rejected", state=ClaimState.REJECTED)
|
||||
session.query.return_value.filter.return_value.all.return_value = [already]
|
||||
|
||||
parsed = SimpleNamespace(set_responses=[_mk_set_response(code="R", scn="991102994")])
|
||||
result = apply_999_rejections(
|
||||
session, parsed,
|
||||
claim_lookup=lambda _pcn: None,
|
||||
batch_envelope_index={"991102994": ["already-rejected"]},
|
||||
)
|
||||
assert result.matched == []
|
||||
assert already.state == ClaimState.REJECTED # untouched
|
||||
@@ -0,0 +1,845 @@
|
||||
"""Tests for :mod:`cyclone.claim_acks` (SP28 auto-linker).
|
||||
|
||||
Two flavours:
|
||||
* Pure-unit tests (no DB) for the walk-through logic of
|
||||
``apply_999_acceptances`` etc. via stubs.
|
||||
* Integration tests against the test DB (autouse conftest) that
|
||||
exercise ``lookup_claims_for_ack_set_response``'s two-pass D10 join,
|
||||
per-AK2 granularity, idempotent re-ingest, and the manual link.
|
||||
|
||||
The 8 named tests from spec §6 are here as
|
||||
``test_999_*`` / ``test_277ca_*`` / ``test_ta1_*`` /
|
||||
``test_reingest_*`` / ``test_manual_link_*`` /
|
||||
``test_unlink_does_not_revert_claim_state``.
|
||||
|
||||
Additional tests cover the D10 two-pass join (Step 2.5 of the plan)
|
||||
and the store facade wiring.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import date, datetime, timezone
|
||||
from decimal import Decimal
|
||||
|
||||
import pytest
|
||||
|
||||
from cyclone import claim_acks as ca
|
||||
from cyclone import db
|
||||
from cyclone.claim_acks import (
|
||||
ClaimAckLinkResult,
|
||||
ClaimAckLinkRow,
|
||||
apply_277ca_acks,
|
||||
apply_999_acceptances,
|
||||
apply_ta1_envelope_link,
|
||||
link_manual,
|
||||
lookup_claims_for_ack_set_response,
|
||||
)
|
||||
from cyclone.db import (
|
||||
Batch,
|
||||
Claim,
|
||||
ClaimState,
|
||||
ClaimAck,
|
||||
)
|
||||
from cyclone.parsers.models import (
|
||||
BatchSummary,
|
||||
ClaimHeader,
|
||||
Envelope,
|
||||
ParseResult,
|
||||
)
|
||||
from cyclone.parsers.models_277ca import (
|
||||
AcknowledgmentHeader,
|
||||
ClaimStatus,
|
||||
ParseResult277CA,
|
||||
)
|
||||
from cyclone.parsers.models_999 import (
|
||||
AcknowledgmentHeader as AckHeader999,
|
||||
ParseResult999,
|
||||
SetAcceptReject,
|
||||
SetFunctionalGroupResponse,
|
||||
)
|
||||
from cyclone.parsers.models_ta1 import ParseResultTa1, Ta1Ack
|
||||
from cyclone.store import store as cycl_store, store
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixture seeds
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed_batch(
|
||||
session,
|
||||
*,
|
||||
batch_id: str = "B1",
|
||||
envelope_control: str = "991102989",
|
||||
sender_id: str = "SENDER",
|
||||
receiver_id: str = "RECEIVER",
|
||||
):
|
||||
"""Insert one Batch row whose raw_result_json envelope carries
|
||||
``envelope_control`` so D10 Pass 1 can resolve it via
|
||||
``batch_envelope_index``."""
|
||||
b = Batch(
|
||||
id=batch_id,
|
||||
kind="837p",
|
||||
input_filename=f"{batch_id}.txt",
|
||||
parsed_at=datetime(2026, 7, 2, 12, 0, tzinfo=timezone.utc),
|
||||
raw_result_json={
|
||||
"envelope": {
|
||||
"sender_id": sender_id,
|
||||
"receiver_id": receiver_id,
|
||||
"control_number": envelope_control,
|
||||
"transaction_date": "2026-07-02",
|
||||
},
|
||||
},
|
||||
)
|
||||
session.add(b)
|
||||
session.commit()
|
||||
session.refresh(b)
|
||||
return b
|
||||
|
||||
|
||||
def _seed_claim(
|
||||
session,
|
||||
*,
|
||||
claim_id: str,
|
||||
batch_id: str,
|
||||
patient_control_number: str | None = None,
|
||||
state: ClaimState = ClaimState.SUBMITTED,
|
||||
):
|
||||
c = Claim(
|
||||
id=claim_id,
|
||||
batch_id=batch_id,
|
||||
patient_control_number=patient_control_number or claim_id,
|
||||
charge_amount=Decimal("100.00"),
|
||||
state=state,
|
||||
)
|
||||
session.add(c)
|
||||
session.commit()
|
||||
session.refresh(c)
|
||||
return c
|
||||
|
||||
|
||||
def _seed_ack_link_index(*, envelope_control_to_batch: dict[str, str]):
|
||||
"""Build a closure compatible with ``lookup_claims_for_ack_set_response``."""
|
||||
|
||||
def _lookup(scn: str) -> str | None:
|
||||
return envelope_control_to_batch.get(scn)
|
||||
|
||||
return _lookup
|
||||
|
||||
|
||||
def _parse_999_two_ak2s() -> ParseResult999:
|
||||
"""Two-AK2 999 — one accepted, one rejected."""
|
||||
sr_accepted = SetFunctionalGroupResponse(
|
||||
ak2=AckHeader999(functional_id_code="837", group_control_number="991102989"),
|
||||
set_control_number="991102989",
|
||||
transaction_set_identifier="837",
|
||||
segment_errors=[],
|
||||
set_accept_reject=SetAcceptReject(code="A"),
|
||||
)
|
||||
sr_rejected = SetFunctionalGroupResponse(
|
||||
ak2=AckHeader999(functional_id_code="837", group_control_number="991102990"),
|
||||
set_control_number="991102990",
|
||||
transaction_set_identifier="837",
|
||||
segment_errors=[],
|
||||
set_accept_reject=SetAcceptReject(code="R"),
|
||||
)
|
||||
return ParseResult999(
|
||||
envelope=Envelope(
|
||||
sender_id="PAYER", receiver_id="SUBMITTER",
|
||||
control_number="000000001", transaction_date=date(2026, 7, 2),
|
||||
),
|
||||
functional_group_acks=[],
|
||||
set_responses=[sr_accepted, sr_rejected],
|
||||
summary=BatchSummary(
|
||||
input_file="two_ak2.txt", control_number="000000001",
|
||||
transaction_date=date(2026, 7, 2),
|
||||
total_claims=2, passed=1, failed=1,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _parse_277ca_one_accepted() -> ParseResult277CA:
|
||||
return ParseResult277CA(
|
||||
envelope=Envelope(
|
||||
sender_id="PAYER", receiver_id="SUBMITTER",
|
||||
control_number="000000077", transaction_date=date(2026, 7, 2),
|
||||
),
|
||||
bht=AcknowledgmentHeader(
|
||||
hierarchical_structure_code="0085",
|
||||
transaction_set_purpose_code="08",
|
||||
reference_identification="REFNUM001",
|
||||
),
|
||||
summary=BatchSummary(
|
||||
input_file="277_one_accepted.txt", control_number="000000077",
|
||||
transaction_date=date(2026, 7, 2),
|
||||
total_claims=1, passed=1, failed=0,
|
||||
),
|
||||
claim_statuses=[
|
||||
ClaimStatus(
|
||||
status_code="A3:19:PR",
|
||||
classification="accepted",
|
||||
payer_claim_control_number="CLAIM001",
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def _parse_277ca_one_rejected() -> ParseResult277CA:
|
||||
return ParseResult277CA(
|
||||
envelope=Envelope(
|
||||
sender_id="PAYER", receiver_id="SUBMITTER",
|
||||
control_number="000000078", transaction_date=date(2026, 7, 2),
|
||||
),
|
||||
bht=AcknowledgmentHeader(
|
||||
hierarchical_structure_code="0085",
|
||||
transaction_set_purpose_code="08",
|
||||
reference_identification="REFNUM002",
|
||||
),
|
||||
summary=BatchSummary(
|
||||
input_file="277_one_rejected.txt", control_number="000000078",
|
||||
transaction_date=date(2026, 7, 2),
|
||||
total_claims=1, passed=0, failed=1,
|
||||
),
|
||||
claim_statuses=[
|
||||
ClaimStatus(
|
||||
status_code="A6:19:PR",
|
||||
classification="rejected",
|
||||
payer_claim_control_number="CLAIM002",
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def _parse_ta1() -> ParseResultTa1:
|
||||
return ParseResultTa1(
|
||||
envelope=Envelope(
|
||||
sender_id="RECEIVER", receiver_id="SENDER",
|
||||
control_number="000000001", transaction_date=date(2026, 7, 2),
|
||||
),
|
||||
ta1=Ta1Ack(
|
||||
control_number="000000001",
|
||||
interchange_date=date(2026, 7, 2),
|
||||
interchange_time="1200",
|
||||
ack_code="A",
|
||||
note_code="000",
|
||||
ack_generated_date=date(2026, 7, 2),
|
||||
),
|
||||
source_batch_id="TA1-000000001",
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Spec §6 tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_999_auto_creates_per_ak2_link_rows():
|
||||
"""Step 2.5 (named after spec §6).
|
||||
|
||||
Two AK2s in one 999 — one accepted, one rejected. They reference
|
||||
two distinct batches (one claim per batch), so per-AK2 granularity
|
||||
produces two link rows with two different set_accept_reject codes.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, batch_id="B-999A", envelope_control="991102989")
|
||||
_seed_claim(s, claim_id="CLM-A", batch_id="B-999A",
|
||||
patient_control_number="t991102989o1cA")
|
||||
_seed_batch(s, batch_id="B-999B", envelope_control="991102990")
|
||||
_seed_claim(s, claim_id="CLM-B", batch_id="B-999B",
|
||||
patient_control_number="t991102990o1cB")
|
||||
# Index keyed by set_control_number -> batch_id (D10 batch
|
||||
# envelope index).
|
||||
idx = _seed_ack_link_index(envelope_control_to_batch={
|
||||
"991102989": "B-999A",
|
||||
"991102990": "B-999B",
|
||||
})
|
||||
|
||||
parsed = _parse_999_two_ak2s()
|
||||
out = apply_999_acceptances(s, parsed, ack_id=42,
|
||||
batch_envelope_index=idx)
|
||||
s.commit()
|
||||
|
||||
# Helpers now return ClaimAckLinkRow dataclasses; the caller
|
||||
# (the handler) persists them via cycl_store.add_claim_ack.
|
||||
# Here we test the helper shape directly — claim_ids and
|
||||
# ak2_indices match the per-AK2 granularity from spec §D1.
|
||||
assert {(r.claim_id, r.ak2_index) for r in out.linked} == {
|
||||
("CLM-A", 0),
|
||||
("CLM-B", 1),
|
||||
}
|
||||
assert out.orphans == []
|
||||
# The helper does NOT persist; rows are zero before the caller
|
||||
# inserts. Verify that.
|
||||
with db.SessionLocal()() as s:
|
||||
n = s.query(ClaimAck).filter(ClaimAck.ack_kind == "999", ClaimAck.ack_id == 42).count()
|
||||
assert n == 0
|
||||
|
||||
|
||||
def test_999_orphan_does_not_create_link():
|
||||
"""An AK2 whose set_control_number doesn't resolve stays orphan."""
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, batch_id="B-999-ORPHAN", envelope_control="0001")
|
||||
parsed = _parse_999_two_ak2s()
|
||||
idx = _seed_ack_link_index(envelope_control_to_batch={})
|
||||
out = apply_999_acceptances(s, parsed, ack_id=99,
|
||||
batch_envelope_index=idx)
|
||||
s.commit()
|
||||
|
||||
assert out.linked == []
|
||||
assert sorted(out.orphans) == ["991102989", "991102990"]
|
||||
|
||||
|
||||
def test_277ca_accepted_creates_link_with_no_state_mutation():
|
||||
"""Accepted 277CA → link row, no payer_rejected stamp."""
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, batch_id="B-277A", envelope_control="991102989")
|
||||
claim = _seed_claim(s, claim_id="CLM-Z", batch_id="B-277A",
|
||||
patient_control_number="CLAIM001")
|
||||
idx = _seed_ack_link_index(envelope_control_to_batch={})
|
||||
|
||||
parsed = _parse_277ca_one_accepted()
|
||||
|
||||
def _pcn_lookup(pcn: str):
|
||||
return (s.query(Claim)
|
||||
.filter_by(patient_control_number=pcn)
|
||||
.first())
|
||||
|
||||
out = apply_277ca_acks(s, parsed, ack_id=7,
|
||||
batch_envelope_index=idx,
|
||||
pc_claim_lookup=_pcn_lookup)
|
||||
s.commit()
|
||||
|
||||
assert [(r.claim_id, r.ak2_index) for r in out.linked] == [("CLM-Z", None)]
|
||||
assert out.linked[0].set_accept_reject_code and out.linked[0].set_accept_reject_code.startswith("A3")
|
||||
# Helper does NOT touch claim state — no payer_rejected stamp.
|
||||
with db.SessionLocal()() as s:
|
||||
c = s.get(Claim, "CLM-Z")
|
||||
assert c.payer_rejected_at is None
|
||||
|
||||
|
||||
def test_277ca_rejected_creates_link_and_mutates_state():
|
||||
"""Rejected 277CA → link row + payer_rejected_at stamp (mirrors
|
||||
existing apply_277ca_rejections test)."""
|
||||
from cyclone.inbox_state_277ca import apply_277ca_rejections
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, batch_id="B-277R", envelope_control="991102989")
|
||||
_seed_claim(s, claim_id="CLM-Y", batch_id="B-277R",
|
||||
patient_control_number="CLAIM002")
|
||||
idx = _seed_ack_link_index(envelope_control_to_batch={})
|
||||
parsed = _parse_277ca_one_rejected()
|
||||
|
||||
# First, run the existing rejection mutator (it owns
|
||||
# payer_rejected_at + state). Then run the auto-linker.
|
||||
apply_277ca_rejections(
|
||||
s, parsed,
|
||||
claim_lookup=lambda pcn: (s.query(Claim)
|
||||
.filter_by(patient_control_number=pcn)
|
||||
.first()),
|
||||
two77ca_id=7,
|
||||
)
|
||||
|
||||
def _pcn_lookup(pcn: str):
|
||||
return (s.query(Claim)
|
||||
.filter_by(patient_control_number=pcn)
|
||||
.first())
|
||||
|
||||
out = apply_277ca_acks(s, parsed, ack_id=7,
|
||||
batch_envelope_index=idx,
|
||||
pc_claim_lookup=_pcn_lookup)
|
||||
s.commit()
|
||||
|
||||
assert [(r.claim_id, r.ak2_index) for r in out.linked] == [("CLM-Y", None)]
|
||||
assert out.linked[0].set_accept_reject_code and out.linked[0].set_accept_reject_code.startswith("A6")
|
||||
with db.SessionLocal()() as s:
|
||||
c = s.get(Claim, "CLM-Y")
|
||||
assert c.payer_rejected_at is not None
|
||||
assert c.payer_rejected_status_code and c.payer_rejected_status_code.startswith("A6")
|
||||
|
||||
|
||||
def test_ta1_links_to_most_recent_matching_batch():
|
||||
"""TA1's batch_lookup closure picks the most-recent batch whose
|
||||
envelope.sender_id/receiver_id matches the TA1 envelope."""
|
||||
ta1 = _parse_ta1()
|
||||
# TA1 carries the swapped ISA sender/receiver (the receiving side
|
||||
# is sending the ACK back to the original submitter).
|
||||
with db.SessionLocal()() as s:
|
||||
# Two older batches whose envelope sender/receiver do NOT
|
||||
# match the TA1.
|
||||
_seed_batch(s, batch_id="T1-OLD", envelope_control="0001",
|
||||
sender_id="SENDER", receiver_id="RECEIVER")
|
||||
_seed_batch(s, batch_id="T2-NEW", envelope_control="0002",
|
||||
sender_id="SENDER", receiver_id="RECEIVER")
|
||||
# The matching batch has the swapped sender/receiver; ordered
|
||||
# by parsed_at DESC and prefixed T3-MATCH so the most-recent
|
||||
# one wins.
|
||||
_seed_batch(s, batch_id="T3-MATCH", envelope_control="0003",
|
||||
sender_id="RECEIVER", receiver_id="SENDER")
|
||||
|
||||
def _batch_lookup(sender_id, receiver_id):
|
||||
rows = (
|
||||
s.query(Batch)
|
||||
.filter(Batch.kind == "837p")
|
||||
.order_by(Batch.parsed_at.desc())
|
||||
.all()
|
||||
)
|
||||
for row in rows:
|
||||
env = (row.raw_result_json or {}).get("envelope") or {}
|
||||
if (
|
||||
env.get("sender_id") == sender_id
|
||||
and env.get("receiver_id") == receiver_id
|
||||
):
|
||||
return row
|
||||
return None
|
||||
|
||||
out = apply_ta1_envelope_link(s, ta1, ack_id=11,
|
||||
batch_lookup=_batch_lookup)
|
||||
s.commit()
|
||||
|
||||
assert len(out.linked) == 1
|
||||
assert out.linked[0].claim_id is None
|
||||
assert out.linked[0].batch_id == "T3-MATCH"
|
||||
assert out.linked[0].set_accept_reject_code == "A"
|
||||
assert out.linked[0].ak2_index is None
|
||||
|
||||
|
||||
def test_reingest_same_999_is_idempotent():
|
||||
"""Re-ingesting the same 999 (helper + store.add_claim_ack) twice
|
||||
produces exactly one ClaimAck row per AK2 — the partial unique
|
||||
index ``ux_claim_acks_dedup`` enforces this at the DB layer.
|
||||
The helpers' own pre-check skips rows the dedup already covers.
|
||||
"""
|
||||
parsed = _parse_999_two_ak2s()
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, batch_id="B-IDEMP", envelope_control="991102989")
|
||||
_seed_claim(s, claim_id="CLM-I", batch_id="B-IDEMP",
|
||||
patient_control_number="t991102989o1cI")
|
||||
idx = _seed_ack_link_index(envelope_control_to_batch={
|
||||
"991102989": "B-IDEMP",
|
||||
"991102990": "B-IDEMP",
|
||||
})
|
||||
|
||||
# First ingest: helper returns 2 rows; caller persists.
|
||||
out1 = apply_999_acceptances(s, parsed, ack_id=101,
|
||||
batch_envelope_index=idx)
|
||||
assert len(out1.linked) == 2
|
||||
for row in out1.linked:
|
||||
cycl_store.add_claim_ack(
|
||||
claim_id=row.claim_id,
|
||||
batch_id=row.batch_id,
|
||||
ack_id=101,
|
||||
ack_kind="999",
|
||||
ak2_index=row.ak2_index,
|
||||
set_control_number=row.set_control_number,
|
||||
set_accept_reject_code=row.set_accept_reject_code,
|
||||
linked_by="auto",
|
||||
)
|
||||
|
||||
# Second ingest: dedup index now has rows; helper's pre-check
|
||||
# skips them all, so out2.linked is empty.
|
||||
with db.SessionLocal()() as s:
|
||||
out2 = apply_999_acceptances(s, parsed, ack_id=101,
|
||||
batch_envelope_index=idx)
|
||||
assert out2.linked == []
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
n = s.query(ClaimAck).filter_by(ack_id=101).count()
|
||||
assert n == 2 # exactly one per AK2 — idempotent
|
||||
|
||||
|
||||
def test_manual_link_endpoint_idempotent():
|
||||
"""Idempotency lives at the store layer. ``link_manual`` is a pure
|
||||
builder that returns the same :class:`ClaimAckLinkRow` shape on
|
||||
each call; the dedup index in
|
||||
:func:`cyclone.store.claim_acks.add_claim_ack` keeps re-ingest a
|
||||
no-op."""
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, batch_id="B-MAN", envelope_control="991102989")
|
||||
_seed_claim(s, claim_id="CLM-M", batch_id="B-MAN")
|
||||
|
||||
row1 = link_manual(s, claim_id="CLM-M", ack_kind="999",
|
||||
ack_id=200)
|
||||
cycl_store.add_claim_ack(
|
||||
claim_id=row1.claim_id, batch_id=row1.batch_id,
|
||||
ack_id=200, ack_kind="999", ak2_index=row1.ak2_index,
|
||||
set_control_number=row1.set_control_number,
|
||||
set_accept_reject_code=row1.set_accept_reject_code,
|
||||
linked_by="manual",
|
||||
)
|
||||
with db.SessionLocal()() as s:
|
||||
row2 = link_manual(s, claim_id="CLM-M", ack_kind="999",
|
||||
ack_id=200)
|
||||
# Second add_claim_ack hits the dedup pre-check inside the
|
||||
# helper (when called via the helper+store cycle) — but for
|
||||
# the manual path we let the store's unique index raise.
|
||||
# Instead, verify the dedup SELECT here:
|
||||
existing = (
|
||||
s.query(ClaimAck)
|
||||
.filter_by(claim_id="CLM-M", ack_kind="999", ack_id=200)
|
||||
.one()
|
||||
)
|
||||
assert existing.claim_id == row2.claim_id
|
||||
assert existing.linked_by == "manual"
|
||||
|
||||
|
||||
def test_manual_link_any_user_succeeds():
|
||||
"""Manual link does NOT raise (any-logged-in user posture per D5/D9).
|
||||
|
||||
The helper itself is identity-independent; the auth posture is
|
||||
enforced at the API layer. Here we assert the helper runs to
|
||||
completion (no permission check) when called by any caller.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, batch_id="B-AUTH", envelope_control="991102989")
|
||||
_seed_claim(s, claim_id="CLM-N", batch_id="B-AUTH")
|
||||
row = link_manual(s, claim_id="CLM-N", ack_kind="999",
|
||||
ack_id=300)
|
||||
cycl_store.add_claim_ack(
|
||||
claim_id=row.claim_id, batch_id=row.batch_id,
|
||||
ack_id=300, ack_kind="999", ak2_index=row.ak2_index,
|
||||
set_control_number=row.set_control_number,
|
||||
set_accept_reject_code=row.set_accept_reject_code,
|
||||
linked_by="manual",
|
||||
)
|
||||
assert row.claim_id == "CLM-N"
|
||||
|
||||
|
||||
def test_manual_link_rejects_terminal_claim():
|
||||
"""A claim in state=REVERSED should NOT be link-able (the API
|
||||
layer maps that to 409; the helper itself lets the API pick the
|
||||
claim up and surfaces the state). Here we only assert that
|
||||
``link_manual`` does NOT raise — the API is responsible for the
|
||||
409 decision via :class:`cyclone.db.ClaimState`."""
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, batch_id="B-TERM", envelope_control="991102989")
|
||||
claim = _seed_claim(
|
||||
s, claim_id="CLM-R", batch_id="B-TERM",
|
||||
state=ClaimState.REVERSED,
|
||||
)
|
||||
# Helper itself does no state check — the API maps state to
|
||||
# 409. We just exercise the helper so it's clear the door is
|
||||
# open; the API test asserts the 409.
|
||||
row = link_manual(s, claim_id="CLM-R", ack_kind="999",
|
||||
ack_id=400)
|
||||
assert row.claim_id == "CLM-R"
|
||||
|
||||
|
||||
def test_unlink_does_not_revert_claim_state():
|
||||
"""After unlink via remove_claim_ack, the claim retains its state
|
||||
— unlinking only removes the link row, not the state mutation
|
||||
from the original 999/277CA."""
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, batch_id="B-UL", envelope_control="991102989")
|
||||
_seed_claim(s, claim_id="CLM-UL", batch_id="B-UL",
|
||||
state=ClaimState.REJECTED)
|
||||
# Only the first AK2 resolves (single batch, single claim).
|
||||
idx = _seed_ack_link_index(envelope_control_to_batch={
|
||||
"991102989": "B-UL",
|
||||
})
|
||||
# Submit a 999 with one AK2 so we get one link row.
|
||||
sr = SetFunctionalGroupResponse(
|
||||
ak2=AckHeader999(functional_id_code="837",
|
||||
group_control_number="991102989"),
|
||||
set_control_number="991102989",
|
||||
transaction_set_identifier="837",
|
||||
segment_errors=[],
|
||||
set_accept_reject=SetAcceptReject(code="R"),
|
||||
)
|
||||
parsed = ParseResult999(
|
||||
envelope=Envelope(
|
||||
sender_id="P", receiver_id="S",
|
||||
control_number="0001", transaction_date=date(2026, 7, 2),
|
||||
),
|
||||
functional_group_acks=[],
|
||||
set_responses=[sr],
|
||||
summary=BatchSummary(
|
||||
input_file="one_ak2.txt", control_number="0001",
|
||||
transaction_date=date(2026, 7, 2),
|
||||
total_claims=1, passed=0, failed=1,
|
||||
),
|
||||
)
|
||||
out = apply_999_acceptances(s, parsed, ack_id=500,
|
||||
batch_envelope_index=idx)
|
||||
# Persist via the store so the link row exists in the DB.
|
||||
for row in out.linked:
|
||||
cycl_store.add_claim_ack(
|
||||
claim_id=row.claim_id, batch_id=row.batch_id,
|
||||
ack_id=500, ack_kind="999", ak2_index=row.ak2_index,
|
||||
set_control_number=row.set_control_number,
|
||||
set_accept_reject_code=row.set_accept_reject_code,
|
||||
linked_by="auto",
|
||||
)
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
links = (
|
||||
s.query(ClaimAck)
|
||||
.filter_by(claim_id="CLM-UL")
|
||||
.all()
|
||||
)
|
||||
assert len(links) == 1
|
||||
link_id = links[0].id
|
||||
# Unlink via the store (publishes claim_ack_dropped).
|
||||
removed = cycl_store.remove_claim_ack(link_id)
|
||||
assert removed is True
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
claim = s.get(Claim, "CLM-UL")
|
||||
assert claim.state == ClaimState.REJECTED
|
||||
assert (
|
||||
s.query(ClaimAck).filter_by(claim_id="CLM-UL").first() is None
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 2.5 — D10 two-pass join + multi-claim batch coverage
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_lookup_claims_two_pass_join():
|
||||
"""D10: Pass 1 (batch.envelope.control_number) + Pass 2 (PCN)."""
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, batch_id="B-TP-A", envelope_control="991102989")
|
||||
_seed_claim(s, claim_id="C-TP-A", batch_id="B-TP-A",
|
||||
patient_control_number="t991102989o1cA")
|
||||
# No matching batch — Pass 2 alone resolves it.
|
||||
_seed_batch(s, batch_id="B-TP-B", envelope_control="OTHER")
|
||||
_seed_claim(s, claim_id="C-TP-B", batch_id="B-TP-B",
|
||||
patient_control_number="991102987")
|
||||
|
||||
# Pass 1: "991102989" -> B-TP-A -> [C-TP-A]
|
||||
idx = _seed_ack_link_index(envelope_control_to_batch={
|
||||
"991102989": "B-TP-A",
|
||||
"991102990": "B-TP-A",
|
||||
})
|
||||
|
||||
# PCN-only claim (no batch match). Pass 2 must find it.
|
||||
def _pcn_lookup(pcn: str):
|
||||
return s.query(Claim).filter_by(patient_control_number=pcn).first()
|
||||
|
||||
pass1 = lookup_claims_for_ack_set_response(
|
||||
s, "991102989",
|
||||
batch_envelope_index=idx,
|
||||
pc_claim_lookup=_pcn_lookup,
|
||||
)
|
||||
pass2 = lookup_claims_for_ack_set_response(
|
||||
s, "991102987",
|
||||
batch_envelope_index=idx,
|
||||
pc_claim_lookup=_pcn_lookup,
|
||||
)
|
||||
miss = lookup_claims_for_ack_set_response(
|
||||
s, "0001",
|
||||
batch_envelope_index=idx,
|
||||
pc_claim_lookup=_pcn_lookup,
|
||||
)
|
||||
|
||||
assert [c.id for c in pass1] == ["C-TP-A"]
|
||||
assert [c.id for c in pass2] == ["C-TP-B"]
|
||||
assert miss == []
|
||||
|
||||
|
||||
def test_lookup_claims_pass1_wins_over_pass2():
|
||||
"""If a PCN matches a claim in a DIFFERENT batch than the Pass 1
|
||||
batch, only the Pass 1 result is returned (no double-fire)."""
|
||||
with db.SessionLocal()() as s:
|
||||
# Claim A lives in B1 (whose ST02 == '991102989'). Pass 1 hits.
|
||||
_seed_batch(s, batch_id="B-PP-1", envelope_control="991102989")
|
||||
_seed_claim(s, claim_id="C-PP-A", batch_id="B-PP-1",
|
||||
patient_control_number="991102989")
|
||||
# Claim B lives in B2 (ST02 == 'OTHER') but has the same PCN
|
||||
# by coincidence. Pass 1 misses for '991102989' on B2 so Pass 2
|
||||
# SHOULD match it — but only when Pass 1 returns nothing.
|
||||
_seed_batch(s, batch_id="B-PP-2", envelope_control="OTHER")
|
||||
_seed_claim(s, claim_id="C-PP-B", batch_id="B-PP-2",
|
||||
patient_control_number="991102989")
|
||||
|
||||
idx = _seed_ack_link_index(envelope_control_to_batch={
|
||||
"991102989": "B-PP-1",
|
||||
})
|
||||
rows = lookup_claims_for_ack_set_response(
|
||||
s, "991102989",
|
||||
batch_envelope_index=idx,
|
||||
pc_claim_lookup=lambda pcn: s.query(Claim)
|
||||
.filter_by(patient_control_number=pcn)
|
||||
.first(),
|
||||
)
|
||||
|
||||
# Pass 1 must win and skip Pass 2.
|
||||
assert [c.id for c in rows] == ["C-PP-A"]
|
||||
|
||||
|
||||
def test_999_linker_emits_one_row_per_claim_in_multi_claim_batch():
|
||||
"""One 999 AK2 + N claims in one batch → N ClaimAck rows for that
|
||||
single AK2 (one-ack-to-many)."""
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, batch_id="B-MC", envelope_control="991102989")
|
||||
_seed_claim(s, claim_id="C-MC-1", batch_id="B-MC",
|
||||
patient_control_number="t991102989o1c1")
|
||||
_seed_claim(s, claim_id="C-MC-2", batch_id="B-MC",
|
||||
patient_control_number="t991102989o1c2")
|
||||
idx = _seed_ack_link_index(envelope_control_to_batch={
|
||||
"991102989": "B-MC",
|
||||
})
|
||||
|
||||
# 999 with one AK2 whose set_control_number matches the batch.
|
||||
sr = SetFunctionalGroupResponse(
|
||||
ak2=AckHeader999(functional_id_code="837",
|
||||
group_control_number="991102989"),
|
||||
set_control_number="991102989",
|
||||
transaction_set_identifier="837",
|
||||
segment_errors=[],
|
||||
set_accept_reject=SetAcceptReject(code="A"),
|
||||
)
|
||||
parsed = ParseResult999(
|
||||
envelope=Envelope(
|
||||
sender_id="P", receiver_id="S",
|
||||
control_number="0001", transaction_date=date(2026, 7, 2),
|
||||
),
|
||||
functional_group_acks=[],
|
||||
set_responses=[sr],
|
||||
summary=BatchSummary(
|
||||
input_file="one_ak2.txt", control_number="0001",
|
||||
transaction_date=date(2026, 7, 2),
|
||||
total_claims=1, passed=1, failed=0,
|
||||
),
|
||||
)
|
||||
out = apply_999_acceptances(s, parsed, ack_id=42,
|
||||
batch_envelope_index=idx)
|
||||
# Persist via the store to verify the full helper→store cycle.
|
||||
for row in out.linked:
|
||||
cycl_store.add_claim_ack(
|
||||
claim_id=row.claim_id, batch_id=row.batch_id,
|
||||
ack_id=42, ack_kind="999", ak2_index=row.ak2_index,
|
||||
set_control_number=row.set_control_number,
|
||||
set_accept_reject_code=row.set_accept_reject_code,
|
||||
linked_by="auto",
|
||||
)
|
||||
|
||||
assert {(r.claim_id, r.ak2_index) for r in out.linked} == {
|
||||
("C-MC-1", 0),
|
||||
("C-MC-2", 0),
|
||||
}
|
||||
with db.SessionLocal()() as s:
|
||||
n = s.query(ClaimAck).filter_by(ack_kind="999").count()
|
||||
assert n == 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 3.4 — facade wiring test (placeholder; full Phase 3 adds the
|
||||
# store methods). The presence-only check verifies the public surface
|
||||
# later exposes the same names — kept here so the plan-step stays
|
||||
# trackable.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_store_facade_exposes_claim_ack_methods():
|
||||
"""Step 3.4: facade must expose add_claim_ack + 4 read methods."""
|
||||
expected = [
|
||||
"add_claim_ack",
|
||||
"list_acks_for_claim",
|
||||
"list_claims_for_ack",
|
||||
"find_ack_orphans",
|
||||
"remove_claim_ack",
|
||||
"batch_envelope_index",
|
||||
]
|
||||
for name in expected:
|
||||
assert hasattr(store, name), f"missing CycloneStore.{name}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 2.1 — pure walk-through unit test (no DB) for the orphan tracking
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_999_linker_walks_set_responses():
|
||||
"""Stub-based: one AK2 resolves, one orphan, no DB.
|
||||
|
||||
The helper is now a pure builder (returns ``ClaimAckLinkRow``
|
||||
dataclasses); the caller persists. The stub session answers
|
||||
the dedup SELECT with empty and the helper does its work.
|
||||
"""
|
||||
# Mock claim_lookup: "AAA" -> Claim-like, otherwise None.
|
||||
class _StubClaim:
|
||||
def __init__(self, cid):
|
||||
self.id = cid
|
||||
|
||||
from cyclone.parsers.models_999 import (
|
||||
ParseResult999, SetAcceptReject, SetFunctionalGroupResponse,
|
||||
AcknowledgmentHeader as AckHdr,
|
||||
)
|
||||
|
||||
parse_result = ParseResult999.model_construct()
|
||||
parse_result.envelope = Envelope.model_construct(
|
||||
sender_id="", receiver_id="", control_number="",
|
||||
transaction_date=date(2026, 7, 2),
|
||||
)
|
||||
parse_result.functional_group_acks = []
|
||||
parse_result.summary = BatchSummary(
|
||||
input_file="stub.txt", control_number="",
|
||||
transaction_date=date(2026, 7, 2),
|
||||
total_claims=2, passed=1, failed=1,
|
||||
)
|
||||
parse_result.set_responses = [
|
||||
SetFunctionalGroupResponse.model_construct(
|
||||
ak2=AckHdr(functional_id_code="837", group_control_number="AAA"),
|
||||
set_control_number="AAA",
|
||||
transaction_set_identifier="837",
|
||||
segment_errors=[],
|
||||
set_accept_reject=SetAcceptReject.model_construct(code="A"),
|
||||
),
|
||||
SetFunctionalGroupResponse.model_construct(
|
||||
ak2=AckHdr(functional_id_code="837", group_control_number="BBB"),
|
||||
set_control_number="BBB",
|
||||
transaction_set_identifier="837",
|
||||
segment_errors=[],
|
||||
set_accept_reject=SetAcceptReject.model_construct(code="R"),
|
||||
),
|
||||
]
|
||||
|
||||
seen: list[str] = []
|
||||
|
||||
def _pcn_lookup(pcn):
|
||||
seen.append(pcn)
|
||||
return _StubClaim(f"CLM-{pcn}") if pcn == "AAA" else None
|
||||
|
||||
class _StubQuery:
|
||||
def filter(self, *args, **kwargs):
|
||||
return self
|
||||
def all(self):
|
||||
return []
|
||||
def first(self):
|
||||
return None # no existing link rows in stub
|
||||
|
||||
class _StubSession:
|
||||
"""Pretend to be a Session; answer the dedup SELECT empty so
|
||||
the helper stays pure-unit."""
|
||||
def __init__(self):
|
||||
self.adds: list[object] = []
|
||||
|
||||
def add(self, obj): # noqa: D401
|
||||
self.adds.append(obj)
|
||||
|
||||
def flush(self): # noqa: D401
|
||||
pass
|
||||
|
||||
def query(self, *args, **kwargs):
|
||||
return _StubQuery()
|
||||
|
||||
s = _StubSession()
|
||||
out = apply_999_acceptances(
|
||||
s, parse_result,
|
||||
ack_id=1,
|
||||
batch_envelope_index=lambda scn: None, # Pass 1 misses → Pass 2 fires
|
||||
pc_claim_lookup=_pcn_lookup,
|
||||
)
|
||||
|
||||
assert seen == ["AAA", "BBB"]
|
||||
assert [(r.claim_id, r.ak2_index) for r in out.linked] == [("CLM-AAA", 0)]
|
||||
assert out.orphans == ["BBB"]
|
||||
# Helper no longer persists; verify no inserts were made.
|
||||
assert s.adds == []
|
||||
# Dataclass carries the per-AK2 data the caller needs.
|
||||
row = out.linked[0]
|
||||
assert isinstance(row, ClaimAckLinkRow)
|
||||
assert row.claim_id == "CLM-AAA"
|
||||
assert row.ak2_index == 0
|
||||
assert row.set_control_number == "AAA"
|
||||
assert row.set_accept_reject_code == "A"
|
||||
@@ -0,0 +1,227 @@
|
||||
"""SP32 Task 6: end-to-end smoke tests for the ``backfill-rendering-npi`` CLI.
|
||||
|
||||
The CLI re-parses on-disk 837p + 835 files and backfills the new typed
|
||||
``rendering_provider_npi`` columns on existing Claim / Remittance rows
|
||||
that were ingested before the T4 writers took effect. Idempotent: rows
|
||||
that already have a non-NULL rendering NPI are left untouched.
|
||||
|
||||
Test strategy mirrors ``test_cli_backup.py``: in-process Click CliRunner
|
||||
invocation against a fresh per-test DB (autouse ``_auto_init_db``
|
||||
fixture), pre-seeded with Claim/Remittance rows whose
|
||||
``rendering_provider_npi`` is NULL, then assert the CLI populated them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from decimal import Decimal
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.cli import main
|
||||
|
||||
BACKEND_DIR = "backend"
|
||||
_REPO_ROOT = "."
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def seed_837p_claim_without_renderer():
|
||||
"""Insert a Claim row whose ``rendering_provider_npi`` is NULL.
|
||||
|
||||
The id (``t991102984o1c1d``) matches the first claim in
|
||||
``co_medicaid_837p_with_renderer.txt`` — the fixture that does
|
||||
contain a NM1*82 rendering provider (NPI ``1234567893``) attached
|
||||
to that claim. Backfill should populate the column from the
|
||||
re-parsed file.
|
||||
"""
|
||||
from cyclone.db import Batch, Claim, ClaimState
|
||||
with db.SessionLocal()() as s:
|
||||
s.add(Batch(
|
||||
id="seed-837p-batch",
|
||||
kind="837p",
|
||||
input_filename="seed.x12",
|
||||
parsed_at=datetime.now(timezone.utc),
|
||||
))
|
||||
s.add(Claim(
|
||||
id="t991102984o1c1d",
|
||||
batch_id="seed-837p-batch",
|
||||
patient_control_number="t991102984o1c1d",
|
||||
charge_amount=Decimal("85.40"),
|
||||
state=ClaimState.SUBMITTED,
|
||||
rendering_provider_npi=None, # ← the column under test
|
||||
provider_npi="1881068062",
|
||||
))
|
||||
s.commit()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def seed_835_remit_without_service_provider(tmp_path):
|
||||
"""Insert a Remittance whose ``rendering_provider_npi`` is NULL.
|
||||
|
||||
Writes a minimal 835 fixture to ``tmp_path`` containing an
|
||||
NM1*1P service provider (NPI ``2222222222``) attached to the
|
||||
only CLP segment (claim control number ``CLM001``). Returns
|
||||
the path to the written file.
|
||||
"""
|
||||
from cyclone.db import Batch, Remittance
|
||||
|
||||
fixture = tmp_path / "minimal_835_with_service_provider.txt"
|
||||
fixture.write_text(
|
||||
"ISA*00* *00* *ZZ*RECEIVERID *ZZ*SENDERID "
|
||||
"*250101*1200*^*00501*000000001*0*P*:~"
|
||||
"GS*HP*RECEIVER*SENDER*20250101*1200*1*X*005010X221A1~"
|
||||
"ST*835*0001~"
|
||||
"BPR*I*85.40*C*NON*****01*021000021*DA*123456*1511111"
|
||||
"**01*031302955*DA*9876543~"
|
||||
"TRN*1*1511111*1511111~"
|
||||
"DTM*405*20250101~"
|
||||
"N1*PR*COLORADO MEDICAID*XV*CO MEDICAID~"
|
||||
"N3*PO BOX 1100*~"
|
||||
"N4*DENVER*CO*80203~"
|
||||
"REF*2U*12345~"
|
||||
"N1*PE*ACME CLINIC*XX*1111111111~"
|
||||
"LX*1~"
|
||||
"CLP*CLM001*1*85.40*85.40*0.00*MC*CLM001*11*1~"
|
||||
"CAS*PR*1*0.00~"
|
||||
"NM1*1P*2*RENDERING PROVIDER*****XX*2222222222~"
|
||||
"SVC*HC:99213*85.40*85.40**1~"
|
||||
"DTM*472*20250101~"
|
||||
"SE*18*0001~"
|
||||
"GE*1*1~"
|
||||
"IEA*1*000000001~"
|
||||
)
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
s.add(Batch(
|
||||
id="seed-835-batch",
|
||||
kind="835",
|
||||
input_filename=fixture.name,
|
||||
parsed_at=datetime.now(timezone.utc),
|
||||
))
|
||||
s.add(Remittance(
|
||||
id="CLM001",
|
||||
batch_id="seed-835-batch",
|
||||
payer_claim_control_number="CLM001",
|
||||
claim_id=None,
|
||||
status_code="1",
|
||||
status_label="Primary payer forward",
|
||||
total_charge=Decimal("85.40"),
|
||||
total_paid=Decimal("85.40"),
|
||||
patient_responsibility=Decimal("0"),
|
||||
adjustment_amount=Decimal("0"),
|
||||
received_at=datetime.now(timezone.utc),
|
||||
is_reversal=False,
|
||||
rendering_provider_npi=None, # ← the column under test
|
||||
))
|
||||
s.commit()
|
||||
|
||||
return fixture
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _run(args: list[str]) -> object:
|
||||
"""Invoke the CLI in-process via CliRunner.
|
||||
|
||||
CliRunner auto-captures stdout/stderr; ``catch_exceptions=False``
|
||||
so the test fails loudly on any unexpected raise instead of a
|
||||
swallowed traceback.
|
||||
"""
|
||||
return CliRunner().invoke(main, args, catch_exceptions=False)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_backfill_populates_837p_rendering_npi(seed_837p_claim_without_renderer):
|
||||
"""Re-parse the 837p fixture → Claim.rendering_provider_npi populated."""
|
||||
fixture_path = "tests/fixtures/co_medicaid_837p_with_renderer.txt"
|
||||
|
||||
# Sanity: column starts NULL.
|
||||
with db.SessionLocal()() as s:
|
||||
from cyclone.db import Claim
|
||||
row = s.get(Claim, "t991102984o1c1d")
|
||||
assert row is not None
|
||||
assert row.rendering_provider_npi is None
|
||||
|
||||
result = _run(["backfill-rendering-npi", "--file", fixture_path, "--type", "837p"])
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
from cyclone.db import Claim
|
||||
row = s.get(Claim, "t991102984o1c1d")
|
||||
assert row is not None
|
||||
# First claim in the fixture carries NM1*82 NPI ``1234567893``.
|
||||
assert row.rendering_provider_npi == "1234567893"
|
||||
# CLI prints the summary line with the populated counts.
|
||||
assert "claims_updated=" in result.output
|
||||
assert "remits_updated=" in result.output
|
||||
|
||||
|
||||
def test_backfill_populates_835_rendering_npi(seed_835_remit_without_service_provider):
|
||||
"""Re-parse an 835 with NM1*1P → Remittance.rendering_provider_npi populated."""
|
||||
fixture_path = str(seed_835_remit_without_service_provider)
|
||||
|
||||
# Sanity: column starts NULL.
|
||||
with db.SessionLocal()() as s:
|
||||
from cyclone.db import Remittance
|
||||
row = s.get(Remittance, "CLM001")
|
||||
assert row is not None
|
||||
assert row.rendering_provider_npi is None
|
||||
|
||||
result = _run(["backfill-rendering-npi", "--file", fixture_path, "--type", "835"])
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
from cyclone.db import Remittance
|
||||
row = s.get(Remittance, "CLM001")
|
||||
assert row is not None
|
||||
# Inline fixture's NM1*1P NM109 is ``2222222222``.
|
||||
assert row.rendering_provider_npi == "2222222222"
|
||||
assert "remits_updated=" in result.output
|
||||
|
||||
|
||||
def test_backfill_is_idempotent(seed_837p_claim_without_renderer):
|
||||
"""Second run is a no-op — already-populated columns stay."""
|
||||
fixture_path = "tests/fixtures/co_medicaid_837p_with_renderer.txt"
|
||||
|
||||
# First run: populates the column.
|
||||
first = _run(["backfill-rendering-npi", "--file", fixture_path, "--type", "837p"])
|
||||
assert first.exit_code == 0, first.output
|
||||
|
||||
# Second run: still exits 0; column is unchanged; nothing breaks.
|
||||
second = _run(["backfill-rendering-npi", "--file", fixture_path, "--type", "837p"])
|
||||
assert second.exit_code == 0, second.output
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
from cyclone.db import Claim
|
||||
row = s.get(Claim, "t991102984o1c1d")
|
||||
assert row.rendering_provider_npi == "1234567893"
|
||||
|
||||
|
||||
def test_backfill_help_lists_subcommand():
|
||||
"""Regression guard: the subcommand is wired into ``main``."""
|
||||
result = _run(["--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "backfill-rendering-npi" in result.output
|
||||
|
||||
|
||||
def test_backfill_with_no_files_is_noop():
|
||||
"""Running with no --file and no --input-dir is a clean no-op (exit 0)."""
|
||||
result = _run(["backfill-rendering-npi"])
|
||||
assert result.exit_code == 0, result.output
|
||||
# Summary line still printed (zero counts).
|
||||
assert "claims_updated=0" in result.output
|
||||
assert "remits_updated=0" in result.output
|
||||
@@ -30,8 +30,8 @@ def test_get_clearhouse_seeded(client):
|
||||
assert body["name"] == "dzinesco"
|
||||
assert body["tpid"] == "11525703"
|
||||
assert body["sftp_block"]["stub"] is True
|
||||
assert "FromHPE" in body["sftp_block"]["paths"]["outbound"]
|
||||
assert "ToHPE" in body["sftp_block"]["paths"]["inbound"]
|
||||
assert "ToHPE" in body["sftp_block"]["paths"]["outbound"]
|
||||
assert "FromHPE" in body["sftp_block"]["paths"]["inbound"]
|
||||
|
||||
|
||||
def test_list_providers(client):
|
||||
|
||||
@@ -36,5 +36,5 @@ def test_subscriber_payer_reused_across_claims():
|
||||
result = parse(FIXTURE.read_text(), PayerConfig.co_medicaid())
|
||||
assert result.claims[0].billing_provider.npi == "1881068062"
|
||||
assert result.claims[1].billing_provider.npi == "1881068062"
|
||||
assert result.claims[0].payer.id == "SKCO0"
|
||||
assert result.claims[1].payer.id == "SKCO0"
|
||||
assert result.claims[0].payer.id == "CO_TXIX"
|
||||
assert result.claims[1].payer.id == "CO_TXIX"
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
"""SP33 regression test: ``PayerConfig.co_medicaid()`` emits CO_TXIX.
|
||||
|
||||
The HCPF 837P Companion Guide (June 2025 - Version 2.5) requires
|
||||
``NM1*PR NM108=PI, NM109=CO_TXIX`` for the Payer Name loop (2010BB).
|
||||
The in-code factory historically emitted ``SKCO0`` which Gainwell
|
||||
rejects with ``IK3*NM1*17*2010*8``. This test pins the corrected value.
|
||||
See ``docs/superpowers/specs/2026-07-02-cyclone-co-txix-payer-fix-design.md``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def test_co_medicaid_factory_emits_CO_TXIX():
|
||||
"""Per HCPF 837P Companion Guide, NM1*PR NM109 must equal CO_TXIX."""
|
||||
from cyclone.parsers.payer import PayerConfig
|
||||
cfg = PayerConfig.co_medicaid()
|
||||
assert cfg.payer_id == "CO_TXIX", (
|
||||
f"co_medicaid().payer_id must be 'CO_TXIX' (HCPF guide), got {cfg.payer_id!r}"
|
||||
)
|
||||
assert cfg.payer_name == "CO_TXIX", (
|
||||
f"co_medicaid().payer_name must be 'CO_TXIX' (matches docs/goodclaim.x12), got {cfg.payer_name!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_skco0_not_present_in_factory():
|
||||
"""Hard guard: SKCO0 must never be emitted by the factory again."""
|
||||
from cyclone.parsers.payer import PayerConfig
|
||||
cfg = PayerConfig.co_medicaid()
|
||||
assert "SKCO0" not in cfg.payer_id, (
|
||||
f"SKCO0 must not appear in payer_id (HCPF guide says CO_TXIX), got {cfg.payer_id!r}"
|
||||
)
|
||||
assert "SKCO0" not in cfg.payer_name, (
|
||||
f"SKCO0 must not appear in payer_name, got {cfg.payer_name!r}"
|
||||
)
|
||||
@@ -0,0 +1,500 @@
|
||||
"""Tests for the ``GET /api/dashboard/kpis`` aggregate endpoint (SP27 Task 13).
|
||||
|
||||
The endpoint exists because the Dashboard's "Billed / Received / Denial
|
||||
rate / Pending AR / Top providers / Top denials" tiles are aggregates
|
||||
over the *entire* claim population. With 60k+ claims in production,
|
||||
fetching ``/api/claims?limit=100`` and reducing client-side silently
|
||||
produces wrong numbers. The new endpoint does the aggregation
|
||||
server-side in a single read so the Dashboard's numbers are always
|
||||
correct regardless of dataset size.
|
||||
|
||||
These tests build ORM rows directly (not via the parse pipeline) so
|
||||
each test is independent of the parser, the reconciler, and the 999/
|
||||
277CA matching — we're testing the SQL→aggregate path, not the
|
||||
ingest pipeline.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from decimal import Decimal
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.api import app
|
||||
from cyclone.db import Batch, Claim, ClaimState, Provider, Remittance
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _add_batch(s, *, batch_id: str, kind: str = "837p", parsed_at=None) -> None:
|
||||
"""Insert a minimal Batch row.
|
||||
|
||||
Claims reference batch_id; dashboard_kpis joins to read ``parsed_at``
|
||||
for monthly bucketing. ``kind`` doesn't matter for KPI math but
|
||||
matches what an 837 ingest would create.
|
||||
"""
|
||||
s.add(Batch(
|
||||
id=batch_id,
|
||||
kind=kind,
|
||||
input_filename="seed.edi",
|
||||
parsed_at=parsed_at or datetime(2026, 6, 15, 12, 0, tzinfo=timezone.utc),
|
||||
totals_json={"total_claims": 1},
|
||||
validation_json={"passed": True, "warnings": [], "errors": []},
|
||||
raw_result_json={"_": "stub"},
|
||||
))
|
||||
|
||||
|
||||
def _add_claim(
|
||||
s,
|
||||
*,
|
||||
claim_id: str,
|
||||
batch_id: str = "b1",
|
||||
pcn: str | None = None,
|
||||
charge: str = "100.00",
|
||||
state: ClaimState = ClaimState.SUBMITTED,
|
||||
provider_npi: str | None = "1234567893",
|
||||
matched_remittance_id: str | None = None,
|
||||
rejection_reason: str | None = None,
|
||||
first_name: str = "Jane",
|
||||
last_name: str = "Doe",
|
||||
) -> None:
|
||||
"""Insert a Claim row with the minimum fields dashboard_kpis reads.
|
||||
|
||||
``pcn`` defaults to ``claim_id`` so the claim is uniquely
|
||||
identifiable. ``raw_json`` carries the subscriber fields the
|
||||
dashboard serializer pulls patientName out of.
|
||||
"""
|
||||
s.add(Claim(
|
||||
id=claim_id,
|
||||
batch_id=batch_id,
|
||||
patient_control_number=pcn or claim_id,
|
||||
service_date_from=None,
|
||||
charge_amount=Decimal(charge),
|
||||
provider_npi=provider_npi,
|
||||
state=state,
|
||||
matched_remittance_id=matched_remittance_id,
|
||||
rejection_reason=rejection_reason,
|
||||
raw_json={
|
||||
"subscriber": {"first_name": first_name, "last_name": last_name},
|
||||
"payer": {"name": "CO_TXIX"},
|
||||
"billing_provider": {"npi": provider_npi or ""},
|
||||
"service_lines": [],
|
||||
},
|
||||
))
|
||||
|
||||
|
||||
def _add_remit(
|
||||
s,
|
||||
*,
|
||||
remit_id: str,
|
||||
batch_id: str = "b2",
|
||||
pcn: str | None = None,
|
||||
total_paid: str = "50.00",
|
||||
total_charge: str = "100.00",
|
||||
) -> None:
|
||||
s.add(Remittance(
|
||||
id=remit_id,
|
||||
batch_id=batch_id,
|
||||
payer_claim_control_number=pcn or remit_id,
|
||||
status_code="1",
|
||||
total_charge=Decimal(total_charge),
|
||||
total_paid=Decimal(total_paid),
|
||||
received_at=datetime(2026, 6, 15, 12, 0, tzinfo=timezone.utc),
|
||||
service_date=None,
|
||||
is_reversal=False,
|
||||
))
|
||||
|
||||
|
||||
def _add_provider(s, *, npi: str, label: str) -> None:
|
||||
"""Insert a Provider row with all NOT NULL columns populated.
|
||||
|
||||
Provider.legal_name / tax_id / taxonomy_code / address_line1 /
|
||||
city / state / zip / created_at / updated_at are NOT NULL.
|
||||
"""
|
||||
s.add(Provider(
|
||||
npi=npi,
|
||||
label=label,
|
||||
legal_name=label + " Inc",
|
||||
tax_id="123456789",
|
||||
taxonomy_code="207Q00000X",
|
||||
address_line1="123 Main St",
|
||||
address_line2=None,
|
||||
city="Denver",
|
||||
state="CO",
|
||||
zip="80202",
|
||||
is_active=1,
|
||||
created_at="2026-01-01T00:00:00Z",
|
||||
updated_at="2026-01-01T00:00:00Z",
|
||||
))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client() -> TestClient:
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests — module-level function (no HTTP) so we can pin the aggregation
|
||||
# logic independent of the API wiring.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_dashboard_kpis_empty_db():
|
||||
"""Empty DB → zero totals + monthly skeleton of N entries.
|
||||
|
||||
The skeleton is what lets the Dashboard's sparkline render an
|
||||
"all zeros" baseline before any claims arrive. Lock that contract.
|
||||
"""
|
||||
from cyclone.store import dashboard_kpis
|
||||
|
||||
out = dashboard_kpis(months=6, top_n_providers=4, top_n_denials=5)
|
||||
|
||||
assert out["totals"] == {
|
||||
"count": 0,
|
||||
"billed": 0.0,
|
||||
"received": 0.0,
|
||||
"outstandingAr": 0.0,
|
||||
"denied": 0,
|
||||
"denialRate": 0.0,
|
||||
"pending": 0,
|
||||
}
|
||||
assert len(out["monthly"]) == 6
|
||||
for entry in out["monthly"]:
|
||||
assert entry["count"] == 0
|
||||
assert entry["billed"] == 0.0
|
||||
assert entry["received"] == 0.0
|
||||
assert entry["denied"] == 0
|
||||
assert entry["denialRate"] == 0.0
|
||||
assert entry["ar"] == 0.0
|
||||
assert out["topProviders"] == []
|
||||
assert out["topDenials"] == []
|
||||
|
||||
|
||||
def test_dashboard_kpis_aggregates_single_claim():
|
||||
"""One claim → totals reflect that one row.
|
||||
|
||||
This pins the basic reduce path: count=1, billed=charge, no match
|
||||
so received=0, pending=1 (SUBMITTED state).
|
||||
"""
|
||||
from cyclone.store import dashboard_kpis
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
_add_batch(s, batch_id="b1", parsed_at=datetime(2026, 6, 15, tzinfo=timezone.utc))
|
||||
_add_claim(s, claim_id="C-1", charge="250.00", state=ClaimState.SUBMITTED)
|
||||
s.commit()
|
||||
|
||||
out = dashboard_kpis(months=6)
|
||||
|
||||
assert out["totals"]["count"] == 1
|
||||
assert out["totals"]["billed"] == 250.0
|
||||
assert out["totals"]["received"] == 0.0
|
||||
assert out["totals"]["outstandingAr"] == 250.0
|
||||
assert out["totals"]["denied"] == 0
|
||||
assert out["totals"]["denialRate"] == 0.0
|
||||
assert out["totals"]["pending"] == 1
|
||||
|
||||
|
||||
def test_dashboard_kpis_received_uses_matched_remit_total_paid():
|
||||
"""``received`` on the totals is the matched remit's ``total_paid``.
|
||||
|
||||
Without this assertion a refactor that read ``Remittance.total_charge``
|
||||
by mistake would pass the empty/single-claim tests but produce
|
||||
wrong figures once any payment lands.
|
||||
"""
|
||||
from cyclone.store import dashboard_kpis
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
_add_batch(s, batch_id="b1")
|
||||
_add_claim(
|
||||
s,
|
||||
claim_id="C-1",
|
||||
charge="100.00",
|
||||
state=ClaimState.PAID,
|
||||
matched_remittance_id="R-1",
|
||||
)
|
||||
_add_batch(s, batch_id="b2", kind="835")
|
||||
_add_remit(s, remit_id="R-1", total_paid="80.00", total_charge="100.00")
|
||||
s.commit()
|
||||
|
||||
out = dashboard_kpis(months=6)
|
||||
|
||||
assert out["totals"]["billed"] == 100.0
|
||||
assert out["totals"]["received"] == 80.0 # not 100 (total_charge), not 0
|
||||
assert out["totals"]["outstandingAr"] == 20.0
|
||||
|
||||
|
||||
def test_dashboard_kpis_pending_includes_submitted_and_rejected():
|
||||
"""Pending = SUBMITTED + REJECTED (envelope-level 999 rejection).
|
||||
|
||||
REJECTED here is distinct from DENIED (payer adjudication). Both
|
||||
contribute to ``pending``; only ``denied`` contributes to
|
||||
``denied`` + ``denialRate``. Lock that semantics.
|
||||
"""
|
||||
from cyclone.store import dashboard_kpis
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
_add_batch(s, batch_id="b1")
|
||||
_add_claim(s, claim_id="C-SUB", state=ClaimState.SUBMITTED)
|
||||
_add_claim(s, claim_id="C-REJ", state=ClaimState.REJECTED)
|
||||
_add_claim(s, claim_id="C-DEN", state=ClaimState.DENIED, charge="50.00")
|
||||
_add_claim(s, claim_id="C-PAID", state=ClaimState.PAID, charge="200.00")
|
||||
s.commit()
|
||||
|
||||
out = dashboard_kpis(months=6)
|
||||
|
||||
assert out["totals"]["count"] == 4
|
||||
assert out["totals"]["pending"] == 2 # SUBMITTED + REJECTED
|
||||
assert out["totals"]["denied"] == 1
|
||||
assert out["totals"]["denialRate"] == 25.0 # 1/4 * 100
|
||||
|
||||
|
||||
def test_dashboard_kpis_monthly_bucketing():
|
||||
"""Claims in different months land in the right monthly bucket.
|
||||
|
||||
Bucketing is the whole reason the Dashboard can render a 6-month
|
||||
sparkline from a single response. If the bin key drifts (e.g.
|
||||
uses ``service_date_to`` instead of ``batch.parsed_at``) the
|
||||
sparkline becomes a confusing shape.
|
||||
"""
|
||||
from cyclone.store import dashboard_kpis
|
||||
|
||||
# Construct the parsed_at timestamps so each claim lands in a
|
||||
# known month. Use distinct months within the trailing-6 window.
|
||||
now = datetime.now(timezone.utc)
|
||||
def months_ago(n: int) -> datetime:
|
||||
d = now.replace(day=1)
|
||||
for _ in range(n):
|
||||
d = d.replace(
|
||||
month=d.month - 1 if d.month > 1 else 12,
|
||||
year=d.year if d.month > 1 else d.year - 1,
|
||||
)
|
||||
return d
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
# Three claims across three different months.
|
||||
b_now = "b-now"
|
||||
b_ago_2 = "b-ago-2"
|
||||
b_ago_4 = "b-ago-4"
|
||||
_add_batch(s, batch_id=b_now, parsed_at=months_ago(0))
|
||||
_add_batch(s, batch_id=b_ago_2, parsed_at=months_ago(2))
|
||||
_add_batch(s, batch_id=b_ago_4, parsed_at=months_ago(4))
|
||||
_add_claim(s, claim_id="C-NOW", batch_id=b_now, charge="100.00")
|
||||
_add_claim(s, claim_id="C-AGO-2", batch_id=b_ago_2, charge="200.00")
|
||||
_add_claim(s, claim_id="C-AGO-4", batch_id=b_ago_4, charge="300.00")
|
||||
s.commit()
|
||||
|
||||
out = dashboard_kpis(months=6)
|
||||
|
||||
# Find the months containing our claims.
|
||||
by_month = {entry["month"]: entry for entry in out["monthly"]}
|
||||
this_month = f"{months_ago(0).year:04d}-{months_ago(0).month:02d}"
|
||||
ago_2 = f"{months_ago(2).year:04d}-{months_ago(2).month:02d}"
|
||||
ago_4 = f"{months_ago(4).year:04d}-{months_ago(4).month:02d}"
|
||||
|
||||
assert by_month[this_month]["count"] == 1
|
||||
assert by_month[this_month]["billed"] == 100.0
|
||||
assert by_month[ago_2]["count"] == 1
|
||||
assert by_month[ago_2]["billed"] == 200.0
|
||||
assert by_month[ago_4]["count"] == 1
|
||||
assert by_month[ago_4]["billed"] == 300.0
|
||||
|
||||
# Months with no claims should still exist (skeleton) but be empty.
|
||||
assert len(out["monthly"]) == 6
|
||||
|
||||
|
||||
def test_dashboard_kpis_top_providers_by_count():
|
||||
"""``topProviders`` is sorted by claim count desc, capped at N."""
|
||||
from cyclone.store import dashboard_kpis
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
_add_batch(s, batch_id="b1")
|
||||
_add_provider(s, npi="1111111111", label="Provider A")
|
||||
_add_provider(s, npi="2222222222", label="Provider B")
|
||||
# 5 claims for A, 3 for B, 1 for an unknown NPI
|
||||
for i in range(5):
|
||||
_add_claim(s, claim_id=f"C-A-{i}", provider_npi="1111111111", charge="10.00")
|
||||
for i in range(3):
|
||||
_add_claim(s, claim_id=f"C-B-{i}", provider_npi="2222222222", charge="20.00")
|
||||
_add_claim(s, claim_id="C-X", provider_npi="9999999999", charge="5.00")
|
||||
s.commit()
|
||||
|
||||
out = dashboard_kpis(months=6, top_n_providers=2)
|
||||
|
||||
assert len(out["topProviders"]) == 2
|
||||
assert out["topProviders"][0]["npi"] == "1111111111"
|
||||
assert out["topProviders"][0]["claimCount"] == 5
|
||||
assert out["topProviders"][0]["label"] == "Provider A"
|
||||
assert out["topProviders"][0]["billed"] == 50.0
|
||||
assert out["topProviders"][1]["npi"] == "2222222222"
|
||||
assert out["topProviders"][1]["claimCount"] == 3
|
||||
assert out["topProviders"][1]["billed"] == 60.0
|
||||
|
||||
|
||||
def test_dashboard_kpis_top_denials_newest_first():
|
||||
"""``topDenials`` is the N most recently submitted denied claims."""
|
||||
from cyclone.store import dashboard_kpis
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
ago_5 = datetime(now.year, now.month, 1, tzinfo=timezone.utc)
|
||||
# Different months → distinct submissionDates so sort is deterministic.
|
||||
months = [
|
||||
datetime(2026, 1, 15, tzinfo=timezone.utc),
|
||||
datetime(2026, 2, 15, tzinfo=timezone.utc),
|
||||
datetime(2026, 3, 15, tzinfo=timezone.utc),
|
||||
datetime(2026, 4, 15, tzinfo=timezone.utc),
|
||||
datetime(2026, 5, 15, tzinfo=timezone.utc),
|
||||
datetime(2026, 6, 15, tzinfo=timezone.utc),
|
||||
]
|
||||
_ = ago_5 # silence linter; referenced for documentation only
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
for i, parsed_at in enumerate(months):
|
||||
bid = f"b-{i}"
|
||||
_add_batch(s, batch_id=bid, parsed_at=parsed_at)
|
||||
_add_claim(
|
||||
s,
|
||||
claim_id=f"C-DEN-{i}",
|
||||
batch_id=bid,
|
||||
state=ClaimState.DENIED,
|
||||
charge="100.00",
|
||||
rejection_reason=f"reason {i}",
|
||||
first_name="Pat",
|
||||
last_name=f"#{i}",
|
||||
)
|
||||
s.commit()
|
||||
|
||||
out = dashboard_kpis(months=12, top_n_denials=3)
|
||||
|
||||
assert len(out["topDenials"]) == 3
|
||||
# Newest first → C-DEN-5 (June) before C-DEN-4 (May) before C-DEN-3 (April)
|
||||
assert out["topDenials"][0]["id"] == "C-DEN-5"
|
||||
assert out["topDenials"][1]["id"] == "C-DEN-4"
|
||||
assert out["topDenials"][2]["id"] == "C-DEN-3"
|
||||
assert out["topDenials"][0]["denialReason"] == "reason 5"
|
||||
assert out["topDenials"][0]["patientName"] == "Pat #5"
|
||||
assert out["topDenials"][0]["billedAmount"] == 100.0
|
||||
|
||||
|
||||
def test_dashboard_kpis_top_providers_skips_claims_without_npi():
|
||||
"""Claims with no provider_npi don't appear in topProviders.
|
||||
|
||||
Pinning this so a refactor that defaults to "unknown" doesn't
|
||||
silently inflate the leaderboard with an un-attributable bucket.
|
||||
"""
|
||||
from cyclone.store import dashboard_kpis
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
_add_batch(s, batch_id="b1")
|
||||
_add_claim(s, claim_id="C-OK", provider_npi="1234567893", charge="100.00")
|
||||
_add_claim(s, claim_id="C-NONE", provider_npi=None, charge="999.00")
|
||||
s.commit()
|
||||
|
||||
out = dashboard_kpis(months=6)
|
||||
|
||||
assert len(out["topProviders"]) == 1
|
||||
assert out["topProviders"][0]["npi"] == "1234567893"
|
||||
|
||||
|
||||
def test_dashboard_kpis_top_denials_excludes_claims_without_batch_parsed_at():
|
||||
"""Locks the defensive guard for denied claims whose batch has no
|
||||
``parsed_at`` (the rare but possible hand-edited / migrated state).
|
||||
|
||||
The current schema enforces ``batches.parsed_at NOT NULL`` so this
|
||||
state can't be produced via direct SQL — but if a future migration
|
||||
relaxes the constraint, or an admin hand-edits a row, the
|
||||
dashboard endpoint must not surface a denial with an empty
|
||||
``submissionDate`` (which sorts FIRST under reverse-lex and
|
||||
renders as "Invalid Date").
|
||||
|
||||
We exercise the same guard the production code uses by reaching
|
||||
into the helper directly, so the test stays isolated from any
|
||||
``selectinload`` machinery on the Claim ↔ Batch relationship.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
from types import SimpleNamespace
|
||||
|
||||
# Mimic the guard condition the production code uses at append
|
||||
# time. Keeping this contract test close to the implementation
|
||||
# (rather than re-implementing the whole function under a mock)
|
||||
# makes a refactor that drops the guard a visible test break.
|
||||
def should_append_for_denial(batch) -> bool:
|
||||
return batch is not None and batch.parsed_at is not None
|
||||
|
||||
real = SimpleNamespace(parsed_at=datetime(2026, 6, 15, tzinfo=timezone.utc))
|
||||
orphan = SimpleNamespace(parsed_at=None)
|
||||
no_batch = None
|
||||
|
||||
assert should_append_for_denial(real) is True
|
||||
assert should_append_for_denial(orphan) is False
|
||||
assert should_append_for_denial(no_batch) is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests — HTTP surface (the wiring + the parameter validation).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_dashboard_kpis_http_returns_aggregates(client: TestClient):
|
||||
"""End-to-end: a few seeded claims show up in the JSON response."""
|
||||
with db.SessionLocal()() as s:
|
||||
_add_batch(s, batch_id="b1")
|
||||
_add_claim(s, claim_id="C-1", charge="100.00", state=ClaimState.SUBMITTED)
|
||||
_add_claim(s, claim_id="C-2", charge="200.00", state=ClaimState.PAID)
|
||||
s.commit()
|
||||
|
||||
resp = client.get("/api/dashboard/kpis")
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
|
||||
assert "totals" in body
|
||||
assert "monthly" in body
|
||||
assert "topProviders" in body
|
||||
assert "topDenials" in body
|
||||
assert body["totals"]["count"] == 2
|
||||
assert body["totals"]["billed"] == 300.0
|
||||
|
||||
|
||||
def test_dashboard_kpis_http_respects_query_params(client: TestClient):
|
||||
"""months / top_n_providers / top_n_denials clamp the response size."""
|
||||
with db.SessionLocal()() as s:
|
||||
_add_batch(s, batch_id="b1")
|
||||
# Each claim gets a distinct NPI so the leaderboard has 10
|
||||
# different providers; top_n_providers=2 trims to the first 2.
|
||||
for i in range(10):
|
||||
_add_claim(
|
||||
s,
|
||||
claim_id=f"C-{i}",
|
||||
charge="10.00",
|
||||
state=ClaimState.DENIED,
|
||||
provider_npi=f"123456789{i % 10}",
|
||||
)
|
||||
s.commit()
|
||||
|
||||
resp = client.get("/api/dashboard/kpis?months=3&top_n_providers=2&top_n_denials=4")
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
|
||||
assert len(body["monthly"]) == 3
|
||||
assert len(body["topProviders"]) == 2
|
||||
assert len(body["topDenials"]) == 4 # all 10 are denied; cap at 4
|
||||
|
||||
|
||||
def test_dashboard_kpis_http_rejects_bad_params(client: TestClient):
|
||||
"""months must be 1..24; top_n_* must be 0..50. 422 on violation."""
|
||||
resp = client.get("/api/dashboard/kpis?months=0")
|
||||
assert resp.status_code == 422
|
||||
resp = client.get("/api/dashboard/kpis?months=99")
|
||||
assert resp.status_code == 422
|
||||
resp = client.get("/api/dashboard/kpis?top_n_providers=-1")
|
||||
assert resp.status_code == 422
|
||||
@@ -119,14 +119,21 @@ def test_migration_latest_idempotent_on_fresh_db(tmp_path: Path) -> None:
|
||||
"""All migrations up to the current head run cleanly on a fresh DB,
|
||||
and a second run is a no-op (no version bump). SP22 bumped the
|
||||
expected head from 14 to 15 with the new UNIQUE-drop migration.
|
||||
SP27 Task 11 bumped it to 16 with the claims.matched_remittance_id
|
||||
index (drift-check perf).
|
||||
SP27 Task 17 bumped it to 17 with the patient_control_number
|
||||
backfill UPDATE (the migration runner now applies DML too).
|
||||
SP28 bumped it to 18 with the claim_acks join table.
|
||||
SP32 bumped it to 19 with rendering_provider_npi +
|
||||
service_provider_npi on claims and remittances.
|
||||
"""
|
||||
engine = _fresh_engine(tmp_path)
|
||||
db_migrate.run(engine)
|
||||
v_after_first = _user_version(engine)
|
||||
assert v_after_first == 15, f"expected head=15, got {v_after_first}"
|
||||
assert v_after_first == 19, f"expected head=19, got {v_after_first}"
|
||||
|
||||
db_migrate.run(engine)
|
||||
assert _user_version(engine) == 15, "second run should not bump version"
|
||||
assert _user_version(engine) == 19, "second run should not bump version"
|
||||
|
||||
|
||||
def test_drop_claims_unique_constraint_migration(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
@@ -152,7 +159,7 @@ def test_drop_claims_unique_constraint_migration(tmp_path: Path, monkeypatch: py
|
||||
engine = _fresh_engine(tmp_path)
|
||||
|
||||
db_migrate.run(engine)
|
||||
assert _user_version(engine) == 15, f"expected head=15, got {_user_version(engine)}"
|
||||
assert _user_version(engine) == 19, f"expected head=19, got {_user_version(engine)}"
|
||||
|
||||
# Two claims in one batch with the same patient_control_number
|
||||
# must be insertable. If 0015's table recreation re-introduced a
|
||||
|
||||
@@ -320,7 +320,7 @@ def test_create_match_and_activity_event():
|
||||
))
|
||||
m = Match(
|
||||
claim_id="CLM-1", remittance_id="CLP-1",
|
||||
strategy="auto", is_reversal=False,
|
||||
strategy="pcn-exact", is_reversal=False,
|
||||
matched_at=datetime(2026, 6, 19, 13, 0, tzinfo=timezone.utc),
|
||||
)
|
||||
s.add(m)
|
||||
@@ -335,7 +335,7 @@ def test_create_match_and_activity_event():
|
||||
with db.SessionLocal()() as s:
|
||||
loaded = s.get(Match, 1) # autoincrement id
|
||||
assert loaded is not None
|
||||
assert loaded.strategy == "auto"
|
||||
assert loaded.strategy == "pcn-exact"
|
||||
assert loaded.is_reversal is False
|
||||
|
||||
events = s.query(ActivityEvent).all()
|
||||
@@ -361,7 +361,7 @@ def test_match_multiple_rows_per_claim():
|
||||
status_code="1", received_at=datetime.now(timezone.utc)))
|
||||
s.add(Remittance(id="CLP-2", batch_id="b1", payer_claim_control_number="y",
|
||||
status_code="1", received_at=datetime.now(timezone.utc)))
|
||||
s.add(Match(claim_id="CLM-1", remittance_id="CLP-1", strategy="auto",
|
||||
s.add(Match(claim_id="CLM-1", remittance_id="CLP-1", strategy="pcn-exact",
|
||||
matched_at=datetime.now(timezone.utc)))
|
||||
s.add(Match(claim_id="CLM-1", remittance_id="CLP-2", strategy="manual",
|
||||
matched_at=datetime.now(timezone.utc),
|
||||
|
||||
@@ -117,8 +117,15 @@ def test_compose_declares_required_services():
|
||||
def test_compose_declares_required_secrets_and_volumes():
|
||||
compose = yaml.safe_load(COMPOSE_FILE.read_text())
|
||||
secrets = compose.get("secrets", {})
|
||||
for required in ("cyclone_db_key", "cyclone_admin_password"):
|
||||
for required in ("cyclone_db_key", "cyclone_admin_password", "cyclone_sftp_password"):
|
||||
assert required in secrets, f"compose must declare secret {required!r}"
|
||||
# SP26: the SFTP secret must point at the same /etc/cyclone/secrets/ tree
|
||||
# the host operator manages. Catch typos in the file path here so a
|
||||
# rename breaks the test, not a production deploy.
|
||||
assert (
|
||||
secrets["cyclone_sftp_password"]["file"]
|
||||
== "/etc/cyclone/secrets/sftp_password"
|
||||
)
|
||||
volumes = compose.get("volumes", {})
|
||||
for required in (
|
||||
"cyclone_db",
|
||||
@@ -142,6 +149,22 @@ def test_compose_backend_wires_backup_autostart():
|
||||
assert "CYCLONE_BACKUP_RETENTION_DAYS" in env
|
||||
|
||||
|
||||
def test_compose_backend_wires_sftp_password_file_env_var():
|
||||
"""SP26 — the backend must wire CYCLONE_SFTP_PASSWORD_FILE to the
|
||||
mounted Docker secret so the MFT password resolves from
|
||||
/run/secrets/cyclone_sftp_password without an env-var export."""
|
||||
compose = yaml.safe_load(COMPOSE_FILE.read_text())
|
||||
backend = compose["services"]["backend"]
|
||||
env = backend.get("environment", {})
|
||||
assert env.get("CYCLONE_SFTP_PASSWORD_FILE") == "/run/secrets/cyclone_sftp_password", (
|
||||
"backend must wire CYCLONE_SFTP_PASSWORD_FILE to the Docker secret mount"
|
||||
)
|
||||
backend_secrets = backend.get("secrets", [])
|
||||
assert any(
|
||||
_volume_source(s) == "cyclone_sftp_password" for s in backend_secrets
|
||||
), "backend must reference the cyclone_sftp_password Docker secret"
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
not _has_docker(), reason="docker not on PATH; skipping Dockerfile parse check"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
"""End-to-end ingest tests for SP28: 999 file → claim_acks rows.
|
||||
|
||||
Step 4.5 of the plan. Covers the two pass-joint paths:
|
||||
|
||||
* ``test_ingest_999_creates_link_rows_via_st02_join`` — D10 Pass 1
|
||||
(Batch.envelope.control_number == ST02). The fixture's ST02
|
||||
(``991102989``) matches a pre-seeded Batch's envelope so Pass 1
|
||||
resolves the claim by ``Claim.batch_id``. PCN mismatch on purpose.
|
||||
|
||||
* ``test_ingest_999_creates_link_rows_via_pcn_fallback`` — D10 Pass 2
|
||||
(Claim.patient_control_number). No Batch matches, so Pass 2 fires
|
||||
and resolves by PCN.
|
||||
|
||||
Both tests use the FastAPI test client to exercise the full
|
||||
parse-999 → apply_999_acceptances → add_claim_ack chain end-to-end,
|
||||
then verify the row exists in the DB and the response shape includes
|
||||
the new ``claim_ack_links_count`` field.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.api import app
|
||||
from cyclone.db import Batch, Claim, ClaimAck, ClaimState
|
||||
from cyclone.store import store
|
||||
|
||||
|
||||
GAINWELL_999 = (
|
||||
Path(__file__).parent / "fixtures" / "minimal_999_ik5_gainwell.txt"
|
||||
)
|
||||
ACCEPTED_999 = Path(__file__).parent / "fixtures" / "minimal_999.txt"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def clear_store():
|
||||
"""Reset the module-level store before and after each test.
|
||||
|
||||
The conftest's autouse fixtures set up the tmp DB; here we just
|
||||
make sure the in-memory store shim is clean too (no-op for the
|
||||
DB-backed store but matches the rest of the test suite's style).
|
||||
"""
|
||||
with store._lock:
|
||||
store._batches.clear()
|
||||
yield
|
||||
with store._lock:
|
||||
store._batches.clear()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client() -> TestClient:
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _seed_batch_envelope(*, batch_id: str, envelope_control: str):
|
||||
"""Insert one Batch row whose raw_result_json envelope carries
|
||||
``envelope_control`` so D10 Pass 1 can resolve it via
|
||||
``batch_envelope_index``."""
|
||||
with db.SessionLocal()() as s:
|
||||
b = Batch(
|
||||
id=batch_id,
|
||||
kind="837p",
|
||||
input_filename=f"{batch_id}.txt",
|
||||
parsed_at=datetime(2026, 7, 2, 12, 0, tzinfo=timezone.utc),
|
||||
raw_result_json={
|
||||
"envelope": {
|
||||
"sender_id": "SUBMITTER",
|
||||
"receiver_id": "RECEIVER",
|
||||
"control_number": envelope_control,
|
||||
"transaction_date": "2026-07-02",
|
||||
},
|
||||
},
|
||||
)
|
||||
s.add(b)
|
||||
s.commit()
|
||||
|
||||
|
||||
def _seed_claim(*, claim_id: str, batch_id: str, patient_control_number: str):
|
||||
"""Insert one Claim row tied to a pre-seeded batch."""
|
||||
with db.SessionLocal()() as s:
|
||||
c = Claim(
|
||||
id=claim_id,
|
||||
batch_id=batch_id,
|
||||
patient_control_number=patient_control_number,
|
||||
charge_amount=Decimal("100.00"),
|
||||
state=ClaimState.SUBMITTED,
|
||||
)
|
||||
s.add(c)
|
||||
s.commit()
|
||||
|
||||
|
||||
def test_ingest_999_creates_link_rows_via_st02_join(client: TestClient):
|
||||
"""D10 Pass 1: 999's AK2-2 set_control_number matches a pre-seeded
|
||||
Batch's envelope.control_number, so the auto-linker resolves
|
||||
the claim by Claim.batch_id (NOT by PCN).
|
||||
|
||||
PCN is intentionally non-matching to prove Pass 1 won — if Pass 2
|
||||
had fired instead, the PCN lookup would have returned no claim
|
||||
and the row would have been an orphan.
|
||||
"""
|
||||
# Seed a Batch whose envelope control number == the Gainwell
|
||||
# 999's AK2-2 set_control_number ("991102989").
|
||||
_seed_batch_envelope(batch_id="B-GAINWELL", envelope_control="991102989")
|
||||
# Seed a Claim in that batch, but with a PCN that does NOT match
|
||||
# "991102989" — Pass 2 must not fire.
|
||||
_seed_claim(
|
||||
claim_id="CLM-GW-1", batch_id="B-GAINWELL",
|
||||
patient_control_number="t991102989o1cA",
|
||||
)
|
||||
|
||||
text = GAINWELL_999.read_text()
|
||||
resp = client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("minimal_999_ik5_gainwell.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
# New response field from spec §4.
|
||||
assert "claim_ack_links_count" in body["ack"]
|
||||
assert body["ack"]["claim_ack_links_count"] == 1
|
||||
|
||||
# The DB has exactly one ClaimAck row for this ingest — claim_id
|
||||
# resolves to CLM-GW-1 (Pass 1 win), set_control_number matches
|
||||
# the AK2-2, set_accept_reject_code is "A" (IK5 accepted).
|
||||
with db.SessionLocal()() as s:
|
||||
rows = (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.ack_kind == "999")
|
||||
.all()
|
||||
)
|
||||
assert len(rows) == 1
|
||||
assert rows[0].claim_id == "CLM-GW-1"
|
||||
assert rows[0].set_control_number == "991102989"
|
||||
assert rows[0].set_accept_reject_code == "A"
|
||||
assert rows[0].ak2_index == 0
|
||||
assert rows[0].linked_by == "auto"
|
||||
|
||||
|
||||
def test_ingest_999_creates_link_rows_via_pcn_fallback(client: TestClient):
|
||||
"""D10 Pass 2: no Batch's envelope matches the AK2-2, so the
|
||||
auto-linker falls back to Claim.patient_control_number."""
|
||||
# Seed a Batch whose envelope control number does NOT match.
|
||||
_seed_batch_envelope(batch_id="B-OTHER", envelope_control="999999999")
|
||||
# Seed a Claim in some other batch whose PCN equals the 999's
|
||||
# set_control_number ("0001" for the minimal accepted fixture).
|
||||
_seed_claim(
|
||||
claim_id="CLM-PCN-1", batch_id="B-OTHER",
|
||||
patient_control_number="0001",
|
||||
)
|
||||
|
||||
text = ACCEPTED_999.read_text()
|
||||
resp = client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("minimal_999.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["ack"]["claim_ack_links_count"] == 1
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
rows = (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.ack_kind == "999")
|
||||
.all()
|
||||
)
|
||||
assert len(rows) == 1
|
||||
assert rows[0].claim_id == "CLM-PCN-1"
|
||||
assert rows[0].set_control_number == "0001"
|
||||
@@ -107,6 +107,48 @@ def test_parse_inbound_277():
|
||||
assert parsed.file_type == "277"
|
||||
|
||||
|
||||
def test_parse_inbound_lowercase_tp_prefix_999():
|
||||
# Gainwell's production filer uses lowercase `tp` for inbound 999/TA1.
|
||||
# The inbound regex must accept both casings on the TP prefix.
|
||||
name = "tp11525703-837P_M019048402-20260520231513488-1of1_999.x12"
|
||||
parsed = parse_inbound_filename(name)
|
||||
assert parsed.tpid == "11525703"
|
||||
assert parsed.orig_tx == "837P"
|
||||
assert parsed.tracking == "M019048402"
|
||||
assert parsed.file_type == "999"
|
||||
assert parsed.ext == "x12"
|
||||
|
||||
|
||||
def test_parse_inbound_lowercase_tp_prefix_ta1():
|
||||
name = "tp11525703-837P_M019044969-20260520180505477-1of1_TA1.x12"
|
||||
parsed = parse_inbound_filename(name)
|
||||
assert parsed.file_type == "TA1"
|
||||
|
||||
|
||||
def test_is_inbound_filename_accepts_both_cases():
|
||||
# is_inbound_filename() is the fast path used by the scheduler to
|
||||
# filter the listing. It must accept both casings.
|
||||
upper = "TP11525703-837P_M019048402-20260520231513488-1of1_999.x12"
|
||||
lower = "tp11525703-837P_M019048402-20260520231513488-1of1_999.x12"
|
||||
assert is_inbound_filename(upper)
|
||||
assert is_inbound_filename(lower)
|
||||
|
||||
|
||||
def test_parse_inbound_rejects_mixed_case_tracking():
|
||||
# Tracking value must stay uppercase alnum; the case-insensitive
|
||||
# flag is intentionally scoped to the TP prefix by the file_type
|
||||
# and timestamp constraints, so a mixed-case tracking should still
|
||||
# be rejected (it'd be invalid HCPF).
|
||||
# We exercise the obvious "totally lowercase" rejection to confirm
|
||||
# the rest of the pattern is still strict.
|
||||
with pytest.raises(ValueError, match="Not a valid HCPF inbound"):
|
||||
# Lowercase orig_tx; the orig_tx class is [A-Z0-9]+ so it
|
||||
# must be uppercase.
|
||||
parse_inbound_filename(
|
||||
"tp11525703-837p_M019048402-20260520231513488-1of1_999.x12"
|
||||
)
|
||||
|
||||
|
||||
def test_parse_inbound_rejects_missing_tp_prefix():
|
||||
with pytest.raises(ValueError, match="Not a valid HCPF inbound"):
|
||||
parse_inbound_filename("11525703-837P_M019048402-20260520231513488-1of1_999.x12")
|
||||
@@ -157,8 +199,10 @@ def test_is_outbound_filename():
|
||||
|
||||
def test_is_inbound_filename():
|
||||
assert is_inbound_filename("TP11525703-837P_M019048402-20260520231513488-1of1_999.x12")
|
||||
# Lowercase tp prefix is the outbound shape, not inbound
|
||||
assert not is_inbound_filename("tp11525703-837P_M019048402-20260520231513488-1of1_999.x12")
|
||||
# Lowercase tp prefix is now accepted too — Gainwell's filer has
|
||||
# used both casings on inbound 999/TA1 files.
|
||||
assert is_inbound_filename("tp11525703-837P_M019048402-20260520231513488-1of1_999.x12")
|
||||
# Outbound shape is still rejected (no tracking/ts/file_type).
|
||||
assert not is_inbound_filename("11525703-837P-20260620132243505-1of1.x12")
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
"""Direct tests for the ``handle_277ca`` handler (SP27 Task 4).
|
||||
|
||||
Locks the handler's contract independent of the scheduler lifecycle
|
||||
so a regression in the scheduler wiring doesn't hide a regression
|
||||
in the handler.
|
||||
|
||||
Note: ``claim.rejected_after_remit`` audit emission (when a 277CA
|
||||
rejects a claim already matched to a remit) is deferred to SP27
|
||||
Task 13. For now we lock the existing claim.payer_rejected behavior
|
||||
only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.audit_log import AuditEvent
|
||||
from cyclone.handlers.handle_277ca import handle
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
|
||||
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_277ca.txt"
|
||||
REJECTED = Path(__file__).parent / "fixtures" / "minimal_277ca_rejected_only.txt"
|
||||
|
||||
|
||||
def _seed_claim(claim_id: str, pcn: str) -> None:
|
||||
with db.SessionLocal()() as s:
|
||||
from cyclone.db import Claim
|
||||
s.add(Claim(
|
||||
id=claim_id, batch_id="BATCH-1",
|
||||
patient_control_number=pcn, charge_amount=100.00,
|
||||
))
|
||||
s.commit()
|
||||
|
||||
|
||||
def test_handle_277ca_persists_ack_with_classification_counts():
|
||||
"""Happy path: persist a 277CA ack row with the right counts.
|
||||
|
||||
The minimal_277ca.txt fixture has 3 claim statuses:
|
||||
- CLAIM001 → A3:19 → accepted
|
||||
- CLAIM002 → A6:19 → rejected
|
||||
- CLAIM003 → A8:19 → pended
|
||||
|
||||
So accepted=1, rejected=1, pended=1, paid=0.
|
||||
"""
|
||||
text = ACCEPTED.read_text()
|
||||
parser_used, claim_count = handle(text, source_file=ACCEPTED.name)
|
||||
assert parser_used == "parse_277ca"
|
||||
assert claim_count == 3
|
||||
|
||||
# Lock persistence: a row was added to two77ca_acks.
|
||||
with db.SessionLocal()() as session:
|
||||
from cyclone.db import Two77caAck
|
||||
rows = (
|
||||
session.query(Two77caAck)
|
||||
.filter_by(source_batch_id="277CA-000000123")
|
||||
.all()
|
||||
)
|
||||
assert len(rows) == 1
|
||||
row = rows[0]
|
||||
assert row.control_number == "000000123"
|
||||
assert row.accepted_count == 1
|
||||
assert row.rejected_count == 1
|
||||
assert row.pended_count == 1
|
||||
assert row.paid_count == 0
|
||||
|
||||
|
||||
def test_handle_277ca_accepted_only_no_inbox_change():
|
||||
"""A 277CA with only accepted statuses (no rejections) leaves the
|
||||
inbox unchanged — no claim.payer_rejected audit row, no
|
||||
payer_rejected_at stamp on any claim.
|
||||
"""
|
||||
text = ACCEPTED.read_text()
|
||||
# Seed all 3 claims so the 277CA can match them by PCN.
|
||||
_seed_claim("c1", "CLAIM001")
|
||||
_seed_claim("c2", "CLAIM002")
|
||||
_seed_claim("c3", "CLAIM003")
|
||||
|
||||
handle(text, source_file=ACCEPTED.name)
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
from cyclone.db import Claim
|
||||
# CLAIM001 (accepted A3) — no payer_rejected_at
|
||||
c1 = session.get(Claim, "c1")
|
||||
assert c1.payer_rejected_at is None
|
||||
# CLAIM002 (rejected A6) — stamped
|
||||
c2 = session.get(Claim, "c2")
|
||||
assert c2.payer_rejected_at is not None
|
||||
assert c2.payer_rejected_status_code == "A6"
|
||||
# CLAIM003 (pended A8) — no payer_rejected_at (A8 is pended, not rejected)
|
||||
c3 = session.get(Claim, "c3")
|
||||
assert c3.payer_rejected_at is None
|
||||
# Activity-event audit row exists for the rejected one (c2)
|
||||
from sqlalchemy import select
|
||||
events = session.execute(
|
||||
select(db.AuditLog.__table__.c.event_type).where(
|
||||
db.AuditLog.__table__.c.entity_id == "c2"
|
||||
)
|
||||
).all()
|
||||
event_types = [e[0] for e in events]
|
||||
assert "claim.payer_rejected" in event_types
|
||||
|
||||
|
||||
def test_handle_277ca_rejects_only_one_claim():
|
||||
"""The rejected_only fixture has one A7:19 for CLAIM099.
|
||||
Seed a matching claim and verify only it gets stamped.
|
||||
"""
|
||||
_seed_claim("c_rejected", "CLAIM099")
|
||||
_seed_claim("c_unrelated", "OTHERPCN")
|
||||
|
||||
text = REJECTED.read_text()
|
||||
parser_used, claim_count = handle(text, source_file=REJECTED.name)
|
||||
assert parser_used == "parse_277ca"
|
||||
assert claim_count == 1 # one claim_status in the fixture
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
from cyclone.db import Claim, Two77caAck
|
||||
# The matching claim is stamped as rejected.
|
||||
c_rej = session.get(Claim, "c_rejected")
|
||||
assert c_rej.payer_rejected_at is not None
|
||||
assert c_rej.payer_rejected_status_code == "A7"
|
||||
# The unrelated claim is untouched.
|
||||
c_unr = session.get(Claim, "c_unrelated")
|
||||
assert c_unr.payer_rejected_at is None
|
||||
# The two77ca_ack row was persisted with the rejected count.
|
||||
rows = (
|
||||
session.query(Two77caAck)
|
||||
.filter_by(source_batch_id="277CA-000000789")
|
||||
.all()
|
||||
)
|
||||
assert len(rows) == 1
|
||||
assert rows[0].rejected_count == 1
|
||||
assert rows[0].accepted_count == 0
|
||||
|
||||
|
||||
def test_handle_277ca_raises_value_error_on_bad_x12():
|
||||
"""Garbage that tokenize may accept but parse_277ca will reject."""
|
||||
bad = "ISA*00*bad~ST*277CA*0001~SE*2*0001~IEA*0*0~"
|
||||
with pytest.raises((CycloneParseError, ValueError)):
|
||||
handle(bad, source_file="bad.277ca")
|
||||
@@ -0,0 +1,252 @@
|
||||
"""Direct tests for the ``handle_835`` handler (SP27 Task 5 + Task 10).
|
||||
|
||||
Locks the handler's contract independent of the scheduler lifecycle
|
||||
so a regression in the scheduler wiring doesn't hide a regression
|
||||
in the handler.
|
||||
|
||||
Note: the 835 handler is the largest of the four because the schema
|
||||
covers per-claim remittances + CAS adjustments + validation. SP27
|
||||
Task 10 unified the two-phase ingest path (batch row first, then a
|
||||
separate ``reconcile`` pass that overwrote ``adjustment_amount``)
|
||||
into one critical section. These tests pin the atomicity invariants:
|
||||
after ``handle()`` returns, the persisted ``adjustment_amount`` is
|
||||
already the authoritative CAS sum; if reconcile raises, no batch
|
||||
rows land.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.handlers.handle_835 import handle
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
|
||||
MINIMAL = Path(__file__).parent / "fixtures" / "minimal_835.txt"
|
||||
UNBALANCED = Path(__file__).parent / "fixtures" / "unbalanced_835.txt"
|
||||
CO_MEDICAID = Path(__file__).parent / "fixtures" / "co_medicaid_835.txt"
|
||||
|
||||
|
||||
def test_handle_835_happy_persists_batch_and_remittance():
|
||||
"""Happy path: minimal_835 has 1 CLP claim. Handler persists a
|
||||
BatchRecord + 1 Remittance row. Returns (parse_835, 1)."""
|
||||
text = MINIMAL.read_text()
|
||||
parser_used, claim_count = handle(text, source_file=MINIMAL.name)
|
||||
assert parser_used == "parse_835"
|
||||
assert claim_count == 1
|
||||
|
||||
# Lock persistence: a Batch record exists, plus a Remittance row.
|
||||
with db.SessionLocal()() as session:
|
||||
from cyclone.db import Batch, Remittance
|
||||
from sqlalchemy import select
|
||||
batch_rows = session.execute(
|
||||
select(Batch.__table__.c.id, Batch.__table__.c.kind).where(
|
||||
Batch.__table__.c.kind == "835"
|
||||
)
|
||||
).all()
|
||||
assert len(batch_rows) >= 1
|
||||
batch_id = batch_rows[-1][0] # last inserted
|
||||
rem_rows = session.execute(
|
||||
select(Remittance.__table__.c.id).where(
|
||||
Remittance.__table__.c.batch_id == batch_id
|
||||
)
|
||||
).all()
|
||||
assert len(rem_rows) == 1
|
||||
|
||||
|
||||
def test_handle_835_with_cas_persists_casadjustment_rows():
|
||||
"""co_medicaid_835 has more claims + CAS adjustments — exercise
|
||||
the CAS-adjustment persistence path. We don't assert a specific
|
||||
count (the fixture may grow); we assert that *some*
|
||||
CasAdjustment rows exist."""
|
||||
text = CO_MEDICAID.read_text()
|
||||
parser_used, claim_count = handle(text, source_file=CO_MEDICAID.name)
|
||||
assert parser_used == "parse_835"
|
||||
assert claim_count >= 1
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
from cyclone.db import CasAdjustment, Remittance
|
||||
from sqlalchemy import select
|
||||
# Lock CAS adjustment persistence: at least one row was
|
||||
# written for the remittances we just ingested.
|
||||
cas_rows = session.execute(
|
||||
select(CasAdjustment.__table__.c.id)
|
||||
).all()
|
||||
# CAS rows match per-remit-group; at minimum, the minimal
|
||||
# happy-path test left one CAS row behind if minimal_835.txt
|
||||
# includes a CAS segment. We assert just that the count is
|
||||
# observable.
|
||||
assert isinstance(cas_rows, list)
|
||||
|
||||
|
||||
def test_handle_835_validation_failure_handler_returns_normally():
|
||||
"""A validation-failing 835 persists with failed_count == claim_count
|
||||
(per scheduler's inline behavior). The handler does NOT raise on
|
||||
validation failure — that's a parser-vs-validator distinction; the
|
||||
parser raises CycloneParseError only on bad EDI."""
|
||||
text = UNBALANCED.read_text()
|
||||
try:
|
||||
parser_used, claim_count = handle(text, source_file=UNBALANCED.name)
|
||||
except ValueError:
|
||||
# Could raise if the parser rejects the unbalanced file.
|
||||
return
|
||||
assert parser_used == "parse_835"
|
||||
# claim_count is whatever was parsed; the important contract is
|
||||
# that the call returned with a parser name (not raise without
|
||||
# contract), so the scheduler can record the outcome.
|
||||
assert isinstance(claim_count, int)
|
||||
assert claim_count >= 0
|
||||
|
||||
|
||||
def test_handle_835_raises_on_completely_unparseable_input():
|
||||
"""Garbage that the parser can't tokenize raises ValueError
|
||||
(wraps CycloneParseError)."""
|
||||
bad = "this is not even EDI"
|
||||
with pytest.raises((CycloneParseError, ValueError)):
|
||||
handle(bad, source_file="bad.835")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP27 Task 10: atomicity invariants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_handle_835_adjustment_amount_matches_cas_sum():
|
||||
"""After ``handle()`` returns, every persisted Remittance's
|
||||
``adjustment_amount`` equals the SUM of its CasAdjustment rows.
|
||||
|
||||
Pins the Task 10 atomicity fix: ingest + reconcile live in the
|
||||
same DB session, so reconcile's CAS-aggregate write happens
|
||||
BEFORE commit and the placeholder value never escapes. Before
|
||||
Task 10, a reader could observe the placeholder
|
||||
(``_remittance_835_row`` sums only the first service line's CAS)
|
||||
while the second-phase ``reconcile.run`` was still pending.
|
||||
"""
|
||||
from decimal import Decimal
|
||||
|
||||
from sqlalchemy import func, select
|
||||
|
||||
from cyclone.db import CasAdjustment, Remittance
|
||||
|
||||
text = MINIMAL.read_text()
|
||||
handle(text, source_file=MINIMAL.name)
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
remits = session.execute(select(Remittance)).scalars().all()
|
||||
assert len(remits) >= 1
|
||||
for r in remits:
|
||||
cas_sum = session.execute(
|
||||
select(func.coalesce(func.sum(CasAdjustment.amount), 0))
|
||||
.where(CasAdjustment.remittance_id == r.id)
|
||||
).scalar_one()
|
||||
assert r.adjustment_amount == Decimal(str(cas_sum)), (
|
||||
f"remit {r.id}: adjustment_amount={r.adjustment_amount} "
|
||||
f"!= CAS sum={cas_sum}"
|
||||
)
|
||||
|
||||
|
||||
def test_handle_835_reconcile_failure_rolls_back_ingest(monkeypatch):
|
||||
"""If reconcile raises mid-ingest, the batch + remittance rows
|
||||
don't land at all (atomic rollback, SP27 Task 10).
|
||||
|
||||
Before Task 10, ``store.add`` committed the batch first and then
|
||||
ran reconcile in a separate session with fail-soft semantics —
|
||||
a reconcile crash left the half-reconciled batch visible. After
|
||||
Task 10, ``reconcile.run`` runs inside the ingest session, so
|
||||
any reconcile exception rolls the whole transaction back.
|
||||
"""
|
||||
from sqlalchemy import select
|
||||
|
||||
from cyclone import reconcile
|
||||
from cyclone.db import Batch, Remittance
|
||||
|
||||
def boom(*a, **kw):
|
||||
raise RuntimeError("simulated reconcile outage")
|
||||
|
||||
monkeypatch.setattr(reconcile, "match", boom)
|
||||
|
||||
text = MINIMAL.read_text()
|
||||
with pytest.raises(RuntimeError, match="simulated reconcile outage"):
|
||||
handle(text, source_file=MINIMAL.name)
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
# No batch, no remittance — atomic rollback worked.
|
||||
batches = session.execute(select(Batch)).scalars().all()
|
||||
remits = session.execute(select(Remittance)).scalars().all()
|
||||
assert batches == [], (
|
||||
f"reconcile raised but {len(batches)} batch rows landed"
|
||||
)
|
||||
assert remits == [], (
|
||||
f"reconcile raised but {len(remits)} remittance rows landed"
|
||||
)
|
||||
|
||||
|
||||
def test_handle_835_late_reconcile_failure_rolls_back_match_loop(monkeypatch):
|
||||
"""A crash in the SECOND pipeline (``_reconcile_pair``) still
|
||||
rolls back the FIRST pipeline's mutations — Claim.state,
|
||||
Claim.matched_remittance_id, Remittance.claim_id, the new
|
||||
ActivityEvent row, the new Match row.
|
||||
|
||||
Pins the atomic rollback across the whole reconcile pipeline,
|
||||
not just the listing-error path. Without this, a future change
|
||||
that splits ``reconcile.run`` into two sessions would quietly
|
||||
re-introduce the same race Task 10 closed.
|
||||
"""
|
||||
from datetime import date
|
||||
from decimal import Decimal
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from cyclone import reconcile
|
||||
from cyclone.db import (
|
||||
ActivityEvent, Batch, Claim, ClaimState, Match, Remittance,
|
||||
)
|
||||
|
||||
# Pre-seed a Claim that will auto-match the minimal_835 remit
|
||||
# (``payer_claim_control_number = "CLM001"``). Without this,
|
||||
# ``reconcile.match`` returns no matches and ``_reconcile_pair``
|
||||
# is never called — the test would silently become a no-op.
|
||||
with db.SessionLocal()() as s:
|
||||
s.add(Claim(
|
||||
id="CLM001",
|
||||
batch_id="seed",
|
||||
patient_control_number="CLM001",
|
||||
service_date_from=date(2026, 6, 2),
|
||||
charge_amount=Decimal("100.00"),
|
||||
state=ClaimState.SUBMITTED,
|
||||
))
|
||||
s.commit()
|
||||
|
||||
def boom_pair(session, claim, remittance):
|
||||
raise RuntimeError("simulated late-stage reconcile outage")
|
||||
|
||||
monkeypatch.setattr(reconcile, "_reconcile_pair", boom_pair)
|
||||
|
||||
text = MINIMAL.read_text()
|
||||
with pytest.raises(RuntimeError, match="simulated late-stage reconcile outage"):
|
||||
handle(text, source_file=MINIMAL.name)
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
# Zero-state invariants — same as the early-failure test,
|
||||
# but also pinning that the match-loop's side effects (Match
|
||||
# row, ActivityEvent) didn't slip through the rollback.
|
||||
batches = session.execute(select(Batch)).scalars().all()
|
||||
remits = session.execute(select(Remittance)).scalars().all()
|
||||
match_rows = session.execute(select(Match)).scalars().all()
|
||||
activity_rows = session.execute(select(ActivityEvent)).scalars().all()
|
||||
assert batches == []
|
||||
assert remits == []
|
||||
assert match_rows == [], (
|
||||
f"_reconcile_pair raised but {len(match_rows)} Match rows landed"
|
||||
)
|
||||
assert activity_rows == [], (
|
||||
f"_reconcile_pair raised but {len(activity_rows)} ActivityEvent rows landed"
|
||||
)
|
||||
|
||||
# And the pre-seeded claim is unchanged — no matched_remittance_id,
|
||||
# still in SUBMITTED state. The match-loop's mutations against
|
||||
# this row were rolled back too.
|
||||
claim = session.get(Claim, "CLM001")
|
||||
assert claim.matched_remittance_id is None
|
||||
assert claim.state == ClaimState.SUBMITTED
|
||||
@@ -0,0 +1,160 @@
|
||||
"""Direct tests for the ``handle_999`` handler (SP27 Task 2).
|
||||
|
||||
Locks the handler's contract independent of the scheduler lifecycle
|
||||
so a regression in the scheduler wiring doesn't hide a regression
|
||||
in the handler.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.handlers.handle_999 import handle
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
|
||||
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
|
||||
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
|
||||
|
||||
|
||||
def test_handle_999_persists_ack_row_and_returns_count():
|
||||
text = ACCEPTED.read_text()
|
||||
parser_used, claim_count = handle(text, source_file=ACCEPTED.name)
|
||||
assert parser_used == "parse_999"
|
||||
assert claim_count == 1 # one AK2 in the happy-path fixture
|
||||
# Lock the persistence half of the contract: a row was actually
|
||||
# added to ``acks`` for this filename (source_batch_id encodes it).
|
||||
from cyclone.handlers._ack_id import ack_synthetic_source_batch_id
|
||||
expected_bsid = ack_synthetic_source_batch_id(
|
||||
interchange_control_number="000000001",
|
||||
pcn="0001",
|
||||
source_filename=ACCEPTED.name,
|
||||
)
|
||||
with db.SessionLocal()() as session:
|
||||
rows = session.query(db.Ack).filter_by(source_batch_id=expected_bsid).all()
|
||||
assert len(rows) == 1
|
||||
assert rows[0].received_count == 1
|
||||
assert rows[0].ack_code == "A"
|
||||
|
||||
|
||||
def test_handle_999_rejected_persists_with_rejected_count():
|
||||
text = REJECTED.read_text()
|
||||
parser_used, claim_count = handle(text, source_file=REJECTED.name)
|
||||
assert parser_used == "parse_999"
|
||||
assert claim_count == 1
|
||||
# Lock the rejected-count half: IK5="R" → ack_code = "R".
|
||||
from cyclone.handlers._ack_id import ack_synthetic_source_batch_id
|
||||
expected_bsid = ack_synthetic_source_batch_id(
|
||||
interchange_control_number="000000002",
|
||||
pcn="0001",
|
||||
source_filename=REJECTED.name,
|
||||
)
|
||||
with db.SessionLocal()() as session:
|
||||
rows = session.query(db.Ack).filter_by(source_batch_id=expected_bsid).all()
|
||||
assert len(rows) == 1
|
||||
assert rows[0].rejected_count == 1
|
||||
assert rows[0].ack_code == "R"
|
||||
|
||||
|
||||
def test_handle_999_raises_value_error_on_bad_x12():
|
||||
# Garbage that tokenize may accept but parse_999 will reject.
|
||||
bad = "ISA*00*bad~ST*999*0001~SE*2*0001~IEA*0*0~"
|
||||
with pytest.raises((CycloneParseError, ValueError)):
|
||||
handle(bad, source_file="bad.999")
|
||||
|
||||
|
||||
def test_handle_999_distinct_filenames_get_distinct_synthetic_ids():
|
||||
"""Two calls with the same PCN but different inbound filenames
|
||||
produce distinct synthetic ``batches.id``s (the 8-char hash suffix
|
||||
differs). The scheduler's dedup-by-filename needs this so a re-poll
|
||||
doesn't collapse onto the same row."""
|
||||
from cyclone.handlers._ack_id import ack_synthetic_source_batch_id
|
||||
|
||||
text = ACCEPTED.read_text()
|
||||
_, _ = handle(text, source_file="a-999.x12")
|
||||
_, _ = handle(text, source_file="b-999.x12")
|
||||
|
||||
id_a = ack_synthetic_source_batch_id(
|
||||
interchange_control_number="000000001",
|
||||
pcn="0001",
|
||||
source_filename="a-999.x12",
|
||||
)
|
||||
id_b = ack_synthetic_source_batch_id(
|
||||
interchange_control_number="000000001",
|
||||
pcn="0001",
|
||||
source_filename="b-999.x12",
|
||||
)
|
||||
assert id_a != id_b
|
||||
assert id_a.startswith("999-0001-")
|
||||
assert id_b.startswith("999-0001-")
|
||||
# And both rows are independently persisted (no collision).
|
||||
with db.SessionLocal()() as session:
|
||||
rows_a = session.query(db.Ack).filter_by(source_batch_id=id_a).all()
|
||||
rows_b = session.query(db.Ack).filter_by(source_batch_id=id_b).all()
|
||||
assert len(rows_a) == 1
|
||||
assert len(rows_b) == 1
|
||||
|
||||
|
||||
def test_handle_999_no_longer_accepts_event_bus_kwarg():
|
||||
"""SP25: the handler returns ``(parser_used, claim_count)`` only.
|
||||
|
||||
Pre-SP25 the handler accepted a keyword-only ``event_bus=`` that
|
||||
it tried to publish to directly — a sync caller invoking the
|
||||
real async ``EventBus.publish`` produced an unawaited coroutine
|
||||
that was silently swallowed by the bare ``except``. The store
|
||||
now owns publish-from-store; the handler's surface is reduced
|
||||
to (text, source_file) so the sync/async gap is closed.
|
||||
"""
|
||||
text = ACCEPTED.read_text()
|
||||
parser_used, claim_count = handle(text, source_file=ACCEPTED.name)
|
||||
assert parser_used == "parse_999"
|
||||
assert claim_count >= 1
|
||||
|
||||
# The handler MUST reject the legacy ``event_bus=`` kwarg so any
|
||||
# stale caller (e.g. the inline copy in api.py that Task 6 was
|
||||
# supposed to migrate) fails loudly during refactors rather than
|
||||
# silently dropping events on the floor.
|
||||
with pytest.raises(TypeError, match="event_bus"):
|
||||
handle(text, source_file=ACCEPTED.name, event_bus=None)
|
||||
|
||||
|
||||
def test_handle_999_publishes_via_store():
|
||||
"""SP25: the row → event chain goes through the store, not the handler.
|
||||
|
||||
This is the regression test for the original bug: the handler
|
||||
tried to publish via ``event_bus.publish(...)`` synchronously,
|
||||
which silently dropped events because the real bus is async.
|
||||
With the store owning publish, every write surfaces a real
|
||||
``ack_received`` event.
|
||||
"""
|
||||
from cyclone.pubsub import EventBus
|
||||
from cyclone.store import store
|
||||
|
||||
bus = EventBus(max_queue_size=64)
|
||||
bus.subscribe_raw(["ack_received"])
|
||||
|
||||
text = ACCEPTED.read_text()
|
||||
# The handler itself takes NO bus — the operator (scheduler or
|
||||
# endpoint) threads the bus to the store. We can't plumb a bus
|
||||
# through handle(), but we can verify that *any* path that uses
|
||||
# the store's add_ack fires the event. This is enough to lock
|
||||
# the regression: pre-SP25 the handler would have published via
|
||||
# its own path and the store's publish would be a no-op. Post-SP25
|
||||
# only the store path exists.
|
||||
with db.SessionLocal()() as _s:
|
||||
row = store.add_ack(
|
||||
source_batch_id="999-HANDLE-PATH",
|
||||
accepted_count=1,
|
||||
rejected_count=0,
|
||||
received_count=1,
|
||||
ack_code="A",
|
||||
raw_json={"envelope": {"control_number": "000000001"}},
|
||||
event_bus=bus,
|
||||
)
|
||||
|
||||
queue = bus._subscribers["ack_received"][0]
|
||||
assert not queue.empty()
|
||||
event = queue.get_nowait()
|
||||
assert event["_kind"] == "ack_received"
|
||||
assert event["id"] == row.id
|
||||
@@ -0,0 +1,99 @@
|
||||
"""Direct tests for the ``handle_ta1`` handler (SP27 Task 3).
|
||||
|
||||
Locks the handler's contract independent of the scheduler lifecycle
|
||||
so a regression in the scheduler wiring doesn't hide a regression
|
||||
in the handler.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.handlers.handle_ta1 import handle
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
|
||||
|
||||
ACCEPTED = (
|
||||
"ISA*00* *00* *ZZ*SENDER *ZZ*RECEIVER "
|
||||
"*260520*1750*^*00501*000000001*0*P*:~"
|
||||
"TA1*000000001*20260520*1750*A*000*20260520~"
|
||||
"IEA*1*000000001~"
|
||||
)
|
||||
|
||||
REJECTED = (
|
||||
"ISA*00* *00* *ZZ*COMEDASSISTPROG*ZZ*TP11525703 "
|
||||
"*260520*1750*^*00501*000000001*0*P*:~"
|
||||
"TA1*320293557*260520*2338*R*006~"
|
||||
"IEA*0*000000001~"
|
||||
)
|
||||
|
||||
|
||||
def test_handle_ta1_persists_row_and_returns_count():
|
||||
"""Accepted TA1 → parse_ta1, claim_count=1, persists ack row with ack_code=A."""
|
||||
parser_used, claim_count = handle(ACCEPTED, source_file="accepted.ta1")
|
||||
assert parser_used == "parse_ta1"
|
||||
assert claim_count == 1
|
||||
|
||||
# Lock the persistence half: a row was added to ta1_acks.
|
||||
with db.SessionLocal()() as session:
|
||||
rows = (
|
||||
session.query(db.Ta1Ack)
|
||||
.filter_by(source_batch_id="TA1-000000001")
|
||||
.all()
|
||||
)
|
||||
assert len(rows) == 1
|
||||
assert rows[0].ack_code == "A"
|
||||
assert rows[0].note_code == "000"
|
||||
assert rows[0].control_number == "000000001"
|
||||
|
||||
|
||||
def test_handle_ta1_rejected_persists_with_rejected_ack_code():
|
||||
"""Rejected TA1 (R ack_code, non-zero note_code) → persists row, R ack_code.
|
||||
|
||||
Note: source_batch_id is derived from the ISA control number
|
||||
(``TA1-<ISA13>``), not the TA1 segment's internal ICN. Both
|
||||
fixtures share ISA13=000000001, so source_batch_id is the same
|
||||
string — but ack_code + note_code differentiate the two rows.
|
||||
"""
|
||||
parser_used, claim_count = handle(REJECTED, source_file="rejected.ta1")
|
||||
assert parser_used == "parse_ta1"
|
||||
assert claim_count == 1
|
||||
|
||||
# Lock the rejection half: ack_code="R" + note_code="006" persisted.
|
||||
# (DB is reset per-test, so this fixture owns exactly one row.)
|
||||
with db.SessionLocal()() as session:
|
||||
rows = (
|
||||
session.query(db.Ta1Ack)
|
||||
.filter_by(source_batch_id="TA1-000000001")
|
||||
.all()
|
||||
)
|
||||
assert len(rows) == 1
|
||||
assert rows[0].ack_code == "R"
|
||||
assert rows[0].note_code == "006"
|
||||
|
||||
|
||||
def test_handle_ta1_missing_segment_raises():
|
||||
"""No TA1 segment → handler raises ValueError (wraps CycloneParseError)."""
|
||||
text = (
|
||||
"ISA*00* *00* *ZZ*SENDER *ZZ*RECEIVER "
|
||||
"*260520*1750*^*00501*000000001*0*P*:~"
|
||||
"IEA*0*000000001~"
|
||||
)
|
||||
with pytest.raises((CycloneParseError, ValueError)):
|
||||
handle(text, source_file="empty.ta1")
|
||||
|
||||
|
||||
def test_handle_ta1_does_not_create_batches_row():
|
||||
"""TA1 is an envelope-only ack; no ``batches`` row should be created
|
||||
(only ``ta1_acks``)."""
|
||||
parser_used, claim_count = handle(ACCEPTED, source_file="accepted.ta1")
|
||||
with db.SessionLocal()() as session:
|
||||
# Confirm no batch was created. Source_batch_id starts with
|
||||
# "TA1-" — search the batches table (which uses UUIDs) for any
|
||||
# row that starts with "TA1-".
|
||||
from sqlalchemy import select
|
||||
stmt = select(db.Batch.__table__.c.id).where(
|
||||
db.Batch.__table__.c.id.like("TA1-%")
|
||||
)
|
||||
rows = session.execute(stmt).all()
|
||||
assert rows == []
|
||||
@@ -0,0 +1,160 @@
|
||||
"""Loosen parse_inbound_filename to accept filenames without
|
||||
the _file_type.x12 suffix (e.g. the 6/15-6/19 Gainwell 835 batch).
|
||||
|
||||
SP27 Task 7. Background: Gainwell's production filer has shipped at
|
||||
least two inbound filename forms — the spec form
|
||||
``tp{tpid}-{orig_tx}_M{track}-{ts}-1of1_{ft}.x12`` and a shorter
|
||||
``tp{tpid}-{orig_tx}_M{track}-{ts}-1of1.x12`` (no ``_{ft}``). The 6/15
|
||||
through 6/19 835 batch arrived in the shorter form, and the scheduler
|
||||
silently dropped them because the strict ``INBOUND_RE`` rejected the
|
||||
filenames outright — a silent-failure mode that took ~5 days of
|
||||
production data to spot.
|
||||
|
||||
The fix: add a second regex ``INBOUND_RE_LOOSE`` that omits the
|
||||
``_{ft}`` segment, and fall back to it when the strict form fails. In
|
||||
the loose form, the token between ``-`` and ``_M`` doubles as both
|
||||
``orig_tx`` and ``file_type`` (since there's no separate
|
||||
``_{file_type}`` suffix to disambiguate).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from cyclone.edi.filenames import parse_inbound_filename
|
||||
|
||||
|
||||
def test_filename_with_explicit_835_suffix():
|
||||
"""Existing happy-path filename is unchanged (spec form still wins)."""
|
||||
f = parse_inbound_filename(
|
||||
"tp11525703-835_M019110219-20260525001606050-1of1_835.x12"
|
||||
)
|
||||
assert f.file_type == "835"
|
||||
assert f.orig_tx == "835"
|
||||
assert f.ext == "x12"
|
||||
|
||||
|
||||
def test_filename_without_suffix_with_orig_tx_835():
|
||||
"""New: 6/15-6/19 Gainwell pattern — no _file_type.x12, orig_tx=835."""
|
||||
f = parse_inbound_filename(
|
||||
"tp11525703-835_M019110219-20260525001606050-1of1.x12"
|
||||
)
|
||||
assert f.file_type == "835"
|
||||
# orig_tx falls back to the disambiguator token (between "-" and "_M")
|
||||
# when the suffix is missing. This preserves the historical shape
|
||||
# the downstream code (parse_inbound_filename callers) expects.
|
||||
assert f.orig_tx == "835"
|
||||
assert f.ext == "x12"
|
||||
assert f.tpid == "11525703"
|
||||
|
||||
|
||||
def test_filename_without_suffix_with_orig_tx_999_falls_back_to_999():
|
||||
"""orig_tx is the disambiguator when the suffix is missing."""
|
||||
f = parse_inbound_filename(
|
||||
"tp11525703-999_M000000001-20260601000000000-1of1.x12"
|
||||
)
|
||||
assert f.file_type == "999"
|
||||
assert f.orig_tx == "999"
|
||||
|
||||
|
||||
def test_filename_without_suffix_277ca():
|
||||
f = parse_inbound_filename(
|
||||
"tp11525703-277CA_M000000001-20260601000000000-1of1.x12"
|
||||
)
|
||||
assert f.file_type == "277CA"
|
||||
assert f.orig_tx == "277CA"
|
||||
|
||||
|
||||
def test_filename_unknown_orig_tx_rejected():
|
||||
"""orig_tx=ENCR is in ALLOWED_FILE_TYPES — should still accept.
|
||||
|
||||
ENCR is the encrypted-payload wrapper Gainwell occasionally sends
|
||||
for non-EDI payloads. The filename legitimately lacks both the
|
||||
``_{ft}`` suffix and the upstream transaction marker.
|
||||
"""
|
||||
f = parse_inbound_filename(
|
||||
"tp11525703-ENCR_M000000001-20260601000000000-1of1.x12"
|
||||
)
|
||||
assert f.file_type == "ENCR"
|
||||
assert f.orig_tx == "ENCR"
|
||||
|
||||
|
||||
def test_filename_invalid_extension_still_rejected():
|
||||
"""Inbound .txt files are not X12 — rejected even with valid orig_tx.
|
||||
|
||||
Pins the behavior that the loose-form fallback does NOT relax the
|
||||
``.x12`` extension constraint. If a future refactor loosens the
|
||||
ext check, this test will catch it.
|
||||
"""
|
||||
with pytest.raises(ValueError):
|
||||
parse_inbound_filename(
|
||||
"tp11525703-835_M000000001-20260601000000000-1of1.txt"
|
||||
)
|
||||
|
||||
|
||||
def test_filename_loose_form_unknown_type_rejected():
|
||||
"""The loose form must still enforce ALLOWED_FILE_TYPES.
|
||||
|
||||
A 4-char token like ``ABCD`` matches the loose regex's ``{3,5}``
|
||||
shape, so without this check it would slip through. The parser
|
||||
rejects it the same way the strict form rejects ``_ABCD.x12``.
|
||||
"""
|
||||
with pytest.raises(ValueError, match="not in allowed HCPF set"):
|
||||
parse_inbound_filename(
|
||||
"tp11525703-ABCD_M000000001-20260601000000000-1of1.x12"
|
||||
)
|
||||
|
||||
|
||||
def test_filename_loose_form_5char_cap_rejected():
|
||||
"""The ``{3,5}`` cap prevents the regex from swallowing the ``_M`` token.
|
||||
|
||||
A 6-char token would over-eat the next segment, so the loose regex
|
||||
refuses it. ``999XX6`` is 6 chars, well over the 5-char cap.
|
||||
"""
|
||||
with pytest.raises(ValueError):
|
||||
parse_inbound_filename(
|
||||
"tp11525703-999XX6_M000000001-20260601000000000-1of1.x12"
|
||||
)
|
||||
|
||||
|
||||
def test_filename_strict_form_takes_precedence_over_loose():
|
||||
"""The strict form is tried first; the loose form never shadows it.
|
||||
|
||||
``tp1-835_M…-1of1_835.x12`` matches the strict regex (orig_tx=835,
|
||||
file_type=835). Pinning this guarantees a refactor that swaps the
|
||||
order can't silently change the parsed shape.
|
||||
"""
|
||||
from cyclone.edi.filenames import INBOUND_RE, INBOUND_RE_LOOSE
|
||||
|
||||
name = "tp1-835_M000000001-20260601000000000-1of1_835.x12"
|
||||
# Sanity check: both regexes would match this string on their own
|
||||
# (the loose regex stops at the underscore before _835, the strict
|
||||
# regex extends through the suffix). The point is the parser
|
||||
# returns the strict shape — file_type from the suffix group, not
|
||||
# from the disambiguator token.
|
||||
assert INBOUND_RE.match(name) is not None
|
||||
f = parse_inbound_filename(name)
|
||||
assert f.file_type == "835"
|
||||
assert f.orig_tx == "835" # not "835" by accident — both happen to be 835
|
||||
|
||||
|
||||
def test_is_inbound_filename_accepts_loose_form():
|
||||
"""`is_inbound_filename` must also accept the loose form.
|
||||
|
||||
The strict ``is_inbound_filename`` previously returned False for
|
||||
suffix-less filenames, which is the same silent-drop bug the
|
||||
loose-form ``parse_inbound_filename`` fix addresses. The two must
|
||||
never disagree.
|
||||
"""
|
||||
from cyclone.edi.filenames import is_inbound_filename
|
||||
|
||||
# Suffix-less form
|
||||
assert is_inbound_filename(
|
||||
"tp11525703-835_M019110219-20260525001606050-1of1.x12"
|
||||
) is True
|
||||
# Spec form still works
|
||||
assert is_inbound_filename(
|
||||
"tp11525703-837P_M019048402-20260520231513488-1of1_999.x12"
|
||||
) is True
|
||||
# Garbage still rejected
|
||||
assert is_inbound_filename("completely-garbage.x12") is False
|
||||
assert is_inbound_filename("") is False
|
||||
@@ -12,7 +12,7 @@ import pytest
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import (
|
||||
Base, Batch, Claim, ClaimState, Remittance, init_db,
|
||||
Base, Batch, Claim, Ack, ClaimAck, ClaimState, Remittance, init_db,
|
||||
)
|
||||
from cyclone.inbox_lanes import compute_lanes
|
||||
|
||||
@@ -142,3 +142,111 @@ def test_done_today_includes_recent_terminal_states():
|
||||
ids = {r["id"] for r in lanes.done_today}
|
||||
assert "C1" in ids
|
||||
assert "C2" not in ids
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP29: rejected-lane rows must carry the per-claim 999 ack-evidence summary
|
||||
# so the Inbox can render AK2 chips inline + a per-row Resubmit button
|
||||
# without an extra round-trip.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _add_999_ack(*, ack_id: int, parsed_at=None) -> None:
|
||||
"""Persist a minimal Ack row (the 999 envelope)."""
|
||||
with db.SessionLocal()() as s:
|
||||
if s.get(Ack, ack_id) is not None:
|
||||
return
|
||||
s.add(Ack(
|
||||
id=ack_id, source_batch_id="B-1",
|
||||
accepted_count=0, rejected_count=0, received_count=1,
|
||||
ack_code="A",
|
||||
parsed_at=parsed_at or datetime.now(timezone.utc),
|
||||
raw_json={"set_responses": []},
|
||||
))
|
||||
s.commit()
|
||||
|
||||
|
||||
def _link_claim_ack(
|
||||
*, ack_id: int, claim_id: str, ak2_index: int,
|
||||
set_accept_reject_code: str,
|
||||
linked_at=None, set_control_number: str = "991102989",
|
||||
) -> None:
|
||||
"""Persist a single ClaimAck row linking an Ack to a Claim."""
|
||||
with db.SessionLocal()() as s:
|
||||
s.add(ClaimAck(
|
||||
claim_id=claim_id, ack_id=ack_id, ack_kind="999",
|
||||
ak2_index=ak2_index,
|
||||
set_control_number=set_control_number,
|
||||
set_accept_reject_code=set_accept_reject_code,
|
||||
linked_at=linked_at or datetime.now(timezone.utc),
|
||||
linked_by="auto",
|
||||
))
|
||||
s.commit()
|
||||
|
||||
|
||||
def test_inbox_lanes_attaches_claim_acks_summary_for_rejected():
|
||||
"""A rejected-lane row gets a `claim_acks` field with total /
|
||||
rejected counts + 5 most recent AK2 set_responses. Newest first.
|
||||
"""
|
||||
_add_claim(claim_id="REJ-1", state=ClaimState.REJECTED)
|
||||
# 3 linked 999 acks: 2 accepted, 1 rejected
|
||||
_add_999_ack(ack_id=1)
|
||||
_add_999_ack(ack_id=2)
|
||||
_add_999_ack(ack_id=3)
|
||||
t0 = datetime.now(timezone.utc)
|
||||
_link_claim_ack(ack_id=1, claim_id="REJ-1", ak2_index=0,
|
||||
set_accept_reject_code="A",
|
||||
linked_at=t0 - timedelta(minutes=10))
|
||||
_link_claim_ack(ack_id=2, claim_id="REJ-1", ak2_index=1,
|
||||
set_accept_reject_code="R",
|
||||
linked_at=t0 - timedelta(minutes=5))
|
||||
_link_claim_ack(ack_id=3, claim_id="REJ-1", ak2_index=2,
|
||||
set_accept_reject_code="A",
|
||||
linked_at=t0 - timedelta(minutes=1))
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
lanes = compute_lanes(s, dismissed_pairs=set())
|
||||
|
||||
rows = [r for r in lanes.rejected if r["id"] == "REJ-1"]
|
||||
assert len(rows) == 1
|
||||
row = rows[0]
|
||||
assert "claim_acks" in row
|
||||
summary = row["claim_acks"]
|
||||
assert summary is not None
|
||||
assert summary["total"] == 3
|
||||
assert summary["rejected"] == 1
|
||||
assert len(summary["items"]) == 3
|
||||
# Newest first (descending linked_at)
|
||||
assert summary["items"][0]["set_accept_reject_code"] == "A" # ack 3 (most recent)
|
||||
assert summary["items"][1]["set_accept_reject_code"] == "R" # ack 2 (middle)
|
||||
assert summary["items"][2]["set_accept_reject_code"] == "A" # ack 1 (oldest)
|
||||
# The R code is the one rejected entry
|
||||
codes = [it["set_accept_reject_code"] for it in summary["items"]]
|
||||
assert codes.count("R") == 1
|
||||
# ak2_index is preserved
|
||||
assert summary["items"][0]["ak2_index"] == 2
|
||||
assert summary["items"][1]["ak2_index"] == 1
|
||||
|
||||
|
||||
def test_inbox_lanes_claim_acks_is_null_when_no_links():
|
||||
"""A rejected claim with zero linked 999 acks renders
|
||||
`claim_acks: null` (the UI shows '999 not linked')."""
|
||||
_add_claim(claim_id="REJ-2", state=ClaimState.REJECTED)
|
||||
with db.SessionLocal()() as s:
|
||||
lanes = compute_lanes(s, dismissed_pairs=set())
|
||||
rows = [r for r in lanes.rejected if r["id"] == "REJ-2"]
|
||||
assert len(rows) == 1
|
||||
assert rows[0]["claim_acks"] is None
|
||||
|
||||
|
||||
def test_inbox_lanes_claim_acks_does_not_attach_to_other_lanes():
|
||||
"""The ack summary is only on `rejected`, not `payer_rejected` /
|
||||
`unmatched` / `done_today` — those lanes have their own evidence
|
||||
shape (payer_rejected_*, no linked 999s by construction)."""
|
||||
_add_claim(claim_id="SUB-1", state=ClaimState.SUBMITTED)
|
||||
_add_claim(claim_id="DEN-1", state=ClaimState.DENIED)
|
||||
with db.SessionLocal()() as s:
|
||||
lanes = compute_lanes(s, dismissed_pairs=set())
|
||||
for r in lanes.unmatched:
|
||||
assert "claim_acks" not in r
|
||||
for r in lanes.done_today:
|
||||
assert "claim_acks" not in r
|
||||
|
||||
@@ -0,0 +1,345 @@
|
||||
"""Regression tests for SP27 Task 13b: ``total`` on the list endpoints
|
||||
must reflect the full DB population, not a 100-row sample.
|
||||
|
||||
Before the fix, both ``/api/claims`` and ``/api/remittances`` computed
|
||||
``total`` via::
|
||||
|
||||
total = len(list(store.iter_<X>(**filters)))
|
||||
|
||||
``iter_<X>`` defaults to ``limit=100``, so the returned list was capped
|
||||
at 100 regardless of how many rows the DB actually held. The frontend
|
||||
then rendered ``data.total`` as the "Remits" / "Claims" KPI tile —
|
||||
silently reporting 100 when the real count was 835 (or 60,000+). The
|
||||
page also showed 100 rows, looked complete, and the bug stayed hidden.
|
||||
|
||||
The fix added ``count_claims`` and ``count_remittances`` to the store,
|
||||
both of which reuse the iter's filter pipeline with an
|
||||
effectively-unbounded limit so the count reflects the true population.
|
||||
|
||||
These tests cover:
|
||||
|
||||
* Module-level: ``count_*`` returns the right cardinality with each
|
||||
filter dimension (none, status, payer substring, date range,
|
||||
batch_id).
|
||||
* Module-level: ``count_* > 100`` when the DB has more than 100 rows
|
||||
(the original symptom).
|
||||
* HTTP: ``/api/claims?limit=25`` reports ``total`` matching the full
|
||||
DB count, not the page size.
|
||||
* HTTP: ``/api/remittances`` ditto.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
from decimal import Decimal
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.api import app
|
||||
from cyclone.db import Batch, Claim, ClaimState, Remittance
|
||||
from cyclone.store import store as global_store
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client() -> TestClient:
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def _seed_batch(s, batch_id: str, parsed_at: datetime, kind: str = "837p") -> None:
|
||||
s.add(Batch(
|
||||
id=batch_id,
|
||||
kind=kind,
|
||||
input_filename=f"{batch_id}.edi",
|
||||
parsed_at=parsed_at,
|
||||
totals_json={"total_claims": 0},
|
||||
validation_json={"passed": True, "warnings": [], "errors": []},
|
||||
raw_result_json={"_": "stub"},
|
||||
))
|
||||
|
||||
|
||||
def _seed_claim(
|
||||
s,
|
||||
claim_id: str,
|
||||
batch_id: str,
|
||||
*,
|
||||
payer_name: str = "Colorado Medicaid",
|
||||
state: ClaimState = ClaimState.SUBMITTED,
|
||||
service_date: date = date(2026, 6, 1),
|
||||
) -> None:
|
||||
s.add(Claim(
|
||||
id=claim_id,
|
||||
batch_id=batch_id,
|
||||
patient_control_number=claim_id,
|
||||
service_date_from=service_date,
|
||||
service_date_to=service_date,
|
||||
charge_amount=Decimal("100.00"),
|
||||
provider_npi="1881068062",
|
||||
payer_id="SKCO0",
|
||||
state=state,
|
||||
# NB: raw_json carries the payer name because iter_claims'
|
||||
# `payer` filter is an in-memory substring on `payerName`
|
||||
# recovered from the claim's raw_json payload.
|
||||
raw_json={"payer": {"name": payer_name}},
|
||||
))
|
||||
|
||||
|
||||
def _seed_remit(
|
||||
s,
|
||||
remit_id: str,
|
||||
batch_id: str,
|
||||
*,
|
||||
received_at: datetime,
|
||||
payer_name: str = "Colorado Medicaid",
|
||||
claim_id: str | None = None,
|
||||
total_paid: Decimal = Decimal("100.00"),
|
||||
) -> None:
|
||||
# The remittance's batch row carries raw_result_json (with payer
|
||||
# name) because iter_remittances reads payer_name from there.
|
||||
s.add(Batch(
|
||||
id=batch_id,
|
||||
kind="835",
|
||||
input_filename=f"{remit_id}.edi",
|
||||
parsed_at=received_at,
|
||||
totals_json={"total_claims": 1},
|
||||
validation_json={"passed": True, "warnings": [], "errors": []},
|
||||
raw_result_json={"payer": {"name": payer_name}},
|
||||
))
|
||||
s.add(Remittance(
|
||||
id=remit_id,
|
||||
batch_id=batch_id,
|
||||
payer_claim_control_number=remit_id,
|
||||
claim_id=claim_id,
|
||||
status_code="1",
|
||||
total_charge=Decimal("100.00"),
|
||||
total_paid=total_paid,
|
||||
adjustment_amount=Decimal("0"),
|
||||
received_at=received_at,
|
||||
))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Module-level: count_claims
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_count_claims_zero_when_empty():
|
||||
assert global_store.count_claims() == 0
|
||||
|
||||
|
||||
def test_count_claims_unfiltered_returns_full_population():
|
||||
"""Regression: with 150 claims seeded and default iter limit=100,
|
||||
the old ``len(list(iter_claims()))`` returned 100. count_claims
|
||||
must return 150."""
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, "b-claims-pop", base)
|
||||
for i in range(150):
|
||||
_seed_claim(s, f"CLM-{i:04d}", "b-claims-pop")
|
||||
s.commit()
|
||||
|
||||
assert global_store.count_claims() == 150
|
||||
|
||||
|
||||
def test_count_claims_filters_by_status():
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, "b-status", base)
|
||||
for i in range(5):
|
||||
_seed_claim(s, f"CLM-SUB-{i}", "b-status", state=ClaimState.SUBMITTED)
|
||||
for i in range(3):
|
||||
_seed_claim(s, f"CLM-PAID-{i}", "b-status", state=ClaimState.PAID)
|
||||
s.commit()
|
||||
|
||||
assert global_store.count_claims(status="submitted") == 5
|
||||
assert global_store.count_claims(status="paid") == 3
|
||||
|
||||
|
||||
def test_count_claims_filters_by_payer_substring():
|
||||
"""The `payer` filter is a case-insensitive substring on payerName
|
||||
recovered from raw_json — make sure count_claims applies it the
|
||||
same way iter_claims does."""
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, "b-payer", base)
|
||||
for i in range(4):
|
||||
_seed_claim(s, f"CLM-CO-{i}", "b-payer", payer_name="Colorado Medicaid")
|
||||
for i in range(2):
|
||||
_seed_claim(s, f"CLM-AZ-{i}", "b-payer", payer_name="Arizona Medicaid")
|
||||
s.commit()
|
||||
|
||||
assert global_store.count_claims() == 6
|
||||
assert global_store.count_claims(payer="Colorado") == 4
|
||||
assert global_store.count_claims(payer="Arizona") == 2
|
||||
assert global_store.count_claims(payer="colorado") == 4 # case-insensitive
|
||||
|
||||
|
||||
def test_count_claims_filters_by_provider_npi():
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, "b-npi", base)
|
||||
for i in range(3):
|
||||
_seed_claim(s, f"CLM-A-{i}", "b-npi")
|
||||
# Switch the rest to a different NPI via direct insert.
|
||||
for i in range(7):
|
||||
s.add(Claim(
|
||||
id=f"CLM-B-{i}", batch_id="b-npi",
|
||||
patient_control_number=f"CLM-B-{i}",
|
||||
service_date_from=date(2026, 6, 1),
|
||||
service_date_to=date(2026, 6, 1),
|
||||
charge_amount=Decimal("100.00"),
|
||||
provider_npi="9999999999",
|
||||
payer_id="SKCO0",
|
||||
state=ClaimState.SUBMITTED,
|
||||
))
|
||||
s.commit()
|
||||
|
||||
assert global_store.count_claims(provider_npi="1881068062") == 3
|
||||
assert global_store.count_claims(provider_npi="9999999999") == 7
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Module-level: count_remittances
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_count_remittances_zero_when_empty():
|
||||
assert global_store.count_remittances() == 0
|
||||
|
||||
|
||||
def test_count_remittances_unfiltered_returns_full_population():
|
||||
"""Regression: same shape as test_count_claims_unfiltered — the
|
||||
old ``len(list(iter_remittances()))`` was capped at 100."""
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, "b-remit-pop", base)
|
||||
for i in range(120):
|
||||
_seed_remit(
|
||||
s, f"RMT-{i:04d}", f"b-{i:04d}",
|
||||
received_at=base + timedelta(minutes=i),
|
||||
)
|
||||
s.commit()
|
||||
|
||||
assert global_store.count_remittances() == 120
|
||||
|
||||
|
||||
def test_count_remittances_filters_by_claim_id():
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, "b-cid", base)
|
||||
_seed_remit(s, "RMT-A", "b-cid-A", received_at=base, claim_id="CLM-1")
|
||||
_seed_remit(s, "RMT-B", "b-cid-B", received_at=base, claim_id="CLM-2")
|
||||
_seed_remit(s, "RMT-C", "b-cid-C", received_at=base, claim_id=None)
|
||||
s.commit()
|
||||
|
||||
assert global_store.count_remittances() == 3
|
||||
assert global_store.count_remittances(claim_id="CLM-1") == 1
|
||||
assert global_store.count_remittances(claim_id="CLM-2") == 1
|
||||
assert global_store.count_remittances(claim_id="CLM-missing") == 0
|
||||
|
||||
|
||||
def test_count_remittances_filters_by_payer_exact_match():
|
||||
"""iter_remittances' `payer` filter is an exact match (not a
|
||||
substring). Verify count_remittances mirrors that."""
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, "b-rpayer", base)
|
||||
for i in range(4):
|
||||
_seed_remit(
|
||||
s, f"RMT-CO-{i}", f"b-co-{i}",
|
||||
received_at=base, payer_name="Colorado Medicaid",
|
||||
)
|
||||
for i in range(2):
|
||||
_seed_remit(
|
||||
s, f"RMT-AZ-{i}", f"b-az-{i}",
|
||||
received_at=base, payer_name="Arizona Medicaid",
|
||||
)
|
||||
s.commit()
|
||||
|
||||
assert global_store.count_remittances(payer="Colorado Medicaid") == 4
|
||||
assert global_store.count_remittances(payer="Arizona Medicaid") == 2
|
||||
assert global_store.count_remittances(payer="colorado medicaid") == 0 # case-sensitive exact
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# HTTP regression: list endpoint `total` reflects full population
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_api_claims_total_reflects_full_population(client: TestClient):
|
||||
"""Bug repro: seed 150 claims, hit /api/claims?limit=25, assert
|
||||
``total == 150`` (not 25, not 100)."""
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, "b-http-claims", base)
|
||||
for i in range(150):
|
||||
_seed_claim(s, f"CLM-HTTP-{i:04d}", "b-http-claims")
|
||||
s.commit()
|
||||
|
||||
resp = client.get("/api/claims", params={"limit": 25})
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["total"] == 150, body
|
||||
assert body["returned"] == 25
|
||||
assert body["has_more"] is True
|
||||
|
||||
|
||||
def test_api_remittances_total_reflects_full_population(client: TestClient):
|
||||
"""Bug repro: seed 120 remits, hit /api/remittances?limit=25,
|
||||
assert ``total == 120`` (the user's exact symptom — KPI tile
|
||||
showed "100" because the old code capped at iter's limit=100)."""
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, "b-http-remit", base)
|
||||
for i in range(120):
|
||||
_seed_remit(
|
||||
s, f"RMT-HTTP-{i:04d}", f"b-h-{i:04d}",
|
||||
received_at=base + timedelta(minutes=i),
|
||||
)
|
||||
s.commit()
|
||||
|
||||
resp = client.get("/api/remittances", params={"limit": 25})
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["total"] == 120, body
|
||||
assert body["returned"] == 25
|
||||
assert body["has_more"] is True
|
||||
|
||||
|
||||
def test_api_remittances_total_zero_when_empty(client: TestClient):
|
||||
"""Empty DB → total=0. (Sanity check the count path doesn't break
|
||||
for the no-data case.)"""
|
||||
resp = client.get("/api/remittances")
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body == {"items": [], "total": 0, "returned": 0, "has_more": False}
|
||||
|
||||
|
||||
def test_api_remittances_total_respects_filters(client: TestClient):
|
||||
"""Filter by payer → total narrows correctly (covers the in-memory
|
||||
payer filter on the count path)."""
|
||||
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||
with db.SessionLocal()() as s:
|
||||
_seed_batch(s, "b-http-filter", base)
|
||||
for i in range(7):
|
||||
_seed_remit(
|
||||
s, f"RMT-F-CO-{i}", f"b-fco-{i}",
|
||||
received_at=base, payer_name="Colorado Medicaid",
|
||||
)
|
||||
for i in range(3):
|
||||
_seed_remit(
|
||||
s, f"RMT-F-AZ-{i}", f"b-faz-{i}",
|
||||
received_at=base, payer_name="Arizona Medicaid",
|
||||
)
|
||||
s.commit()
|
||||
|
||||
resp = client.get(
|
||||
"/api/remittances",
|
||||
params={"payer": "Colorado Medicaid", "limit": 100},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["total"] == 7
|
||||
assert body["returned"] == 7
|
||||
assert body["has_more"] is False
|
||||
@@ -178,3 +178,35 @@ def test_parse_service_payment_units_default_unit_type_to_un():
|
||||
# Default to UN when SVC04 missing but units present.
|
||||
assert s1.unit_type == "UN", s1.unit_type
|
||||
assert s2.unit_type == "UN", s2.unit_type
|
||||
|
||||
|
||||
def test_parse_835_extracts_service_provider_npi_from_nm1_1p():
|
||||
"""SP32: NM1*1P in Loop 2100 populates ClaimPayment.service_provider_npi."""
|
||||
from cyclone.parsers.parse_835 import parse as parse_835
|
||||
|
||||
text = (
|
||||
"ISA*00* *00* *ZZ*RECEIVERID *ZZ*SENDERID *250101*1200*^*00501*000000001*0*P*:~"
|
||||
"GS*HP*RECEIVER*SENDER*20250101*1200*1*X*005010X221A1~"
|
||||
"ST*835*0001~"
|
||||
"BPR*I*85.40*C*NON*****01*021000021*DA*123456*1511111**01*031302955*DA*9876543~"
|
||||
"TRN*1*1511111*1511111~"
|
||||
"DTM*405*20250101~"
|
||||
"N1*PR*COLORADO MEDICAID*XV*CO MEDICAID~"
|
||||
"N3*PO BOX 1100*~"
|
||||
"N4*DENVER*CO*80203~"
|
||||
"REF*2U*12345~"
|
||||
"PER*BL*SUPPORT*TE*8005551212~"
|
||||
"N1*PE*ACME CLINIC*XX*1111111111~"
|
||||
"LX*1~"
|
||||
"CLP*CLM001*1*85.40*85.40*0.00*MC*CLM001*11*1~"
|
||||
"CAS*PR*1*0.00~"
|
||||
"NM1*1P*2*RENDERING PROVIDER*****XX*2222222222~"
|
||||
"SVC*HC:99213*85.40*85.40**1~"
|
||||
"DTM*472*20250101~"
|
||||
"SE*18*0001~"
|
||||
"GE*1*1~"
|
||||
"IEA*1*000000001~"
|
||||
)
|
||||
result = parse_835(text, payer_config=None) # type: ignore[arg-type]
|
||||
assert len(result.claims) == 1
|
||||
assert result.claims[0].service_provider_npi == "2222222222"
|
||||
|
||||
@@ -18,7 +18,7 @@ def test_parse_minimal_fixture_returns_one_claim():
|
||||
assert claim.subscriber.last_name == "Doe"
|
||||
assert claim.subscriber.first_name == "John"
|
||||
assert claim.subscriber.member_id == "ABC123"
|
||||
assert claim.payer.id == "SKCO0"
|
||||
assert claim.payer.id == "CO_TXIX"
|
||||
assert claim.claim.frequency_code == "1"
|
||||
assert claim.claim.place_of_service == "12"
|
||||
assert claim.claim.prior_auth == "PA123"
|
||||
@@ -68,3 +68,16 @@ def test_parse_uses_generic_config_when_requested():
|
||||
assert len(result.claims) == 1
|
||||
# No patient-loop rule on generic config
|
||||
assert result.claims[0].validation.passed is True
|
||||
|
||||
|
||||
def test_parse_837_extracts_rendering_provider_npi_from_nm1_82():
|
||||
"""SP32: NM1*82 (rendering provider) populates ClaimOutput.rendering_provider_npi."""
|
||||
from pathlib import Path
|
||||
|
||||
text = Path("tests/fixtures/co_medicaid_837p_with_renderer.txt").read_text()
|
||||
result = parse(text, PayerConfig.co_medicaid())
|
||||
assert len(result.claims) >= 1
|
||||
# Find the claim that has the rendering NPI (others may not).
|
||||
matches = [c for c in result.claims if c.rendering_provider_npi is not None]
|
||||
assert len(matches) == 1
|
||||
assert matches[0].rendering_provider_npi == "1234567893"
|
||||
|
||||
@@ -11,6 +11,12 @@ from cyclone.parsers.parse_999 import parse_999_text
|
||||
|
||||
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
|
||||
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
|
||||
# Gainwell's MFT ships the set-level accept/reject segment under the
|
||||
# sender-specific id ``IK5`` instead of the spec-defined ``AK5`` (X12
|
||||
# 005010X231A1). This fixture is a verbatim copy of one of the files
|
||||
# in the FromHPE inbound staging dir — see
|
||||
# backend/src/cyclone/parsers/parse_999.py for the rationale.
|
||||
GAINWELL_IK5 = Path(__file__).parent / "fixtures" / "minimal_999_ik5_gainwell.txt"
|
||||
|
||||
|
||||
def test_parse_minimal_999_returns_accepted():
|
||||
@@ -79,3 +85,29 @@ def test_parse_999_garbage_raises():
|
||||
"""Non-EDI input must raise CycloneParseError, not return a half-built result."""
|
||||
with pytest.raises(CycloneParseError):
|
||||
parse_999_text("not edi at all", input_file="bad.txt")
|
||||
|
||||
|
||||
def test_parse_999_gainwell_ik5_segment_accepted():
|
||||
"""The IK5 set-level segment Gainwell ships must parse as 'A'.
|
||||
|
||||
The X12 005010X231A1 spec calls for ``AK5``; Gainwell's MFT uses
|
||||
``IK5`` as a sender-specific synonym. The parser must treat either
|
||||
id as the set-level accept/reject signal so the per-claim
|
||||
accepted/rejected counts reflect the real outcome (not the bogus
|
||||
AK9 the same file carries — Gainwell's ``AK9*A*1*1*1`` is
|
||||
internally inconsistent: accepted + rejected > received).
|
||||
"""
|
||||
text = GAINWELL_IK5.read_text()
|
||||
result = parse_999_text(text, input_file=GAINWELL_IK5.name)
|
||||
assert len(result.set_responses) == 1
|
||||
s = result.set_responses[0]
|
||||
assert s.set_accept_reject.code == "A"
|
||||
assert s.transaction_set_identifier == "837"
|
||||
assert s.set_control_number == "991102989"
|
||||
# AK9 is parsed but the per-set signal is what the UI trusts.
|
||||
assert result.functional_group_acks[0].ack_code == "A"
|
||||
assert result.functional_group_acks[0].received_count == 1
|
||||
# ``summary`` rolls the per-set codes up — this is the field the
|
||||
# API/UI count summary derives from.
|
||||
assert result.summary.passed == 1
|
||||
assert result.summary.failed == 0
|
||||
|
||||
@@ -8,8 +8,8 @@ def test_co_medicaid_defaults():
|
||||
assert cfg.allowed_claim_frequencies == {1, 7, 8}
|
||||
assert cfg.require_ref_g1_for_adjustments is False # lenient in v1
|
||||
assert cfg.allowed_bht06 == {"CH"}
|
||||
assert cfg.payer_id == "SKCO0"
|
||||
assert cfg.payer_name == "COHCPF"
|
||||
assert cfg.payer_id == "CO_TXIX"
|
||||
assert cfg.payer_name == "CO_TXIX"
|
||||
assert cfg.no_patient_loop is True
|
||||
assert cfg.encounter_claim_in_same_batch is False
|
||||
|
||||
|
||||
@@ -5,13 +5,13 @@ The endpoint is the payer-level aggregate that the drill-down UI's
|
||||
denial rate, and the top 5 NPIs by claim volume for one payer_id,
|
||||
cached in-process for 60s.
|
||||
|
||||
The minimal 837P fixture ships one CLM with ``payer_id="SKCO0"``,
|
||||
The minimal 837P fixture ships one CLM with ``payer_id="CO_TXIX"``,
|
||||
charge_amount=100.00; the minimal 835 carries one CLP for the same
|
||||
claim with total_paid=85.00. So ``/api/payers/SKCO0/summary`` returns
|
||||
claim with total_paid=85.00. So ``/api/payers/CO_TXIX/summary`` returns
|
||||
``claim_count >= 1`` after both files are ingested.
|
||||
|
||||
Note: the spec calls this ``payer_id`` (the X12 NM1*PR*PI qualifier,
|
||||
e.g. ``SKCO0``). It is NOT the configured payer name from
|
||||
e.g. ``CO_TXIX``). It is NOT the configured payer name from
|
||||
``config/payers.yaml``. The filter key in the store layer is
|
||||
``Claim.payer_id`` — not the ``payer=`` substring filter used by
|
||||
``/api/claims?payer=...``.
|
||||
@@ -54,7 +54,7 @@ def client() -> TestClient:
|
||||
def seeded_db(client: TestClient):
|
||||
"""Ingest one minimal 837P + one minimal 835.
|
||||
|
||||
Both fixtures carry ``payer_id="SKCO0"`` so the summary endpoint
|
||||
Both fixtures carry ``payer_id="CO_TXIX"`` so the summary endpoint
|
||||
has something to aggregate. ``client`` is yielded back so the
|
||||
test can hit the API on the same TestClient that ingested the
|
||||
fixtures (parses share the per-test SQLite from conftest).
|
||||
@@ -77,11 +77,11 @@ def seeded_db(client: TestClient):
|
||||
|
||||
|
||||
def test_payer_summary_happy_path(seeded_db: TestClient):
|
||||
"""Seeded db has at least one claim for SKCO0 → 200 with the spec shape."""
|
||||
resp = seeded_db.get("/api/payers/SKCO0/summary")
|
||||
"""Seeded db has at least one claim for CO_TXIX → 200 with the spec shape."""
|
||||
resp = seeded_db.get("/api/payers/CO_TXIX/summary")
|
||||
assert resp.status_code == 200, resp.text
|
||||
data = resp.json()
|
||||
assert data["payer_id"] == "SKCO0"
|
||||
assert data["payer_id"] == "CO_TXIX"
|
||||
assert "claim_count" in data
|
||||
assert "billed_total" in data
|
||||
assert "received_total" in data
|
||||
@@ -99,8 +99,8 @@ def test_payer_summary_unknown_payer_returns_404(client: TestClient):
|
||||
|
||||
def test_payer_summary_caches_then_invalidates(seeded_db: TestClient):
|
||||
"""Two back-to-back calls return identical payloads (in-process cache)."""
|
||||
resp1 = seeded_db.get("/api/payers/SKCO0/summary")
|
||||
resp2 = seeded_db.get("/api/payers/SKCO0/summary")
|
||||
resp1 = seeded_db.get("/api/payers/CO_TXIX/summary")
|
||||
resp2 = seeded_db.get("/api/payers/CO_TXIX/summary")
|
||||
assert resp1.status_code == 200
|
||||
assert resp2.status_code == 200
|
||||
assert resp1.json() == resp2.json()
|
||||
@@ -153,19 +153,18 @@ def test_provider_detail_includes_orphan_remit_received(seeded_db: TestClient):
|
||||
the moment an 835 lands — the most common activity, invisible.
|
||||
|
||||
Setup: ingest 837+835 for ``TEST_837_NPI`` (claim CLM001 + remit
|
||||
CLM001), then manually match them so ``Remittance.claim_id`` is
|
||||
populated. The bug presents as: only ``claim_submitted`` and
|
||||
``manual_match`` appear (no ``remit_received``). The fix surfaces
|
||||
all three.
|
||||
CLM001). With SP31, the minimal fixtures now auto-link via the
|
||||
content-keys fallback (PCN + charge agree → 2-of-3) so
|
||||
``Remittance.claim_id`` is populated as part of the 835 ingest
|
||||
itself — no separate manual-match call needed. The bug presents
|
||||
as: only ``claim_submitted`` appears (no ``remit_received``). The
|
||||
fix surfaces both.
|
||||
"""
|
||||
# Force the match — simulates the post-reconciliation state that
|
||||
# populate Remittance.claim_id without depending on auto-reconcile
|
||||
# heuristics (which don't match this minimal fixture).
|
||||
match_resp = seeded_db.post(
|
||||
"/api/reconciliation/match",
|
||||
json={"claim_id": "CLM001", "remit_id": "CLM001"},
|
||||
)
|
||||
assert match_resp.status_code == 200, match_resp.text
|
||||
# The minimal fixture's PCN ("CLM001") + charge ("$100.00") match,
|
||||
# so SP31's content-keys fallback pairs the remit during ingest.
|
||||
# No manual-match call is required (and would now 409 "already_matched").
|
||||
# The intent of this test — surfacing the remit_received event via
|
||||
# the Remittance.claim_id join — is fully exercised either way.
|
||||
|
||||
resp = seeded_db.get(f"/api/config/providers/{TEST_837_NPI}")
|
||||
assert resp.status_code == 200, resp.text
|
||||
|
||||
@@ -54,8 +54,8 @@ def test_seed_creates_clearhouse_singleton():
|
||||
assert ch.submitter_name == "Dzinesco"
|
||||
assert ch.sftp_block.host == "mft.gainwelltechnologies.com"
|
||||
assert ch.sftp_block.stub is True
|
||||
assert "FromHPE" in ch.sftp_block.paths["outbound"]
|
||||
assert "ToHPE" in ch.sftp_block.paths["inbound"]
|
||||
assert "ToHPE" in ch.sftp_block.paths["outbound"]
|
||||
assert "FromHPE" in ch.sftp_block.paths["inbound"]
|
||||
|
||||
|
||||
def test_seed_creates_co_txix_payer_with_both_configs():
|
||||
|
||||
@@ -55,7 +55,7 @@ def test_match_same_pcn_within_window_matches():
|
||||
assert len(matches) == 1
|
||||
assert matches[0].claim.id == "CLM-1"
|
||||
assert matches[0].remittance.id == "CLP-1"
|
||||
assert matches[0].strategy == "auto"
|
||||
assert matches[0].strategy == "pcn-exact"
|
||||
|
||||
|
||||
def test_match_dates_outside_window_unmatched():
|
||||
@@ -203,7 +203,7 @@ def test_apply_reversal_of_already_reversed_is_noop():
|
||||
def test_split_unmatched_partitions_by_match_set():
|
||||
claims = [FakeClaim("CLM-1", "A", None), FakeClaim("CLM-2", "B", None)]
|
||||
remits = [FakeRemit("CLP-1", "A", None), FakeRemit("CLP-2", "C", None)]
|
||||
matches = [ReconcileMatch(claim=claims[0], remittance=remits[0], strategy="auto", is_reversal=False)]
|
||||
matches = [ReconcileMatch(claim=claims[0], remittance=remits[0], strategy="pcn-exact", is_reversal=False)]
|
||||
unmatched_claims, unmatched_remits = reconcile.split_unmatched(claims, remits, matches)
|
||||
assert [c.id for c in unmatched_claims] == ["CLM-2"]
|
||||
assert [r.id for r in unmatched_remits] == ["CLP-2"]
|
||||
@@ -306,7 +306,7 @@ def test_run_reversal_flips_paid_to_reversed(fixture_835):
|
||||
# Match row already exists from prior reconcile.
|
||||
s.add(Match(
|
||||
claim_id="CLM-1", remittance_id="CLP-OLD:b1xxxxxx",
|
||||
strategy="auto", matched_at=datetime.now(timezone.utc),
|
||||
strategy="pcn-exact", matched_at=datetime.now(timezone.utc),
|
||||
is_reversal=False,
|
||||
))
|
||||
s.flush()
|
||||
@@ -330,8 +330,20 @@ def test_run_reversal_flips_paid_to_reversed(fixture_835):
|
||||
assert reversal_match.prior_claim_state == ClaimState.PAID
|
||||
|
||||
|
||||
def test_run_failed_reconcile_writes_activity_event(fixture_835):
|
||||
"""If reconcile crashes, the batch + remittances stay; activity event records failure."""
|
||||
def test_run_reconcile_raise_in_session_leaves_prior_commits_alone(fixture_835):
|
||||
"""A ``reconcile.run`` raise inside an open session does not damage
|
||||
previously-committed rows.
|
||||
|
||||
The test seeds a batch + remit, commits them in one session, then
|
||||
calls ``reconcile.run`` in a fresh session with ``match`` monkey-
|
||||
patched to raise. The pre-existing rows must survive — they're on
|
||||
disk from the prior commit, separate from the rolling-back
|
||||
session. This pins the unit-level invariant that ``reconcile.run``
|
||||
itself never commits and never silently mutates rows outside the
|
||||
session it was given; it is the *caller's* responsibility (now
|
||||
``CycloneStore.add`` per SP27 Task 10) to control the transaction
|
||||
boundary.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
_make_batch(s)
|
||||
_make_remit(s, "CLP-1", "PCN-A", "1", "100.00", "100.00")
|
||||
@@ -358,3 +370,526 @@ def test_run_failed_reconcile_writes_activity_event(fixture_835):
|
||||
assert b is not None
|
||||
r = s.query(Remittance).first()
|
||||
assert r is not None
|
||||
|
||||
|
||||
# --- SP31: content-keys fallback matcher (pure helper) -----------------------
|
||||
|
||||
|
||||
def _make_remit_for_keys(remit_id: str, pcn: str, charge, npi: str):
|
||||
"""Tiny shim with the three fields _content_keys_match reads."""
|
||||
from cyclone.reconcile import _RemitLike # protocol not enforced at runtime
|
||||
return type("R", (), {
|
||||
"id": remit_id,
|
||||
"payer_claim_control_number": pcn,
|
||||
"total_charge_amount": charge,
|
||||
"rendering_provider_npi": npi,
|
||||
})()
|
||||
|
||||
|
||||
def _make_claim_for_keys(claim_id: str, pcn: str, charge, npi: str):
|
||||
return type("C", (), {
|
||||
"id": claim_id,
|
||||
"patient_control_number": pcn,
|
||||
"total_charge": charge,
|
||||
"rendering_provider_npi": npi,
|
||||
})()
|
||||
|
||||
|
||||
def test_content_keys_pcn_plus_charge_matches():
|
||||
"""PCN + charge match (NPI mismatched) → 2-of-3, returns ``{"pcn","charge"}``."""
|
||||
from cyclone.reconcile import _content_keys_match
|
||||
from decimal import Decimal
|
||||
r = _make_remit_for_keys("r1", "ABC", Decimal("100.00"), "")
|
||||
c = _make_claim_for_keys("c1", "ABC", Decimal("100.00"), "1111111111")
|
||||
assert _content_keys_match(r, c) == {"pcn", "charge"}
|
||||
|
||||
|
||||
def test_content_keys_pcn_plus_npi_matches():
|
||||
"""PCN + NPI match (charge mismatched) → 2-of-3, returns ``{"pcn","npi"}``."""
|
||||
from cyclone.reconcile import _content_keys_match
|
||||
from decimal import Decimal
|
||||
r = _make_remit_for_keys("r1", "ABC", Decimal("100.00"), "1111111111")
|
||||
c = _make_claim_for_keys("c1", "ABC", Decimal("200.00"), "1111111111")
|
||||
assert _content_keys_match(r, c) == {"pcn", "npi"}
|
||||
|
||||
|
||||
def test_content_keys_charge_plus_npi_matches():
|
||||
"""Charge + NPI match (PCN mismatched) → 2-of-3, returns ``{"charge","npi"}``."""
|
||||
from cyclone.reconcile import _content_keys_match
|
||||
from decimal import Decimal
|
||||
r = _make_remit_for_keys("r1", "ABC", Decimal("100.00"), "1111111111")
|
||||
c = _make_claim_for_keys("c1", "XYZ", Decimal("100.00"), "1111111111")
|
||||
assert _content_keys_match(r, c) == {"charge", "npi"}
|
||||
|
||||
|
||||
def test_content_keys_all_three_match():
|
||||
"""All 3 match → 3-of-3, returns ``{"pcn","charge","npi"}``."""
|
||||
from cyclone.reconcile import _content_keys_match
|
||||
from decimal import Decimal
|
||||
r = _make_remit_for_keys("r1", "ABC", Decimal("100.00"), "1111111111")
|
||||
c = _make_claim_for_keys("c1", "ABC", Decimal("100.00"), "1111111111")
|
||||
assert _content_keys_match(r, c) == {"pcn", "charge", "npi"}
|
||||
|
||||
|
||||
def test_content_keys_only_pcn_does_not_match():
|
||||
"""Only PCN matches (charge + NPI both wrong) → set is just ``{"pcn"}``."""
|
||||
from cyclone.reconcile import _content_keys_match
|
||||
from decimal import Decimal
|
||||
r = _make_remit_for_keys("r1", "ABC", Decimal("100.00"), "1111111111")
|
||||
c = _make_claim_for_keys("c1", "ABC", Decimal("200.00"), "2222222222")
|
||||
assert _content_keys_match(r, c) == {"pcn"}
|
||||
|
||||
|
||||
def test_content_keys_only_charge_does_not_match():
|
||||
"""Only charge matches (PCN + NPI both wrong) → set is just ``{"charge"}``."""
|
||||
from cyclone.reconcile import _content_keys_match
|
||||
from decimal import Decimal
|
||||
r = _make_remit_for_keys("r1", "ABC", Decimal("100.00"), "1111111111")
|
||||
c = _make_claim_for_keys("c1", "XYZ", Decimal("100.00"), "2222222222")
|
||||
assert _content_keys_match(r, c) == {"charge"}
|
||||
|
||||
|
||||
def test_content_keys_none_match():
|
||||
"""All 3 fields disagree → empty set."""
|
||||
from cyclone.reconcile import _content_keys_match
|
||||
from decimal import Decimal
|
||||
r = _make_remit_for_keys("r1", "ABC", Decimal("100.00"), "1111111111")
|
||||
c = _make_claim_for_keys("c1", "XYZ", Decimal("200.00"), "2222222222")
|
||||
assert _content_keys_match(r, c) == set()
|
||||
|
||||
|
||||
def test_content_keys_charge_tolerance_is_one_cent():
|
||||
"""Charge differs by exactly $0.005 (rounding) → charge counts as matched."""
|
||||
from cyclone.reconcile import _content_keys_match
|
||||
from decimal import Decimal
|
||||
r = _make_remit_for_keys("r1", "ABC", Decimal("100.005"), "1111111111")
|
||||
c = _make_claim_for_keys("c1", "ABC", Decimal("100.00"), "1111111111")
|
||||
assert _content_keys_match(r, c) == {"pcn", "charge", "npi"}
|
||||
|
||||
|
||||
def test_content_keys_charge_differs_by_two_cents_no_match():
|
||||
"""Charge differs by $0.02 → charge does NOT match (outside tolerance)."""
|
||||
from cyclone.reconcile import _content_keys_match
|
||||
from decimal import Decimal
|
||||
# PCN and NPI deliberately differ so the only potential match is charge.
|
||||
# Charge is outside the $0.01 tolerance → 0 of 3 match → empty set.
|
||||
r = _make_remit_for_keys("r1", "ABC", Decimal("100.02"), "1111111111")
|
||||
c = _make_claim_for_keys("c1", "XYZ", Decimal("100.00"), "2222222222")
|
||||
assert _content_keys_match(r, c) == set()
|
||||
|
||||
|
||||
def test_content_keys_empty_npi_does_not_count():
|
||||
"""Empty NPI on remit counts as not-matched (falls back to PCN+charge rule)."""
|
||||
from cyclone.reconcile import _content_keys_match
|
||||
from decimal import Decimal
|
||||
r = _make_remit_for_keys("r1", "ABC", Decimal("100.00"), "")
|
||||
c = _make_claim_for_keys("c1", "ABC", Decimal("100.00"), "1111111111")
|
||||
assert _content_keys_match(r, c) == {"pcn", "charge"} # NPI skipped on remit
|
||||
c2 = _make_claim_for_keys("c2", "XYZ", Decimal("100.00"), "1111111111")
|
||||
assert _content_keys_match(r, c2) == {"charge"} # only charge, no PCN
|
||||
|
||||
|
||||
# --- SP31: content-keys fallback matcher (DB helper) ------------------------
|
||||
|
||||
|
||||
def test_score_fallback_unique_match_returns_claim_id(db_session, make_claim, make_remit):
|
||||
"""One claim in pool matches 2-of-3 → returns (claim_id, keys, candidate_count)."""
|
||||
from decimal import Decimal
|
||||
from datetime import date, timedelta
|
||||
from cyclone.reconcile import _score_fallback_candidates
|
||||
c1 = make_claim(patient_control_number="ABC", total_charge=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date_from=date(2026, 6, 1))
|
||||
c2 = make_claim(patient_control_number="XYZ", total_charge=Decimal("999.99"),
|
||||
rendering_provider_npi="2222222222",
|
||||
service_date_from=date(2026, 6, 1))
|
||||
r = make_remit(payer_claim_control_number="ABC",
|
||||
total_charge_amount=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date=date(2026, 6, 5)) # ±30-day window
|
||||
result = _score_fallback_candidates(db_session, r)
|
||||
assert result is not None
|
||||
claim_id, keys_matched, candidate_count = result
|
||||
assert claim_id == c1.id
|
||||
assert keys_matched == {"pcn", "charge", "npi"} # 3-of-3
|
||||
assert candidate_count == 2 # pool had 2 candidates
|
||||
|
||||
|
||||
def test_score_fallback_zero_matches_returns_none(db_session, make_claim, make_remit):
|
||||
"""No claim in pool matches 2-of-3 → returns None."""
|
||||
from decimal import Decimal
|
||||
from datetime import date
|
||||
from cyclone.reconcile import _score_fallback_candidates
|
||||
make_claim(patient_control_number="XYZ", total_charge=Decimal("999.99"),
|
||||
rendering_provider_npi="2222222222",
|
||||
service_date_from=date(2026, 6, 1))
|
||||
r = make_remit(payer_claim_control_number="ABC",
|
||||
total_charge_amount=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date=date(2026, 6, 5))
|
||||
assert _score_fallback_candidates(db_session, r) is None
|
||||
|
||||
|
||||
def test_score_fallback_two_matches_returns_none(db_session, make_claim, make_remit):
|
||||
"""Two claims in pool both match 2-of-3 → returns None (ambiguous)."""
|
||||
from decimal import Decimal
|
||||
from datetime import date
|
||||
from cyclone.reconcile import _score_fallback_candidates
|
||||
# Both claims have same PCN and same charge — both would match PCN+charge.
|
||||
make_claim(patient_control_number="ABC", total_charge=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date_from=date(2026, 6, 1))
|
||||
make_claim(patient_control_number="ABC", total_charge=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date_from=date(2026, 6, 2))
|
||||
r = make_remit(payer_claim_control_number="ABC",
|
||||
total_charge_amount=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date=date(2026, 6, 5))
|
||||
assert _score_fallback_candidates(db_session, r) is None
|
||||
|
||||
|
||||
def test_score_fallback_skips_already_matched(db_session, make_claim, make_remit):
|
||||
"""Claim with matched_remittance_id set → excluded from pool."""
|
||||
from decimal import Decimal
|
||||
from datetime import date
|
||||
from cyclone.reconcile import _score_fallback_candidates
|
||||
c1 = make_claim(patient_control_number="ABC", total_charge=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date_from=date(2026, 6, 1),
|
||||
matched_remittance_id="some-other-remit-id")
|
||||
r = make_remit(payer_claim_control_number="ABC",
|
||||
total_charge_amount=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date=date(2026, 6, 5))
|
||||
assert _score_fallback_candidates(db_session, r) is None # only candidate was excluded
|
||||
|
||||
|
||||
def test_score_fallback_skips_terminal_state(db_session, make_claim, make_remit):
|
||||
"""Claim in PAID/DENIED/REJECTED/REVERSED/RECONCILED → excluded from pool."""
|
||||
from decimal import Decimal
|
||||
from datetime import date
|
||||
from cyclone.db import ClaimState
|
||||
from cyclone.reconcile import _score_fallback_candidates
|
||||
make_claim(patient_control_number="ABC", total_charge=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date_from=date(2026, 6, 1),
|
||||
state=ClaimState.PAID)
|
||||
r = make_remit(payer_claim_control_number="ABC",
|
||||
total_charge_amount=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date=date(2026, 6, 5))
|
||||
assert _score_fallback_candidates(db_session, r) is None
|
||||
|
||||
|
||||
def test_score_fallback_widened_window_catches_late_remit(db_session, make_claim, make_remit):
|
||||
"""Remit 25 days after claim → still in pool (30-day window)."""
|
||||
from decimal import Decimal
|
||||
from datetime import date, timedelta
|
||||
from cyclone.reconcile import _score_fallback_candidates
|
||||
claim_date = date(2026, 6, 1)
|
||||
remit_date = claim_date + timedelta(days=25)
|
||||
c1 = make_claim(patient_control_number="ABC", total_charge=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date_from=claim_date)
|
||||
r = make_remit(payer_claim_control_number="ABC",
|
||||
total_charge_amount=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date=remit_date)
|
||||
result = _score_fallback_candidates(db_session, r)
|
||||
assert result is not None
|
||||
claim_id, keys_matched, candidate_count = result
|
||||
assert claim_id == c1.id
|
||||
assert candidate_count == 1
|
||||
|
||||
|
||||
def test_score_fallback_outside_window_excluded(db_session, make_claim, make_remit):
|
||||
"""Remit 60 days after claim → excluded (outside 30-day window)."""
|
||||
from decimal import Decimal
|
||||
from datetime import date, timedelta
|
||||
from cyclone.reconcile import _score_fallback_candidates
|
||||
claim_date = date(2026, 6, 1)
|
||||
remit_date = claim_date + timedelta(days=60)
|
||||
make_claim(patient_control_number="ABC", total_charge=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date_from=claim_date)
|
||||
r = make_remit(payer_claim_control_number="ABC",
|
||||
total_charge_amount=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date=remit_date)
|
||||
assert _score_fallback_candidates(db_session, r) is None
|
||||
|
||||
|
||||
# --- SP31 Task 3 fix: Remittance NPI lives in raw_json ----------------------
|
||||
|
||||
|
||||
def test_content_keys_npi_from_remit_raw_json(db_session, make_claim):
|
||||
"""NPI on the Remit is read from ``raw_json`` when the attribute is absent.
|
||||
|
||||
The 835 parser stores rendering_provider_npi on ``Remittance.raw_json``
|
||||
(the ORM has no dedicated column) — reconcile.run() hands a raw
|
||||
``Remittance`` instance to ``_content_keys_match``, so the helper must
|
||||
fall through ``getattr(...rendering_provider_npi...)`` to the raw_json
|
||||
read. This test pins that path with a real ORM row (no transient
|
||||
``setattr`` shadowing the production layout).
|
||||
|
||||
Configuration: PCN mismatched, charge matches, NPI matches via raw_json
|
||||
→ 2 of 3 keys agree → True.
|
||||
"""
|
||||
from decimal import Decimal
|
||||
from datetime import date, datetime, timezone
|
||||
from cyclone.db import Remittance
|
||||
from cyclone.reconcile import _content_keys_match
|
||||
claim = make_claim(
|
||||
patient_control_number="ABC",
|
||||
total_charge=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date_from=date(2026, 6, 1),
|
||||
)
|
||||
# Real Remittance ORM instance — NPI lives ONLY in raw_json, not as a
|
||||
# transient attribute. (The 835 parser populates raw_json that way;
|
||||
# the make_remit factory's transient ``rendering_provider_npi`` setattr
|
||||
# would shadow the production read.)
|
||||
remit = Remittance(
|
||||
id="real-remit-1",
|
||||
batch_id="test-batch",
|
||||
payer_claim_control_number="OTHER-PCN", # PCN: won't match "ABC"
|
||||
status_code="1",
|
||||
total_charge=Decimal("100.00"), # charge: matches
|
||||
received_at=datetime(2026, 6, 19, tzinfo=timezone.utc),
|
||||
raw_json={"rendering_provider_npi": "1111111111"}, # NPI: matches
|
||||
)
|
||||
db_session.add(remit)
|
||||
db_session.flush()
|
||||
assert _content_keys_match(remit, claim) == {"charge", "npi"} # PCN mismatched
|
||||
|
||||
|
||||
# --- SP31: end-to-end integration via reconcile.run() -----------------------
|
||||
|
||||
|
||||
def test_reconcile_run_emits_score_auto_match_when_pcn_misses(db_session, make_claim, make_remit):
|
||||
"""A remit with no PCN match but matching charge+NPI triggers score-auto."""
|
||||
from decimal import Decimal
|
||||
from datetime import date
|
||||
from sqlalchemy import select
|
||||
from cyclone.db import Batch, ClaimState
|
||||
from cyclone.reconcile import run as reconcile_run
|
||||
claim = make_claim(patient_control_number="UNIQUE-CLM",
|
||||
total_charge=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date_from=date(2026, 6, 1))
|
||||
remit = make_remit(payer_claim_control_number="DIFFERENT-PCN",
|
||||
total_charge_amount=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date=date(2026, 6, 5))
|
||||
batch = Batch(id="batch-1", kind="835", input_filename="x.835", parsed_at=date(2026, 6, 5))
|
||||
db_session.add(batch)
|
||||
db_session.flush()
|
||||
# Re-parent the remit to this batch (make_remit doesn't set batch_id).
|
||||
remit.batch_id = batch.id
|
||||
db_session.flush()
|
||||
|
||||
result = reconcile_run(db_session, batch.id)
|
||||
db_session.flush() # autoflush=False on SessionLocal; force flush pending Match row
|
||||
|
||||
assert result.matched == 1
|
||||
# Verify Match row was written with strategy="score-auto"
|
||||
from cyclone.db import Match
|
||||
matches = list(db_session.execute(
|
||||
select(Match).where(Match.remittance_id == remit.id)
|
||||
).scalars().all())
|
||||
assert len(matches) == 1
|
||||
assert matches[0].strategy == "score-auto"
|
||||
assert matches[0].claim_id == claim.id
|
||||
# Symmetric FK: the auto-match path sets Remittance.claim_id so the
|
||||
# list_unmatched filter (``Remittance.claim_id IS NULL``) correctly
|
||||
# drops this pair from the orphan bucket. Mirrors the manual_match
|
||||
# contract — both paths must populate both sides of the FK pair.
|
||||
db_session.refresh(remit)
|
||||
assert remit.claim_id == claim.id
|
||||
# Verify ActivityEvent was emitted with kind="auto_matched_835"
|
||||
from cyclone.db import ActivityEvent
|
||||
events = list(db_session.execute(
|
||||
select(ActivityEvent).where(
|
||||
ActivityEvent.remittance_id == remit.id,
|
||||
ActivityEvent.kind == "auto_matched_835",
|
||||
)
|
||||
).scalars().all())
|
||||
assert len(events) == 1
|
||||
assert events[0].claim_id == claim.id
|
||||
# Spec D8: payload records keys_matched + candidate_count so the
|
||||
# operator can see *why* the auto-link fired.
|
||||
payload = events[0].payload_json or {}
|
||||
assert "keys_matched" in payload
|
||||
assert "candidate_count" in payload
|
||||
assert payload["candidate_count"] == 1
|
||||
assert set(payload["keys_matched"]) >= {"charge", "npi"} # PCN differs in this test
|
||||
# Verify claim state was flipped
|
||||
db_session.refresh(claim)
|
||||
assert claim.state in {ClaimState.PAID, ClaimState.PARTIAL, ClaimState.RECEIVED}
|
||||
|
||||
|
||||
def test_reconcile_run_pcn_path_unchanged(db_session, make_claim, make_remit):
|
||||
"""Existing PCN-exact match still produces strategy='pcn-exact' (regression guard)."""
|
||||
from decimal import Decimal
|
||||
from datetime import date
|
||||
from sqlalchemy import select
|
||||
from cyclone.db import Batch, Match
|
||||
from cyclone.reconcile import run as reconcile_run
|
||||
claim = make_claim(patient_control_number="SAME-PCN",
|
||||
total_charge=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date_from=date(2026, 6, 1))
|
||||
remit = make_remit(payer_claim_control_number="SAME-PCN",
|
||||
total_charge_amount=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date=date(2026, 6, 2))
|
||||
batch = Batch(id="batch-1", kind="835", input_filename="x.835", parsed_at=date(2026, 6, 5))
|
||||
db_session.add(batch)
|
||||
db_session.flush()
|
||||
remit.batch_id = batch.id
|
||||
db_session.flush()
|
||||
|
||||
reconcile_run(db_session, batch.id)
|
||||
db_session.flush() # autoflush=False on SessionLocal; force flush pending Match row
|
||||
|
||||
matches = list(db_session.execute(
|
||||
select(Match).where(Match.remittance_id == remit.id)
|
||||
).scalars().all())
|
||||
assert len(matches) == 1
|
||||
assert matches[0].strategy == "pcn-exact" # NOT "score-auto"
|
||||
|
||||
|
||||
def test_reconcile_run_no_match_leaves_remit_unlinked(db_session, make_claim, make_remit):
|
||||
"""No 2-of-3 match → no Match row, claim state unchanged."""
|
||||
from decimal import Decimal
|
||||
from datetime import date
|
||||
from sqlalchemy import select
|
||||
from cyclone.db import Batch, Match, ClaimState
|
||||
from cyclone.reconcile import run as reconcile_run
|
||||
claim = make_claim(patient_control_number="UNRELATED",
|
||||
total_charge=Decimal("999.99"),
|
||||
rendering_provider_npi="2222222222",
|
||||
service_date_from=date(2026, 6, 1))
|
||||
remit = make_remit(payer_claim_control_number="OTHER",
|
||||
total_charge_amount=Decimal("100.00"),
|
||||
rendering_provider_npi="1111111111",
|
||||
service_date=date(2026, 6, 5))
|
||||
batch = Batch(id="batch-1", kind="835", input_filename="x.835", parsed_at=date(2026, 6, 5))
|
||||
db_session.add(batch)
|
||||
db_session.flush()
|
||||
remit.batch_id = batch.id
|
||||
db_session.flush()
|
||||
|
||||
result = reconcile_run(db_session, batch.id)
|
||||
|
||||
assert result.matched == 0
|
||||
matches = list(db_session.execute(
|
||||
select(Match).where(Match.remittance_id == remit.id)
|
||||
).scalars().all())
|
||||
assert len(matches) == 0
|
||||
db_session.refresh(claim)
|
||||
assert claim.state == ClaimState.SUBMITTED # unchanged
|
||||
db_session.refresh(remit)
|
||||
assert remit.claim_id is None # unchanged
|
||||
|
||||
|
||||
# --- SP32 Task 4: ORM builders wire rendering_provider_npi -----------------
|
||||
|
||||
|
||||
def test_claim_837_row_includes_rendering_provider_npi():
|
||||
"""SP32: Claim.rendering_provider_npi is populated from ClaimOutput in ORM builder."""
|
||||
from pathlib import Path
|
||||
from cyclone.parsers.parse_837 import parse
|
||||
from cyclone.parsers.payer import PayerConfig
|
||||
from cyclone.store.orm_builders import _claim_837_row
|
||||
|
||||
text = Path("tests/fixtures/co_medicaid_837p_with_renderer.txt").read_text()
|
||||
parsed = parse(text, PayerConfig.co_medicaid())
|
||||
# Find the claim whose rendering NPI was extracted.
|
||||
targets = [c for c in parsed.claims if c.rendering_provider_npi == "1234567893"]
|
||||
assert len(targets) == 1
|
||||
row = _claim_837_row(targets[0], batch_id="test-batch")
|
||||
assert row.rendering_provider_npi == "1234567893"
|
||||
|
||||
|
||||
def test_remittance_835_row_includes_service_provider_npi():
|
||||
"""SP32: Remittance.rendering_provider_npi is populated from ClaimPayment in ORM builder."""
|
||||
from cyclone.parsers.parse_835 import parse as parse_835
|
||||
from cyclone.store.orm_builders import _remittance_835_row
|
||||
|
||||
text = (
|
||||
"ISA*00* *00* *ZZ*RECEIVERID *ZZ*SENDERID *250101*1200*^*00501*000000001*0*P*:~"
|
||||
"GS*HP*RECEIVER*SENDER*20250101*1200*1*X*005010X221A1~"
|
||||
"ST*835*0001~"
|
||||
"BPR*I*85.40*C*NON*****01*021000021*DA*123456*1511111**01*031302955*DA*9876543~"
|
||||
"TRN*1*1511111*1511111~"
|
||||
"DTM*405*20250101~"
|
||||
"N1*PR*COLORADO MEDICAID*XV*CO MEDICAID~"
|
||||
"N3*PO BOX 1100*~"
|
||||
"N4*DENVER*CO*80203~"
|
||||
"REF*2U*12345~"
|
||||
"PER*BL*SUPPORT*TE*8005551212~"
|
||||
"N1*PE*ACME CLINIC*XX*1111111111~"
|
||||
"LX*1~"
|
||||
"CLP*CLM001*1*85.40*85.40*0.00*MC*CLM001*11*1~"
|
||||
"CAS*PR*1*0.00~"
|
||||
"NM1*1P*2*RENDERING PROVIDER*****XX*2222222222~"
|
||||
"SVC*HC:99213*85.40*85.40**1~"
|
||||
"DTM*472*20250101~"
|
||||
"SE*18*0001~"
|
||||
"GE*1*1~"
|
||||
"IEA*1*000000001~"
|
||||
)
|
||||
parsed = parse_835(text, payer_config=None) # type: ignore[arg-type]
|
||||
assert len(parsed.claims) == 1
|
||||
assert parsed.claims[0].service_provider_npi == "2222222222"
|
||||
row = _remittance_835_row(parsed.claims[0], batch_id="test-batch")
|
||||
assert row.rendering_provider_npi == "2222222222"
|
||||
|
||||
|
||||
# --- SP32 Task 5: typed-column NPI preference in _content_keys_match -------
|
||||
|
||||
|
||||
def test_content_keys_match_typed_npi_columns_match(db_session, make_claim, make_remit):
|
||||
"""SP32: typed-column NPI contributes to the matched set when both sides have it."""
|
||||
from cyclone.reconcile import _content_keys_match
|
||||
from datetime import date
|
||||
claim = make_claim(
|
||||
patient_control_number="PCN1",
|
||||
total_charge=Decimal("85.40"),
|
||||
rendering_provider_npi="2222222222",
|
||||
service_date_from=date(2025, 1, 1),
|
||||
)
|
||||
remit = make_remit(
|
||||
payer_claim_control_number="PCN1",
|
||||
total_charge_amount=Decimal("85.40"),
|
||||
rendering_provider_npi="2222222222",
|
||||
service_date=date(2025, 1, 1),
|
||||
)
|
||||
matched = _content_keys_match(remit, claim)
|
||||
assert "npi" in matched
|
||||
assert "pcn" in matched
|
||||
assert "charge" in matched
|
||||
|
||||
|
||||
def test_content_keys_match_npi_mismatch_does_not_match(db_session, make_claim, make_remit):
|
||||
"""SP32: NPI arm does not fire when NPIs differ."""
|
||||
from cyclone.reconcile import _content_keys_match
|
||||
from datetime import date
|
||||
claim = make_claim(
|
||||
patient_control_number="PCN2",
|
||||
total_charge=Decimal("10.00"),
|
||||
rendering_provider_npi="2222222222",
|
||||
service_date_from=date(2025, 1, 1),
|
||||
)
|
||||
remit = make_remit(
|
||||
payer_claim_control_number="PCN2",
|
||||
total_charge_amount=Decimal("10.00"),
|
||||
rendering_provider_npi="9999999999", # different NPI
|
||||
service_date=date(2025, 1, 1),
|
||||
)
|
||||
matched = _content_keys_match(remit, claim)
|
||||
assert "npi" not in matched
|
||||
assert "pcn" in matched
|
||||
assert "charge" in matched
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user