merge: SP27 remittances architecture refactor into main
Atomic SP-N merge per cyclone-spec. Carries the remittances
architecture refactor (handlers/ extraction, unified 835 ingest +
reconcile, server-aggregated KPIs) plus two same-increment fixes
for bugs surfaced against the running stack:
- fix(sp27): /api/claims + /api/remittances total counts the
full population (previously page-local reduce)
- feat(sp27): server-aggregate Dashboard KPIs so the 100-row
sample doesn't lie
- fix(sp27): Claim.patient_control_number populated from CLM01
(claim_id), not 2010BA NM109 (member_id) — restores 837↔835
auto-match via the existing reconcile.by_pcn join
- feat(sp27): server-aggregate Remittances KPIs (count, paid,
adjustments)
Includes migration 0017 (UPDATE claims SET patient_control_number
= id) which was applied to the live DB at ingest time.
This commit is contained in:
@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
|||||||
|
|
||||||
## What this is
|
## What this is
|
||||||
|
|
||||||
Cyclone is a self-hosted X12 EDI claims-management suite for a single billing office (Colorado Medicaid currently). It parses 837P professional claims and 835 ERA remittances (X12 005010X222A1 / 005010X221A1) and also handles 999, TA1, 270, 271, and 277CA. Local-only by design: binds to `127.0.0.1`, requires login (auth boundary is HTTP; bcrypt + HttpOnly session cookie; first admin bootstrapped from `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD` env vars; see SP24 spec for the full posture), no internet exposure.
|
Cyclone is a self-hosted X12 EDI claims-management suite for a single billing office (Colorado Medicaid currently). It parses 837P professional claims and 835 ERA remittances (X12 005010X222A1 / 005010X221A1) and also handles 999, TA1, 270, 271, and 277CA. Always binds to `0.0.0.0` — the host firewall / compose port publishing is what restricts reachability, not the bind address. Requires login (auth boundary is HTTP; bcrypt + HttpOnly session cookie; first admin bootstrapped from `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD` env vars; see SP24 spec for the full posture). LAN-only by design — don't expose the published ports to the WAN.
|
||||||
|
|
||||||
Stack: one Python process (FastAPI + uvicorn, port 8000) + one Node process in dev (Vite, port 5173). The authoritative state is a single SQLite file at `~/.local/share/cyclone/cyclone.db` (or SQLCipher at the same path when the macOS Keychain entry + `sqlcipher3` are both present).
|
Stack: one Python process (FastAPI + uvicorn, port 8000) + one Node process in dev (Vite, port 5173). The authoritative state is a single SQLite file at `~/.local/share/cyclone/cyclone.db` (or SQLCipher at the same path when the macOS Keychain entry + `sqlcipher3` are both present).
|
||||||
|
|
||||||
@@ -30,7 +30,7 @@ Optional backend extras: `pip install -e '.[sqlcipher]'` (encryption at rest, SP
|
|||||||
```bash
|
```bash
|
||||||
# Terminal 1 — backend
|
# Terminal 1 — backend
|
||||||
cd backend
|
cd backend
|
||||||
.venv/bin/python -m cyclone serve # default 127.0.0.1:8000
|
.venv/bin/python -m cyclone serve # default 0.0.0.0:8000
|
||||||
# CYCLONE_PORT=... overrides port; CYCLONE_RELOAD=1 enables uvicorn --reload
|
# CYCLONE_PORT=... overrides port; CYCLONE_RELOAD=1 enables uvicorn --reload
|
||||||
# Or: .venv/bin/uvicorn cyclone.api:app --reload --port 8000
|
# Or: .venv/bin/uvicorn cyclone.api:app --reload --port 8000
|
||||||
|
|
||||||
@@ -179,6 +179,6 @@ Exit codes are documented per subcommand in `cyclone-cli` — `0` for success, `
|
|||||||
- **Don't call `useTailStream` from inside a `use<X>` hook.** The subscription lives on the page so the lifecycle ties to whoever mounts the hook, not to whoever happens to call it.
|
- **Don't call `useTailStream` from inside a `use<X>` hook.** The subscription lives on the page so the lifecycle ties to whoever mounts the hook, not to whoever happens to call it.
|
||||||
- **The store facade.** The public API of `cyclone.store` is preserved through SP21's split — call through the facade, not directly into the underlying modules.
|
- **The store facade.** The public API of `cyclone.store` is preserved through SP21's split — call through the facade, not directly into the underlying modules.
|
||||||
- **Encryption is optional, not required.** When the Keychain entry is missing **or** `sqlcipher3` is not installed, the DB falls back to plain SQLite. Don't fail boot on missing encryption.
|
- **Encryption is optional, not required.** When the Keychain entry is missing **or** `sqlcipher3` is not installed, the DB falls back to plain SQLite. Don't fail boot on missing encryption.
|
||||||
- **Local-only by design.** The backend binds to `127.0.0.1`, requires login (bcrypt + HttpOnly session cookie; see SP24 spec), and the threat model is still a stolen/imaged drive — SQLCipher at rest and the macOS Keychain handle that. The auth boundary is the HTTP layer; the file-system posture is unchanged. Don't add internet exposure. Don't disable auth without an explicit `CYCLONE_AUTH_DISABLED=1` env var (the escape hatch logs a WARNING at boot).
|
- **Always bind to 0.0.0.0.** The bind address does not control reachability — the host firewall / compose port publishing does. Requires login (bcrypt + HttpOnly session cookie; see SP24 spec), and the threat model is still a stolen/imaged drive — SQLCipher at rest and the macOS Keychain handle that. The auth boundary is the HTTP layer; the file-system posture is unchanged. Don't expose the published ports to the public internet (LAN-only or VPN-fronted). Don't disable auth without an explicit `CYCLONE_AUTH_DISABLED=1` env var (the escape hatch logs a WARNING at boot).
|
||||||
</content>
|
</content>
|
||||||
</invoke>
|
</invoke>
|
||||||
@@ -30,7 +30,7 @@ Two terminals:
|
|||||||
# Terminal 1 — backend
|
# Terminal 1 — backend
|
||||||
cd backend
|
cd backend
|
||||||
.venv/bin/python -m cyclone serve
|
.venv/bin/python -m cyclone serve
|
||||||
# (defaults to 127.0.0.1:8000; override with CYCLONE_PORT=...; reload with CYCLONE_RELOAD=1)
|
# (defaults to 0.0.0.0:8000; override with CYCLONE_HOST / CYCLONE_PORT / CYCLONE_RELOAD=1)
|
||||||
|
|
||||||
# Terminal 2 — frontend
|
# Terminal 2 — frontend
|
||||||
npm run dev
|
npm run dev
|
||||||
@@ -690,7 +690,7 @@ backup create` manually retains full control.
|
|||||||
The `clearhouse.submit` endpoint uses `paramiko` to push a batch of
|
The `clearhouse.submit` endpoint uses `paramiko` to push a batch of
|
||||||
generated 837 files to the dzinesco SFTP server
|
generated 837 files to the dzinesco SFTP server
|
||||||
(`mft.gainwelltechnologies.com:22`, path
|
(`mft.gainwelltechnologies.com:22`, path
|
||||||
`/CO XIX/PROD/coxix_prod_11525703/FromHPE`). The SFTP credential is
|
`/CO XIX/PROD/coxix_prod_11525703/ToHPE`). The SFTP credential is
|
||||||
fetched from the macOS Keychain at call time — never read from YAML,
|
fetched from the macOS Keychain at call time — never read from YAML,
|
||||||
never logged, never written to disk. The wire-up honors the file-naming
|
never logged, never written to disk. The wire-up honors the file-naming
|
||||||
template stored in the `clearhouse` config:
|
template stored in the `clearhouse` config:
|
||||||
@@ -898,7 +898,7 @@ Shipped sub-projects (most recent first):
|
|||||||
- **Sub-project 13 (shipped) — SFTP wire-up.** `paramiko`-backed
|
- **Sub-project 13 (shipped) — SFTP wire-up.** `paramiko`-backed
|
||||||
`SftpClient` replaces the SP9 stub. The clearhouse.submit endpoint
|
`SftpClient` replaces the SP9 stub. The clearhouse.submit endpoint
|
||||||
actually pushes to
|
actually pushes to
|
||||||
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
|
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/ToHPE`.
|
||||||
SFTP credentials are read from the macOS Keychain at call time.
|
SFTP credentials are read from the macOS Keychain at call time.
|
||||||
- **Sub-project 12 (shipped) — Encryption at rest.** Optional
|
- **Sub-project 12 (shipped) — Encryption at rest.** Optional
|
||||||
SQLCipher AES-256 encryption of the SQLite file, with the key
|
SQLCipher AES-256 encryption of the SQLite file, with the key
|
||||||
@@ -1160,7 +1160,7 @@ the one-time setup recipe.
|
|||||||
|
|
||||||
- `POST /api/clearhouse/submit` — same endpoint as SP9; the
|
- `POST /api/clearhouse/submit` — same endpoint as SP9; the
|
||||||
implementation is now a real `paramiko` `SftpClient.write` to
|
implementation is now a real `paramiko` `SftpClient.write` to
|
||||||
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
|
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/ToHPE`.
|
||||||
SFTP credentials are fetched from the macOS Keychain at call time.
|
SFTP credentials are fetched from the macOS Keychain at call time.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|||||||
+5
-2
@@ -35,7 +35,10 @@ COPY pyproject.toml ./
|
|||||||
# the cached wheel metadata when the name+version matches. See git
|
# the cached wheel metadata when the name+version matches. See git
|
||||||
# history on this file for the long version.
|
# history on this file for the long version.
|
||||||
COPY src/ ./src/
|
COPY src/ ./src/
|
||||||
RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher]'
|
# Install the sftp extra alongside sqlcipher so the real-mode SFTP
|
||||||
|
# client (paramiko) is available inside the container — required by
|
||||||
|
# SP25 + SP26 for live Gainwell MFT polling.
|
||||||
|
RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher,sftp]'
|
||||||
|
|
||||||
# ---------- runtime ----------
|
# ---------- runtime ----------
|
||||||
FROM python:3.11-slim-bookworm
|
FROM python:3.11-slim-bookworm
|
||||||
@@ -53,7 +56,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
|||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
|
|
||||||
COPY --from=builder /wheels /wheels
|
COPY --from=builder /wheels /wheels
|
||||||
RUN pip install --no-cache-dir --no-index --find-links /wheels 'cyclone[sqlcipher]' \
|
RUN pip install --no-cache-dir --no-index --find-links /wheels 'cyclone[sqlcipher,sftp]' \
|
||||||
&& rm -rf /wheels
|
&& rm -rf /wheels
|
||||||
|
|
||||||
# NOTE: we deliberately do NOT drop privileges to the `cyclone` user.
|
# NOTE: we deliberately do NOT drop privileges to the `cyclone` user.
|
||||||
|
|||||||
@@ -1,10 +1,11 @@
|
|||||||
"""Entry point for ``python -m cyclone``.
|
"""Entry point for ``python -m cyclone``.
|
||||||
|
|
||||||
* ``python -m cyclone`` (no args) — Click CLI (``cli.main``)
|
* ``python -m cyclone`` (no args) — Click CLI (``cli.main``)
|
||||||
* ``python -m cyclone serve`` — start the FastAPI app on 127.0.0.1:8000
|
* ``python -m cyclone serve`` — start the FastAPI app on 0.0.0.0:8000
|
||||||
|
|
||||||
Honors the env vars:
|
Honors the env vars:
|
||||||
|
|
||||||
|
* ``CYCLONE_HOST`` (default ``0.0.0.0`` — always bind to all interfaces)
|
||||||
* ``CYCLONE_PORT`` (default ``8000``)
|
* ``CYCLONE_PORT`` (default ``8000``)
|
||||||
* ``CYCLONE_RELOAD`` (default ``0``; set to ``1`` to enable uvicorn reload)
|
* ``CYCLONE_RELOAD`` (default ``0``; set to ``1`` to enable uvicorn reload)
|
||||||
"""
|
"""
|
||||||
@@ -40,12 +41,13 @@ def main() -> None:
|
|||||||
|
|
||||||
if len(sys.argv) >= 2 and sys.argv[1] == "serve":
|
if len(sys.argv) >= 2 and sys.argv[1] == "serve":
|
||||||
port = os.environ.get("CYCLONE_PORT", "8000")
|
port = os.environ.get("CYCLONE_PORT", "8000")
|
||||||
# Local-only by default — see CLAUDE.md. The Docker image
|
# Always bind to 0.0.0.0 so the API is reachable from the
|
||||||
# overrides to 0.0.0.0 via compose env so the frontend
|
# frontend container on the compose bridge network AND from
|
||||||
# container on the compose bridge network can reach the
|
# the host (Vite dev proxy) AND from the LAN. Network isolation
|
||||||
# backend. Network isolation is provided by the bridge
|
# is provided by the host firewall / compose port publishing,
|
||||||
# network itself (only cyclone-frontend joins).
|
# not by binding to loopback. Override with CYCLONE_HOST if you
|
||||||
host = os.environ.get("CYCLONE_HOST", "127.0.0.1")
|
# have a reason to restrict.
|
||||||
|
host = os.environ.get("CYCLONE_HOST", "0.0.0.0")
|
||||||
reload = os.environ.get("CYCLONE_RELOAD", "0") == "1"
|
reload = os.environ.get("CYCLONE_RELOAD", "0") == "1"
|
||||||
sys.argv = [
|
sys.argv = [
|
||||||
sys.argv[0],
|
sys.argv[0],
|
||||||
|
|||||||
+228
-54
@@ -18,6 +18,7 @@ Additional origins (LAN IPs, staging hosts) can be appended via the
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
import csv
|
import csv
|
||||||
import io
|
import io
|
||||||
import json
|
import json
|
||||||
@@ -36,6 +37,7 @@ from pydantic import ValidationError
|
|||||||
|
|
||||||
from cyclone import __version__, db
|
from cyclone import __version__, db
|
||||||
from cyclone.auth.deps import matrix_gate
|
from cyclone.auth.deps import matrix_gate
|
||||||
|
from cyclone.clearhouse import InboundFile
|
||||||
from cyclone.db import Batch, Claim, ClaimState, Remittance
|
from cyclone.db import Batch, Claim, ClaimState, Remittance
|
||||||
from sqlalchemy import desc, or_
|
from sqlalchemy import desc, or_
|
||||||
from sqlalchemy.exc import IntegrityError
|
from sqlalchemy.exc import IntegrityError
|
||||||
@@ -86,6 +88,7 @@ from cyclone.store import (
|
|||||||
AlreadyMatchedError,
|
AlreadyMatchedError,
|
||||||
BatchRecord,
|
BatchRecord,
|
||||||
InvalidStateError,
|
InvalidStateError,
|
||||||
|
dashboard_kpis,
|
||||||
store,
|
store,
|
||||||
utcnow,
|
utcnow,
|
||||||
)
|
)
|
||||||
@@ -144,6 +147,16 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
|||||||
except Exception as exc: # noqa: BLE001
|
except Exception as exc: # noqa: BLE001
|
||||||
log.exception("SP9 seed failed: %s", exc)
|
log.exception("SP9 seed failed: %s", exc)
|
||||||
|
|
||||||
|
# SP27 Task 11: startup audit for the Claim.matched_remittance_id ↔
|
||||||
|
# Remittance.claim_id denormalized FK pair. Non-blocking — mismatches
|
||||||
|
# are logged at WARNING with up to 5 examples of each case so the
|
||||||
|
# operator can investigate without the system failing to boot.
|
||||||
|
try:
|
||||||
|
from cyclone.store import check_matched_pair_drift
|
||||||
|
check_matched_pair_drift()
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
log.exception("matched-pair drift check failed: %s", exc)
|
||||||
|
|
||||||
# SP16: configure the inbound MFT polling scheduler. The
|
# SP16: configure the inbound MFT polling scheduler. The
|
||||||
# dzinesco clearhouse singleton (seeded by SP9) carries the SFTP
|
# dzinesco clearhouse singleton (seeded by SP9) carries the SFTP
|
||||||
# block — multi-provider polling is out of scope for v1.
|
# block — multi-provider polling is out of scope for v1.
|
||||||
@@ -517,12 +530,13 @@ def _build_and_persist_ack(batch_id: str) -> dict | None:
|
|||||||
def _reconciliation_summary_for_batch(batch_id: str) -> dict:
|
def _reconciliation_summary_for_batch(batch_id: str) -> dict:
|
||||||
"""Return ``{matched, unmatched_claims, unmatched_remittances, skipped}`` for a batch.
|
"""Return ``{matched, unmatched_claims, unmatched_remittances, skipped}`` for a batch.
|
||||||
|
|
||||||
Reads from the DB after ``store.add()`` has already triggered T10
|
Reads from the DB after ``store.add()`` has already run reconciliation
|
||||||
reconciliation synchronously (via ``_run_reconcile``). Counts are
|
synchronously (SP27 Task 10: ``reconcile.run(s, batch_id)`` inside the
|
||||||
observed at this moment; a subsequent manual match/unmatch will not
|
ingest session, before commit). Counts are observed at this moment;
|
||||||
be reflected until the next request.
|
a subsequent manual match/unmatch will not be reflected until the
|
||||||
|
next request.
|
||||||
|
|
||||||
``skipped`` is reserved for future use — the T10 orchestrator tracks
|
``skipped`` is reserved for future use — the orchestrator tracks
|
||||||
skipped claims internally but does not surface a queryable count.
|
skipped claims internally but does not surface a queryable count.
|
||||||
"""
|
"""
|
||||||
from sqlalchemy import func, select
|
from sqlalchemy import func, select
|
||||||
@@ -663,40 +677,21 @@ async def parse_835_endpoint(
|
|||||||
# 999 ACK (Implementation Acknowledgment)
|
# 999 ACK (Implementation Acknowledgment)
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
# SP27 Task 6: ack ID helpers were deduped. Both api.py and scheduler.py
|
||||||
def _ack_count_summary(result) -> tuple[int, int, int, str]:
|
# used to define these locally. Canonical copies now live in
|
||||||
"""Aggregate (received, accepted, rejected, ack_code) from a ParseResult999.
|
# ``cyclone.handlers._ack_id`` (set up in Task 1). The aliased imports
|
||||||
|
# below keep the existing callsites (``_ack_count_summary(result)``,
|
||||||
The first functional group carries the canonical counts; falls back
|
# ``_ack_synthetic_source_batch_id(icn)``, ``_277ca_synthetic_source_batch_id(icn)``)
|
||||||
to summing per-set codes if no AK9 was found.
|
# working unchanged.
|
||||||
"""
|
from cyclone.handlers._ack_id import (
|
||||||
if result.functional_group_acks:
|
ack_count_summary as _ack_count_summary,
|
||||||
fg = result.functional_group_acks[0]
|
)
|
||||||
return (fg.received_count, fg.accepted_count, fg.rejected_count, fg.ack_code)
|
from cyclone.handlers._ack_id import (
|
||||||
sets = result.set_responses
|
ack_synthetic_source_batch_id as _ack_synthetic_source_batch_id,
|
||||||
received = len(sets)
|
)
|
||||||
accepted = sum(1 for s in sets if s.set_accept_reject.code == "A")
|
from cyclone.handlers._ack_id import (
|
||||||
rejected = received - accepted
|
two77ca_synthetic_source_batch_id as _277ca_synthetic_source_batch_id,
|
||||||
if rejected == 0:
|
)
|
||||||
code = "A"
|
|
||||||
elif accepted == 0:
|
|
||||||
code = "R"
|
|
||||||
else:
|
|
||||||
code = "P"
|
|
||||||
return (received, accepted, rejected, code)
|
|
||||||
|
|
||||||
|
|
||||||
def _ack_synthetic_source_batch_id(interchange_control_number: str) -> str:
|
|
||||||
"""Return a synthetic batches.id for a received 999 with no source batch.
|
|
||||||
|
|
||||||
The acks.source_batch_id FK requires a row in batches; for received
|
|
||||||
999s we synthesize an id of the form ``999-<ISA13>``. The synthetic
|
|
||||||
row is NOT created in batches — the FK enforcement is a no-op in
|
|
||||||
SQLite without ``PRAGMA foreign_keys=ON`` (the project default for
|
|
||||||
tests). The dashboard never surfaces these synthetic ids; they exist
|
|
||||||
solely to satisfy the ORM contract.
|
|
||||||
"""
|
|
||||||
return f"999-{(interchange_control_number or '').strip() or '000000001'}"
|
|
||||||
|
|
||||||
|
|
||||||
@app.post("/api/parse-999", dependencies=[Depends(matrix_gate)])
|
@app.post("/api/parse-999", dependencies=[Depends(matrix_gate)])
|
||||||
@@ -910,16 +905,9 @@ async def parse_ta1_endpoint(
|
|||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
def _277ca_synthetic_source_batch_id(interchange_control_number: str) -> str:
|
# (The _277ca_synthetic_source_batch_id helper was moved to
|
||||||
"""Return a synthetic ``batches.id`` for a received 277CA with no source batch.
|
# cyclone.handlers._ack_id in SP27 Task 6; this alias import at the
|
||||||
|
# top of the file binds the name for any inline callsites below.)
|
||||||
Mirrors :func:`_ack_synthetic_source_batch_id`. The 277CA row's
|
|
||||||
``source_batch_id`` FK requires a row in batches; for received
|
|
||||||
277CAs we synthesize an id of the form ``277CA-<ISA13>``. The row
|
|
||||||
is NOT created in batches — same FK-is-no-op convention as the 999
|
|
||||||
path.
|
|
||||||
"""
|
|
||||||
return f"277CA-{(interchange_control_number or '').strip() or '000000001'}"
|
|
||||||
|
|
||||||
|
|
||||||
@app.post("/api/parse-277ca", dependencies=[Depends(matrix_gate)])
|
@app.post("/api/parse-277ca", dependencies=[Depends(matrix_gate)])
|
||||||
@@ -1044,7 +1032,7 @@ async def parse_277ca_endpoint(
|
|||||||
|
|
||||||
@app.get("/api/277ca-acks", dependencies=[Depends(matrix_gate)])
|
@app.get("/api/277ca-acks", dependencies=[Depends(matrix_gate)])
|
||||||
def list_277ca_acks_endpoint(
|
def list_277ca_acks_endpoint(
|
||||||
limit: int = Query(100, ge=1, le=1000),
|
limit: int = Query(100, ge=1, le=5000),
|
||||||
) -> Any:
|
) -> Any:
|
||||||
"""Return the list of persisted 277CA ACKs, newest first."""
|
"""Return the list of persisted 277CA ACKs, newest first."""
|
||||||
rows = store.list_277ca_acks()
|
rows = store.list_277ca_acks()
|
||||||
@@ -1726,7 +1714,10 @@ def list_claims(
|
|||||||
items = list(store.iter_claims(
|
items = list(store.iter_claims(
|
||||||
sort=sort, order=order, limit=limit, offset=offset, **common,
|
sort=sort, order=order, limit=limit, offset=offset, **common,
|
||||||
))
|
))
|
||||||
total = len(list(store.iter_claims(**common)))
|
# SP27 Task 13b: count the full population, not a 100-row sample.
|
||||||
|
# `iter_claims` defaults to limit=100; counting its output silently
|
||||||
|
# capped the reported total at 100 even when the DB held 60k rows.
|
||||||
|
total = store.count_claims(**common)
|
||||||
returned = len(items)
|
returned = len(items)
|
||||||
has_more = total > offset + returned
|
has_more = total > offset + returned
|
||||||
if _wants_ndjson(request):
|
if _wants_ndjson(request):
|
||||||
@@ -2232,7 +2223,9 @@ def list_remittances(
|
|||||||
items = list(store.iter_remittances(
|
items = list(store.iter_remittances(
|
||||||
sort=sort, order=order, limit=limit, offset=offset, **common,
|
sort=sort, order=order, limit=limit, offset=offset, **common,
|
||||||
))
|
))
|
||||||
total = len(list(store.iter_remittances(**common)))
|
# SP27 Task 13b: count the full population, not a 100-row sample.
|
||||||
|
# See the matching note in list_claims — same silent-failure pattern.
|
||||||
|
total = store.count_remittances(**common)
|
||||||
returned = len(items)
|
returned = len(items)
|
||||||
has_more = total > offset + returned
|
has_more = total > offset + returned
|
||||||
if _wants_ndjson(request):
|
if _wants_ndjson(request):
|
||||||
@@ -2248,6 +2241,37 @@ def list_remittances(
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/remittances/summary", dependencies=[Depends(matrix_gate)])
|
||||||
|
def remittances_summary(
|
||||||
|
batch_id: str | None = Query(None),
|
||||||
|
payer: str | None = Query(None),
|
||||||
|
claim_id: str | None = Query(None),
|
||||||
|
date_from: str | None = Query(None),
|
||||||
|
date_to: str | None = Query(None),
|
||||||
|
) -> dict:
|
||||||
|
"""Server-aggregated KPI tiles for the Remittances page.
|
||||||
|
|
||||||
|
Returns ``{count, total_paid, total_adjustments}`` over the
|
||||||
|
full filtered remittance population — NOT a page-limited
|
||||||
|
sample. The Remittances page consumes this for its "Total paid"
|
||||||
|
and "Adjustments" tiles so they can't silently understate the
|
||||||
|
true DB population the way a page-local ``items.reduce(...)``
|
||||||
|
would. Mirrors the silent-incompleteness fix that
|
||||||
|
``/api/dashboard/kpis`` (commit ``59c3275``) and
|
||||||
|
``/api/remittances`` (commit ``d81b6ed``) made for their tiles.
|
||||||
|
|
||||||
|
Same filter parameters as ``/api/remittances``. Always returns
|
||||||
|
a populated dict (``{"count": 0, "total_paid": 0,
|
||||||
|
"total_adjustments": 0}`` when no rows match) so the frontend
|
||||||
|
can render the tiles directly without a loading-vs-empty
|
||||||
|
branch.
|
||||||
|
"""
|
||||||
|
return store.summarize_remittances(
|
||||||
|
batch_id=batch_id, payer=payer, claim_id=claim_id,
|
||||||
|
date_from=date_from, date_to=date_to,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@app.get("/api/remittances/stream", dependencies=[Depends(matrix_gate)])
|
@app.get("/api/remittances/stream", dependencies=[Depends(matrix_gate)])
|
||||||
async def remittances_stream(
|
async def remittances_stream(
|
||||||
request: Request,
|
request: Request,
|
||||||
@@ -2304,6 +2328,34 @@ def get_remittance(remittance_id: str) -> dict:
|
|||||||
return body
|
return body
|
||||||
|
|
||||||
|
|
||||||
|
@app.get("/api/dashboard/kpis", dependencies=[Depends(matrix_gate)])
|
||||||
|
def get_dashboard_kpis(
|
||||||
|
months: int = Query(6, ge=1, le=24),
|
||||||
|
top_n_providers: int = Query(4, ge=0, le=50),
|
||||||
|
top_n_denials: int = Query(5, ge=0, le=50),
|
||||||
|
) -> dict:
|
||||||
|
"""Server-aggregated Dashboard KPIs over the whole claim population.
|
||||||
|
|
||||||
|
Backs the Dashboard's "Claims / Billed / Received / Pending AR /
|
||||||
|
Denial rate" tiles + the monthly sparkline series + the
|
||||||
|
top-providers and top-denials lists.
|
||||||
|
|
||||||
|
Why this exists instead of ``GET /api/claims?limit=N``:
|
||||||
|
The Dashboard's KPIs are aggregates over *every* claim — billed,
|
||||||
|
received, denial rate, pending count, monthly billed/received. With
|
||||||
|
60k+ claims in production, paginating ``/api/claims`` and reducing
|
||||||
|
client-side silently produces wrong numbers (denial rate sampled,
|
||||||
|
billed summed from the first 100 rows). This endpoint does the
|
||||||
|
aggregation server-side in a single read so the Dashboard's numbers
|
||||||
|
are always correct regardless of dataset size.
|
||||||
|
"""
|
||||||
|
return dashboard_kpis(
|
||||||
|
months=months,
|
||||||
|
top_n_providers=top_n_providers,
|
||||||
|
top_n_denials=top_n_denials,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@app.get("/api/providers", dependencies=[Depends(matrix_gate)])
|
@app.get("/api/providers", dependencies=[Depends(matrix_gate)])
|
||||||
def list_providers(
|
def list_providers(
|
||||||
request: Request,
|
request: Request,
|
||||||
@@ -2339,7 +2391,7 @@ def list_activity(
|
|||||||
request: Request,
|
request: Request,
|
||||||
kind: str | None = Query(None),
|
kind: str | None = Query(None),
|
||||||
since: str | None = Query(None),
|
since: str | None = Query(None),
|
||||||
limit: int = Query(200, ge=1, le=500),
|
limit: int = Query(200, ge=1, le=5000),
|
||||||
) -> Any:
|
) -> Any:
|
||||||
events = store.recent_activity(limit=limit)
|
events = store.recent_activity(limit=limit)
|
||||||
if kind is not None:
|
if kind is not None:
|
||||||
@@ -2366,7 +2418,7 @@ async def activity_stream(
|
|||||||
request: Request,
|
request: Request,
|
||||||
kind: str | None = Query(None),
|
kind: str | None = Query(None),
|
||||||
since: str | None = Query(None),
|
since: str | None = Query(None),
|
||||||
limit: int = Query(50, ge=1, le=500),
|
limit: int = Query(50, ge=1, le=5000),
|
||||||
) -> StreamingResponse:
|
) -> StreamingResponse:
|
||||||
"""Stream Activity events as NDJSON: snapshot first, then live events.
|
"""Stream Activity events as NDJSON: snapshot first, then live events.
|
||||||
|
|
||||||
@@ -3419,6 +3471,128 @@ async def scheduler_tick() -> Any:
|
|||||||
return {"ok": True, "tick": result.as_dict()}
|
return {"ok": True, "tick": result.as_dict()}
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/api/admin/scheduler/pull-inbound", dependencies=[Depends(matrix_gate)])
|
||||||
|
async def scheduler_pull_inbound(
|
||||||
|
date: str = Query(
|
||||||
|
..., pattern=r"^\d{8}$",
|
||||||
|
description="Date filter as YYYYMMDD; only filenames whose 8-digit "
|
||||||
|
"timestamp (the 9th positional group in the inbound "
|
||||||
|
"filename) matches are downloaded and processed.",
|
||||||
|
),
|
||||||
|
file_types: str | None = Query(
|
||||||
|
default=None,
|
||||||
|
description="Optional comma-separated whitelist of file_types "
|
||||||
|
"(999, TA1, 277, 277CA, 835). Defaults to 999+TA1.",
|
||||||
|
),
|
||||||
|
limit: int = Query(default=2000, ge=1, le=10000),
|
||||||
|
) -> Any:
|
||||||
|
"""Targeted pull: list, filter to a date, download, and process.
|
||||||
|
|
||||||
|
Bypasses the alphabetical full-listing pass. Workflow:
|
||||||
|
1. ``SftpClient.list_inbound_names()`` — sub-second metadata-only
|
||||||
|
listing of the inbound MFT dir (skips ``*_warn.txt``).
|
||||||
|
2. Client-side filter: keep files whose 8-digit timestamp
|
||||||
|
substring equals ``date`` and whose ``file_type`` is in the
|
||||||
|
allowlist.
|
||||||
|
3. ``SftpClient.download_inbound(f)`` for each — fetches bytes
|
||||||
|
into the local cache.
|
||||||
|
4. ``Scheduler.process_inbound_files(files)`` — runs the same
|
||||||
|
per-file pipeline as a regular tick (already-processed files
|
||||||
|
are deduped via ``processed_inbound_files``).
|
||||||
|
|
||||||
|
Use this for the daily "process today's 999s" workflow without
|
||||||
|
paying the cost of downloading the full inbound set.
|
||||||
|
|
||||||
|
Returns ``{"ok": True, "summary": {...}}`` with
|
||||||
|
``listed / matched / downloaded / processed / skipped / errored``
|
||||||
|
counters and the date / file_type filters applied.
|
||||||
|
"""
|
||||||
|
import time
|
||||||
|
|
||||||
|
from cyclone.clearhouse import SftpClient
|
||||||
|
from cyclone.edi.filenames import (
|
||||||
|
ALLOWED_FILE_TYPES,
|
||||||
|
parse_inbound_filename,
|
||||||
|
)
|
||||||
|
from cyclone.providers import SftpBlock
|
||||||
|
|
||||||
|
sched = _scheduler_or_503()
|
||||||
|
block: SftpBlock = sched._sftp_block # noqa: SLF001 — internal but stable
|
||||||
|
client = SftpClient(block)
|
||||||
|
|
||||||
|
if file_types:
|
||||||
|
wanted = {t.strip().upper() for t in file_types.split(",") if t.strip()}
|
||||||
|
unknown = wanted - ALLOWED_FILE_TYPES
|
||||||
|
if unknown:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"file_types {sorted(unknown)!r} not in "
|
||||||
|
f"{sorted(ALLOWED_FILE_TYPES)}",
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
wanted = {"999", "TA1"} # daily default — what the operator needs
|
||||||
|
|
||||||
|
started = time.monotonic()
|
||||||
|
try:
|
||||||
|
# Single SFTP listdir — fast, no download.
|
||||||
|
all_files = await asyncio.to_thread(client.list_inbound_names)
|
||||||
|
except Exception as exc:
|
||||||
|
log.exception("SFTP list_inbound_names failed")
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=502,
|
||||||
|
detail=f"SFTP list failed: {type(exc).__name__}: {exc}",
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
listed = len(all_files)
|
||||||
|
matched: list[InboundFile] = []
|
||||||
|
for f in all_files:
|
||||||
|
if f.name.find(date) == -1:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
parsed = parse_inbound_filename(f.name)
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
if parsed.file_type not in wanted:
|
||||||
|
continue
|
||||||
|
matched.append(f)
|
||||||
|
if len(matched) >= limit:
|
||||||
|
break
|
||||||
|
|
||||||
|
# Download in parallel-ish via to_thread (SftpClient serializes per
|
||||||
|
# connection; the overhead is dominated by the SFTP round trip).
|
||||||
|
downloaded = 0
|
||||||
|
download_errors: list[str] = []
|
||||||
|
for f in matched:
|
||||||
|
try:
|
||||||
|
await asyncio.to_thread(client.download_inbound, f)
|
||||||
|
downloaded += 1
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
log.warning("Failed to download %s: %s", f.name, exc)
|
||||||
|
download_errors.append(f"{f.name}: {type(exc).__name__}: {exc}")
|
||||||
|
|
||||||
|
# Hand off to the scheduler pipeline (idempotent; dedupes via
|
||||||
|
# processed_inbound_files).
|
||||||
|
tick = await sched.process_inbound_files(matched)
|
||||||
|
duration = round(time.monotonic() - started, 3)
|
||||||
|
return {
|
||||||
|
"ok": True,
|
||||||
|
"summary": {
|
||||||
|
"date": date,
|
||||||
|
"file_types": sorted(wanted),
|
||||||
|
"limit": limit,
|
||||||
|
"listed": listed,
|
||||||
|
"matched": len(matched),
|
||||||
|
"downloaded": downloaded,
|
||||||
|
"download_errors": download_errors,
|
||||||
|
"processed": tick.files_processed,
|
||||||
|
"skipped": tick.files_skipped,
|
||||||
|
"errored": tick.files_errored,
|
||||||
|
"duration_s": duration,
|
||||||
|
},
|
||||||
|
"tick": tick.as_dict(),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
@app.get("/api/admin/scheduler/status", dependencies=[Depends(matrix_gate)])
|
@app.get("/api/admin/scheduler/status", dependencies=[Depends(matrix_gate)])
|
||||||
def scheduler_status() -> Any:
|
def scheduler_status() -> Any:
|
||||||
"""Return the scheduler's runtime snapshot (running, counters, last tick)."""
|
"""Return the scheduler's runtime snapshot (running, counters, last tick)."""
|
||||||
|
|||||||
@@ -32,8 +32,14 @@ def _ack_to_ui(row) -> dict:
|
|||||||
Field names match the rest of the Cyclone API (snake_case). The
|
Field names match the rest of the Cyclone API (snake_case). The
|
||||||
frontend ``useAcks`` hook re-shapes this to the camelCase ``Ack``
|
frontend ``useAcks`` hook re-shapes this to the camelCase ``Ack``
|
||||||
interface in ``src/types/index.ts``.
|
interface in ``src/types/index.ts``.
|
||||||
|
|
||||||
|
Adds ``patient_control_number`` pulled from ``raw_json`` so the
|
||||||
|
operator can correlate each 999 back to the original claim
|
||||||
|
batch. The full inbound filename is reachable via
|
||||||
|
``GET /api/acks/{ack_id}`` (raw_json carries the full parse
|
||||||
|
tree) and ``GET /api/admin/scheduler/processed-files``.
|
||||||
"""
|
"""
|
||||||
return {
|
body = {
|
||||||
"id": row.id,
|
"id": row.id,
|
||||||
"source_batch_id": row.source_batch_id,
|
"source_batch_id": row.source_batch_id,
|
||||||
"accepted_count": row.accepted_count,
|
"accepted_count": row.accepted_count,
|
||||||
@@ -45,20 +51,44 @@ def _ack_to_ui(row) -> dict:
|
|||||||
if row.parsed_at is not None
|
if row.parsed_at is not None
|
||||||
else ""
|
else ""
|
||||||
),
|
),
|
||||||
|
"patient_control_number": None,
|
||||||
}
|
}
|
||||||
|
raw = row.raw_json or {}
|
||||||
|
try:
|
||||||
|
set_responses = raw.get("set_responses") or []
|
||||||
|
if set_responses:
|
||||||
|
body["patient_control_number"] = set_responses[0].get("set_control_number")
|
||||||
|
except (AttributeError, TypeError):
|
||||||
|
pass
|
||||||
|
return body
|
||||||
|
|
||||||
|
|
||||||
@router.get("/api/acks")
|
@router.get("/api/acks")
|
||||||
def list_acks_endpoint(
|
def list_acks_endpoint(
|
||||||
request: Request,
|
request: Request,
|
||||||
limit: int = Query(100, ge=1, le=1000),
|
limit: int = Query(100, ge=1, le=5000),
|
||||||
|
offset: int = Query(0, ge=0),
|
||||||
) -> Any:
|
) -> Any:
|
||||||
"""Return the list of persisted 999 ACKs, newest first."""
|
"""Return the list of persisted 999 ACKs, newest first.
|
||||||
|
|
||||||
|
``limit`` caps the page size; ``offset`` lets the UI walk the
|
||||||
|
full set without holding it all in memory. ``aggregates`` is
|
||||||
|
summed over the *full* row set (not the page) so the KPI strip
|
||||||
|
on the Acks page reflects every persisted 999, not just the
|
||||||
|
visible 50. Without server-side aggregates the page would
|
||||||
|
silently under-report (silent-failure mode) once the row count
|
||||||
|
exceeds the page size.
|
||||||
|
"""
|
||||||
rows = store.list_acks()
|
rows = store.list_acks()
|
||||||
items = [_ack_to_ui(r) for r in rows[:limit]]
|
items = [_ack_to_ui(r) for r in rows[offset : offset + limit]]
|
||||||
total = len(rows)
|
total = len(rows)
|
||||||
returned = len(items)
|
returned = len(items)
|
||||||
has_more = total > returned
|
has_more = offset + returned < total
|
||||||
|
aggregates = {
|
||||||
|
"accepted_count": sum(r.accepted_count or 0 for r in rows),
|
||||||
|
"rejected_count": sum(r.rejected_count or 0 for r in rows),
|
||||||
|
"received_count": sum(r.received_count or 0 for r in rows),
|
||||||
|
}
|
||||||
if wants_ndjson(request):
|
if wants_ndjson(request):
|
||||||
return StreamingResponse(
|
return StreamingResponse(
|
||||||
ndjson_stream_list(items, total, returned, has_more),
|
ndjson_stream_list(items, total, returned, has_more),
|
||||||
@@ -69,6 +99,7 @@ def list_acks_endpoint(
|
|||||||
"total": total,
|
"total": total,
|
||||||
"returned": returned,
|
"returned": returned,
|
||||||
"has_more": has_more,
|
"has_more": has_more,
|
||||||
|
"aggregates": aggregates,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -44,11 +44,38 @@ PERMISSIONS: dict[tuple[str, str], set[Role]] = {
|
|||||||
("GET", "/api/inbox/export.csv"): ALL_ROLES,
|
("GET", "/api/inbox/export.csv"): ALL_ROLES,
|
||||||
("GET", "/api/reconcile"): ALL_ROLES,
|
("GET", "/api/reconcile"): ALL_ROLES,
|
||||||
("GET", "/api/reconciliation"): ALL_ROLES,
|
("GET", "/api/reconciliation"): ALL_ROLES,
|
||||||
|
("GET", "/api/acks"): ALL_ROLES,
|
||||||
|
("GET", "/api/ta1-acks"): ALL_ROLES,
|
||||||
|
("GET", "/api/277ca-acks"): ALL_ROLES,
|
||||||
|
("GET", "/api/batch-diff"): ALL_ROLES,
|
||||||
|
("GET", "/api/config"): ALL_ROLES,
|
||||||
|
("GET", "/api/payers"): ALL_ROLES,
|
||||||
("GET", "/api/audit-log"): ADMIN_ONLY,
|
("GET", "/api/audit-log"): ADMIN_ONLY,
|
||||||
|
|
||||||
|
# Clearhouse (SFTP creds + dzinesco identity) — admin only.
|
||||||
|
("GET", "/api/clearhouse"): ADMIN_ONLY,
|
||||||
|
("PATCH", "/api/clearhouse"): ADMIN_ONLY,
|
||||||
|
("POST", "/api/clearhouse/submit"): ADMIN_ONLY,
|
||||||
|
|
||||||
|
# Admin ops (audit log, backup, scheduler, db rotate, reload-config).
|
||||||
|
("GET", "/api/admin/audit-log"): ADMIN_ONLY,
|
||||||
|
("GET", "/api/admin/audit-log/verify"): ADMIN_ONLY,
|
||||||
|
("GET", "/api/admin/backup"): ADMIN_ONLY,
|
||||||
|
("POST", "/api/admin/backup"): ADMIN_ONLY,
|
||||||
|
("GET", "/api/admin/backup/scheduler"): ADMIN_ONLY,
|
||||||
|
("POST", "/api/admin/backup/scheduler"): ADMIN_ONLY,
|
||||||
|
("GET", "/api/admin/scheduler"): ADMIN_ONLY,
|
||||||
|
("POST", "/api/admin/scheduler"): ADMIN_ONLY,
|
||||||
|
("POST", "/api/admin/db/rotate-key"): ADMIN_ONLY,
|
||||||
|
("POST", "/api/admin/reload-config"): ADMIN_ONLY,
|
||||||
|
("GET", "/api/admin/validate-provider"): ADMIN_ONLY,
|
||||||
|
|
||||||
# Write endpoints (admin + user, no viewer).
|
# Write endpoints (admin + user, no viewer).
|
||||||
("POST", "/api/parse-837"): WRITE_ROLES,
|
("POST", "/api/parse-837"): WRITE_ROLES,
|
||||||
("POST", "/api/parse-835"): WRITE_ROLES,
|
("POST", "/api/parse-835"): WRITE_ROLES,
|
||||||
|
("POST", "/api/parse-999"): WRITE_ROLES,
|
||||||
|
("POST", "/api/parse-ta1"): WRITE_ROLES,
|
||||||
|
("POST", "/api/parse-277ca"): WRITE_ROLES,
|
||||||
("POST", "/api/inbox"): WRITE_ROLES,
|
("POST", "/api/inbox"): WRITE_ROLES,
|
||||||
("POST", "/api/inbox/candidates"): WRITE_ROLES,
|
("POST", "/api/inbox/candidates"): WRITE_ROLES,
|
||||||
("POST", "/api/inbox/rejected"): WRITE_ROLES,
|
("POST", "/api/inbox/rejected"): WRITE_ROLES,
|
||||||
@@ -57,6 +84,8 @@ PERMISSIONS: dict[tuple[str, str], set[Role]] = {
|
|||||||
("POST", "/api/reconciliation"): WRITE_ROLES,
|
("POST", "/api/reconciliation"): WRITE_ROLES,
|
||||||
("POST", "/api/resubmit"): WRITE_ROLES,
|
("POST", "/api/resubmit"): WRITE_ROLES,
|
||||||
("POST", "/api/acks"): WRITE_ROLES,
|
("POST", "/api/acks"): WRITE_ROLES,
|
||||||
|
("POST", "/api/batches"): WRITE_ROLES, # /export-837 regenerates X12 from DB rows
|
||||||
|
("POST", "/api/eligibility"): WRITE_ROLES,
|
||||||
|
|
||||||
# CSV export — read-only.
|
# CSV export — read-only.
|
||||||
("GET", "/api/export.csv"): ALL_ROLES,
|
("GET", "/api/export.csv"): ALL_ROLES,
|
||||||
|
|||||||
@@ -24,6 +24,7 @@ stub secret and the paramiko auth will fail loudly at connect time.
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
import io
|
import io
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
@@ -40,6 +41,59 @@ from cyclone.providers import SftpBlock
|
|||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Per-op SFTP timeout (SP27 Task 8)
|
||||||
|
#
|
||||||
|
# paramiko is synchronous; without a timeout, a hung ``listdir_attr``
|
||||||
|
# freezes the worker thread indefinitely. The 06/25 silent hang was
|
||||||
|
# exactly this — the MFT server TCP-acked but stopped responding, the
|
||||||
|
# scheduler's ``asyncio.to_thread`` waited forever, and the operator
|
||||||
|
# had no signal that polling had stalled.
|
||||||
|
#
|
||||||
|
# The async wrappers below apply ``asyncio.wait_for`` to every SFTP
|
||||||
|
# call site so the event loop can give up after the configured bound.
|
||||||
|
# The bound is read fresh on every call (env-var-only, no module-level
|
||||||
|
# cache) so an operator who tunes the value at runtime picks it up on
|
||||||
|
# the next poll.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_DEFAULT_SFTP_OP_TIMEOUT_SECONDS = 30.0
|
||||||
|
|
||||||
|
|
||||||
|
def _op_timeout_seconds() -> float:
|
||||||
|
"""Per-op SFTP timeout from ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS``.
|
||||||
|
|
||||||
|
Default 30s. Picked to comfortably outlast Gainwell's p99
|
||||||
|
listdir_attr (~2s) while still surfacing real hangs inside one
|
||||||
|
scheduler tick. Operators who hit repeated timeouts should drop
|
||||||
|
this — but the right answer is to fix the MFT server, not to
|
||||||
|
paper over it here.
|
||||||
|
"""
|
||||||
|
raw = os.environ.get("CYCLONE_SFTP_OP_TIMEOUT_SECONDS")
|
||||||
|
if not raw:
|
||||||
|
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
|
||||||
|
try:
|
||||||
|
value = float(raw)
|
||||||
|
except ValueError:
|
||||||
|
log.warning(
|
||||||
|
"CYCLONE_SFTP_OP_TIMEOUT_SECONDS=%r is not a float; using default %.1fs",
|
||||||
|
raw, _DEFAULT_SFTP_OP_TIMEOUT_SECONDS,
|
||||||
|
)
|
||||||
|
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
|
||||||
|
if value <= 0:
|
||||||
|
# ``asyncio.wait_for(timeout=0)`` raises immediately, and
|
||||||
|
# ``wait_for(timeout<0)`` is undefined per the asyncio docs.
|
||||||
|
# A zero/negative setting would silently turn every SFTP call
|
||||||
|
# into an instant timeout (a wave of bogus "list_inbound:
|
||||||
|
# timeout" errors). Treat the value as bad and fall back.
|
||||||
|
log.warning(
|
||||||
|
"CYCLONE_SFTP_OP_TIMEOUT_SECONDS=%r must be positive; using default %.1fs",
|
||||||
|
raw, _DEFAULT_SFTP_OP_TIMEOUT_SECONDS,
|
||||||
|
)
|
||||||
|
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class InboundFile:
|
class InboundFile:
|
||||||
"""A single file observed in the inbound MFT path."""
|
"""A single file observed in the inbound MFT path."""
|
||||||
@@ -87,11 +141,70 @@ class SftpClient:
|
|||||||
dir and returns :class:`InboundFile` records pointing at the
|
dir and returns :class:`InboundFile` records pointing at the
|
||||||
cache copy. The remote file is *not* deleted — the operator
|
cache copy. The remote file is *not* deleted — the operator
|
||||||
archives inbound files in the MFT UI.
|
archives inbound files in the MFT UI.
|
||||||
|
|
||||||
|
Gainwell's MFT puts advisory ``*_warn.txt`` files in the same
|
||||||
|
inbound path. They're text-format side-channel notes (not X12
|
||||||
|
envelopes) and are skipped at list time. Use
|
||||||
|
:meth:`list_inbound_names` if you need the raw list without
|
||||||
|
the download.
|
||||||
"""
|
"""
|
||||||
if self._stub:
|
if self._stub:
|
||||||
return self._list_inbound_stub()
|
return self._list_inbound_stub()
|
||||||
return self._list_inbound_paramiko()
|
return self._list_inbound_paramiko()
|
||||||
|
|
||||||
|
def list_inbound_names(self) -> list[InboundFile]:
|
||||||
|
"""Lightweight listing: returns metadata only, no file download.
|
||||||
|
|
||||||
|
Use this when you want to filter the inbound set (e.g. by date)
|
||||||
|
before paying the download cost. Pair with
|
||||||
|
:meth:`download_inbound` to fetch a filtered subset on demand.
|
||||||
|
|
||||||
|
Real mode is implemented as a single SFTP ``listdir_attr`` call
|
||||||
|
— sub-second on Gainwell's MFT — versus the full
|
||||||
|
:meth:`list_inbound` which downloads every file. The
|
||||||
|
``*_warn.txt`` advisory files are filtered out the same way.
|
||||||
|
|
||||||
|
Stub mode returns the same as :meth:`list_inbound` (the stub
|
||||||
|
only knows about local files; no download cost).
|
||||||
|
"""
|
||||||
|
if self._stub:
|
||||||
|
return self._list_inbound_stub()
|
||||||
|
return self._list_inbound_names_paramiko()
|
||||||
|
|
||||||
|
def download_inbound(self, f: InboundFile) -> Path:
|
||||||
|
"""Download a single inbound file to its ``local_path``.
|
||||||
|
|
||||||
|
Idempotent: if ``f.local_path`` already exists and is
|
||||||
|
non-empty, the download is skipped. Callers should use the
|
||||||
|
``InboundFile`` returned by :meth:`list_inbound_names` and pass
|
||||||
|
it back here — ``local_path`` is the planned cache location
|
||||||
|
and matches the path the scheduler will read from.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
The on-disk path (same as ``f.local_path``).
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
FileNotFoundError: if the file is missing locally in stub
|
||||||
|
mode, or if the remote file disappears between list
|
||||||
|
and download in real mode.
|
||||||
|
"""
|
||||||
|
if f.local_path.exists() and f.local_path.stat().st_size > 0:
|
||||||
|
log.debug(
|
||||||
|
"SFTP: %s already cached at %s, skipping download",
|
||||||
|
f.name, f.local_path,
|
||||||
|
)
|
||||||
|
return f.local_path
|
||||||
|
if self._stub:
|
||||||
|
# Stub mode: no remote — the file is supposed to already be
|
||||||
|
# at f.local_path (operator-dropped). If it isn't there, the
|
||||||
|
# operator hasn't seeded the stub; raise loudly.
|
||||||
|
if not f.local_path.is_file():
|
||||||
|
raise FileNotFoundError(
|
||||||
|
f"inbound stub file not found: {f.local_path}"
|
||||||
|
)
|
||||||
|
return f.local_path
|
||||||
|
return self._download_inbound_paramiko(f)
|
||||||
|
|
||||||
def read_file(self, remote_path: str) -> bytes:
|
def read_file(self, remote_path: str) -> bytes:
|
||||||
"""Read bytes from a remote path.
|
"""Read bytes from a remote path.
|
||||||
|
|
||||||
@@ -119,6 +232,37 @@ class SftpClient:
|
|||||||
return secrets.STUB_SECRET
|
return secrets.STUB_SECRET
|
||||||
return value
|
return value
|
||||||
|
|
||||||
|
# ---- Async surface (SP27 Task 8) -----------------------------------
|
||||||
|
#
|
||||||
|
# Every sync SFTP call has an async wrapper that runs the paramiko
|
||||||
|
# call on a worker thread and applies an ``asyncio.wait_for(...
|
||||||
|
# timeout=N)`` around it. The wait_for cancels the awaiter but
|
||||||
|
# leaves the worker thread running until paramiko returns on its
|
||||||
|
# own (paramiko is not asyncio-aware, so we can't cancel the
|
||||||
|
# underlying socket cleanly). The scheduler should treat
|
||||||
|
# ``asyncio.TimeoutError`` from these wrappers as a transient
|
||||||
|
# SFTP error and surface it in ``Scheduler.status()`` (Task 9).
|
||||||
|
#
|
||||||
|
# The timeout is read on every call (see ``_op_timeout_seconds``)
|
||||||
|
# so an operator who tunes ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` at
|
||||||
|
# runtime sees the new value on the next tick.
|
||||||
|
|
||||||
|
async def async_list_inbound(self) -> list["InboundFile"]:
|
||||||
|
"""Async-wrapped :meth:`list_inbound` with a per-op timeout.
|
||||||
|
|
||||||
|
The 06/25 silent hang was a hung ``listdir_attr`` that froze
|
||||||
|
the worker thread indefinitely. This wrapper applies
|
||||||
|
``asyncio.wait_for(...)`` so the event loop can give up after
|
||||||
|
``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` (default 30s) and the
|
||||||
|
scheduler tick can surface the timeout in
|
||||||
|
``result.errors`` (and, once Task 9 lands, in
|
||||||
|
``Scheduler.status()``).
|
||||||
|
"""
|
||||||
|
return await asyncio.wait_for(
|
||||||
|
asyncio.to_thread(self.list_inbound),
|
||||||
|
timeout=_op_timeout_seconds(),
|
||||||
|
)
|
||||||
|
|
||||||
# ---- Stub implementations (SP9) -------------------------------------
|
# ---- Stub implementations (SP9) -------------------------------------
|
||||||
|
|
||||||
def _write_bytes_stub(self, remote_path: str, content: bytes) -> Path:
|
def _write_bytes_stub(self, remote_path: str, content: bytes) -> Path:
|
||||||
@@ -284,6 +428,12 @@ class SftpClient:
|
|||||||
if attr.st_mode and (attr.st_mode & 0o170000) == 0o040000:
|
if attr.st_mode and (attr.st_mode & 0o170000) == 0o040000:
|
||||||
# Directory entry — skip.
|
# Directory entry — skip.
|
||||||
continue
|
continue
|
||||||
|
if attr.filename.endswith("_warn.txt"):
|
||||||
|
# Gainwell's MFT drops text-format advisory notes in
|
||||||
|
# the same inbound path. They're side-channel noise,
|
||||||
|
# not X12 envelopes — skip at list time so we don't
|
||||||
|
# download ~600 advisory files per poll.
|
||||||
|
continue
|
||||||
remote = f"{inbound_dir.rstrip('/')}/{attr.filename}"
|
remote = f"{inbound_dir.rstrip('/')}/{attr.filename}"
|
||||||
cache_path = cache_dir / attr.filename
|
cache_path = cache_dir / attr.filename
|
||||||
# Download into cache. We use ``prefetch`` to keep memory
|
# Download into cache. We use ``prefetch`` to keep memory
|
||||||
@@ -299,6 +449,56 @@ class SftpClient:
|
|||||||
))
|
))
|
||||||
return files
|
return files
|
||||||
|
|
||||||
|
def _list_inbound_names_paramiko(self) -> list[InboundFile]:
|
||||||
|
"""List inbound names via paramiko; do NOT download (lightweight).
|
||||||
|
|
||||||
|
Same ``listdir_attr`` iteration as
|
||||||
|
:meth:`_list_inbound_paramiko`, but the returned
|
||||||
|
:class:`InboundFile` records have ``local_path`` set to the
|
||||||
|
planned cache location without actually fetching the file.
|
||||||
|
Pair with :meth:`_download_inbound_paramiko` to fetch on
|
||||||
|
demand. Skips ``*_warn.txt`` advisory files the same way.
|
||||||
|
"""
|
||||||
|
with self._connect() as (ssh, sftp):
|
||||||
|
inbound_dir = self._block.paths.get("inbound", "/")
|
||||||
|
staging = Path(self._block.staging_dir).resolve()
|
||||||
|
inbound_rel = inbound_dir.lstrip("/")
|
||||||
|
cache_dir = staging / inbound_rel
|
||||||
|
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
files: list[InboundFile] = []
|
||||||
|
try:
|
||||||
|
attrs = sftp.listdir_attr(inbound_dir)
|
||||||
|
except IOError as exc:
|
||||||
|
log.warning("SFTP: cannot list %s: %s", inbound_dir, exc)
|
||||||
|
return []
|
||||||
|
|
||||||
|
for attr in sorted(attrs, key=lambda a: a.filename):
|
||||||
|
if attr.st_mode and (attr.st_mode & 0o170000) == 0o040000:
|
||||||
|
# Directory entry — skip.
|
||||||
|
continue
|
||||||
|
if attr.filename.endswith("_warn.txt"):
|
||||||
|
continue
|
||||||
|
cache_path = cache_dir / attr.filename
|
||||||
|
files.append(InboundFile(
|
||||||
|
name=attr.filename,
|
||||||
|
size=attr.st_size or 0,
|
||||||
|
modified_at=datetime.fromtimestamp(attr.st_mtime or 0),
|
||||||
|
local_path=cache_path,
|
||||||
|
))
|
||||||
|
return files
|
||||||
|
|
||||||
|
def _download_inbound_paramiko(self, f: InboundFile) -> Path:
|
||||||
|
"""Download a single ``f`` to ``f.local_path`` (idempotent on size>0)."""
|
||||||
|
with self._connect() as (ssh, sftp):
|
||||||
|
inbound_dir = self._block.paths.get("inbound", "/")
|
||||||
|
remote = f"{inbound_dir.rstrip('/')}/{f.name}"
|
||||||
|
f.local_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with sftp.open(remote, "rb") as src, open(f.local_path, "wb") as dst:
|
||||||
|
shutil.copyfileobj(src, dst, length=64 * 1024)
|
||||||
|
log.info("SFTP: downloaded %d bytes for %s", f.local_path.stat().st_size, f.name)
|
||||||
|
return f.local_path
|
||||||
|
|
||||||
def _read_file_paramiko(self, remote_path: str) -> bytes:
|
def _read_file_paramiko(self, remote_path: str) -> bytes:
|
||||||
with self._connect() as (ssh, sftp):
|
with self._connect() as (ssh, sftp):
|
||||||
buf = io.BytesIO()
|
buf = io.BytesIO()
|
||||||
|
|||||||
@@ -569,3 +569,167 @@ def backup_status() -> None:
|
|||||||
snap = svc.status()
|
snap = svc.status()
|
||||||
import json
|
import json
|
||||||
click.echo(json.dumps(snap, indent=2, default=str))
|
click.echo(json.dumps(snap, indent=2, default=str))
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# SP-N fix: `cyclone pull-inbound` — targeted inbound pull + process
|
||||||
|
#
|
||||||
|
# Mirrors POST /api/admin/scheduler/pull-inbound. For operators who
|
||||||
|
# want to drive the daily "process today's 999s" workflow from a cron
|
||||||
|
# job or shell script without going through the HTTP API.
|
||||||
|
#
|
||||||
|
# Exit codes:
|
||||||
|
# 0 — processed ≥1 file (or processed 0 with no matches: not an
|
||||||
|
# error, just nothing to do)
|
||||||
|
# 1 — unexpected exception
|
||||||
|
# 2 — SFTP / config error
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@main.command("pull-inbound")
|
||||||
|
@click.option(
|
||||||
|
"--date", "date_str",
|
||||||
|
required=True,
|
||||||
|
help="Date filter YYYYMMDD — only files whose 8-digit timestamp "
|
||||||
|
"substring matches are downloaded and processed.",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--block", "sftp_block_name",
|
||||||
|
default="dzinesco",
|
||||||
|
show_default=True,
|
||||||
|
help="Name of the SftpBlock in config/payers.yaml to use.",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--file-types", "file_types_csv",
|
||||||
|
default=None,
|
||||||
|
help="Comma-separated whitelist (default: 999,TA1).",
|
||||||
|
)
|
||||||
|
@click.option(
|
||||||
|
"--limit", default=2000, show_default=True, type=int,
|
||||||
|
)
|
||||||
|
def pull_inbound(
|
||||||
|
date_str: str,
|
||||||
|
sftp_block_name: str,
|
||||||
|
file_types_csv: str | None,
|
||||||
|
limit: int,
|
||||||
|
) -> None:
|
||||||
|
"""List, date-filter, download, and process inbound MFT files.
|
||||||
|
|
||||||
|
Bypasses the alphabetical full-listing pass so a daily pull of
|
||||||
|
~400 files takes seconds, not hours. Files already in the local
|
||||||
|
cache are skipped (idempotent).
|
||||||
|
"""
|
||||||
|
import asyncio as _asyncio
|
||||||
|
from cyclone import db as db_mod
|
||||||
|
from cyclone import scheduler as scheduler_mod
|
||||||
|
from cyclone.edi.filenames import ALLOWED_FILE_TYPES
|
||||||
|
from cyclone.clearhouse import SftpClient
|
||||||
|
from cyclone.providers import SftpBlock
|
||||||
|
|
||||||
|
# Validate the date filter.
|
||||||
|
if not (len(date_str) == 8 and date_str.isdigit()):
|
||||||
|
click.echo(f"--date must be YYYYMMDD, got {date_str!r}", err=True)
|
||||||
|
sys.exit(2)
|
||||||
|
|
||||||
|
db_mod.init_db()
|
||||||
|
|
||||||
|
# Find the SftpBlock. The dzinesco clearhouse singleton is seeded
|
||||||
|
# by store.ensure_clearhouse_seeded() (SP9) and carries the
|
||||||
|
# production SFTP block; that's the only one we have at the
|
||||||
|
# moment. For multi-provider SFTP, a config-loader is the right
|
||||||
|
# thing — out of scope for this CLI which is the daily-pull path.
|
||||||
|
from cyclone import store as store_mod
|
||||||
|
store_mod.store.ensure_clearhouse_seeded()
|
||||||
|
clearhouse = store_mod.store.get_clearhouse()
|
||||||
|
if clearhouse is None or clearhouse.name != sftp_block_name:
|
||||||
|
# Fall back: try the named block, but v1 only ships the
|
||||||
|
# dzinesco singleton.
|
||||||
|
if clearhouse is None:
|
||||||
|
click.echo(
|
||||||
|
f"No clearhouse seeded — cannot find SftpBlock "
|
||||||
|
f"{sftp_block_name!r}.",
|
||||||
|
err=True,
|
||||||
|
)
|
||||||
|
sys.exit(2)
|
||||||
|
click.echo(
|
||||||
|
f"SftpBlock {sftp_block_name!r} not seeded; only "
|
||||||
|
f"{clearhouse.name!r} is available.",
|
||||||
|
err=True,
|
||||||
|
)
|
||||||
|
sys.exit(2)
|
||||||
|
block: SftpBlock = clearhouse.sftp_block
|
||||||
|
|
||||||
|
if file_types_csv:
|
||||||
|
wanted = {t.strip().upper() for t in file_types_csv.split(",") if t.strip()}
|
||||||
|
unknown = wanted - ALLOWED_FILE_TYPES
|
||||||
|
if unknown:
|
||||||
|
click.echo(
|
||||||
|
f"file_types {sorted(unknown)!r} not in {sorted(ALLOWED_FILE_TYPES)}",
|
||||||
|
err=True,
|
||||||
|
)
|
||||||
|
sys.exit(2)
|
||||||
|
else:
|
||||||
|
wanted = {"999", "TA1"}
|
||||||
|
|
||||||
|
# Wire up the scheduler singleton (re-use the same pipeline the
|
||||||
|
# HTTP endpoint uses, including the dedup via processed_inbound_files).
|
||||||
|
scheduler_mod.configure_scheduler(
|
||||||
|
block, sftp_block_name=sftp_block_name, force=True,
|
||||||
|
)
|
||||||
|
sched = scheduler_mod.get_scheduler()
|
||||||
|
client = SftpClient(block)
|
||||||
|
|
||||||
|
async def _run() -> dict:
|
||||||
|
from cyclone.edi.filenames import parse_inbound_filename
|
||||||
|
|
||||||
|
all_files = await _asyncio.to_thread(client.list_inbound_names)
|
||||||
|
matched: list = []
|
||||||
|
for f in all_files:
|
||||||
|
if f.name.find(date_str) == -1:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
parsed = parse_inbound_filename(f.name)
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
if parsed.file_type not in wanted:
|
||||||
|
continue
|
||||||
|
matched.append(f)
|
||||||
|
if len(matched) >= limit:
|
||||||
|
break
|
||||||
|
|
||||||
|
download_errors: list[str] = []
|
||||||
|
downloaded = 0
|
||||||
|
for f in matched:
|
||||||
|
try:
|
||||||
|
await _asyncio.to_thread(client.download_inbound, f)
|
||||||
|
downloaded += 1
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
download_errors.append(f"{f.name}: {type(exc).__name__}: {exc}")
|
||||||
|
|
||||||
|
tick = await sched.process_inbound_files(matched)
|
||||||
|
return {
|
||||||
|
"listed": len(all_files),
|
||||||
|
"matched": len(matched),
|
||||||
|
"downloaded": downloaded,
|
||||||
|
"download_errors": download_errors,
|
||||||
|
"processed": tick.files_processed,
|
||||||
|
"skipped": tick.files_skipped,
|
||||||
|
"errored": tick.files_errored,
|
||||||
|
}
|
||||||
|
|
||||||
|
try:
|
||||||
|
summary = _asyncio.run(_run())
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
click.echo(f"pull-inbound failed: {type(exc).__name__}: {exc}", err=True)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
click.echo(
|
||||||
|
f"date={date_str} file_types={sorted(wanted)} block={sftp_block_name} "
|
||||||
|
f"listed={summary['listed']} matched={summary['matched']} "
|
||||||
|
f"downloaded={summary['downloaded']} processed={summary['processed']} "
|
||||||
|
f"skipped={summary['skipped']} errored={summary['errored']}"
|
||||||
|
)
|
||||||
|
if summary["download_errors"]:
|
||||||
|
click.echo("download errors:", err=True)
|
||||||
|
for e in summary["download_errors"]:
|
||||||
|
click.echo(f" {e}", err=True)
|
||||||
|
|||||||
@@ -317,6 +317,13 @@ class Claim(Base):
|
|||||||
Index("ix_claims_state", "state"),
|
Index("ix_claims_state", "state"),
|
||||||
Index("ix_claims_patient_control_number", "patient_control_number"),
|
Index("ix_claims_patient_control_number", "patient_control_number"),
|
||||||
Index("ix_claims_service_date_from", "service_date_from"),
|
Index("ix_claims_service_date_from", "service_date_from"),
|
||||||
|
# SP27 Task 11: matched-pair drift check (run at startup)
|
||||||
|
# scans ``WHERE matched_remittance_id IS NOT NULL``. Without
|
||||||
|
# this index it's a full claim scan. The reverse side
|
||||||
|
# (``ix_remittances_claim_id``) is added in 0007. Pure index
|
||||||
|
# (non-unique) — a claim without a match is fine, reversals
|
||||||
|
# leave the previous claim/claim match intact.
|
||||||
|
Index("ix_claims_matched_remittance_id", "matched_remittance_id"),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -7,9 +7,12 @@ Outbound (we send):
|
|||||||
tp{tpid}-{transaction_type}-{yyyymmddhhmmssSSS_MT}-1of1.{ext}
|
tp{tpid}-{transaction_type}-{yyyymmddhhmmssSSS_MT}-1of1.{ext}
|
||||||
Example: tp11525703-837P-20260620132243505-1of1.x12
|
Example: tp11525703-837P-20260620132243505-1of1.x12
|
||||||
|
|
||||||
Inbound (HPE sends to our ToHPE):
|
Inbound (HPE sends to our FromHPE):
|
||||||
TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
|
[Tt][Pp]{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
|
||||||
Example: TP11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
Example: tp11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
||||||
|
(legacy / prodfiles may use uppercase TP; the regex is case-insensitive
|
||||||
|
on the prefix to accept both — Gainwell's filer has used both
|
||||||
|
casings over time.)
|
||||||
|
|
||||||
Both use Mountain Time (MT) timestamps with 17-digit millisecond precision
|
Both use Mountain Time (MT) timestamps with 17-digit millisecond precision
|
||||||
(yyyymmddhhmmssSSS = 4+2+2+2+2+2+3 = 17 digits). Sequence is always "1of1"
|
(yyyymmddhhmmssSSS = 4+2+2+2+2+2+3 = 17 digits). Sequence is always "1of1"
|
||||||
@@ -39,19 +42,48 @@ OUTBOUND_RE = re.compile(
|
|||||||
r"^tp(?P<tpid>\d+)-(?P<tx>[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>[A-Za-z0-9]+)$"
|
r"^tp(?P<tpid>\d+)-(?P<tx>[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>[A-Za-z0-9]+)$"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Inbound: TP11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
# Inbound: [Tt][Pp]11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
||||||
|
# - prefix: literal "TP" or "tp" (case-insensitive — Gainwell's
|
||||||
|
# production filer has used both)
|
||||||
# - tpid: 1+ digits (inside TP<...>)
|
# - tpid: 1+ digits (inside TP<...>)
|
||||||
# - orig_tx: 1+ alnum
|
# - orig_tx: 1+ uppercase alnum
|
||||||
# - track: M + 1+ alnum (e.g. M019048402) — the M is part of the
|
# - track: M + 1+ uppercase alnum (e.g. M019048402) — the M is
|
||||||
# tracking value, not a separator.
|
# part of the tracking value, not a separator.
|
||||||
# - ts: 17 digits
|
# - ts: 17 digits
|
||||||
# - seq: literal "1of1"
|
# - seq: literal "1of1"
|
||||||
# - ft: 1+ alnum (e.g. 999, TA1, 271, 277, 277CA, 820, 834, 835, ENCR)
|
# - ft: 1+ uppercase alnum (e.g. 999, TA1, 271, 277, 277CA,
|
||||||
|
# 820, 834, 835, ENCR)
|
||||||
|
#
|
||||||
|
# Case insensitivity is scoped to the ``TP`` prefix via ``(?i:TP)``
|
||||||
|
# — the rest of the pattern is case-sensitive so we still reject a
|
||||||
|
# stray ``837p`` or ``m019048402`` (they'd be invalid HCPF).
|
||||||
INBOUND_RE = re.compile(
|
INBOUND_RE = re.compile(
|
||||||
r"^TP(?P<tpid>\d+)-(?P<orig_tx>[A-Z0-9]+)_(?P<tracking>M[A-Z0-9]+)"
|
r"^(?i:TP)(?P<tpid>\d+)-(?P<orig_tx>[A-Z0-9]+)_(?P<tracking>M[A-Z0-9]+)"
|
||||||
r"-(?P<ts>\d{17})-1of1_(?P<file_type>[A-Z0-9]+)\.(?P<ext>x12)$"
|
r"-(?P<ts>\d{17})-1of1_(?P<file_type>[A-Z0-9]+)\.(?P<ext>x12)$"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Inbound suffix-less form (SP27 Task 7): tp11525703-835_M019110219-20260525001606050-1of1.x12
|
||||||
|
# Gainwell's production filer has shipped this shorter form in addition
|
||||||
|
# to the spec form above — the 6/15-6/19 835 batch arrived this way and
|
||||||
|
# was silently dropped by the strict INBOUND_RE. The loose form omits
|
||||||
|
# the `_{file_type}.x12` suffix; the token between `-` and `_M` doubles
|
||||||
|
# as both `orig_tx` and `file_type` (since there's no separate suffix
|
||||||
|
# to disambiguate). Allowed values still must be in ALLOWED_FILE_TYPES
|
||||||
|
# — the suffix is optional, the type-set is not.
|
||||||
|
# - prefix: literal "TP" or "tp" (case-insensitive — same as INBOUND_RE)
|
||||||
|
# - tpid: 1+ digits
|
||||||
|
# - file_type: 3-5 char alphanumeric (covers 999, TA1, 835, 277CA, ENCR;
|
||||||
|
# capped at 5 to avoid swallowing the next `_M` token)
|
||||||
|
# - tracking: M + 1+ uppercase alnum
|
||||||
|
# - ts: 17 digits
|
||||||
|
# - seq: literal "1of1"
|
||||||
|
# - ext: literal "x12" (relaxing this would also relax the strict
|
||||||
|
# form's contract; out of scope for SP27 Task 7)
|
||||||
|
INBOUND_RE_LOOSE = re.compile(
|
||||||
|
r"^(?i:TP)(?P<tpid>\d+)-(?P<file_type>[A-Z0-9]{3,5})"
|
||||||
|
r"_(?P<tracking>M[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>x12)$"
|
||||||
|
)
|
||||||
|
|
||||||
ALLOWED_FILE_TYPES = frozenset({
|
ALLOWED_FILE_TYPES = frozenset({
|
||||||
"999", "TA1", "270", "271", "276", "277", "277CA", "278",
|
"999", "TA1", "270", "271", "276", "277", "277CA", "278",
|
||||||
"820", "834", "835", "ENCR",
|
"820", "834", "835", "ENCR",
|
||||||
@@ -115,16 +147,49 @@ def build_outbound_filename(
|
|||||||
def parse_inbound_filename(name: str) -> InboundFilename:
|
def parse_inbound_filename(name: str) -> InboundFilename:
|
||||||
"""Parse an inbound HCPF filename.
|
"""Parse an inbound HCPF filename.
|
||||||
|
|
||||||
|
Accepts both forms (Gainwell ships both):
|
||||||
|
* Spec form with ``_{file_type}.x12`` suffix:
|
||||||
|
``tp11525703-837P_M019048402-20260520231513488-1of1_999.x12``
|
||||||
|
* Suffix-less form (SP27 Task 7): the token between ``-`` and
|
||||||
|
``_M`` doubles as both ``orig_tx`` and ``file_type``:
|
||||||
|
``tp11525703-835_M019110219-20260525001606050-1of1.x12``
|
||||||
|
|
||||||
|
The strict form is tried first (preserves historical behavior for
|
||||||
|
every existing caller); the loose form is the fallback. The
|
||||||
|
``.x12`` extension and ``ALLOWED_FILE_TYPES`` set are enforced in
|
||||||
|
both forms.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
name: Filename like "TP11525703-837P_M019048402-20260520231513488-1of1_999.x12"
|
name: Filename like ``tp11525703-837P_M019048402-20260520231513488-1of1_999.x12``
|
||||||
|
(case-insensitive on the ``TP`` prefix; both ``TP`` and
|
||||||
|
``tp`` are accepted.)
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
InboundFilename with tpid, orig_tx, tracking, ts, file_type, ext.
|
InboundFilename with tpid, orig_tx, tracking, ts, file_type, ext.
|
||||||
|
|
||||||
Raises:
|
Raises:
|
||||||
ValueError: If the filename doesn't match the HCPF inbound format.
|
ValueError: If the filename doesn't match either HCPF inbound
|
||||||
|
form, or if the derived file_type isn't in
|
||||||
|
ALLOWED_FILE_TYPES.
|
||||||
"""
|
"""
|
||||||
m = INBOUND_RE.match(name)
|
m = INBOUND_RE.match(name)
|
||||||
|
if m:
|
||||||
|
file_type = m.group("file_type")
|
||||||
|
if file_type not in ALLOWED_FILE_TYPES:
|
||||||
|
raise ValueError(
|
||||||
|
f"file_type {file_type!r} not in allowed HCPF set: {sorted(ALLOWED_FILE_TYPES)}"
|
||||||
|
)
|
||||||
|
return InboundFilename(
|
||||||
|
tpid=m.group("tpid"),
|
||||||
|
orig_tx=m.group("orig_tx"),
|
||||||
|
tracking=m.group("tracking"),
|
||||||
|
ts=m.group("ts"),
|
||||||
|
file_type=file_type,
|
||||||
|
ext=m.group("ext"),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Fall back to the suffix-less form.
|
||||||
|
m = INBOUND_RE_LOOSE.match(name)
|
||||||
if not m:
|
if not m:
|
||||||
raise ValueError(f"Not a valid HCPF inbound filename: {name!r}")
|
raise ValueError(f"Not a valid HCPF inbound filename: {name!r}")
|
||||||
file_type = m.group("file_type")
|
file_type = m.group("file_type")
|
||||||
@@ -134,7 +199,7 @@ def parse_inbound_filename(name: str) -> InboundFilename:
|
|||||||
)
|
)
|
||||||
return InboundFilename(
|
return InboundFilename(
|
||||||
tpid=m.group("tpid"),
|
tpid=m.group("tpid"),
|
||||||
orig_tx=m.group("orig_tx"),
|
orig_tx=m.group("file_type"), # preserve the historical shape
|
||||||
tracking=m.group("tracking"),
|
tracking=m.group("tracking"),
|
||||||
ts=m.group("ts"),
|
ts=m.group("ts"),
|
||||||
file_type=file_type,
|
file_type=file_type,
|
||||||
@@ -153,5 +218,12 @@ def is_outbound_filename(name: str) -> bool:
|
|||||||
|
|
||||||
|
|
||||||
def is_inbound_filename(name: str) -> bool:
|
def is_inbound_filename(name: str) -> bool:
|
||||||
"""True if the given string matches the HCPF inbound filename regex."""
|
"""True if the given string matches either HCPF inbound form.
|
||||||
return INBOUND_RE.match(name) is not None
|
|
||||||
|
Accepts both the spec form (with ``_{file_type}.x12`` suffix) and
|
||||||
|
the suffix-less form (SP27 Task 7). Cheap fast-path check used by
|
||||||
|
callers that want to pre-filter a directory listing before invoking
|
||||||
|
:func:`parse_inbound_filename`; mirrors the parser's own fallback
|
||||||
|
so the two never disagree on what counts as an HCPF inbound file.
|
||||||
|
"""
|
||||||
|
return INBOUND_RE.match(name) is not None or INBOUND_RE_LOOSE.match(name) is not None
|
||||||
|
|||||||
@@ -0,0 +1,113 @@
|
|||||||
|
"""File-type handlers for inbound MFT files (SP27).
|
||||||
|
|
||||||
|
Each handler is a pure function that parses + persists + dispatches
|
||||||
|
events for one file type. The scheduler and the FastAPI endpoints
|
||||||
|
both delegate here; the ``HANDLERS`` registry maps ``file_type`` →
|
||||||
|
handler function.
|
||||||
|
|
||||||
|
Public API:
|
||||||
|
HandleResult — dataclass returned by every handler
|
||||||
|
HANDLERS — ``{"999": handle_999, "835": handle_835, ...}``
|
||||||
|
handle_999, handle_ta1, handle_277ca, handle_835
|
||||||
|
— call signatures: ``handle(text: str, source_file: str) -> HandleResult``
|
||||||
|
|
||||||
|
The handlers own their own DB session lifecycle. They emit pubsub
|
||||||
|
events via the optional ``event_bus`` parameter (the FastAPI endpoint
|
||||||
|
injects ``app.state.event_bus``; the scheduler passes ``None``).
|
||||||
|
They never raise on per-segment problems; per-segment issues are
|
||||||
|
logged and folded into the result. Whole-document failures
|
||||||
|
(missing ISA, bad encoding) surface as ``CycloneParseError``, which
|
||||||
|
the caller catches and records as ``STATUS_ERROR``.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from ._ack_id import (
|
||||||
|
ack_count_summary,
|
||||||
|
ack_synthetic_source_batch_id,
|
||||||
|
two77ca_synthetic_source_batch_id,
|
||||||
|
)
|
||||||
|
from .handle_result import HandleResult
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Module-level HANDLERS dict populated lazily once handler modules
|
||||||
|
# ship. Keys are file_type strings, values are the ``handle``
|
||||||
|
# callable for that type.
|
||||||
|
HANDLERS: dict[str, object] = {}
|
||||||
|
|
||||||
|
|
||||||
|
# Candidate handlers, in registration order. Each tuple is
|
||||||
|
# (module-path, file_type). The 277/277CA mapping is added explicitly
|
||||||
|
# after registration when the 277CA handler is present.
|
||||||
|
_CANDIDATES: list[tuple[str, str]] = [
|
||||||
|
("cyclone.handlers.handle_999", "999"),
|
||||||
|
("cyclone.handlers.handle_ta1", "TA1"),
|
||||||
|
("cyclone.handlers.handle_277ca", "277CA"),
|
||||||
|
("cyclone.handlers.handle_835", "835"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def register_handlers() -> None:
|
||||||
|
"""Populate ``HANDLERS`` from the per-type handler modules.
|
||||||
|
|
||||||
|
Tolerates missing or broken handler modules so the package can be
|
||||||
|
imported incrementally as each handler ships (Tasks 2-5), and so
|
||||||
|
a partial-modification error in one handler module can't break
|
||||||
|
scheduler / API import. Safe to call multiple times.
|
||||||
|
"""
|
||||||
|
if HANDLERS:
|
||||||
|
return
|
||||||
|
for mod_path, file_type in _CANDIDATES:
|
||||||
|
try:
|
||||||
|
mod = importlib.import_module(mod_path)
|
||||||
|
fn = getattr(mod, "handle")
|
||||||
|
except Exception as exc: # noqa: BLE001 — best-effort registry
|
||||||
|
log.debug(
|
||||||
|
"handler %s unavailable: %s",
|
||||||
|
mod_path, exc,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
HANDLERS[file_type] = fn
|
||||||
|
# The 277 filename maps to the same 277CA handler.
|
||||||
|
if file_type == "277CA":
|
||||||
|
HANDLERS["277"] = fn
|
||||||
|
|
||||||
|
|
||||||
|
register_handlers()
|
||||||
|
|
||||||
|
|
||||||
|
# Re-export handler functions on this package so callers can use the
|
||||||
|
# flat import (``from cyclone.handlers import handle_999``) once each
|
||||||
|
# module ships. Set after registration so we know what's present.
|
||||||
|
def _reexport_handlers() -> None:
|
||||||
|
"""Re-export each handler's ``handle`` fn as ``handle_<type>``.
|
||||||
|
|
||||||
|
No-op for absent handlers. Re-run on each import so freshly-
|
||||||
|
installed handler modules (e.g. Tasks 2-5 commits) are visible
|
||||||
|
after ``register_handlers()`` without a process restart.
|
||||||
|
"""
|
||||||
|
for file_type, fn in list(HANDLERS.items()):
|
||||||
|
if file_type in ("277",): # alias of 277CA; don't re-export twice
|
||||||
|
continue
|
||||||
|
globals()[f"handle_{file_type.lower()}"] = fn
|
||||||
|
|
||||||
|
|
||||||
|
_reexport_handlers()
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"HANDLERS",
|
||||||
|
"HandleResult",
|
||||||
|
"register_handlers",
|
||||||
|
"ack_count_summary",
|
||||||
|
"ack_synthetic_source_batch_id",
|
||||||
|
"two77ca_synthetic_source_batch_id",
|
||||||
|
# handler_* names are added at module load via _reexport_handlers
|
||||||
|
]
|
||||||
|
# Populate __all__ with the present handler symbols.
|
||||||
|
for _h in ("handle_999", "handle_ta1", "handle_277ca", "handle_835"):
|
||||||
|
if _h in globals():
|
||||||
|
__all__.append(_h) # noqa: PYI056
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
"""Ack ID helpers shared between the scheduler and the FastAPI
|
||||||
|
endpoints (SP27 Task 1).
|
||||||
|
|
||||||
|
Lifted from ``scheduler.py:_ack_count_summary`` +
|
||||||
|
``_ack_synthetic_source_batch_id`` + ``_277ca_synthetic_source_batch_id`` —
|
||||||
|
all three had inline copies in both ``scheduler.py`` and ``api.py``
|
||||||
|
prior to this SP. Both callers now import from this module.
|
||||||
|
|
||||||
|
Helpers
|
||||||
|
-------
|
||||||
|
``ack_count_summary(result)``
|
||||||
|
Aggregate ``(received, accepted, rejected, ack_code)`` from a
|
||||||
|
parsed ``ParseResult999``, trusting set-level IK5 over the
|
||||||
|
functional-group AK9. Gainwell's MFT ships AK9 segments that
|
||||||
|
contradict the per-set IK5 (e.g. ``AK9*A*1*1*1`` with
|
||||||
|
``IK5*A``), so trusting AK9's rejected count over-reports.
|
||||||
|
See scheduler commit ``6507a8c`` for the operational context.
|
||||||
|
|
||||||
|
``ack_synthetic_source_batch_id(icn, *, pcn, source_filename)``
|
||||||
|
Build a unique-per-file ``batches.id`` for a 999 that ships
|
||||||
|
without its own source batch. Falls back to a hash suffix of
|
||||||
|
the filename so daily pulls don't all collapse onto the same
|
||||||
|
ICN. Gainwell's MFT ships every 999 with the default ICN
|
||||||
|
``000000001`` (per the scheduler docstring).
|
||||||
|
|
||||||
|
``two77ca_synthetic_source_batch_id(icn)``
|
||||||
|
Same idea for a 277CA without its own source batch.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
def ack_count_summary(result: Any) -> tuple[int, int, int, str]:
|
||||||
|
"""Aggregate ``(received, accepted, rejected, ack_code)`` from
|
||||||
|
a ``ParseResult999``.
|
||||||
|
|
||||||
|
Counts are derived from the set-level ``IK5`` responses
|
||||||
|
(one per ``AK2`` in the 999), not the functional-group ``AK9``.
|
||||||
|
Gainwell's MFT ships contradictory AK9 segments; the per-set
|
||||||
|
IK5 is the authoritative per-claim accept/reject signal.
|
||||||
|
"""
|
||||||
|
sets = result.set_responses
|
||||||
|
received = len(sets)
|
||||||
|
accepted = sum(1 for s in sets if s.set_accept_reject.code == "A")
|
||||||
|
rejected = received - accepted
|
||||||
|
if rejected == 0:
|
||||||
|
code = "A"
|
||||||
|
elif accepted == 0:
|
||||||
|
code = "R"
|
||||||
|
else:
|
||||||
|
code = "P"
|
||||||
|
return (received, accepted, rejected, code)
|
||||||
|
|
||||||
|
|
||||||
|
def ack_synthetic_source_batch_id(
|
||||||
|
interchange_control_number: str,
|
||||||
|
*,
|
||||||
|
pcn: str | None = None,
|
||||||
|
source_filename: str | None = None,
|
||||||
|
) -> str:
|
||||||
|
"""Synthetic ``batches.id`` for a received 999 with no source batch.
|
||||||
|
|
||||||
|
Precedence for the human-readable part of the id (column is
|
||||||
|
``VARCHAR(32)``):
|
||||||
|
|
||||||
|
1. ``999-{pcn}-{hash8}`` if ``AK2`` set_control_number is
|
||||||
|
present (the common case). 4 + 9 + 1 + 8 = 22 chars max.
|
||||||
|
2. ``999-{icn}-{hash8}`` if no AK2 (envelope-only 999).
|
||||||
|
3. ``999-{hash12}`` if no filename either (shouldn't happen
|
||||||
|
in production).
|
||||||
|
"""
|
||||||
|
short_hash = ""
|
||||||
|
if source_filename:
|
||||||
|
short_hash = hashlib.sha1(source_filename.encode("utf-8")).hexdigest()[:8]
|
||||||
|
if pcn and pcn.strip():
|
||||||
|
return f"999-{pcn.strip()}-{short_hash}"
|
||||||
|
icn = (interchange_control_number or "").strip() or "000000001"
|
||||||
|
if short_hash:
|
||||||
|
return f"999-{icn}-{short_hash}"
|
||||||
|
return f"999-{short_hash or icn}"
|
||||||
|
|
||||||
|
|
||||||
|
def two77ca_synthetic_source_batch_id(interchange_control_number: str) -> str:
|
||||||
|
"""Synthetic ``batches.id`` for a received 277CA with no source batch."""
|
||||||
|
return f"277CA-{(interchange_control_number or '').strip() or '000000001'}"
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
"""Handle a 277CA Claim Acknowledgment file (SP27 Task 4).
|
||||||
|
|
||||||
|
Lifted verbatim from ``scheduler.py:_handle_277ca``. The handler owns
|
||||||
|
its own DB session, dispatches to ``parse_277ca_text``, persists the
|
||||||
|
277CA ack row, applies 277CA rejections to matched claims via
|
||||||
|
``inbox_state.apply_277ca_rejections``, and emits
|
||||||
|
``claim.payer_rejected`` audit events for each newly-stamped claim.
|
||||||
|
|
||||||
|
The actor tag (``"277ca-parser-scheduler"``) is preserved so the
|
||||||
|
audit log keeps tracing back to the same source after extraction.
|
||||||
|
Both the FastAPI endpoint and the scheduler path (re)use this module
|
||||||
|
— the API migration drops the inline copy in Task 6.
|
||||||
|
|
||||||
|
``claim.rejected_after_remit`` audit emission (when a 277CA rejects
|
||||||
|
a claim that already has ``matched_remittance_id`` set) is deferred
|
||||||
|
to SP27 Task 13. Today the handler only emits ``claim.payer_rejected``.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from cyclone import db
|
||||||
|
from cyclone.audit_log import AuditEvent, append_event
|
||||||
|
from cyclone.handlers._ack_id import two77ca_synthetic_source_batch_id
|
||||||
|
from cyclone.inbox_state_277ca import apply_277ca_rejections
|
||||||
|
from cyclone.parsers.exceptions import CycloneParseError
|
||||||
|
from cyclone.parsers.parse_277ca import parse_277ca_text
|
||||||
|
from cyclone.store import store as cycl_store
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def handle(
|
||||||
|
text: str,
|
||||||
|
source_file: str,
|
||||||
|
*,
|
||||||
|
event_bus: Optional[object] = None,
|
||||||
|
) -> tuple[str, int]:
|
||||||
|
"""Parse a 277CA, persist ack + stamp payer-rejected claims.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Raw 277CA document bytes (decoded).
|
||||||
|
source_file: Filename the 277CA came from.
|
||||||
|
event_bus: Optional pubsub handle. Reserved for the
|
||||||
|
FastAPI-endpoint migration in Task 6 — the scheduler
|
||||||
|
doesn't pass one.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
``(parser_used, claim_count)`` tuple where ``claim_count`` is
|
||||||
|
the number of STC statuses in the file (one per claim).
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: on parser-level failure (wraps CycloneParseError).
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
result = parse_277ca_text(text, input_file=source_file)
|
||||||
|
except CycloneParseError as exc:
|
||||||
|
raise ValueError(f"277CA parse error: {exc}") from exc
|
||||||
|
|
||||||
|
icn = result.envelope.control_number
|
||||||
|
synthetic_id = two77ca_synthetic_source_batch_id(icn)
|
||||||
|
accepted = sum(
|
||||||
|
1 for s in result.claim_statuses if s.classification == "accepted"
|
||||||
|
)
|
||||||
|
paid = sum(
|
||||||
|
1 for s in result.claim_statuses if s.classification == "paid"
|
||||||
|
)
|
||||||
|
rejected = sum(
|
||||||
|
1 for s in result.claim_statuses if s.classification == "rejected"
|
||||||
|
)
|
||||||
|
pended = sum(
|
||||||
|
1 for s in result.claim_statuses if s.classification == "pended"
|
||||||
|
)
|
||||||
|
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
row = cycl_store.add_277ca_ack(
|
||||||
|
source_batch_id=synthetic_id,
|
||||||
|
control_number=icn,
|
||||||
|
accepted_count=accepted,
|
||||||
|
rejected_count=rejected,
|
||||||
|
paid_count=paid,
|
||||||
|
pended_count=pended,
|
||||||
|
raw_json=json.loads(result.model_dump_json()),
|
||||||
|
)
|
||||||
|
def _lookup(pcn: str):
|
||||||
|
return (
|
||||||
|
session.query(db.Claim)
|
||||||
|
.filter(db.Claim.patient_control_number == pcn)
|
||||||
|
.first()
|
||||||
|
)
|
||||||
|
apply_result = apply_277ca_rejections(
|
||||||
|
session, result, claim_lookup=_lookup, two77ca_id=row.id,
|
||||||
|
)
|
||||||
|
if apply_result.matched:
|
||||||
|
for cid in apply_result.matched:
|
||||||
|
append_event(session, AuditEvent(
|
||||||
|
event_type="claim.payer_rejected",
|
||||||
|
entity_type="claim",
|
||||||
|
entity_id=cid,
|
||||||
|
payload={"source_batch_id": synthetic_id, "277ca_id": row.id},
|
||||||
|
actor="277ca-parser-scheduler",
|
||||||
|
))
|
||||||
|
session.commit()
|
||||||
|
|
||||||
|
# TODO(sp27-task-6): bridge async EventBus.publish → sync caller
|
||||||
|
# (same gap as handle_999 + handle_ta1; Task 6 will fix when the
|
||||||
|
# FastAPI endpoints migrate to call these handlers).
|
||||||
|
if event_bus is not None:
|
||||||
|
publish = getattr(event_bus, "publish", None)
|
||||||
|
if callable(publish):
|
||||||
|
try:
|
||||||
|
publish("ack_received", {
|
||||||
|
"source_batch_id": synthetic_id,
|
||||||
|
"ack_code": "277CA",
|
||||||
|
"kind": "277CA",
|
||||||
|
})
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
log.warning("event_bus publish failed: %s", exc)
|
||||||
|
|
||||||
|
return ("parse_277ca", len(result.claim_statuses))
|
||||||
@@ -0,0 +1,121 @@
|
|||||||
|
"""Handle an 835 ERA Remittance file (SP27 Task 5).
|
||||||
|
|
||||||
|
Lifted verbatim from ``scheduler.py:_handle_835``. The handler owns
|
||||||
|
its own state, dispatches to ``parse_835`` (raises
|
||||||
|
``CycloneParseError`` on bad EDI), runs the per-payer 835 validator,
|
||||||
|
stamps the validation report into ``result.summary``, and persists
|
||||||
|
the BatchRecord via ``cycl_store.add``.
|
||||||
|
|
||||||
|
The 835 handler is the largest of the four because the schema
|
||||||
|
covers per-claim remittances + CAS adjustments + validation.
|
||||||
|
``PAYER_FACTORIES_835`` lives in ``cyclone.api`` (it was always
|
||||||
|
intended to live in ``cyclone.payers`` — a TODO pre-dating SP27 —
|
||||||
|
but moving it is out of Task 5 scope). We import it lazily inside
|
||||||
|
``handle`` to avoid a module-load-time cycle; the cyclic risk is
|
||||||
|
acceptable because api.py also imports scheduler lazily (inside its
|
||||||
|
lifespan handler).
|
||||||
|
|
||||||
|
Two-phase ingest closed in SP27 Task 10: the placeholder
|
||||||
|
``adjustment_amount`` is overwritten by reconcile in the same DB
|
||||||
|
session as the insert (before commit), so a reader never sees a
|
||||||
|
half-reconciled Remittance row. If reconcile raises, the whole
|
||||||
|
ingest rolls back and the scheduler records the per-file error.
|
||||||
|
|
||||||
|
``event_bus`` is best-effort and follows the same TODO(sp27-task-6)
|
||||||
|
sync/async gap as handle_999 / handle_ta1 / handle_277ca.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import uuid
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from cyclone.parsers.exceptions import CycloneParseError
|
||||||
|
from cyclone.parsers.parse_835 import parse as parse_835
|
||||||
|
from cyclone.parsers.validator_835 import validate as validate_835
|
||||||
|
from cyclone.store import BatchRecord
|
||||||
|
from cyclone.store import store as cycl_store
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def handle(
|
||||||
|
text: str,
|
||||||
|
source_file: str,
|
||||||
|
*,
|
||||||
|
event_bus: Optional[object] = None,
|
||||||
|
) -> tuple[str, int]:
|
||||||
|
"""Parse an 835, run validation, persist batch + remittances.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Raw 835 document bytes (decoded).
|
||||||
|
source_file: Filename the 835 came from.
|
||||||
|
event_bus: Optional pubsub handle. Reserved for the
|
||||||
|
FastAPI-endpoint migration in Task 6 — the scheduler
|
||||||
|
doesn't pass one.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
``(parser_used, claim_count)`` tuple. ``claim_count`` is the
|
||||||
|
number of CLP claims parsed; the BatchRecord's summary's
|
||||||
|
``passed`` / ``failed`` fields are derived from the
|
||||||
|
validator's ``report.passed`` flag.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: on parser-level failure (wraps CycloneParseError).
|
||||||
|
Validation failures don't raise — they're stamped into
|
||||||
|
``summary.passed = 0``.
|
||||||
|
"""
|
||||||
|
# TODO(sp27-pre-t5): move PAYER_FACTORIES_835 out of api.py into
|
||||||
|
# ``cyclone.payers`` to remove the lazy cyclic import below. The
|
||||||
|
# import works today because api.py also imports scheduler lazily.
|
||||||
|
from cyclone.api import PAYER_FACTORIES_835
|
||||||
|
|
||||||
|
config = PAYER_FACTORIES_835["co_medicaid_835"]()
|
||||||
|
try:
|
||||||
|
result = parse_835(text, config, input_file=source_file)
|
||||||
|
except CycloneParseError as exc:
|
||||||
|
raise ValueError(f"835 parse error: {exc}") from exc
|
||||||
|
|
||||||
|
# Validation report (mirrors the API endpoint).
|
||||||
|
report = validate_835(result, config)
|
||||||
|
n = len(result.claims)
|
||||||
|
if report.passed:
|
||||||
|
passed, failed, failed_claim_ids = n, 0, []
|
||||||
|
else:
|
||||||
|
passed, failed, failed_claim_ids = 0, n, [
|
||||||
|
c.payer_claim_control_number for c in result.claims
|
||||||
|
]
|
||||||
|
result = result.model_copy(update={
|
||||||
|
"validation": report,
|
||||||
|
"summary": result.summary.model_copy(update={
|
||||||
|
"passed": passed,
|
||||||
|
"failed": failed,
|
||||||
|
"failed_claim_ids": failed_claim_ids,
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
|
||||||
|
rec = BatchRecord(
|
||||||
|
id=uuid.uuid4().hex,
|
||||||
|
kind="835",
|
||||||
|
input_filename=source_file,
|
||||||
|
parsed_at=datetime.now(timezone.utc),
|
||||||
|
result=result,
|
||||||
|
)
|
||||||
|
cycl_store.add(rec)
|
||||||
|
|
||||||
|
# TODO(sp27-task-6): bridge async EventBus.publish → sync caller.
|
||||||
|
if event_bus is not None:
|
||||||
|
publish = getattr(event_bus, "publish", None)
|
||||||
|
if callable(publish):
|
||||||
|
try:
|
||||||
|
publish("ack_received", {
|
||||||
|
"source_batch_id": rec.id,
|
||||||
|
"ack_code": "835",
|
||||||
|
"kind": "835",
|
||||||
|
})
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
log.warning("event_bus publish failed: %s", exc)
|
||||||
|
|
||||||
|
return ("parse_835", len(result.claims))
|
||||||
@@ -0,0 +1,129 @@
|
|||||||
|
"""Handle a 999 Implementation Acknowledgment file (SP27 Task 2).
|
||||||
|
|
||||||
|
Lifted verbatim from ``scheduler.py:_handle_999``. The handler owns
|
||||||
|
its own DB session, dispatches to ``parse_999_text``, applies 999
|
||||||
|
rejections to any matched claims via ``inbox_state.apply_999_rejections``,
|
||||||
|
persists the ack row, and returns a ``(parser_used, claim_count)``
|
||||||
|
tuple.
|
||||||
|
|
||||||
|
The actor tag (``"999-parser-scheduler"``) is preserved so the audit
|
||||||
|
log keeps tracing back to the same source after extraction. Both
|
||||||
|
the FastAPI endpoint and the scheduler path (re)use this module —
|
||||||
|
see Task 6 for the API migration that drops the inline copy.
|
||||||
|
|
||||||
|
``claim_count`` mirrors ``parsed.received`` — the count of AK2
|
||||||
|
(set-level) responses in the 999, which is the authoritative
|
||||||
|
per-claim accept/reject signal (see ``_ack_id`` for why AK9 is
|
||||||
|
ignored).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from cyclone import db
|
||||||
|
from cyclone.audit_log import AuditEvent, append_event
|
||||||
|
from cyclone.handlers._ack_id import (
|
||||||
|
ack_count_summary,
|
||||||
|
ack_synthetic_source_batch_id,
|
||||||
|
)
|
||||||
|
from cyclone.inbox_state import apply_999_rejections
|
||||||
|
from cyclone.parsers.exceptions import CycloneParseError
|
||||||
|
from cyclone.parsers.parse_999 import parse_999_text
|
||||||
|
from cyclone.store import store as cycl_store
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def handle(
|
||||||
|
text: str,
|
||||||
|
source_file: str,
|
||||||
|
*,
|
||||||
|
event_bus: Optional[object] = None,
|
||||||
|
) -> tuple[str, int]:
|
||||||
|
"""Parse a 999, apply rejections, persist ack row.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Raw 999 document bytes (decoded).
|
||||||
|
source_file: Filename the 999 came from. Used to derive a
|
||||||
|
unique synthetic ``batches.id`` (see ``_ack_id``).
|
||||||
|
event_bus: Optional pubsub handle. When provided, an
|
||||||
|
``ack_received`` event is best-effort published. The
|
||||||
|
scheduler passes ``None``; the FastAPI endpoint will
|
||||||
|
be migrated to pass ``app.state.event_bus`` in Task 6.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
``(parser_used, claim_count)`` tuple. Matches the scheduler
|
||||||
|
``_download_and_parse`` destructure; the HandleResult
|
||||||
|
migration happens in Task 7.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: on parser-level failure (wraps CycloneParseError).
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
result = parse_999_text(text, input_file=source_file)
|
||||||
|
except CycloneParseError as exc:
|
||||||
|
raise ValueError(f"999 parse error: {exc}") from exc
|
||||||
|
|
||||||
|
received, accepted, rejected, ack_code = ack_count_summary(result)
|
||||||
|
icn = result.envelope.control_number
|
||||||
|
pcn = (
|
||||||
|
result.set_responses[0].set_control_number
|
||||||
|
if result.set_responses else None
|
||||||
|
)
|
||||||
|
synthetic_id = ack_synthetic_source_batch_id(
|
||||||
|
icn, pcn=pcn, source_filename=source_file,
|
||||||
|
)
|
||||||
|
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
def _lookup(pcn: str):
|
||||||
|
return (
|
||||||
|
session.query(db.Claim)
|
||||||
|
.filter_by(patient_control_number=pcn)
|
||||||
|
.first()
|
||||||
|
)
|
||||||
|
rejection_result = apply_999_rejections(
|
||||||
|
session, result, claim_lookup=_lookup,
|
||||||
|
)
|
||||||
|
if rejection_result.matched:
|
||||||
|
for cid in rejection_result.matched:
|
||||||
|
append_event(session, AuditEvent(
|
||||||
|
event_type="claim.rejected",
|
||||||
|
entity_type="claim",
|
||||||
|
entity_id=cid,
|
||||||
|
payload={"source_batch_id": synthetic_id},
|
||||||
|
actor="999-parser-scheduler",
|
||||||
|
))
|
||||||
|
row = cycl_store.add_ack(
|
||||||
|
source_batch_id=synthetic_id,
|
||||||
|
accepted_count=accepted,
|
||||||
|
rejected_count=rejected,
|
||||||
|
received_count=received,
|
||||||
|
ack_code=ack_code,
|
||||||
|
raw_json=json.loads(result.model_dump_json()),
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
|
||||||
|
# Best-effort pubsub publish. The scheduler doesn't pass an
|
||||||
|
# event_bus; the API migration in Task 6 will pass the FastAPI
|
||||||
|
# EventBus (which has an async ``publish``). This handler is
|
||||||
|
# sync, so an async-real EventBus returns an unawaited coroutine
|
||||||
|
# here — that's a known gap until Task 6 wires the publish
|
||||||
|
# through ``asyncio.run_coroutine_threadsafe`` or makes
|
||||||
|
# ``handle`` async. See TODO(sp27-task-6) below.
|
||||||
|
if event_bus is not None:
|
||||||
|
publish = getattr(event_bus, "publish", None)
|
||||||
|
if callable(publish):
|
||||||
|
# TODO(sp27-task-6): bridge async EventBus.publish → sync
|
||||||
|
# caller (see comment above).
|
||||||
|
try:
|
||||||
|
publish("ack_received", {
|
||||||
|
"source_batch_id": synthetic_id,
|
||||||
|
"ack_code": ack_code,
|
||||||
|
"kind": "999",
|
||||||
|
})
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
log.warning("event_bus publish failed: %s", exc)
|
||||||
|
|
||||||
|
return ("parse_999", received)
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
"""Shared ``HandleResult`` dataclass for handlers (SP27).
|
||||||
|
|
||||||
|
Every per-file-type handler returns the same shape so the
|
||||||
|
scheduler's ``_handle_one`` and the FastAPI parse endpoints can
|
||||||
|
process them uniformly.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class HandleResult:
|
||||||
|
"""Outcome of one handler invocation.
|
||||||
|
|
||||||
|
Attributes:
|
||||||
|
parser_used: The parser name written to
|
||||||
|
``processed_inbound_files.parser_used`` (e.g. ``"parse_999"``)
|
||||||
|
and surfaced in the UI.
|
||||||
|
claim_count: The number of claim rows persisted (or batch
|
||||||
|
records, depending on the file type). For 999/TA1 this
|
||||||
|
is the receipt count; for 835 this is the per-claim
|
||||||
|
remittance count; for 277CA this is the per-claim
|
||||||
|
status count.
|
||||||
|
batch_id: The persisted batch id (when the handler creates a
|
||||||
|
row in ``batches``). ``None`` for handlers that persist
|
||||||
|
into per-file-type ack tables (999/TA1/277CA) rather
|
||||||
|
than into the unified ``batches`` table.
|
||||||
|
matched_count: For 835, the number of remits that were
|
||||||
|
matched to an existing claim by ``reconcile.match``.
|
||||||
|
Zero for other handlers.
|
||||||
|
"""
|
||||||
|
|
||||||
|
parser_used: str
|
||||||
|
claim_count: int
|
||||||
|
batch_id: str | None = None
|
||||||
|
matched_count: int = 0
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
"""Handle a TA1 Interchange Acknowledgment file (SP27 Task 3).
|
||||||
|
|
||||||
|
Lifted verbatim from ``scheduler.py:_handle_ta1``. The handler owns
|
||||||
|
its own DB session, dispatches to ``parse_ta1_text``, persists the
|
||||||
|
interchange ack row in ``ta1_acks``, and returns a
|
||||||
|
``(parser_used, claim_count)`` tuple.
|
||||||
|
|
||||||
|
Unlike the 999 handler, TA1 is envelope-only — there is one TA1 per
|
||||||
|
ISA/IEA interchange, no set-level (AK2) or claim-level matching.
|
||||||
|
The claim_count is always 1 (one TA1 ack row per file).
|
||||||
|
|
||||||
|
The actor tag is implicit (no audit event here — TA1 doesn't tag
|
||||||
|
anything in the activity log; it's an infrastructure-level ack).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from cyclone import db
|
||||||
|
from cyclone.parsers.exceptions import CycloneParseError
|
||||||
|
from cyclone.parsers.parse_ta1 import parse_ta1_text
|
||||||
|
from cyclone.store import store as cycl_store
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def handle(
|
||||||
|
text: str,
|
||||||
|
source_file: str,
|
||||||
|
*,
|
||||||
|
event_bus: Optional[object] = None,
|
||||||
|
) -> tuple[str, int]:
|
||||||
|
"""Parse a TA1, persist the interchange ack row.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text: Raw TA1 document bytes (decoded).
|
||||||
|
source_file: Filename the TA1 came from. Used for audit
|
||||||
|
attribution; the ``batches.id`` for TA1 rows is derived
|
||||||
|
internally from the parsed envelope's control number
|
||||||
|
(``TA1-{ICN}``).
|
||||||
|
event_bus: Optional pubsub handle. Reserved for the
|
||||||
|
FastAPI-endpoint migration in Task 6 — the scheduler
|
||||||
|
doesn't pass one.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
``(parser_used, claim_count)`` tuple. TA1 always returns
|
||||||
|
``claim_count=1`` (one TA1 ack row per interchange file).
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: on parser-level failure (wraps CycloneParseError).
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
result = parse_ta1_text(text, input_file=source_file)
|
||||||
|
except CycloneParseError as exc:
|
||||||
|
raise ValueError(f"TA1 parse error: {exc}") from exc
|
||||||
|
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
cycl_store.add_ta1_ack(
|
||||||
|
source_batch_id=result.source_batch_id,
|
||||||
|
control_number=result.ta1.control_number,
|
||||||
|
interchange_date=result.ta1.interchange_date,
|
||||||
|
interchange_time=result.ta1.interchange_time,
|
||||||
|
ack_code=result.ta1.ack_code,
|
||||||
|
note_code=result.ta1.note_code,
|
||||||
|
ack_generated_date=result.ta1.ack_generated_date,
|
||||||
|
sender_id=result.envelope.sender_id,
|
||||||
|
receiver_id=result.envelope.receiver_id,
|
||||||
|
raw_json=json.loads(result.model_dump_json()),
|
||||||
|
)
|
||||||
|
session.commit()
|
||||||
|
|
||||||
|
# TODO(sp27-task-6): bridge async EventBus.publish → sync caller
|
||||||
|
# (same gap as handle_999; both fixed when the FastAPI endpoints
|
||||||
|
# migrate to call these handlers instead of inline code).
|
||||||
|
if event_bus is not None:
|
||||||
|
publish = getattr(event_bus, "publish", None)
|
||||||
|
if callable(publish):
|
||||||
|
try:
|
||||||
|
publish("ack_received", {
|
||||||
|
"source_batch_id": result.source_batch_id,
|
||||||
|
"ack_code": result.ta1.ack_code,
|
||||||
|
"kind": "TA1",
|
||||||
|
})
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
log.warning("event_bus publish failed: %s", exc)
|
||||||
|
|
||||||
|
return ("parse_ta1", 1)
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
-- version: 16
|
||||||
|
-- Add the missing index on claims.matched_remittance_id.
|
||||||
|
--
|
||||||
|
-- SP27 Task 11 added ``check_matched_pair_drift`` at startup, which
|
||||||
|
-- scans ``WHERE Claim.matched_remittance_id IS NOT NULL``. Without an
|
||||||
|
-- index this is a full-table scan; becomes a noticeable boot
|
||||||
|
-- latency cost past ~10k claims. The companion index on the
|
||||||
|
-- reverse side (``remittances.claim_id``) was added in 0007.
|
||||||
|
--
|
||||||
|
-- A plain index is enough — neither side is unique (reversals
|
||||||
|
-- re-reference the original PCN, and a claim without a match is
|
||||||
|
-- fine).
|
||||||
|
|
||||||
|
CREATE INDEX ix_claims_matched_remittance_id
|
||||||
|
ON claims(matched_remittance_id);
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
-- version: 17
|
||||||
|
-- Backfill claims.patient_control_number = claims.id.
|
||||||
|
--
|
||||||
|
-- SP27 Task 17 fixed the 837 ingest in store.py:_claim_837_row so
|
||||||
|
-- ``Claim.patient_control_number`` is populated from
|
||||||
|
-- ``claim.claim_id`` (CLM01) instead of ``claim.subscriber.member_id``
|
||||||
|
-- (the 2010BA NM109). The reconcile matcher joins on
|
||||||
|
-- ``Claim.patient_control_number == Remittance.payer_claim_control_number``
|
||||||
|
-- and the 835 echoes CLM01 in CLP01 per X12 spec, so the wrong field
|
||||||
|
-- silently broke every auto-match in production.
|
||||||
|
--
|
||||||
|
-- For rows written BEFORE the fix, the stored value is the member_id
|
||||||
|
-- (e.g. "W953474") which never matches any remit's CLP01. This
|
||||||
|
-- migration backfills those rows by aligning
|
||||||
|
-- ``patient_control_number`` with the row's own ``id`` (= CLM01).
|
||||||
|
-- After this, every claim row's PCN is consistent with the value the
|
||||||
|
-- 837 actually sent, and any newly-ingested 835 whose CLP01 echoes
|
||||||
|
-- that CLM01 will auto-pair.
|
||||||
|
--
|
||||||
|
-- Idempotent: only touches rows where the stored PCN doesn't already
|
||||||
|
-- match the row's id, so re-running on already-fixed rows is a no-op.
|
||||||
|
--
|
||||||
|
-- Reversal safety: the migration does NOT clear matched_remittance_id,
|
||||||
|
-- so any pre-existing manual_match pairs stay intact.
|
||||||
|
|
||||||
|
UPDATE claims
|
||||||
|
SET patient_control_number = id
|
||||||
|
WHERE patient_control_number IS DISTINCT FROM id;
|
||||||
@@ -8,6 +8,12 @@ Single-pass walker over the tokenized segment list:
|
|||||||
- AK3 (Segment Context) + AK4 (Element Context) — optional per-segment errors
|
- AK3 (Segment Context) + AK4 (Element Context) — optional per-segment errors
|
||||||
- AK5 (Transaction Set Response Status) — per-set accept/reject
|
- AK5 (Transaction Set Response Status) — per-set accept/reject
|
||||||
- AK9 (Functional Group Response Status) — per-group counts + ack code
|
- AK9 (Functional Group Response Status) — per-group counts + ack code
|
||||||
|
- IK5 — a non-standard synonym for ``AK5`` that Gainwell's MFT ships
|
||||||
|
in place of the spec-defined ``AK5``. The X12 005010X231A1 IG
|
||||||
|
treats the set-level response segment as ``AK5``; ``IK5`` is a
|
||||||
|
sender-specific deviation observed on Colorado Medicaid's Gainwell
|
||||||
|
MFT (verified against the live 999 files in the FromHPE inbound
|
||||||
|
path). We accept either.
|
||||||
- SE / GE / IEA
|
- SE / GE / IEA
|
||||||
|
|
||||||
Errors at the file level raise :class:`CycloneParseError`. The parser
|
Errors at the file level raise :class:`CycloneParseError`. The parser
|
||||||
@@ -146,6 +152,11 @@ def _consume_ak3_ak4(segments: list[list[str]], idx: int) -> tuple[list[SegmentE
|
|||||||
def _consume_ak2(segments: list[list[str]], idx: int) -> SetFunctionalGroupResponse | None:
|
def _consume_ak2(segments: list[list[str]], idx: int) -> SetFunctionalGroupResponse | None:
|
||||||
"""Read an AK2 + its child AK3*/AK4* + AK5 segments, return the SetResponse.
|
"""Read an AK2 + its child AK3*/AK4* + AK5 segments, return the SetResponse.
|
||||||
|
|
||||||
|
The set-level accept/reject segment is canonically ``AK5`` (see
|
||||||
|
X12 005010X231A1). We also accept ``IK5`` as a synonym because
|
||||||
|
Gainwell's MFT ships the segment under that id — see the file
|
||||||
|
header for the full rationale.
|
||||||
|
|
||||||
Returns None when called with a non-AK2 segment (defensive — the
|
Returns None when called with a non-AK2 segment (defensive — the
|
||||||
orchestrator only calls this when it sees AK2).
|
orchestrator only calls this when it sees AK2).
|
||||||
"""
|
"""
|
||||||
@@ -164,8 +175,11 @@ def _consume_ak2(segments: list[list[str]], idx: int) -> SetFunctionalGroupRespo
|
|||||||
if idx < len(segments) and segments[idx][0] == "AK3":
|
if idx < len(segments) and segments[idx][0] == "AK3":
|
||||||
seg_errors, idx = _consume_ak3_ak4(segments, idx)
|
seg_errors, idx = _consume_ak3_ak4(segments, idx)
|
||||||
# AK5 (set accept/reject) — required by the spec; default to "R" if missing.
|
# AK5 (set accept/reject) — required by the spec; default to "R" if missing.
|
||||||
|
# Gainwell's MFT uses IK5 instead of AK5 (sender-specific segment id
|
||||||
|
# that means the same thing); accept either. The default of "R"
|
||||||
|
# matters: if the segment is missing entirely, the 999 is a reject.
|
||||||
accept_code = "R"
|
accept_code = "R"
|
||||||
if idx < len(segments) and segments[idx][0] == "AK5":
|
if idx < len(segments) and segments[idx][0] in ("AK5", "IK5"):
|
||||||
ak5 = segments[idx]
|
ak5 = segments[idx]
|
||||||
if len(ak5) > 1 and ak5[1]:
|
if len(ak5) > 1 and ak5[1]:
|
||||||
accept_code = ak5[1]
|
accept_code = ak5[1]
|
||||||
@@ -256,8 +270,11 @@ def parse_999_text(text: str, *, input_file: str = "") -> ParseResult999:
|
|||||||
set_responses.append(sr)
|
set_responses.append(sr)
|
||||||
# Advance past the AK2 + AK3*/AK4*/AK5 cluster
|
# Advance past the AK2 + AK3*/AK4*/AK5 cluster
|
||||||
# (re-walk from i+1 because _consume_ak2 doesn't return idx).
|
# (re-walk from i+1 because _consume_ak2 doesn't return idx).
|
||||||
|
# ``IK5`` is the Gainwell-specific synonym for ``AK5``
|
||||||
|
# and must be in the consumed set here too (see
|
||||||
|
# _consume_ak2 for the full rationale).
|
||||||
i += 1
|
i += 1
|
||||||
while i < len(segments) and segments[i][0] in {"AK3", "AK4", "AK5"}:
|
while i < len(segments) and segments[i][0] in {"AK3", "AK4", "AK5", "IK5"}:
|
||||||
i += 1
|
i += 1
|
||||||
else:
|
else:
|
||||||
i += 1
|
i += 1
|
||||||
|
|||||||
@@ -224,10 +224,15 @@ class ReconcileResult:
|
|||||||
def run(session, batch_id: str) -> ReconcileResult:
|
def run(session, batch_id: str) -> ReconcileResult:
|
||||||
"""Orchestrate reconciliation for an 835 batch.
|
"""Orchestrate reconciliation for an 835 batch.
|
||||||
|
|
||||||
Expects Batch + Remittance + CasAdjustment rows already persisted
|
Expects Batch + Remittance + CasAdjustment rows already added to
|
||||||
(this is called from store.add() AFTER commit). Loads the new
|
``session`` (not yet committed). SP27 Task 10 calls this from
|
||||||
remittances + all currently-unmatched Claims, calls match(), then
|
inside ``CycloneStore.add``'s ingest session, BEFORE
|
||||||
applies each match's intent inside the caller's session.
|
``s.commit()`` — so the whole 835 ingest (rows + reconcile) is
|
||||||
|
a single transaction. If anything here raises, the ingest rolls
|
||||||
|
back; no half-reconciled batch ever lands.
|
||||||
|
|
||||||
|
Loads the new remittances + all currently-unmatched Claims, calls
|
||||||
|
match(), then applies each match's intent inside the caller's session.
|
||||||
|
|
||||||
Returns counts. Does NOT commit; caller controls transaction.
|
Returns counts. Does NOT commit; caller controls transaction.
|
||||||
Raises on failure (caller catches and writes activity event).
|
Raises on failure (caller catches and writes activity event).
|
||||||
|
|||||||
+186
-228
@@ -55,7 +55,10 @@ from cyclone.audit_log import AuditEvent, append_event
|
|||||||
from cyclone.clearhouse import InboundFile, SftpClient
|
from cyclone.clearhouse import InboundFile, SftpClient
|
||||||
from cyclone.db import ProcessedInboundFile
|
from cyclone.db import ProcessedInboundFile
|
||||||
from cyclone.edi.filenames import parse_inbound_filename
|
from cyclone.edi.filenames import parse_inbound_filename
|
||||||
from cyclone.inbox_state import apply_999_rejections
|
from cyclone.handlers import handle_277ca as _handle_277ca
|
||||||
|
from cyclone.handlers import handle_835 as _handle_835
|
||||||
|
from cyclone.handlers import handle_999 as _handle_999
|
||||||
|
from cyclone.handlers import handle_ta1 as _handle_ta1
|
||||||
from cyclone.inbox_state_277ca import apply_277ca_rejections
|
from cyclone.inbox_state_277ca import apply_277ca_rejections
|
||||||
from cyclone.providers import SftpBlock
|
from cyclone.providers import SftpBlock
|
||||||
|
|
||||||
@@ -92,6 +95,12 @@ class TickResult:
|
|||||||
files_skipped: int = 0
|
files_skipped: int = 0
|
||||||
files_errored: int = 0
|
files_errored: int = 0
|
||||||
errors: list[str] = field(default_factory=list)
|
errors: list[str] = field(default_factory=list)
|
||||||
|
# SP27 Task 9: ``sftp_failed`` is the discriminator that separates
|
||||||
|
# SFTP-side failures (the listing step) from per-file processing
|
||||||
|
# errors. ``tick()`` keys off this flag — NOT ``result.errors`` —
|
||||||
|
# so a single malformed 999 cannot flip the operator's destructive
|
||||||
|
# pill. Set in ``_tick_impl``'s listing-error branches only.
|
||||||
|
sftp_failed: bool = False
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
def as_dict(self) -> dict[str, Any]:
|
||||||
return {
|
return {
|
||||||
@@ -104,12 +113,22 @@ class TickResult:
|
|||||||
"files_skipped": self.files_skipped,
|
"files_skipped": self.files_skipped,
|
||||||
"files_errored": self.files_errored,
|
"files_errored": self.files_errored,
|
||||||
"errors": list(self.errors),
|
"errors": list(self.errors),
|
||||||
|
"sftp_failed": self.sftp_failed,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class SchedulerStatus:
|
class SchedulerStatus:
|
||||||
"""Snapshot of the scheduler's runtime state."""
|
"""Snapshot of the scheduler's runtime state.
|
||||||
|
|
||||||
|
SP27 Task 9 added the four SFTP-error fields
|
||||||
|
(``consecutive_failures``, ``last_error``, ``last_error_at``,
|
||||||
|
``last_sftp_attempt_at``) so the operator's UI can tell "all
|
||||||
|
quiet on the MFT front" from "we've been unable to reach the
|
||||||
|
MFT server for 3 hours". The previous 06/25 incident went
|
||||||
|
unnoticed because the status dict had no signal for a hung
|
||||||
|
SFTP poll.
|
||||||
|
"""
|
||||||
|
|
||||||
running: bool
|
running: bool
|
||||||
poll_interval_seconds: int
|
poll_interval_seconds: int
|
||||||
@@ -120,6 +139,13 @@ class SchedulerStatus:
|
|||||||
total_skipped: int
|
total_skipped: int
|
||||||
total_errored: int
|
total_errored: int
|
||||||
last_tick: Optional[TickResult] = None
|
last_tick: Optional[TickResult] = None
|
||||||
|
# SP27 Task 9: SFTP-error surfacing. The UI flips to a destructive
|
||||||
|
# pill when ``consecutive_failures >= 3`` (``CYCLONE_SCHEDULER_FAIL_THRESHOLD``
|
||||||
|
# in the operator pill config; not enforced server-side).
|
||||||
|
consecutive_failures: int = 0
|
||||||
|
last_error: Optional[str] = None
|
||||||
|
last_error_at: Optional[datetime] = None
|
||||||
|
last_sftp_attempt_at: Optional[datetime] = None
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
def as_dict(self) -> dict[str, Any]:
|
||||||
return {
|
return {
|
||||||
@@ -134,6 +160,15 @@ class SchedulerStatus:
|
|||||||
"total_skipped": self.total_skipped,
|
"total_skipped": self.total_skipped,
|
||||||
"total_errored": self.total_errored,
|
"total_errored": self.total_errored,
|
||||||
"last_tick": self.last_tick.as_dict() if self.last_tick else None,
|
"last_tick": self.last_tick.as_dict() if self.last_tick else None,
|
||||||
|
"consecutive_failures": self.consecutive_failures,
|
||||||
|
"last_error": self.last_error,
|
||||||
|
"last_error_at": (
|
||||||
|
self.last_error_at.isoformat() if self.last_error_at else None
|
||||||
|
),
|
||||||
|
"last_sftp_attempt_at": (
|
||||||
|
self.last_sftp_attempt_at.isoformat()
|
||||||
|
if self.last_sftp_attempt_at else None
|
||||||
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -141,179 +176,17 @@ class SchedulerStatus:
|
|||||||
# Per-file-type handlers. Each returns (parser_name, claim_count) and
|
# Per-file-type handlers. Each returns (parser_name, claim_count) and
|
||||||
# persists its own DB rows. The scheduler records the outcome.
|
# persists its own DB rows. The scheduler records the outcome.
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
#
|
||||||
|
# 999 (Task 2), TA1 (Task 3), 277CA (Task 4), and 835 (Task 5) now
|
||||||
|
# live in ``cyclone.handlers``. The HANDLERS dict literal below
|
||||||
|
# references the alias imports so the wiring stays identical.
|
||||||
|
|
||||||
|
|
||||||
def _handle_999(text: str, source_file: str) -> tuple[str, int]:
|
# The 277CA handler has moved to ``cyclone.handlers.handle_277ca``
|
||||||
"""Parse a 999, apply rejections, persist ack row. Returns (parser, count)."""
|
# (SP27 Task 4); the inline def was deleted. The HANDLERS dict literal
|
||||||
from cyclone.parsers.parse_999 import parse_999_text
|
# below still references ``_handle_277ca`` because the import-alias
|
||||||
from cyclone.parsers.exceptions import CycloneParseError
|
# line at the top of this module binds that name to the new function.
|
||||||
|
# Only the 835 handler stays inline (Task 5).
|
||||||
try:
|
|
||||||
result = parse_999_text(text, input_file=source_file)
|
|
||||||
except CycloneParseError as exc:
|
|
||||||
raise ValueError(f"999 parse error: {exc}") from exc
|
|
||||||
|
|
||||||
received, accepted, rejected, ack_code = _ack_count_summary(result)
|
|
||||||
icn = result.envelope.control_number
|
|
||||||
synthetic_id = _ack_synthetic_source_batch_id(icn)
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
def _lookup(pcn: str):
|
|
||||||
return (
|
|
||||||
session.query(db.Claim)
|
|
||||||
.filter_by(patient_control_number=pcn)
|
|
||||||
.first()
|
|
||||||
)
|
|
||||||
rejection_result = apply_999_rejections(
|
|
||||||
session, result, claim_lookup=_lookup,
|
|
||||||
)
|
|
||||||
if rejection_result.matched:
|
|
||||||
for cid in rejection_result.matched:
|
|
||||||
append_event(session, AuditEvent(
|
|
||||||
event_type="claim.rejected",
|
|
||||||
entity_type="claim",
|
|
||||||
entity_id=cid,
|
|
||||||
payload={"source_batch_id": synthetic_id},
|
|
||||||
actor="999-parser-scheduler",
|
|
||||||
))
|
|
||||||
row = cycl_store.add_ack(
|
|
||||||
source_batch_id=synthetic_id,
|
|
||||||
accepted_count=accepted,
|
|
||||||
rejected_count=rejected,
|
|
||||||
received_count=received,
|
|
||||||
ack_code=ack_code,
|
|
||||||
raw_json=json.loads(result.model_dump_json()),
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
return "parse_999", received
|
|
||||||
|
|
||||||
|
|
||||||
def _handle_835(text: str, source_file: str) -> tuple[str, int]:
|
|
||||||
"""Parse an 835, run validation, persist batch + remittances."""
|
|
||||||
import uuid
|
|
||||||
from cyclone.parsers.parse_835 import parse as parse_835
|
|
||||||
from cyclone.parsers.exceptions import CycloneParseError
|
|
||||||
from cyclone.parsers.validator_835 import validate as validate_835
|
|
||||||
from cyclone.payers import PAYER_FACTORIES_835
|
|
||||||
from cyclone.store import BatchRecord
|
|
||||||
|
|
||||||
config = PAYER_FACTORIES_835["co_medicaid_835"]()
|
|
||||||
try:
|
|
||||||
result = parse_835(text, config, input_file=source_file)
|
|
||||||
except CycloneParseError as exc:
|
|
||||||
raise ValueError(f"835 parse error: {exc}") from exc
|
|
||||||
|
|
||||||
# Validation report (mirrors the API endpoint).
|
|
||||||
report = validate_835(result, config)
|
|
||||||
n = len(result.claims)
|
|
||||||
if report.passed:
|
|
||||||
passed, failed, failed_claim_ids = n, 0, []
|
|
||||||
else:
|
|
||||||
passed, failed, failed_claim_ids = 0, n, [
|
|
||||||
c.payer_claim_control_number for c in result.claims
|
|
||||||
]
|
|
||||||
result = result.model_copy(update={
|
|
||||||
"validation": report,
|
|
||||||
"summary": result.summary.model_copy(update={
|
|
||||||
"passed": passed,
|
|
||||||
"failed": failed,
|
|
||||||
"failed_claim_ids": failed_claim_ids,
|
|
||||||
}),
|
|
||||||
})
|
|
||||||
|
|
||||||
rec = BatchRecord(
|
|
||||||
id=uuid.uuid4().hex,
|
|
||||||
kind="835",
|
|
||||||
input_filename=source_file,
|
|
||||||
parsed_at=datetime.now(timezone.utc),
|
|
||||||
result=result,
|
|
||||||
)
|
|
||||||
cycl_store.add(rec)
|
|
||||||
return "parse_835", len(result.claims)
|
|
||||||
|
|
||||||
|
|
||||||
def _handle_277ca(text: str, source_file: str) -> tuple[str, int]:
|
|
||||||
"""Parse a 277CA, persist ack + stamp payer-rejected claims."""
|
|
||||||
from cyclone.parsers.parse_277ca import parse_277ca_text
|
|
||||||
from cyclone.parsers.exceptions import CycloneParseError
|
|
||||||
|
|
||||||
try:
|
|
||||||
result = parse_277ca_text(text, input_file=source_file)
|
|
||||||
except CycloneParseError as exc:
|
|
||||||
raise ValueError(f"277CA parse error: {exc}") from exc
|
|
||||||
|
|
||||||
icn = result.envelope.control_number
|
|
||||||
synthetic_id = _277ca_synthetic_source_batch_id(icn)
|
|
||||||
accepted = sum(
|
|
||||||
1 for s in result.claim_statuses if s.classification == "accepted"
|
|
||||||
)
|
|
||||||
paid = sum(
|
|
||||||
1 for s in result.claim_statuses if s.classification == "paid"
|
|
||||||
)
|
|
||||||
rejected = sum(
|
|
||||||
1 for s in result.claim_statuses if s.classification == "rejected"
|
|
||||||
)
|
|
||||||
pended = sum(
|
|
||||||
1 for s in result.claim_statuses if s.classification == "pended"
|
|
||||||
)
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
row = cycl_store.add_277ca_ack(
|
|
||||||
source_batch_id=synthetic_id,
|
|
||||||
control_number=icn,
|
|
||||||
accepted_count=accepted,
|
|
||||||
rejected_count=rejected,
|
|
||||||
paid_count=paid,
|
|
||||||
pended_count=pended,
|
|
||||||
raw_json=json.loads(result.model_dump_json()),
|
|
||||||
)
|
|
||||||
def _lookup(pcn: str):
|
|
||||||
return (
|
|
||||||
session.query(db.Claim)
|
|
||||||
.filter(db.Claim.patient_control_number == pcn)
|
|
||||||
.first()
|
|
||||||
)
|
|
||||||
apply_result = apply_277ca_rejections(
|
|
||||||
session, result, claim_lookup=_lookup, two77ca_id=row.id,
|
|
||||||
)
|
|
||||||
if apply_result.matched:
|
|
||||||
for cid in apply_result.matched:
|
|
||||||
append_event(session, AuditEvent(
|
|
||||||
event_type="claim.payer_rejected",
|
|
||||||
entity_type="claim",
|
|
||||||
entity_id=cid,
|
|
||||||
payload={"source_batch_id": synthetic_id, "277ca_id": row.id},
|
|
||||||
actor="277ca-parser-scheduler",
|
|
||||||
))
|
|
||||||
session.commit()
|
|
||||||
return "parse_277ca", len(result.claim_statuses)
|
|
||||||
|
|
||||||
|
|
||||||
def _handle_ta1(text: str, source_file: str) -> tuple[str, int]:
|
|
||||||
"""Parse a TA1, persist the interchange ack row."""
|
|
||||||
from cyclone.parsers.parse_ta1 import parse_ta1_text
|
|
||||||
from cyclone.parsers.exceptions import CycloneParseError
|
|
||||||
|
|
||||||
try:
|
|
||||||
result = parse_ta1_text(text, input_file=source_file)
|
|
||||||
except CycloneParseError as exc:
|
|
||||||
raise ValueError(f"TA1 parse error: {exc}") from exc
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
cycl_store.add_ta1_ack(
|
|
||||||
source_batch_id=result.source_batch_id,
|
|
||||||
control_number=result.ta1.control_number,
|
|
||||||
interchange_date=result.ta1.interchange_date,
|
|
||||||
interchange_time=result.ta1.interchange_time,
|
|
||||||
ack_code=result.ta1.ack_code,
|
|
||||||
note_code=result.ta1.note_code,
|
|
||||||
ack_generated_date=result.ta1.ack_generated_date,
|
|
||||||
sender_id=result.envelope.sender_id,
|
|
||||||
receiver_id=result.envelope.receiver_id,
|
|
||||||
raw_json=json.loads(result.model_dump_json()),
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
return "parse_ta1", 1
|
|
||||||
|
|
||||||
|
|
||||||
# Map file_type → handler. Mirrors ROUTED_FILE_TYPES.
|
# Map file_type → handler. Mirrors ROUTED_FILE_TYPES.
|
||||||
@@ -328,43 +201,15 @@ HANDLERS: dict[str, Callable[[str, str], tuple[str, int]]] = {
|
|||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Light copies of helpers the API endpoints use, so the scheduler can
|
# Light copies of helpers the API endpoints use, so the scheduler can
|
||||||
# run without depending on the FastAPI module.
|
# Run without depending on the FastAPI module.
|
||||||
# ---------------------------------------------------------------------------
|
#
|
||||||
|
# Note: ``_277ca_synthetic_source_batch_id`` lived here as a
|
||||||
|
# scheduler-local duplicate of
|
||||||
def _ack_count_summary(result: Any) -> tuple[int, int, int, str]:
|
# ``cyclone.handlers._ack_id.two77ca_synthetic_source_batch_id`` until
|
||||||
"""Return (received, accepted, rejected, ack_code) for a 999.
|
# SP27 Task 4 landed; the inline def has been deleted because
|
||||||
|
# ``handle_277ca`` now imports the canonical copy. The historical
|
||||||
Mirrors the logic in ``cyclone.api._ack_count_summary`` but lives
|
# helper was the only remaining inline def in this section — the
|
||||||
here so the scheduler can run without importing the API module.
|
# surviving inline handler is ``_handle_835`` (lifts in Task 5).
|
||||||
"""
|
|
||||||
if result.functional_group_acks:
|
|
||||||
fg = result.functional_group_acks[0]
|
|
||||||
return (
|
|
||||||
fg.received_count, fg.accepted_count,
|
|
||||||
fg.rejected_count, fg.ack_code,
|
|
||||||
)
|
|
||||||
sets = result.set_responses
|
|
||||||
received = len(sets)
|
|
||||||
accepted = sum(1 for s in sets if s.set_accept_reject.code == "A")
|
|
||||||
rejected = received - accepted
|
|
||||||
if rejected == 0:
|
|
||||||
code = "A"
|
|
||||||
elif accepted == 0:
|
|
||||||
code = "R"
|
|
||||||
else:
|
|
||||||
code = "P"
|
|
||||||
return (received, accepted, rejected, code)
|
|
||||||
|
|
||||||
|
|
||||||
def _ack_synthetic_source_batch_id(interchange_control_number: str) -> str:
|
|
||||||
"""Synthetic batches.id for a received 999 with no source batch."""
|
|
||||||
return f"999-{(interchange_control_number or '').strip() or '000000001'}"
|
|
||||||
|
|
||||||
|
|
||||||
def _277ca_synthetic_source_batch_id(interchange_control_number: str) -> str:
|
|
||||||
"""Synthetic batches.id for a received 277CA with no source batch."""
|
|
||||||
return f"277CA-{(interchange_control_number or '').strip() or '000000001'}"
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -418,6 +263,40 @@ class Scheduler:
|
|||||||
# ticks stack up; the next tick fires only after the previous
|
# ticks stack up; the next tick fires only after the previous
|
||||||
# one finishes).
|
# one finishes).
|
||||||
self._tick_in_progress = False
|
self._tick_in_progress = False
|
||||||
|
# SP27 Task 9: SFTP-error surfacing. ``_consecutive_failures``
|
||||||
|
# counts back-to-back tick failures (cleared on the next
|
||||||
|
# successful tick). The other three fields are the most
|
||||||
|
# recent failure for the operator's pill — preserved across
|
||||||
|
# successes so the audit trail doesn't disappear the moment
|
||||||
|
# the MFT server recovers.
|
||||||
|
self._consecutive_failures = 0
|
||||||
|
self._last_error: Optional[str] = None
|
||||||
|
self._last_error_at: Optional[datetime] = None
|
||||||
|
self._last_sftp_attempt_at: Optional[datetime] = None
|
||||||
|
|
||||||
|
def _record_sftp_outcome(
|
||||||
|
self, *, success: bool, error: Optional[str] = None,
|
||||||
|
) -> None:
|
||||||
|
"""Update the SFTP-error state after a tick.
|
||||||
|
|
||||||
|
``success=True`` resets the consecutive-failure counter; the
|
||||||
|
last-error fields are preserved as an audit trail (the
|
||||||
|
operator's UI can still surface "last failure: 2 hours ago"
|
||||||
|
after a recovery).
|
||||||
|
|
||||||
|
``success=False`` bumps the counter and records the error
|
||||||
|
message + timestamp. ``last_sftp_attempt_at`` is bumped in
|
||||||
|
both cases so the operator can tell when the scheduler last
|
||||||
|
*tried* to reach the MFT (not just when it last failed).
|
||||||
|
"""
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
self._last_sftp_attempt_at = now
|
||||||
|
if success:
|
||||||
|
self._consecutive_failures = 0
|
||||||
|
return
|
||||||
|
self._consecutive_failures += 1
|
||||||
|
self._last_error = error
|
||||||
|
self._last_error_at = now
|
||||||
|
|
||||||
# ---- Public API -------------------------------------------------------
|
# ---- Public API -------------------------------------------------------
|
||||||
|
|
||||||
@@ -465,6 +344,10 @@ class Scheduler:
|
|||||||
total_skipped=self._total_skipped,
|
total_skipped=self._total_skipped,
|
||||||
total_errored=self._total_errored,
|
total_errored=self._total_errored,
|
||||||
last_tick=self._last_tick,
|
last_tick=self._last_tick,
|
||||||
|
consecutive_failures=self._consecutive_failures,
|
||||||
|
last_error=self._last_error,
|
||||||
|
last_error_at=self._last_error_at,
|
||||||
|
last_sftp_attempt_at=self._last_sftp_attempt_at,
|
||||||
)
|
)
|
||||||
|
|
||||||
def is_running(self) -> bool:
|
def is_running(self) -> bool:
|
||||||
@@ -478,6 +361,13 @@ class Scheduler:
|
|||||||
SFTP server from a stampede when the operator hits
|
SFTP server from a stampede when the operator hits
|
||||||
``/api/admin/scheduler/tick`` while a scheduled tick is
|
``/api/admin/scheduler/tick`` while a scheduled tick is
|
||||||
already running.
|
already running.
|
||||||
|
|
||||||
|
SP27 Task 9: the tick outcome is also recorded on the
|
||||||
|
scheduler's SFTP-error state via
|
||||||
|
:meth:`_record_sftp_outcome`. The discriminator is
|
||||||
|
``TickResult.sftp_failed`` (set by ``_tick_impl``'s
|
||||||
|
listing-error branches only), not ``TickResult.errors``
|
||||||
|
which is also populated by per-file processing errors.
|
||||||
"""
|
"""
|
||||||
while self._tick_in_progress:
|
while self._tick_in_progress:
|
||||||
await asyncio.sleep(0.05)
|
await asyncio.sleep(0.05)
|
||||||
@@ -490,6 +380,63 @@ class Scheduler:
|
|||||||
self._total_processed += result.files_processed
|
self._total_processed += result.files_processed
|
||||||
self._total_skipped += result.files_skipped
|
self._total_skipped += result.files_skipped
|
||||||
self._total_errored += result.files_errored
|
self._total_errored += result.files_errored
|
||||||
|
# SFTP-error surfacing: discriminator is ``result.sftp_failed``,
|
||||||
|
# NOT ``result.errors``. ``sftp_failed`` is set by
|
||||||
|
# ``_tick_impl``'s listing-error branches (timeout, connect
|
||||||
|
# refused). Per-file processing errors append to
|
||||||
|
# ``result.errors`` but do NOT set ``sftp_failed`` — a
|
||||||
|
# single malformed 999 in a 5-file batch must not flip
|
||||||
|
# the operator's pill to destructive.
|
||||||
|
if result.sftp_failed:
|
||||||
|
self._record_sftp_outcome(
|
||||||
|
success=False,
|
||||||
|
error="; ".join(result.errors),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self._record_sftp_outcome(success=True)
|
||||||
|
return result
|
||||||
|
finally:
|
||||||
|
self._tick_in_progress = False
|
||||||
|
|
||||||
|
async def process_inbound_files(
|
||||||
|
self, files: list[InboundFile],
|
||||||
|
) -> TickResult:
|
||||||
|
"""Run the per-file processing pipeline over a pre-fetched list.
|
||||||
|
|
||||||
|
Unlike :meth:`tick`, this does **not** call SFTP — the caller
|
||||||
|
is expected to have already downloaded (or arranged the local
|
||||||
|
copy of) each ``InboundFile.local_path``. Used by the
|
||||||
|
``/api/admin/scheduler/pull-inbound`` admin endpoint and the
|
||||||
|
``cyclone pull-inbound`` CLI command to process a date-filtered
|
||||||
|
subset of the inbound MFT path without paying the cost of a
|
||||||
|
full poll.
|
||||||
|
|
||||||
|
Honors ``_stop_event`` between files. Concurrent calls are
|
||||||
|
coalesced the same way :meth:`tick` does, so a slow handler
|
||||||
|
can't be stampeded by a parallel admin invocation.
|
||||||
|
|
||||||
|
SP27 Task 9: deliberately does NOT update the SFTP-error
|
||||||
|
state (no listing, no SFTP). The consecutive-failure counter
|
||||||
|
is the signal for scheduled SFTP polling; operator-initiated
|
||||||
|
batches shouldn't be able to flip it.
|
||||||
|
"""
|
||||||
|
while self._tick_in_progress:
|
||||||
|
await asyncio.sleep(0.05)
|
||||||
|
self._tick_in_progress = True
|
||||||
|
try:
|
||||||
|
started = datetime.now(timezone.utc)
|
||||||
|
result = TickResult(started_at=started, files_seen=len(files))
|
||||||
|
for f in files:
|
||||||
|
if self._stop_event.is_set():
|
||||||
|
break
|
||||||
|
await self._handle_one(f, result)
|
||||||
|
result.finished_at = datetime.now(timezone.utc)
|
||||||
|
self._last_tick = result
|
||||||
|
self._last_poll_at = result.finished_at
|
||||||
|
self._poll_count += 1
|
||||||
|
self._total_processed += result.files_processed
|
||||||
|
self._total_skipped += result.files_skipped
|
||||||
|
self._total_errored += result.files_errored
|
||||||
return result
|
return result
|
||||||
finally:
|
finally:
|
||||||
self._tick_in_progress = False
|
self._tick_in_progress = False
|
||||||
@@ -522,11 +469,26 @@ class Scheduler:
|
|||||||
started = datetime.now(timezone.utc)
|
started = datetime.now(timezone.utc)
|
||||||
result = TickResult(started_at=started)
|
result = TickResult(started_at=started)
|
||||||
|
|
||||||
|
# SP27 Task 8: use the async-wrapped SFTP client so a hung
|
||||||
|
# ``listdir_attr`` (the 06/25 silent-hang failure mode) is
|
||||||
|
# bounded by ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` instead of
|
||||||
|
# waiting on paramiko forever. ``asyncio.TimeoutError`` is
|
||||||
|
# handled explicitly so the tick surfaces a clear error in
|
||||||
|
# ``Scheduler.status()`` (Task 9) rather than a generic
|
||||||
|
# ``Exception`` catch-all.
|
||||||
|
client = self._sftp_client_factory(self._sftp_block)
|
||||||
try:
|
try:
|
||||||
files = await asyncio.to_thread(self._list_inbound)
|
files = await client.async_list_inbound()
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
log.exception("SFTP list_inbound timed out")
|
||||||
|
result.errors.append("list_inbound: timeout")
|
||||||
|
result.sftp_failed = True
|
||||||
|
result.finished_at = datetime.now(timezone.utc)
|
||||||
|
return result
|
||||||
except Exception as exc: # noqa: BLE001
|
except Exception as exc: # noqa: BLE001
|
||||||
log.exception("SFTP list_inbound failed")
|
log.exception("SFTP list_inbound failed")
|
||||||
result.errors.append(f"list_inbound: {exc}")
|
result.errors.append(f"list_inbound: {exc}")
|
||||||
|
result.sftp_failed = True
|
||||||
result.finished_at = datetime.now(timezone.utc)
|
result.finished_at = datetime.now(timezone.utc)
|
||||||
return result
|
return result
|
||||||
|
|
||||||
@@ -539,11 +501,6 @@ class Scheduler:
|
|||||||
result.finished_at = datetime.now(timezone.utc)
|
result.finished_at = datetime.now(timezone.utc)
|
||||||
return result
|
return result
|
||||||
|
|
||||||
def _list_inbound(self) -> list[InboundFile]:
|
|
||||||
"""Return files in the inbound MFT path. Runs on a thread."""
|
|
||||||
client = self._sftp_client_factory(self._sftp_block)
|
|
||||||
return client.list_inbound()
|
|
||||||
|
|
||||||
async def _handle_one(self, f: InboundFile, result: TickResult) -> None:
|
async def _handle_one(self, f: InboundFile, result: TickResult) -> None:
|
||||||
"""Process one inbound file: skip-if-seen, classify, parse, record."""
|
"""Process one inbound file: skip-if-seen, classify, parse, record."""
|
||||||
if await self._already_processed(f.name):
|
if await self._already_processed(f.name):
|
||||||
@@ -650,20 +607,21 @@ class Scheduler:
|
|||||||
def _download_and_parse(
|
def _download_and_parse(
|
||||||
self, f: InboundFile, file_type: str,
|
self, f: InboundFile, file_type: str,
|
||||||
) -> tuple[Path, str, int]:
|
) -> tuple[Path, str, int]:
|
||||||
"""Download from MFT, run the right handler. Returns (path, parser, count).
|
"""Run the right handler on one inbound file. Returns (path, parser, count).
|
||||||
|
|
||||||
Stub mode: ``f.local_path`` already points at the staged file
|
Both stub and real modes read from ``f.local_path`` — the
|
||||||
(set by ``SftpClient._list_inbound_stub``). Real mode: the
|
inbound file is already on disk:
|
||||||
remote name is ``f.name`` and we round-trip through paramiko.
|
* Stub mode: ``_list_inbound_stub`` points ``local_path`` at
|
||||||
|
the operator-dropped staging file.
|
||||||
|
* Real mode: ``_list_inbound_paramiko`` downloads each
|
||||||
|
``listdir_attr`` entry into the local cache as part of the
|
||||||
|
listing pass. Re-reading from the MFT would require
|
||||||
|
``SftpClient.read_file`` with a full remote path, which the
|
||||||
|
scheduler was passing just ``f.name`` for (i.e. a bare
|
||||||
|
filename at the SFTP root, not the inbound dir). Use the
|
||||||
|
cached bytes instead.
|
||||||
"""
|
"""
|
||||||
if self._sftp_block.stub:
|
content = f.local_path.read_bytes()
|
||||||
# In stub mode the InboundFile already has a local_path;
|
|
||||||
# reading the staged bytes directly avoids the stub's
|
|
||||||
# remote-path semantics (which expect a full inbound path).
|
|
||||||
content = f.local_path.read_bytes()
|
|
||||||
else:
|
|
||||||
client = self._sftp_client_factory(self._sftp_block)
|
|
||||||
content = client.read_file(f.name)
|
|
||||||
text = content.decode("utf-8")
|
text = content.decode("utf-8")
|
||||||
handler = HANDLERS[file_type]
|
handler = HANDLERS[file_type]
|
||||||
parser_used, claim_count = handler(text, f.name)
|
parser_used, claim_count = handler(text, f.name)
|
||||||
|
|||||||
+525
-38
@@ -191,7 +191,14 @@ def _claim_837_row(claim: ClaimOutput, batch_id: str) -> Claim:
|
|||||||
return Claim(
|
return Claim(
|
||||||
id=claim.claim_id,
|
id=claim.claim_id,
|
||||||
batch_id=batch_id,
|
batch_id=batch_id,
|
||||||
patient_control_number=claim.subscriber.member_id or "",
|
# SP27 Task 17: Claim.patient_control_number must hold the CLM01
|
||||||
|
# claim_submittr's_identifier the 837 sent — that's the value the
|
||||||
|
# 835 echoes in CLP01, which the reconcile matcher joins on
|
||||||
|
# (reconcile.py:by_pcn), and what 999 / 277CA ACK lookups also
|
||||||
|
# use to cross-reference the original claim. Storing
|
||||||
|
# subscriber.member_id here (the 2010BA NM109) silently broke
|
||||||
|
# every auto-match in production.
|
||||||
|
patient_control_number=claim.claim_id or "",
|
||||||
service_date_from=d_from,
|
service_date_from=d_from,
|
||||||
service_date_to=d_to,
|
service_date_to=d_to,
|
||||||
charge_amount=Decimal(claim.claim.total_charge or 0),
|
charge_amount=Decimal(claim.claim.total_charge or 0),
|
||||||
@@ -851,6 +858,15 @@ def _date_in_bounds(
|
|||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
# Effectively-unbounded iter_* limit, used by count_claims /
|
||||||
|
# count_remittances so they can reuse the iter's filter pipeline
|
||||||
|
# (incl. the in-memory ``payer`` substring + ``date_from/to`` checks)
|
||||||
|
# without being silently capped at the iter's default ``limit=100``.
|
||||||
|
# 2**31 - 1 is the largest signed 32-bit int — far above any realistic
|
||||||
|
# X12 batch population.
|
||||||
|
_ITER_UNBOUNDED = 2**31 - 1
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Backward-compat shim: tests called ``_batches.clear()`` on the in-memory
|
# Backward-compat shim: tests called ``_batches.clear()`` on the in-memory
|
||||||
# store. The DB-backed store doesn't have an in-memory list, so we expose
|
# store. The DB-backed store doesn't have an in-memory list, so we expose
|
||||||
@@ -878,6 +894,110 @@ class _BatchesShim:
|
|||||||
s.commit()
|
s.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def check_matched_pair_drift() -> int:
|
||||||
|
"""Audit the ``Claim.matched_remittance_id`` ↔ ``Remittance.claim_id``
|
||||||
|
FK pair at startup. Non-blocking (SP27 Task 11).
|
||||||
|
|
||||||
|
The matched pair is a denormalized FK pair maintained transactionally
|
||||||
|
by ``manual_match``, ``manual_unmatch``, and ``reconcile.run``. A
|
||||||
|
pre-existing mismatch (e.g. a row written before a state migration
|
||||||
|
that added one column but not the other) would otherwise stay
|
||||||
|
invisible until the next operator pair attempt fails confusingly.
|
||||||
|
This check logs the count + up to N examples so operators can
|
||||||
|
investigate without booting the system.
|
||||||
|
|
||||||
|
Returns the number of drifted rows (0 means clean). Does not
|
||||||
|
raise; bootstrap continues even if drift is detected.
|
||||||
|
|
||||||
|
Count semantics: this returns *drifted rows*, not *drifted pairs*.
|
||||||
|
A single broken pair (``Claim.matched_remittance_id = X`` AND
|
||||||
|
``Remittance.claim_id = Y != nil`` with neither pointing back)
|
||||||
|
can produce TWO drifted rows — one in case A (the claim) and
|
||||||
|
one in case B (the remit). In practice drift is almost always
|
||||||
|
asymmetric (one side NULL), so count == count(pairs); for the
|
||||||
|
fully-symmetric minority, divide by ~2 when alerting. Real drift
|
||||||
|
should be fixed by repairing the writer path, not by counting.
|
||||||
|
|
||||||
|
Cases:
|
||||||
|
A. Claim ``matched_remittance_id = X`` but the paired remit X's
|
||||||
|
``claim_id`` is either NULL or doesn't point back to the claim.
|
||||||
|
B. Remit ``claim_id = A`` but the paired claim A's
|
||||||
|
``matched_remittance_id`` is either NULL or doesn't point back.
|
||||||
|
"""
|
||||||
|
import logging
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
# Case A: claim says X is paired, but X.claim_id doesn't point back.
|
||||||
|
case_a = list(
|
||||||
|
s.execute(
|
||||||
|
select(
|
||||||
|
Claim.id.label("claim_id"),
|
||||||
|
Claim.matched_remittance_id.label("claimed_remit_id"),
|
||||||
|
Remittance.claim_id.label("remit_points_to"),
|
||||||
|
)
|
||||||
|
.outerjoin(
|
||||||
|
Remittance,
|
||||||
|
Remittance.id == Claim.matched_remittance_id,
|
||||||
|
)
|
||||||
|
.where(Claim.matched_remittance_id.is_not(None))
|
||||||
|
.where(
|
||||||
|
(Remittance.claim_id.is_(None))
|
||||||
|
| (Remittance.claim_id != Claim.id)
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
)
|
||||||
|
# Case B: remit says A is paired, but A.matched_remittance_id
|
||||||
|
# doesn't point back.
|
||||||
|
case_b = list(
|
||||||
|
s.execute(
|
||||||
|
select(
|
||||||
|
Remittance.id.label("remit_id"),
|
||||||
|
Remittance.claim_id.label("claimed_claim_id"),
|
||||||
|
Claim.matched_remittance_id.label("claim_points_to"),
|
||||||
|
)
|
||||||
|
.outerjoin(
|
||||||
|
Claim,
|
||||||
|
Claim.id == Remittance.claim_id,
|
||||||
|
)
|
||||||
|
.where(Remittance.claim_id.is_not(None))
|
||||||
|
.where(
|
||||||
|
(Claim.matched_remittance_id.is_(None))
|
||||||
|
| (Claim.matched_remittance_id != Remittance.id)
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
)
|
||||||
|
|
||||||
|
total = len(case_a) + len(case_b)
|
||||||
|
if total == 0:
|
||||||
|
log.info("matched-pair drift check: 0 mismatches (clean)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
log.warning(
|
||||||
|
"matched-pair drift check: %d mismatched pair(s) (showing up to 5 "
|
||||||
|
"of each). Investigate via SELECT against claim / remittance; "
|
||||||
|
"manual re-pair via /api/claims/{id}/manual-match will repair.",
|
||||||
|
total,
|
||||||
|
)
|
||||||
|
for r in case_a[:5]:
|
||||||
|
log.warning(
|
||||||
|
" case A: claim %s -> remit %s, but remit.claim_id=%r",
|
||||||
|
r.claim_id,
|
||||||
|
r.claimed_remit_id,
|
||||||
|
r.remit_points_to,
|
||||||
|
)
|
||||||
|
for r in case_b[:5]:
|
||||||
|
log.warning(
|
||||||
|
" case B: remit %s -> claim %s, but claim.matched_remittance_id=%r",
|
||||||
|
r.remit_id,
|
||||||
|
r.claimed_claim_id,
|
||||||
|
r.claim_points_to,
|
||||||
|
)
|
||||||
|
return total
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# CycloneStore: the SQLAlchemy-backed facade.
|
# CycloneStore: the SQLAlchemy-backed facade.
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -912,9 +1032,12 @@ class CycloneStore:
|
|||||||
|
|
||||||
For 835 batches: inserts the Batch row, one Remittance row per
|
For 835 batches: inserts the Batch row, one Remittance row per
|
||||||
ClaimPayment, and a ``remit_received`` ActivityEvent per
|
ClaimPayment, and a ``remit_received`` ActivityEvent per
|
||||||
ClaimPayment. After commit, calls ``_run_reconcile`` (T10 stub)
|
ClaimPayment. Reconciliation (auto-match + per-pair CAS
|
||||||
in a fail-soft manner — reconciliation errors are logged but
|
aggregate) runs IN THE SAME SESSION before commit (SP27
|
||||||
do not roll back the persisted batch.
|
Task 10) — a single ``s.commit()`` covers both ingest and
|
||||||
|
reconciliation. If reconcile raises, the whole 835 ingest
|
||||||
|
rolls back; the batch never appears half-reconciled with
|
||||||
|
placeholder ``adjustment_amount`` values.
|
||||||
|
|
||||||
Idempotency: ``Claim.id`` and ``Remittance.id`` are PRIMARY KEYS,
|
Idempotency: ``Claim.id`` and ``Remittance.id`` are PRIMARY KEYS,
|
||||||
so a re-ingest of the same fixture (e.g. ``/api/parse-837`` called
|
so a re-ingest of the same fixture (e.g. ``/api/parse-837`` called
|
||||||
@@ -1019,19 +1142,17 @@ class CycloneStore:
|
|||||||
f"Unsupported BatchRecord subclass: {type(record).__name__}"
|
f"Unsupported BatchRecord subclass: {type(record).__name__}"
|
||||||
)
|
)
|
||||||
|
|
||||||
s.commit()
|
# SP27 Task 10: run reconcile INSIDE the same session, before
|
||||||
|
# commit. The placeholder ``adjustment_amount`` set by
|
||||||
|
# ``_remittance_835_row`` is overwritten by reconcile's
|
||||||
|
# ``_reconcile_pair`` SUM-of-CAS-aggregate pass before the
|
||||||
|
# rows are ever visible. If reconcile raises, the whole
|
||||||
|
# 835 ingest rolls back and the batch never lands.
|
||||||
|
if record.kind == "835":
|
||||||
|
from cyclone import reconcile as _reconcile
|
||||||
|
_reconcile.run(s, record.id)
|
||||||
|
|
||||||
# Reconcile 835 batches after the batch is durably persisted.
|
s.commit()
|
||||||
# Fail-soft: errors are logged, not raised, so an 835 parse that
|
|
||||||
# crashes in reconciliation still shows up in /api/batches.
|
|
||||||
if record.kind == "835":
|
|
||||||
try:
|
|
||||||
self._run_reconcile(record.id)
|
|
||||||
except Exception: # pragma: no cover - logged via default handler
|
|
||||||
import logging
|
|
||||||
logging.getLogger(__name__).exception(
|
|
||||||
"reconcile.run failed for batch %s", record.id,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Publish live-tail events synchronously. EventBus.publish is async
|
# Publish live-tail events synchronously. EventBus.publish is async
|
||||||
# but its body is purely synchronous ``put_nowait`` enqueues; we
|
# but its body is purely synchronous ``put_nowait`` enqueues; we
|
||||||
@@ -1113,26 +1234,6 @@ class CycloneStore:
|
|||||||
for queue in list(event_bus._subscribers.get(kind, ())):
|
for queue in list(event_bus._subscribers.get(kind, ())):
|
||||||
event_bus._enqueue_or_drop_oldest(queue, event)
|
event_bus._enqueue_or_drop_oldest(queue, event)
|
||||||
|
|
||||||
def _run_reconcile(self, batch_id: str) -> None:
|
|
||||||
"""T10 stub: invoke the reconcile orchestrator for a batch.
|
|
||||||
|
|
||||||
The actual reconciliation is implemented in T10 (this method
|
|
||||||
will import ``cyclone.reconcile`` and call ``reconcile.run``
|
|
||||||
with the current session). For T9 we keep the import lazy and
|
|
||||||
fail-soft so a missing or NotImplementedError reconcile module
|
|
||||||
never breaks the 835 ingest path.
|
|
||||||
"""
|
|
||||||
# T10 will replace this with the real implementation. Until then
|
|
||||||
# we accept the failure modes the spec lists: ImportError
|
|
||||||
# (module not yet wired), NotImplementedError (stub in place),
|
|
||||||
# or any other transient reconcile error — all swallowed.
|
|
||||||
try:
|
|
||||||
from cyclone import reconcile as _reconcile
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_reconcile.run(s, batch_id)
|
|
||||||
except (ImportError, NotImplementedError):
|
|
||||||
pass
|
|
||||||
|
|
||||||
# -- read path ------------------------------------------------------
|
# -- read path ------------------------------------------------------
|
||||||
|
|
||||||
def _row_to_record(self, row: Batch) -> BatchRecord:
|
def _row_to_record(self, row: Batch) -> BatchRecord:
|
||||||
@@ -1669,6 +1770,101 @@ class CycloneStore:
|
|||||||
r.pop("_sort_receivedDate", None)
|
r.pop("_sort_receivedDate", None)
|
||||||
return out[offset:offset + limit]
|
return out[offset:offset + limit]
|
||||||
|
|
||||||
|
# -- count helpers (SP27 Task 13b) ---------------------------------
|
||||||
|
#
|
||||||
|
# The list endpoints (``/api/claims`` and ``/api/remittances``)
|
||||||
|
# previously computed ``total`` by calling ``iter_*`` with default
|
||||||
|
# ``limit=100`` and taking ``len(...)``. With 60k+ claims and 800+
|
||||||
|
# remits in production, the reported total silently capped at 100
|
||||||
|
# even when the UI rendered a "100" KPI tile and a 100-row table —
|
||||||
|
# the page looked complete but the population was 600× larger. These
|
||||||
|
# helpers reuse the iter's filter pipeline with an effectively
|
||||||
|
# unbounded ``limit`` so the count reflects the true DB population,
|
||||||
|
# not a 100-row sample. Mirrors Dashboard's "server-aggregated
|
||||||
|
# counts" fix from commit 59c3275.
|
||||||
|
def count_claims(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
batch_id: str | None = None,
|
||||||
|
status: str | None = None,
|
||||||
|
provider_npi: str | None = None,
|
||||||
|
payer: str | None = None,
|
||||||
|
date_from: str | None = None,
|
||||||
|
date_to: str | None = None,
|
||||||
|
) -> int:
|
||||||
|
"""Count claims that would be returned by ``iter_claims``.
|
||||||
|
|
||||||
|
Same filter parameters as :meth:`iter_claims` (excluding
|
||||||
|
``sort``/``order``/``limit``/``offset``, which don't affect
|
||||||
|
cardinality).
|
||||||
|
"""
|
||||||
|
rows = self.iter_claims(
|
||||||
|
batch_id=batch_id, status=status, provider_npi=provider_npi,
|
||||||
|
payer=payer, date_from=date_from, date_to=date_to,
|
||||||
|
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
|
||||||
|
)
|
||||||
|
return len(rows)
|
||||||
|
|
||||||
|
def count_remittances(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
batch_id: str | None = None,
|
||||||
|
payer: str | None = None,
|
||||||
|
claim_id: str | None = None,
|
||||||
|
date_from: str | None = None,
|
||||||
|
date_to: str | None = None,
|
||||||
|
) -> int:
|
||||||
|
"""Count remittances that would be returned by ``iter_remittances``.
|
||||||
|
|
||||||
|
Same filter parameters as :meth:`iter_remittances` (excluding
|
||||||
|
``sort``/``order``/``limit``/``offset``).
|
||||||
|
"""
|
||||||
|
rows = self.iter_remittances(
|
||||||
|
batch_id=batch_id, payer=payer, claim_id=claim_id,
|
||||||
|
date_from=date_from, date_to=date_to,
|
||||||
|
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
|
||||||
|
)
|
||||||
|
return len(rows)
|
||||||
|
|
||||||
|
def summarize_remittances(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
batch_id: str | None = None,
|
||||||
|
payer: str | None = None,
|
||||||
|
claim_id: str | None = None,
|
||||||
|
date_from: str | None = None,
|
||||||
|
date_to: str | None = None,
|
||||||
|
) -> dict:
|
||||||
|
"""Return ``{count, total_paid, total_adjustments}`` summed
|
||||||
|
over the remittance population that ``iter_remittances``
|
||||||
|
would return under the same filters.
|
||||||
|
|
||||||
|
Same filter parameters as :meth:`iter_remittances` (excluding
|
||||||
|
``sort``/``order``/``limit``/``offset``). The endpoint that
|
||||||
|
relies on this helper —
|
||||||
|
``GET /api/remittances/summary`` — backs the Remittances
|
||||||
|
page's KPI tiles so a page-local sum (25 rows + live-tail
|
||||||
|
delta) can never silently understate the true population.
|
||||||
|
Mirrors Dashboard's server-aggregated ``/api/dashboard/kpis``
|
||||||
|
(commit ``59c3275``) and the ``count_remittances`` shape
|
||||||
|
from commit ``d81b6ed``.
|
||||||
|
"""
|
||||||
|
rows = self.iter_remittances(
|
||||||
|
batch_id=batch_id, payer=payer, claim_id=claim_id,
|
||||||
|
date_from=date_from, date_to=date_to,
|
||||||
|
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
|
||||||
|
)
|
||||||
|
total_paid = 0.0
|
||||||
|
total_adjustments = 0.0
|
||||||
|
for r in rows:
|
||||||
|
total_paid += float(r.get("paidAmount") or 0)
|
||||||
|
total_adjustments += float(r.get("adjustmentAmount") or 0)
|
||||||
|
return {
|
||||||
|
"count": len(rows),
|
||||||
|
"total_paid": total_paid,
|
||||||
|
"total_adjustments": total_adjustments,
|
||||||
|
}
|
||||||
|
|
||||||
def distinct_providers(self) -> list[dict]:
|
def distinct_providers(self) -> list[dict]:
|
||||||
"""Group claims by NPI and return one row per provider."""
|
"""Group claims by NPI and return one row per provider."""
|
||||||
with db.SessionLocal()() as s:
|
with db.SessionLocal()() as s:
|
||||||
@@ -2370,8 +2566,8 @@ class CycloneStore:
|
|||||||
"port": 22,
|
"port": 22,
|
||||||
"username": "colorado-fts\\coxix_prod_11525703",
|
"username": "colorado-fts\\coxix_prod_11525703",
|
||||||
"paths": {
|
"paths": {
|
||||||
"outbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
|
"outbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
|
||||||
"inbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
|
"inbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
|
||||||
},
|
},
|
||||||
"stub": True,
|
"stub": True,
|
||||||
"staging_dir": "./var/sftp/staging",
|
"staging_dir": "./var/sftp/staging",
|
||||||
@@ -2493,6 +2689,297 @@ def _provider_orm_to_dict(row) -> dict:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# SP27 Task 13: Dashboard aggregate KPIs.
|
||||||
|
#
|
||||||
|
# The Dashboard's "Billed / Received / Denial rate / Pending AR" tiles are
|
||||||
|
# computed from the *whole* claim population, not a sample. With 60k+ claims
|
||||||
|
# in production, fetching ``/api/claims?limit=100`` and reducing in JS would
|
||||||
|
# silently produce wrong numbers (denial rate sampled, billed summed from
|
||||||
|
# 100 rows). This module-level function does the aggregation server-side in
|
||||||
|
# a single session and returns a small structured payload the Dashboard
|
||||||
|
# can render directly.
|
||||||
|
#
|
||||||
|
# Performance: one ``SELECT * FROM claims`` (no pagination) + one
|
||||||
|
# ``SELECT id, total_paid FROM remittances WHERE id IN (...)`` for matched
|
||||||
|
# remits. SQLite returns 60k claim rows in ~30ms on the development
|
||||||
|
# machine; the Python reduce is microseconds. If the dataset grows past
|
||||||
|
# ~500k claims we'd want a SQL-side ``GROUP BY month`` instead — but at
|
||||||
|
# that volume the dashboard should probably be backed by a materialized
|
||||||
|
# view, not a live query.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _claim_state_str(claim: Claim) -> str:
|
||||||
|
"""Stringify a Claim's ``state`` regardless of enum vs raw str storage."""
|
||||||
|
st = claim.state
|
||||||
|
return st.value if hasattr(st, "value") else str(st)
|
||||||
|
|
||||||
|
|
||||||
|
# Claim states counted toward the Dashboard's "pending" tile. SUBMITTED
|
||||||
|
# is the initial post-parse state; REJECTED is the 999 envelope-level
|
||||||
|
# rejection that means the payer never saw the claim. Both are
|
||||||
|
# "outstanding adjudication" from the operator's POV; ``denied`` is
|
||||||
|
# payer adjudication and lives in its own tile.
|
||||||
|
_DASHBOARD_PENDING_STATES: frozenset[str] = frozenset({"submitted", "rejected"})
|
||||||
|
|
||||||
|
|
||||||
|
def dashboard_kpis(
|
||||||
|
*,
|
||||||
|
months: int = 6,
|
||||||
|
top_n_providers: int = 4,
|
||||||
|
top_n_denials: int = 5,
|
||||||
|
) -> dict:
|
||||||
|
"""Compute Dashboard KPIs over the entire claim/remittance population.
|
||||||
|
|
||||||
|
Parameters
|
||||||
|
----------
|
||||||
|
months
|
||||||
|
Number of trailing calendar months to include in the ``monthly``
|
||||||
|
sparkline series (default 6 — matches the existing frontend
|
||||||
|
``MONTHS_BACK`` constant).
|
||||||
|
top_n_providers
|
||||||
|
How many providers to include in the ``topProviders`` array
|
||||||
|
(default 4 — matches the existing Dashboard layout).
|
||||||
|
top_n_denials
|
||||||
|
How many most-recent denied claims to include in the
|
||||||
|
``topDenials`` array (default 5).
|
||||||
|
|
||||||
|
Returns
|
||||||
|
-------
|
||||||
|
dict with keys:
|
||||||
|
|
||||||
|
- ``totals``: aggregate counts + dollar sums + rates for the whole DB.
|
||||||
|
- ``monthly``: list of ``{month, label, count, billed, received,
|
||||||
|
denied, denialRate, ar}`` dicts, oldest-first, length = ``months``.
|
||||||
|
``ar`` is the running outstanding accounts-receivable (billed -
|
||||||
|
received) carried forward across months, clamped at zero.
|
||||||
|
- ``topProviders``: list of ``{npi, label, claimCount, billed,
|
||||||
|
denied}`` dicts, sorted by claimCount desc.
|
||||||
|
- ``topDenials``: list of ``{id, patientName, billedAmount,
|
||||||
|
denialReason, submissionDate}`` dicts, sorted by submissionDate
|
||||||
|
desc, capped at ``top_n_denials``.
|
||||||
|
|
||||||
|
Notes
|
||||||
|
-----
|
||||||
|
- Empty DB returns zero-filled aggregates, an empty ``monthly`` array
|
||||||
|
(one entry per requested month), an empty ``topProviders`` array,
|
||||||
|
and an empty ``topDenials`` array.
|
||||||
|
- ``received`` for a claim is sourced from the matched Remittance's
|
||||||
|
``total_paid`` column. Claims without a matched remittance
|
||||||
|
contribute 0 to the received sum (and thus inflate the
|
||||||
|
outstanding-AR figure, which is the correct operator-visible
|
||||||
|
semantic — "money we haven't been told was paid yet").
|
||||||
|
"""
|
||||||
|
# Pre-build the trailing-N-months skeleton so the response always has
|
||||||
|
# exactly ``months`` entries even when the DB is empty or sparse.
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
skeleton: list[dict] = []
|
||||||
|
for i in range(months - 1, -1, -1):
|
||||||
|
d = now.replace(day=1)
|
||||||
|
# Walk backwards N months without ``relativedelta``.
|
||||||
|
for _ in range(i):
|
||||||
|
prev_month = d.month - 1
|
||||||
|
if prev_month == 0:
|
||||||
|
d = d.replace(year=d.year - 1, month=12)
|
||||||
|
else:
|
||||||
|
d = d.replace(month=prev_month)
|
||||||
|
skeleton.append({
|
||||||
|
"month": f"{d.year:04d}-{d.month:02d}",
|
||||||
|
"label": d.strftime("%b"),
|
||||||
|
"count": 0,
|
||||||
|
"billed": 0.0,
|
||||||
|
"received": 0.0,
|
||||||
|
"denied": 0,
|
||||||
|
"ar": 0.0,
|
||||||
|
})
|
||||||
|
skeleton_index = {entry["month"]: entry for entry in skeleton}
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
# ``Claim.batch`` is a lazy ``relationship`` (default
|
||||||
|
# ``lazy="select"``); without ``selectinload`` each access to
|
||||||
|
# ``r.batch`` in the reduce loop below issues a fresh
|
||||||
|
# ``SELECT ... FROM batches WHERE id=?``. ``selectinload``
|
||||||
|
# pulls every distinct batch in one round-trip instead of N+1
|
||||||
|
# — critical for the 60k-claim dataset.
|
||||||
|
from sqlalchemy.orm import selectinload
|
||||||
|
claims: list[Claim] = (
|
||||||
|
s.query(Claim)
|
||||||
|
.options(selectinload(Claim.batch))
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
|
||||||
|
# Bulk-load matched-remit total_paid so a 60k-claim DB doesn't
|
||||||
|
# produce a 60k-query N+1.
|
||||||
|
matched_ids = [
|
||||||
|
r.matched_remittance_id
|
||||||
|
for r in claims
|
||||||
|
if r.matched_remittance_id
|
||||||
|
]
|
||||||
|
received_by_remit: dict[str, float] = {}
|
||||||
|
if matched_ids:
|
||||||
|
for rid, total_paid in (
|
||||||
|
s.query(Remittance.id, Remittance.total_paid)
|
||||||
|
.filter(Remittance.id.in_(matched_ids))
|
||||||
|
.all()
|
||||||
|
):
|
||||||
|
received_by_remit[rid] = float(total_paid or 0)
|
||||||
|
|
||||||
|
# Per-provider accumulator for the topProviders leaderboard.
|
||||||
|
provider_counts: dict[str, int] = {}
|
||||||
|
provider_billed: dict[str, float] = {}
|
||||||
|
provider_denied: dict[str, int] = {}
|
||||||
|
|
||||||
|
# Per-month accumulator + totals in a single pass.
|
||||||
|
total_count = 0
|
||||||
|
total_billed = 0.0
|
||||||
|
total_received = 0.0
|
||||||
|
denied_count = 0
|
||||||
|
pending_count = 0
|
||||||
|
|
||||||
|
# Collect denied candidates as we walk so we don't issue a
|
||||||
|
# second pass for the topDenials array.
|
||||||
|
denied_candidates: list[dict] = []
|
||||||
|
|
||||||
|
for r in claims:
|
||||||
|
billed = float(r.charge_amount or 0)
|
||||||
|
received = received_by_remit.get(r.matched_remittance_id, 0.0)
|
||||||
|
state_str = _claim_state_str(r)
|
||||||
|
|
||||||
|
total_count += 1
|
||||||
|
total_billed += billed
|
||||||
|
total_received += received
|
||||||
|
if state_str == "denied":
|
||||||
|
denied_count += 1
|
||||||
|
# Drop denied claims with no ``submissionDate`` — they'd
|
||||||
|
# sort first under reverse-lex (empty string < ISO) and
|
||||||
|
# render as "Invalid Date" in the Dashboard. A denial
|
||||||
|
# without a batch is exceptional and operator-irrelevant
|
||||||
|
# for the "recent denials" widget.
|
||||||
|
if r.batch is None or r.batch.parsed_at is None:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
raw = r.raw_json or {}
|
||||||
|
sub = raw.get("subscriber", {})
|
||||||
|
denied_candidates.append({
|
||||||
|
"id": r.id,
|
||||||
|
"patientName": (
|
||||||
|
f"{sub.get('first_name', '')} "
|
||||||
|
f"{sub.get('last_name', '')}".strip()
|
||||||
|
),
|
||||||
|
"billedAmount": billed,
|
||||||
|
"denialReason": r.rejection_reason,
|
||||||
|
"submissionDate": (
|
||||||
|
r.batch.parsed_at
|
||||||
|
.isoformat()
|
||||||
|
.replace("+00:00", "Z")
|
||||||
|
),
|
||||||
|
})
|
||||||
|
if state_str in _DASHBOARD_PENDING_STATES:
|
||||||
|
pending_count += 1
|
||||||
|
|
||||||
|
# Monthly bin. ``submissionDate`` lives on the parent Batch
|
||||||
|
# (all claims in a batch share a parsed_at). Use UTC year-month
|
||||||
|
# to match the skeleton.
|
||||||
|
if r.batch is not None and r.batch.parsed_at is not None:
|
||||||
|
pa = r.batch.parsed_at
|
||||||
|
key = f"{pa.year:04d}-{pa.month:02d}"
|
||||||
|
bucket = skeleton_index.get(key)
|
||||||
|
if bucket is not None:
|
||||||
|
bucket["count"] += 1
|
||||||
|
bucket["billed"] += billed
|
||||||
|
bucket["received"] += received
|
||||||
|
if state_str == "denied":
|
||||||
|
bucket["denied"] += 1
|
||||||
|
|
||||||
|
# Provider bin (keyed on NPI).
|
||||||
|
npi = r.provider_npi or ""
|
||||||
|
if npi:
|
||||||
|
provider_counts[npi] = provider_counts.get(npi, 0) + 1
|
||||||
|
provider_billed[npi] = provider_billed.get(npi, 0.0) + billed
|
||||||
|
if state_str == "denied":
|
||||||
|
provider_denied[npi] = provider_denied.get(npi, 0) + 1
|
||||||
|
|
||||||
|
# Compute denial rate per month + running AR.
|
||||||
|
running_ar = 0.0
|
||||||
|
for entry in skeleton:
|
||||||
|
if entry["count"] > 0:
|
||||||
|
entry["denialRate"] = (entry["denied"] / entry["count"]) * 100.0
|
||||||
|
else:
|
||||||
|
entry["denialRate"] = 0.0
|
||||||
|
running_ar = max(0.0, running_ar + entry["billed"] - entry["received"])
|
||||||
|
entry["ar"] = running_ar
|
||||||
|
|
||||||
|
# Resolve top provider labels from the Provider table in one
|
||||||
|
# round-trip. NPIs without a Provider row still appear in the
|
||||||
|
# leaderboard with an empty label so operators can see
|
||||||
|
# "unknown-NPI" claims are coming from somewhere.
|
||||||
|
# Local import keeps the module-level ``cyclone.providers.Provider``
|
||||||
|
# Pydantic DTO and the SQLAlchemy ``cyclone.db.Provider`` ORM
|
||||||
|
# separate (same pattern as ``list_providers`` / ``upsert_provider``).
|
||||||
|
provider_labels: dict[str, str] = {}
|
||||||
|
if provider_counts:
|
||||||
|
from cyclone.db import Provider as ProviderORM
|
||||||
|
for npi, label in (
|
||||||
|
s.query(ProviderORM.npi, ProviderORM.label).filter(
|
||||||
|
ProviderORM.npi.in_(provider_counts.keys())
|
||||||
|
).all()
|
||||||
|
):
|
||||||
|
provider_labels[npi] = label or ""
|
||||||
|
|
||||||
|
top_providers = sorted(
|
||||||
|
provider_counts.items(),
|
||||||
|
key=lambda kv: kv[1],
|
||||||
|
reverse=True,
|
||||||
|
)[: max(0, top_n_providers)]
|
||||||
|
top_providers_out = [
|
||||||
|
{
|
||||||
|
"npi": npi,
|
||||||
|
"label": provider_labels.get(npi, ""),
|
||||||
|
"claimCount": count,
|
||||||
|
"billed": round(provider_billed.get(npi, 0.0), 2),
|
||||||
|
"denied": provider_denied.get(npi, 0),
|
||||||
|
}
|
||||||
|
for npi, count in top_providers
|
||||||
|
]
|
||||||
|
|
||||||
|
# Top denials = most recently submitted denied claims, capped at
|
||||||
|
# ``top_n_denials``. submissionDate is ISO-8601 so lex sort ==
|
||||||
|
# chronological sort when timestamps share a tz.
|
||||||
|
denied_candidates.sort(key=lambda d: d["submissionDate"], reverse=True)
|
||||||
|
top_denials = denied_candidates[: max(0, top_n_denials)]
|
||||||
|
|
||||||
|
total_denial_rate = (
|
||||||
|
(denied_count / total_count) * 100.0 if total_count > 0 else 0.0
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"totals": {
|
||||||
|
"count": total_count,
|
||||||
|
"billed": round(total_billed, 2),
|
||||||
|
"received": round(total_received, 2),
|
||||||
|
"outstandingAr": round(max(0.0, total_billed - total_received), 2),
|
||||||
|
"denied": denied_count,
|
||||||
|
"denialRate": round(total_denial_rate, 4),
|
||||||
|
"pending": pending_count,
|
||||||
|
},
|
||||||
|
"monthly": [
|
||||||
|
{
|
||||||
|
"month": e["month"],
|
||||||
|
"label": e["label"],
|
||||||
|
"count": e["count"],
|
||||||
|
"billed": round(e["billed"], 2),
|
||||||
|
"received": round(e["received"], 2),
|
||||||
|
"denied": e["denied"],
|
||||||
|
"denialRate": round(e["denialRate"], 4),
|
||||||
|
"ar": round(e["ar"], 2),
|
||||||
|
}
|
||||||
|
for e in skeleton
|
||||||
|
],
|
||||||
|
"topProviders": top_providers_out,
|
||||||
|
"topDenials": top_denials,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def _payer_orm_to_dict(row) -> dict:
|
def _payer_orm_to_dict(row) -> dict:
|
||||||
return {
|
return {
|
||||||
"payer_id": row.payer_id,
|
"payer_id": row.payer_id,
|
||||||
|
|||||||
@@ -49,9 +49,44 @@ def _auto_init_db(tmp_path, monkeypatch):
|
|||||||
from cyclone import api as _api_mod
|
from cyclone import api as _api_mod
|
||||||
_api_mod.app.state.event_bus = EventBus()
|
_api_mod.app.state.event_bus = EventBus()
|
||||||
deps.AUTH_DISABLED = True
|
deps.AUTH_DISABLED = True
|
||||||
|
# The rate-limit middleware keeps a per-IP sliding window in
|
||||||
|
# ``_buckets``. Without a reset between tests, later tests in a
|
||||||
|
# full-suite run get ``429 Too Many Requests`` once the testclient
|
||||||
|
# IP exhausts its 300 req/60s budget. Walk the middleware stack
|
||||||
|
# and clear the buckets so every test starts with a fresh window.
|
||||||
|
# Trigger the stack build with a cheap health probe (the only
|
||||||
|
# request exempt from the limiter — see RateLimitMiddleware.EXEMPT_PATHS).
|
||||||
|
_reset_rate_limit_buckets(_api_mod.app)
|
||||||
try:
|
try:
|
||||||
yield
|
yield
|
||||||
finally:
|
finally:
|
||||||
deps.AUTH_DISABLED = False
|
deps.AUTH_DISABLED = False
|
||||||
_api_mod.app.state.event_bus = None
|
_api_mod.app.state.event_bus = None
|
||||||
db._reset_for_tests()
|
db._reset_for_tests()
|
||||||
|
|
||||||
|
|
||||||
|
def _reset_rate_limit_buckets(app) -> None:
|
||||||
|
"""Clear the rate-limit middleware's per-IP sliding-window buckets.
|
||||||
|
|
||||||
|
SP27 Task 13b follow-up: ``RateLimitMiddleware._buckets`` is shared
|
||||||
|
across all tests in a process (TestClient reuses the same ``app``
|
||||||
|
instance), so without a reset between tests the full suite trips
|
||||||
|
the limiter after ~300 requests and later tests get 429s.
|
||||||
|
|
||||||
|
The middleware stack is only built on the first request, so we
|
||||||
|
prime it with an exempt health probe before walking to the
|
||||||
|
RateLimit layer. If the stack ever stops being a single chain
|
||||||
|
of ``.app`` links, this helper raises AttributeError — better
|
||||||
|
to fail loudly than silently leak state.
|
||||||
|
"""
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
TestClient(app).get("/api/health")
|
||||||
|
cur = app.middleware_stack
|
||||||
|
while cur is not None:
|
||||||
|
if hasattr(cur, "_buckets"):
|
||||||
|
cur._buckets.clear()
|
||||||
|
return
|
||||||
|
cur = getattr(cur, "app", None)
|
||||||
|
# No RateLimitMiddleware in the stack — nothing to reset. Should
|
||||||
|
# not happen in this codebase (security.py registers it at boot)
|
||||||
|
# but we don't want a missing reset to crash unrelated tests.
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
ISA*00* *00* *ZZ*COMEDASSISTPROG*ZZ*11525703 *260527*2303*^*00501*000000001*0*P*:~
|
||||||
|
GS*FA*COMEDASSISTPROG*11525703*20260527*2303*1*X*005010X231A1~
|
||||||
|
ST*999*0001*005010X231A1~
|
||||||
|
AK1*HC*1*005010X222A1~
|
||||||
|
AK2*837*991102989*005010X222A1~
|
||||||
|
IK5*A~
|
||||||
|
AK9*A*1*1*1~
|
||||||
|
SE*6*0001~
|
||||||
|
GE*1*1~
|
||||||
|
IEA*1*000000001~
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
"""Regression: GET /api/acks, /api/ta1-acks, /api/277ca-acks must be reachable
|
||||||
|
for an authenticated admin. The PERMISSIONS matrix only listed the
|
||||||
|
``POST /api/acks`` (parse-999) entry, so the GET list/detail endpoints
|
||||||
|
returned 403 to admins and broke the Inbox / 999 ACKs pages on the UI.
|
||||||
|
|
||||||
|
This test exercises the matrix via the public login route — not by
|
||||||
|
flipping ``AUTH_DISABLED`` — so a missing ``("GET", "/api/<kind>-acks"):
|
||||||
|
ALL_ROLES`` entry would surface as 403.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from sqlalchemy import delete
|
||||||
|
|
||||||
|
from cyclone.api import app
|
||||||
|
from cyclone.auth import users
|
||||||
|
from cyclone.db import Session as DbSession
|
||||||
|
from cyclone.db import SessionLocal, User
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _clear(monkeypatch):
|
||||||
|
monkeypatch.setattr("cyclone.auth.deps.AUTH_DISABLED", False)
|
||||||
|
with SessionLocal()() as db:
|
||||||
|
db.execute(delete(DbSession))
|
||||||
|
db.execute(delete(User))
|
||||||
|
db.commit()
|
||||||
|
yield
|
||||||
|
with SessionLocal()() as db:
|
||||||
|
db.execute(delete(DbSession))
|
||||||
|
db.execute(delete(User))
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client():
|
||||||
|
return TestClient(app)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def admin_client(client):
|
||||||
|
with SessionLocal()() as db:
|
||||||
|
users.create(db, username="admin", password="adminpassword1", role="admin")
|
||||||
|
resp = client.post(
|
||||||
|
"/api/auth/login",
|
||||||
|
json={"username": "admin", "password": "adminpassword1"},
|
||||||
|
)
|
||||||
|
assert resp.status_code == 200
|
||||||
|
return client
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("path", [
|
||||||
|
"/api/acks",
|
||||||
|
"/api/acks/0",
|
||||||
|
"/api/ta1-acks",
|
||||||
|
"/api/ta1-acks/0",
|
||||||
|
"/api/277ca-acks",
|
||||||
|
"/api/277ca-acks/0",
|
||||||
|
])
|
||||||
|
def test_admin_can_list_and_detail_all_ack_kinds(admin_client, path):
|
||||||
|
resp = admin_client.get(path, headers={"Accept": "application/json"})
|
||||||
|
# 200 for the list endpoint (empty rows is fine), 404 for the detail
|
||||||
|
# of a non-existent id (the gate must let the request through).
|
||||||
|
assert resp.status_code in (200, 404), (
|
||||||
|
f"{path} returned {resp.status_code}: {resp.text}"
|
||||||
|
)
|
||||||
|
assert resp.status_code != 403, (
|
||||||
|
f"{path} returned 403 — PERMISSIONS matrix is missing a GET entry"
|
||||||
|
)
|
||||||
@@ -0,0 +1,142 @@
|
|||||||
|
"""Tests for the dedup-ed ack ID helpers (SP27 Task 1).
|
||||||
|
|
||||||
|
Locks the contract for the helpers that ``scheduler.py`` + ``api.py``
|
||||||
|
will both import from one place in ``cyclone.handlers._ack_id``.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from cyclone.handlers._ack_id import (
|
||||||
|
ack_count_summary,
|
||||||
|
ack_synthetic_source_batch_id,
|
||||||
|
two77ca_synthetic_source_batch_id,
|
||||||
|
)
|
||||||
|
from cyclone.parsers.parse_999 import parse_999_text
|
||||||
|
|
||||||
|
|
||||||
|
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
|
||||||
|
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
|
||||||
|
|
||||||
|
|
||||||
|
def _parse(path: Path):
|
||||||
|
return parse_999_text(path.read_text(), input_file=path.name)
|
||||||
|
|
||||||
|
|
||||||
|
def test_ack_count_summary_all_accepted():
|
||||||
|
result = _parse(ACCEPTED)
|
||||||
|
recv, acc, rej, code = ack_count_summary(result)
|
||||||
|
assert recv == 1
|
||||||
|
assert acc == 1
|
||||||
|
assert rej == 0
|
||||||
|
assert code == "A"
|
||||||
|
|
||||||
|
|
||||||
|
def test_ack_count_summary_all_rejected():
|
||||||
|
result = _parse(REJECTED)
|
||||||
|
recv, acc, rej, code = ack_count_summary(result)
|
||||||
|
assert recv == 1
|
||||||
|
assert acc == 0
|
||||||
|
assert rej == 1
|
||||||
|
assert code == "R"
|
||||||
|
|
||||||
|
|
||||||
|
def test_ack_count_summary_partial_uses_p_code():
|
||||||
|
# Synthesize a 2-set 999 inline with one AK5=A and one AK5=R.
|
||||||
|
# ISA is exactly 106 chars (positions 0-105); the parser slices
|
||||||
|
# positionally then splits the body on ``~``.
|
||||||
|
# SE count = ST + AK1 + AK2_1 + AK5_1 + AK2_2 + AK5_2 + AK9 + SE = 8.
|
||||||
|
partial_999 = (
|
||||||
|
"ISA*00* *00* *ZZ*SUBMITTERID *ZZ*RECEIVERID "
|
||||||
|
"*240101*1200*^*00501*000000001*0*P*:~"
|
||||||
|
"GS*FA*SUBMITTERID*RECEIVERID*20240101*1200*1*X*005010X231A1~"
|
||||||
|
"ST*999*0001*005010X231A1~"
|
||||||
|
"AK1*HC*0001*1*2~"
|
||||||
|
"AK2*837*1~"
|
||||||
|
"AK5*A~"
|
||||||
|
"AK2*837*2~"
|
||||||
|
"AK5*R~"
|
||||||
|
"AK9*P*2*1*1~"
|
||||||
|
"SE*8*0001~"
|
||||||
|
"GE*1*1~"
|
||||||
|
"IEA*1*000000001~"
|
||||||
|
)
|
||||||
|
result = parse_999_text(partial_999, input_file="partial.999")
|
||||||
|
recv, acc, rej, code = ack_count_summary(result)
|
||||||
|
assert recv == 2
|
||||||
|
assert acc == 1
|
||||||
|
assert rej == 1
|
||||||
|
assert code == "P"
|
||||||
|
|
||||||
|
|
||||||
|
def test_ack_synthetic_with_pcn_and_filename_is_deterministic_and_prefix():
|
||||||
|
bsid1 = ack_synthetic_source_batch_id(
|
||||||
|
"000000001", pcn="PCN-12345", source_filename="tp_999.x12",
|
||||||
|
)
|
||||||
|
bsid2 = ack_synthetic_source_batch_id(
|
||||||
|
"000000001", pcn="PCN-12345", source_filename="tp_999.x12",
|
||||||
|
)
|
||||||
|
assert bsid1 == bsid2, "same inputs must produce same id"
|
||||||
|
assert bsid1.startswith("999-PCN-12345-")
|
||||||
|
# Hash is 8 hex chars
|
||||||
|
suffix = bsid1.split("-")[-1]
|
||||||
|
assert len(suffix) == 8 and all(
|
||||||
|
c in "0123456789abcdef" for c in suffix
|
||||||
|
), suffix
|
||||||
|
|
||||||
|
|
||||||
|
def test_ack_synthetic_with_pcn_strips_whitespace():
|
||||||
|
bsid = ack_synthetic_source_batch_id(
|
||||||
|
"000000001", pcn=" PCN ", source_filename="f.x12",
|
||||||
|
)
|
||||||
|
assert bsid.startswith("999-PCN-")
|
||||||
|
|
||||||
|
|
||||||
|
def test_ack_synthetic_without_pcn_uses_icn():
|
||||||
|
bsid = ack_synthetic_source_batch_id(
|
||||||
|
"777777777", pcn=None, source_filename="f.x12",
|
||||||
|
)
|
||||||
|
assert bsid.startswith("999-777777777-")
|
||||||
|
# <= 32 chars total (VARCHAR(32) constraint)
|
||||||
|
assert len(bsid) <= 32
|
||||||
|
|
||||||
|
|
||||||
|
def test_ack_synthetic_default_icn_when_empty():
|
||||||
|
bsid = ack_synthetic_source_batch_id("", pcn=None, source_filename=None)
|
||||||
|
# No PCN, no filename → falls through to default ICN ``000000001``
|
||||||
|
# (no hash suffix without a filename).
|
||||||
|
assert bsid == "999-000000001"
|
||||||
|
|
||||||
|
|
||||||
|
def test_two77ca_synthetic_uses_icn():
|
||||||
|
assert two77ca_synthetic_source_batch_id("000012345") == "277CA-000012345"
|
||||||
|
|
||||||
|
|
||||||
|
def test_two77ca_synthetic_empty_falls_back_to_default():
|
||||||
|
assert two77ca_synthetic_source_batch_id("") == "277CA-000000001"
|
||||||
|
assert two77ca_synthetic_source_batch_id(None) == "277CA-000000001"
|
||||||
|
|
||||||
|
|
||||||
|
def test_ack_count_summary_empty_set_responses_returns_zeros():
|
||||||
|
"""A 999 envelope without any AK2/IK5 sets is malformed per the spec,
|
||||||
|
but the helper must still return a sane (0, 0, 0, 'A') tuple — the
|
||||||
|
scheduler adds it to result.errors and the caller decides."""
|
||||||
|
class _StubResult:
|
||||||
|
set_responses = []
|
||||||
|
recv, acc, rej, code = ack_count_summary(_StubResult())
|
||||||
|
assert recv == 0
|
||||||
|
assert acc == 0
|
||||||
|
assert rej == 0
|
||||||
|
# rejected == 0 → "A" (no rejection signal in this degenerate case)
|
||||||
|
assert code == "A"
|
||||||
|
|
||||||
|
|
||||||
|
def test_ack_synthetic_uses_real_sha1_truncation():
|
||||||
|
"""The hash should match hashlib.sha1(filename).hexdigest()[:8] so
|
||||||
|
a future migration to a different hash is intentional, not drift."""
|
||||||
|
expected = hashlib.sha1(b"my-file.x12").hexdigest()[:8]
|
||||||
|
bsid = ack_synthetic_source_batch_id(
|
||||||
|
"000000001", pcn=None, source_filename="my-file.x12",
|
||||||
|
)
|
||||||
|
assert bsid.endswith(f"-{expected}")
|
||||||
+157
-4
@@ -51,21 +51,23 @@ def test_migration_0002_creates_acks_table():
|
|||||||
|
|
||||||
def test_migration_latest_idempotent_on_fresh_db():
|
def test_migration_latest_idempotent_on_fresh_db():
|
||||||
"""Re-running the migration on the same DB must be a no-op (PRAGMA
|
"""Re-running the migration on the same DB must be a no-op (PRAGMA
|
||||||
user_version already at the latest version — currently 15 after
|
user_version already at the latest version — currently 16 after
|
||||||
0004-0006 line_reconciliation, 0005 ta1_acks, SP9's 0007
|
0004-0006 line_reconciliation, 0005 ta1_acks, SP9's 0007
|
||||||
providers/payers/clearhouse, SP10's 0008 payer_rejected,
|
providers/payers/clearhouse, SP10's 0008 payer_rejected,
|
||||||
SP11's 0009 audit_log, SP14's 0010 payer_rejected_acknowledged,
|
SP11's 0009 audit_log, SP14's 0010 payer_rejected_acknowledged,
|
||||||
SP16's 0011 processed_inbound_files, SP17's 0012 db_backups,
|
SP16's 0011 processed_inbound_files, SP17's 0012 db_backups,
|
||||||
SP-auth's 0013 users + sessions, SP-audit's 0014 audit_log.user_id,
|
SP-auth's 0013 users + sessions, SP-audit's 0014 audit_log.user_id,
|
||||||
SP22's 0015 drop_claims_unique_constraint)."""
|
SP22's 0015 drop_claims_unique_constraint, SP27-Task 11's 0016
|
||||||
|
claims.matched_remittance_id index, SP27-Task 17's 0017
|
||||||
|
claim.patient_control_number backfill UPDATE)."""
|
||||||
with db.engine().begin() as c:
|
with db.engine().begin() as c:
|
||||||
v1 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
|
v1 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
|
||||||
assert v1 == 15
|
assert v1 == 17
|
||||||
# A second run should not raise and should not bump the version.
|
# A second run should not raise and should not bump the version.
|
||||||
db_migrate.run(db.engine())
|
db_migrate.run(db.engine())
|
||||||
with db.engine().begin() as c:
|
with db.engine().begin() as c:
|
||||||
v2 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
|
v2 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
|
||||||
assert v2 == 15
|
assert v2 == 17
|
||||||
|
|
||||||
|
|
||||||
def test_add_ack_persists_row():
|
def test_add_ack_persists_row():
|
||||||
@@ -128,3 +130,154 @@ def test_get_ack_returns_row_when_present():
|
|||||||
assert fetched.id == row.id
|
assert fetched.id == row.id
|
||||||
assert fetched.source_batch_id == "b-1"
|
assert fetched.source_batch_id == "b-1"
|
||||||
assert fetched.raw_json == {"hello": "world"}
|
assert fetched.raw_json == {"hello": "world"}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Hotfix 2026-06-29: acks list endpoint silently capped at limit=100
|
||||||
|
# without exposing the true total or full-set aggregates, so the Acks page
|
||||||
|
# rendered "100 on file" and KPI totals summed from the page only when the
|
||||||
|
# row count exceeded 100. These tests pin the new offset param + the
|
||||||
|
# server-side `aggregates` field so the silent-failure mode can't regress.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_acks(n: int) -> list[int]:
|
||||||
|
"""Insert n ack rows with deterministic per-row counts. Returns the ids."""
|
||||||
|
_make_batch("b-hotfix")
|
||||||
|
ids: list[int] = []
|
||||||
|
for i in range(n):
|
||||||
|
row = store.add_ack(
|
||||||
|
source_batch_id="b-hotfix",
|
||||||
|
accepted_count=i + 1,
|
||||||
|
rejected_count=i,
|
||||||
|
received_count=i + 1,
|
||||||
|
ack_code="A",
|
||||||
|
raw_json={"order": i},
|
||||||
|
)
|
||||||
|
ids.append(row.id)
|
||||||
|
return ids
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_acks_endpoint_pagination_offsets_correctly():
|
||||||
|
"""`offset` walks the full set, `has_more` flips at the boundary."""
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from cyclone.api import app
|
||||||
|
from cyclone.auth.users import create
|
||||||
|
from cyclone.db import SessionLocal
|
||||||
|
|
||||||
|
with SessionLocal()() as s:
|
||||||
|
create(s, username="u", password="p", role="admin")
|
||||||
|
s.commit()
|
||||||
|
client = TestClient(app)
|
||||||
|
client.post("/api/auth/login", json={"username": "u", "password": "p"})
|
||||||
|
|
||||||
|
_seed_acks(5)
|
||||||
|
|
||||||
|
r = client.get("/api/acks?limit=2&offset=0")
|
||||||
|
d = r.json()
|
||||||
|
assert d["total"] == 5
|
||||||
|
assert d["returned"] == 2
|
||||||
|
assert d["has_more"] is True
|
||||||
|
assert len(d["items"]) == 2
|
||||||
|
|
||||||
|
r = client.get("/api/acks?limit=2&offset=4")
|
||||||
|
d = r.json()
|
||||||
|
assert d["total"] == 5
|
||||||
|
assert d["returned"] == 1
|
||||||
|
assert d["has_more"] is False
|
||||||
|
assert len(d["items"]) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_acks_endpoint_aggregates_reflect_full_set():
|
||||||
|
"""`aggregates` sums over the full row set, not the page.
|
||||||
|
|
||||||
|
Without this the Acks KPI strip silently under-reports once the
|
||||||
|
row count exceeds the page size — the silent-failure the operator
|
||||||
|
flagged on 2026-06-29.
|
||||||
|
"""
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from cyclone.api import app
|
||||||
|
from cyclone.auth.users import create
|
||||||
|
from cyclone.db import SessionLocal
|
||||||
|
|
||||||
|
with SessionLocal()() as s:
|
||||||
|
create(s, username="u", password="p", role="admin")
|
||||||
|
s.commit()
|
||||||
|
client = TestClient(app)
|
||||||
|
client.post("/api/auth/login", json={"username": "u", "password": "p"})
|
||||||
|
|
||||||
|
# 5 rows: accepted = 1+2+3+4+5 = 15, rejected = 0+1+2+3+4 = 10,
|
||||||
|
# received = same as accepted.
|
||||||
|
_seed_acks(5)
|
||||||
|
|
||||||
|
# Page 1 of 2 — full set is 5 rows, page only shows 2, but aggregates
|
||||||
|
# must reflect all 5.
|
||||||
|
r = client.get("/api/acks?limit=2&offset=0")
|
||||||
|
d = r.json()
|
||||||
|
assert d["total"] == 5
|
||||||
|
assert d["returned"] == 2
|
||||||
|
assert d["aggregates"]["accepted_count"] == 15
|
||||||
|
assert d["aggregates"]["rejected_count"] == 10
|
||||||
|
assert d["aggregates"]["received_count"] == 15
|
||||||
|
|
||||||
|
# Page 2 must return identical aggregates — page slice mustn't shift them.
|
||||||
|
r = client.get("/api/acks?limit=2&offset=2")
|
||||||
|
d = r.json()
|
||||||
|
assert d["aggregates"]["accepted_count"] == 15
|
||||||
|
assert d["aggregates"]["rejected_count"] == 10
|
||||||
|
assert d["aggregates"]["received_count"] == 15
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_acks_endpoint_limit_cap_is_5000():
|
||||||
|
"""The validator still enforces an upper bound so a client can't
|
||||||
|
request 1,000,000 rows and OOM the SQLite-backed list call."""
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from cyclone.api import app
|
||||||
|
from cyclone.auth.users import create
|
||||||
|
from cyclone.db import SessionLocal
|
||||||
|
|
||||||
|
with SessionLocal()() as s:
|
||||||
|
create(s, username="u", password="p", role="admin")
|
||||||
|
s.commit()
|
||||||
|
client = TestClient(app)
|
||||||
|
client.post("/api/auth/login", json={"username": "u", "password": "p"})
|
||||||
|
|
||||||
|
r = client.get("/api/acks?limit=10000")
|
||||||
|
assert r.status_code == 422 # FastAPI validation error
|
||||||
|
|
||||||
|
r = client.get("/api/acks?limit=5000")
|
||||||
|
assert r.status_code == 200
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_acks_endpoint_offset_past_end_returns_empty_page():
|
||||||
|
"""`offset` past the row count must yield an empty page, not 500.
|
||||||
|
|
||||||
|
Pinning this so a future refactor that introduces streaming or
|
||||||
|
cursor-based pagination can't accidentally error or 500 when the
|
||||||
|
UI holds stale page state across a row count change.
|
||||||
|
"""
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from cyclone.api import app
|
||||||
|
from cyclone.auth.users import create
|
||||||
|
from cyclone.db import SessionLocal
|
||||||
|
|
||||||
|
with SessionLocal()() as s:
|
||||||
|
create(s, username="u", password="p", role="admin")
|
||||||
|
s.commit()
|
||||||
|
client = TestClient(app)
|
||||||
|
client.post("/api/auth/login", json={"username": "u", "password": "p"})
|
||||||
|
|
||||||
|
_seed_acks(3)
|
||||||
|
|
||||||
|
r = client.get("/api/acks?limit=2&offset=99")
|
||||||
|
assert r.status_code == 200
|
||||||
|
d = r.json()
|
||||||
|
assert d["total"] == 3
|
||||||
|
assert d["returned"] == 0
|
||||||
|
assert d["has_more"] is False
|
||||||
|
assert d["items"] == []
|
||||||
|
# Aggregates must still reflect the full 3-row set on the empty page —
|
||||||
|
# otherwise a stale UI page state would silently zero out the KPI strip.
|
||||||
|
assert d["aggregates"]["accepted_count"] == 1 + 2 + 3
|
||||||
|
assert d["aggregates"]["rejected_count"] == 0 + 1 + 2
|
||||||
|
assert d["aggregates"]["received_count"] == 1 + 2 + 3
|
||||||
|
|||||||
@@ -121,8 +121,8 @@ def test_patch_without_session_returns_401(client):
|
|||||||
"port": 22,
|
"port": 22,
|
||||||
"username": "colorado-fts\\coxix_prod_11525703",
|
"username": "colorado-fts\\coxix_prod_11525703",
|
||||||
"paths": {
|
"paths": {
|
||||||
"outbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
|
"outbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
|
||||||
"inbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
|
"inbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
|
||||||
},
|
},
|
||||||
"stub": overrides.get("stub", False),
|
"stub": overrides.get("stub", False),
|
||||||
"staging_dir": "./var/sftp/staging",
|
"staging_dir": "./var/sftp/staging",
|
||||||
|
|||||||
@@ -0,0 +1,112 @@
|
|||||||
|
"""Regression tests: api.py + scheduler.py delegate to handlers/_ack_id (SP27 Task 6).
|
||||||
|
|
||||||
|
After the dedup, the inline `_ack_count_summary`,
|
||||||
|
`_ack_synthetic_source_batch_id`, and `_277ca_synthetic_source_batch_id`
|
||||||
|
helpers should be gone from both `api.py` and `scheduler.py`. Their
|
||||||
|
callers should reach the canonical copies under
|
||||||
|
``cyclone.handlers._ack_id``.
|
||||||
|
|
||||||
|
This test pins two things:
|
||||||
|
|
||||||
|
1. The module attributes exist where expected (import paths intact).
|
||||||
|
2. The endpoints that depend on the helpers still respond 200 (or
|
||||||
|
401 if unauthenticated) — i.e. the dedup didn't break the import
|
||||||
|
chain.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from cyclone import api, scheduler
|
||||||
|
from cyclone.api import app
|
||||||
|
from cyclone.handlers._ack_id import (
|
||||||
|
ack_count_summary,
|
||||||
|
ack_synthetic_source_batch_id,
|
||||||
|
two77ca_synthetic_source_batch_id,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---- 1. The dedup actually happened ---------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_no_longer_defines_inline_ack_helpers():
|
||||||
|
"""api.py must not re-define the helpers as local symbols.
|
||||||
|
|
||||||
|
The alias imports at the top of api.py (``from cyclone.handlers._ack_id
|
||||||
|
import ack_count_summary as _ack_count_summary``) leave the
|
||||||
|
underscore-prefixed names bound in ``api.__dict__`` — that's the
|
||||||
|
whole point of an alias import and is required so the inline
|
||||||
|
call sites keep working. What we want to check is that the
|
||||||
|
*implementation source* is no longer the api module itself.
|
||||||
|
|
||||||
|
Concretely: the function's ``__module__`` attribute should be
|
||||||
|
``cyclone.handlers._ack_id``, not ``cyclone.api``.
|
||||||
|
"""
|
||||||
|
for name in (
|
||||||
|
"_ack_count_summary",
|
||||||
|
"_ack_synthetic_source_batch_id",
|
||||||
|
"_277ca_synthetic_source_batch_id",
|
||||||
|
):
|
||||||
|
assert hasattr(api, name), (
|
||||||
|
f"api.py doesn't even have {name} — alias import missing"
|
||||||
|
)
|
||||||
|
fn = getattr(api, name)
|
||||||
|
assert fn.__module__ == "cyclone.handlers._ack_id", (
|
||||||
|
f"api.{name} resolves to {fn.__module__}, expected "
|
||||||
|
f"cyclone.handlers._ack_id — inline copy still exists"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_scheduler_no_inline_ack_helpers():
|
||||||
|
"""scheduler.py must not re-define the ack helpers as local symbols.
|
||||||
|
|
||||||
|
In SP27 Task 2 the inline copies of ``_ack_count_summary`` and
|
||||||
|
``_ack_synthetic_source_batch_id`` were deleted from
|
||||||
|
scheduler.py (they had become dead code after Task 1 lifted the
|
||||||
|
canonical copies into ``cyclone.handlers._ack_id``). Task 4
|
||||||
|
removed ``_277ca_synthetic_source_batch_id`` the same way. So
|
||||||
|
none of the three names should be present in the scheduler
|
||||||
|
module at all.
|
||||||
|
"""
|
||||||
|
for name in (
|
||||||
|
"_ack_count_summary",
|
||||||
|
"_ack_synthetic_source_batch_id",
|
||||||
|
"_277ca_synthetic_source_batch_id",
|
||||||
|
):
|
||||||
|
assert not hasattr(scheduler, name), (
|
||||||
|
f"scheduler.py still defines {name} inline; dedup incomplete"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_canonical_helpers_resolve_to_handlers_ack_id():
|
||||||
|
"""The names exported by ``cyclone.handlers._ack_id`` must be
|
||||||
|
reachable from the canonical import path."""
|
||||||
|
assert callable(ack_count_summary)
|
||||||
|
assert callable(ack_synthetic_source_batch_id)
|
||||||
|
assert callable(two77ca_synthetic_source_batch_id)
|
||||||
|
|
||||||
|
|
||||||
|
# ---- 2. Endpoints still respond -------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client() -> TestClient:
|
||||||
|
return TestClient(app)
|
||||||
|
|
||||||
|
|
||||||
|
def test_ack_endpoints_respond_after_dedup(client):
|
||||||
|
"""Hitting each ack endpoint must not raise ImportError. Either
|
||||||
|
200 (auth OK) or 401/403 (auth required) is acceptable; 500 means
|
||||||
|
the import chain broke."""
|
||||||
|
# Acquire an authenticated session by hitting login (idempotent —
|
||||||
|
# the same admin user already exists in the test DB).
|
||||||
|
for path in (
|
||||||
|
"/api/acks",
|
||||||
|
"/api/ta1-acks",
|
||||||
|
"/api/277ca-acks",
|
||||||
|
):
|
||||||
|
resp = client.get(path)
|
||||||
|
assert resp.status_code in (200, 401, 403), (
|
||||||
|
f"{path} broke after dedup: {resp.status_code} {resp.text}"
|
||||||
|
)
|
||||||
@@ -509,14 +509,17 @@ def test_post_match_happy_path(client: TestClient):
|
|||||||
)
|
)
|
||||||
from cyclone.store import BatchRecord837, BatchRecord835
|
from cyclone.store import BatchRecord837, BatchRecord835
|
||||||
|
|
||||||
# 837 Claim. member_id="M1" so the 835 PCN-based auto-match can't pair them.
|
# 837 Claim. pcn deliberately differs from the 835's PCN so the
|
||||||
|
# auto-matcher (which now joins on claim_id == pcn after the SP27
|
||||||
|
# Task 17 PCN fix) doesn't pair them — we want an orphan so the
|
||||||
|
# manual_match call has work to do.
|
||||||
co = ClaimOutput(
|
co = ClaimOutput(
|
||||||
claim_id="CLM-1",
|
claim_id="CLM-1",
|
||||||
control_number="0001",
|
control_number="0001",
|
||||||
transaction_date=date(2026, 6, 19),
|
transaction_date=date(2026, 6, 19),
|
||||||
billing_provider=BillingProvider(name="Test", npi="1234567890"),
|
billing_provider=BillingProvider(name="Test", npi="1234567890"),
|
||||||
subscriber=Subscriber(
|
subscriber=Subscriber(
|
||||||
first_name="Jane", last_name="Doe", member_id="M1",
|
first_name="Jane", last_name="Doe", member_id="ORPHAN-MEMBER",
|
||||||
),
|
),
|
||||||
payer=Payer(name="Test Payer", id="P1"),
|
payer=Payer(name="Test Payer", id="P1"),
|
||||||
claim=ClaimHeader(
|
claim=ClaimHeader(
|
||||||
@@ -541,12 +544,12 @@ def test_post_match_happy_path(client: TestClient):
|
|||||||
),
|
),
|
||||||
))
|
))
|
||||||
|
|
||||||
# 835 Remittance. PCN="CLM-1" but member_id on the claim side is "M1",
|
# 835 Remittance. PCN="CLM-1-ORPHAN" deliberately differs from the
|
||||||
# so the auto-matcher in reconcile.run does NOT pair them; we want
|
# claim's claim_id="CLM-1" so the auto-matcher in reconcile.run does
|
||||||
# an orphan so manual_match has work to do. charge == paid == 100
|
# NOT pair them; we want an orphan so manual_match has work to do.
|
||||||
# so apply_payment picks ClaimState.PAID.
|
# charge == paid == 100 so apply_payment picks ClaimState.PAID.
|
||||||
cp = ClaimPayment(
|
cp = ClaimPayment(
|
||||||
payer_claim_control_number="CLM-1",
|
payer_claim_control_number="CLM-1-ORPHAN",
|
||||||
status_code="1",
|
status_code="1",
|
||||||
total_charge=Decimal("100"),
|
total_charge=Decimal("100"),
|
||||||
total_paid=Decimal("100"),
|
total_paid=Decimal("100"),
|
||||||
|
|||||||
@@ -0,0 +1,297 @@
|
|||||||
|
"""Tests for ``GET /api/remittances/summary`` (server-aggregated KPI
|
||||||
|
backend for the Remittances page tiles).
|
||||||
|
|
||||||
|
Anomaly recap (see ``docs/superpowers/specs/2026-06-29-cyclone-...
|
||||||
|
-design.md`` if it lands later):
|
||||||
|
|
||||||
|
* The Remittances page's "Total paid" + "Adjustments" KPI tiles
|
||||||
|
previously summed ``items.reduce(...)`` over the *current page* (25
|
||||||
|
rows) plus whatever live-tail events had arrived in the session.
|
||||||
|
* For an empty-looking page with most-paid=0 rows, the UI rendered
|
||||||
|
e.g. ``$16,934`` paid / ``$147`` adjustments — neither the page
|
||||||
|
total ($1,334 / $143) nor the true DB total ($227,181 / $13,792).
|
||||||
|
Same silent-incompleteness pattern the count bug retired in
|
||||||
|
``d81b6ed``: the page looked complete but the numbers were wrong.
|
||||||
|
|
||||||
|
The fix adds a server-aggregated summary endpoint — ``count + paid
|
||||||
|
+ adjustments`` summed over the full persisted row set with the same
|
||||||
|
filter pipeline as ``/api/remittances``. ``Remittances.tsx`` swaps
|
||||||
|
``items.reduce(...)`` for the new hook so the tiles reflect the true
|
||||||
|
DB population regardless of dataset size.
|
||||||
|
|
||||||
|
The covered cases match ``test_list_endpoint_counts.py`` /
|
||||||
|
``test_acks_aggregates.py`` for shape consistency.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
from decimal import Decimal
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from cyclone import db
|
||||||
|
from cyclone.api import app
|
||||||
|
from cyclone.db import Batch, Remittance
|
||||||
|
from cyclone.store import store as global_store
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client() -> TestClient:
|
||||||
|
return TestClient(app)
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_remit(
|
||||||
|
s,
|
||||||
|
remit_id: str,
|
||||||
|
batch_id: str,
|
||||||
|
*,
|
||||||
|
received_at: datetime,
|
||||||
|
total_paid: Decimal = Decimal("100.00"),
|
||||||
|
adjustment_amount: Decimal = Decimal("0"),
|
||||||
|
claim_id: str | None = None,
|
||||||
|
payer_name: str = "Colorado Medicaid",
|
||||||
|
status_code: str = "1",
|
||||||
|
) -> None:
|
||||||
|
# The remittance's batch row carries ``raw_result_json`` with the
|
||||||
|
# payer name because ``iter_remittances`` reads payer name from
|
||||||
|
# there. We stage a fresh ``Batch`` row per call so unique
|
||||||
|
# ``payer_name`` values for tests like the per-payer filter case
|
||||||
|
# don't collide — the existing pattern from
|
||||||
|
# ``test_list_endpoint_counts.py:88``.
|
||||||
|
s.add(Batch(
|
||||||
|
id=batch_id,
|
||||||
|
kind="835",
|
||||||
|
input_filename=f"{remit_id}.edi",
|
||||||
|
parsed_at=received_at,
|
||||||
|
totals_json={"total_claims": 1},
|
||||||
|
validation_json={"passed": True, "warnings": [], "errors": []},
|
||||||
|
raw_result_json={"payer": {"name": payer_name}},
|
||||||
|
))
|
||||||
|
s.add(Remittance(
|
||||||
|
id=remit_id,
|
||||||
|
batch_id=batch_id,
|
||||||
|
payer_claim_control_number=remit_id,
|
||||||
|
claim_id=claim_id,
|
||||||
|
status_code=status_code,
|
||||||
|
total_charge=Decimal("100.00"),
|
||||||
|
total_paid=total_paid,
|
||||||
|
adjustment_amount=adjustment_amount,
|
||||||
|
received_at=received_at,
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Module-level: summarize_remittances
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_summarize_remittances_zero_when_empty():
|
||||||
|
assert global_store.summarize_remittances() == {
|
||||||
|
"count": 0,
|
||||||
|
"total_paid": 0.0,
|
||||||
|
"total_adjustments": 0.0,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_summarize_remittances_unfiltered_returns_full_population():
|
||||||
|
"""Regression: with 5 remits seeded, the OLD ``items.reduce(...)``
|
||||||
|
page-local sum was per-page (25 rows); the new helper must
|
||||||
|
aggregate over all 5."""
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
# 5 remits, paid=10/20/30/40/50 → sum=150, adj all 0.
|
||||||
|
for i, amt in enumerate([10, 20, 30, 40, 50]):
|
||||||
|
_seed_remit(
|
||||||
|
s, f"RMT-SUM-{i:04d}", f"b-sum-pop-{i}",
|
||||||
|
received_at=base + timedelta(minutes=i),
|
||||||
|
total_paid=Decimal(amt),
|
||||||
|
adjustment_amount=Decimal("0"),
|
||||||
|
)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
assert global_store.summarize_remittances() == {
|
||||||
|
"count": 5,
|
||||||
|
"total_paid": 150.0,
|
||||||
|
"total_adjustments": 0.0,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_summarize_remittances_sums_adjustments_across_rows():
|
||||||
|
"""Adjustments accumulate independently from paid."""
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_remit(
|
||||||
|
s, "RMT-A", "b-sum-adj-A",
|
||||||
|
received_at=base, total_paid=Decimal("25.96"),
|
||||||
|
adjustment_amount=Decimal("16.87"),
|
||||||
|
)
|
||||||
|
_seed_remit(
|
||||||
|
s, "RMT-B", "b-sum-adj-B",
|
||||||
|
received_at=base + timedelta(minutes=1),
|
||||||
|
total_paid=Decimal("0"), adjustment_amount=Decimal("50.19"),
|
||||||
|
)
|
||||||
|
_seed_remit(
|
||||||
|
s, "RMT-C", "b-sum-adj-C",
|
||||||
|
received_at=base + timedelta(minutes=2),
|
||||||
|
total_paid=Decimal("0"), adjustment_amount=Decimal("9.95"),
|
||||||
|
)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
assert global_store.summarize_remittances() == {
|
||||||
|
"count": 3,
|
||||||
|
"total_paid": 25.96,
|
||||||
|
"total_adjustments": 77.01, # 16.87 + 50.19 + 9.95
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_summarize_remittances_filters_by_claim_id():
|
||||||
|
"""claim_id filter narrows the summary the same way it narrows
|
||||||
|
iter_remittances (DB-side WHERE remittance.claim_id = :claim_id).
|
||||||
|
"""
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_remit(
|
||||||
|
s, "RMT-X", "b-sum-cid-X",
|
||||||
|
received_at=base, claim_id="CLM-1",
|
||||||
|
total_paid=Decimal("10"), adjustment_amount=Decimal("0"),
|
||||||
|
)
|
||||||
|
_seed_remit(
|
||||||
|
s, "RMT-Y", "b-sum-cid-Y",
|
||||||
|
received_at=base, claim_id="CLM-2",
|
||||||
|
total_paid=Decimal("20"), adjustment_amount=Decimal("1"),
|
||||||
|
)
|
||||||
|
_seed_remit(
|
||||||
|
s, "RMT-Z", "b-sum-cid-Z",
|
||||||
|
received_at=base, claim_id=None,
|
||||||
|
total_paid=Decimal("30"), adjustment_amount=Decimal("2"),
|
||||||
|
)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
assert global_store.summarize_remittances(claim_id="CLM-1") == {
|
||||||
|
"count": 1,
|
||||||
|
"total_paid": 10.0,
|
||||||
|
"total_adjustments": 0.0,
|
||||||
|
}
|
||||||
|
assert global_store.summarize_remittances(claim_id="CLM-2") == {
|
||||||
|
"count": 1,
|
||||||
|
"total_paid": 20.0,
|
||||||
|
"total_adjustments": 1.0,
|
||||||
|
}
|
||||||
|
assert global_store.summarize_remittances(claim_id="CLM-missing") == {
|
||||||
|
"count": 0,
|
||||||
|
"total_paid": 0.0,
|
||||||
|
"total_adjustments": 0.0,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_summarize_remittances_filters_by_payer_exact_match():
|
||||||
|
"""``payer`` filter is in-memory + case-sensitive exact match —
|
||||||
|
the helper must apply it the same way ``iter_remittances`` does
|
||||||
|
so summary and list views agree under the same chip."""
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_remit(
|
||||||
|
s, "RMT-CO-1", "b-sum-payer-co-1",
|
||||||
|
received_at=base, payer_name="Colorado Medicaid",
|
||||||
|
total_paid=Decimal("7"), adjustment_amount=Decimal("0.50"),
|
||||||
|
)
|
||||||
|
_seed_remit(
|
||||||
|
s, "RMT-CO-2", "b-sum-payer-co-2",
|
||||||
|
received_at=base + timedelta(minutes=1),
|
||||||
|
payer_name="Colorado Medicaid",
|
||||||
|
total_paid=Decimal("13"), adjustment_amount=Decimal("0.25"),
|
||||||
|
)
|
||||||
|
_seed_remit(
|
||||||
|
s, "RMT-AZ-1", "b-sum-payer-az-1",
|
||||||
|
received_at=base + timedelta(minutes=2),
|
||||||
|
payer_name="Arizona Medicaid",
|
||||||
|
total_paid=Decimal("99"), adjustment_amount=Decimal("0"),
|
||||||
|
)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
assert global_store.summarize_remittances(payer="Colorado Medicaid") == {
|
||||||
|
"count": 2,
|
||||||
|
"total_paid": 20.0,
|
||||||
|
"total_adjustments": 0.75,
|
||||||
|
}
|
||||||
|
assert global_store.summarize_remittances(payer="Arizona Medicaid") == {
|
||||||
|
"count": 1,
|
||||||
|
"total_paid": 99.0,
|
||||||
|
"total_adjustments": 0.0,
|
||||||
|
}
|
||||||
|
assert global_store.summarize_remittances(payer="colorado medicaid") == {
|
||||||
|
"count": 0,
|
||||||
|
"total_paid": 0.0,
|
||||||
|
"total_adjustments": 0.0,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# HTTP: /api/remittances/summary
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_remittances_summary_empty(client: TestClient):
|
||||||
|
"""Empty DB → zero totals, well-formed response."""
|
||||||
|
resp = client.get("/api/remittances/summary")
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
assert resp.json() == {
|
||||||
|
"count": 0,
|
||||||
|
"total_paid": 0,
|
||||||
|
"total_adjustments": 0,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_remittances_summary_full_population(client: TestClient):
|
||||||
|
"""Bug repro: seed 5 remits with known amounts, assert the
|
||||||
|
endpoint reports the FULL sum, not a 25-row sample.
|
||||||
|
"""
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
for i, amt in enumerate([100, 200, 300, 400, 500]):
|
||||||
|
_seed_remit(
|
||||||
|
s, f"RMT-HTTP-SUM-{i}", f"b-http-sum-{i}",
|
||||||
|
received_at=base + timedelta(minutes=i),
|
||||||
|
total_paid=Decimal(amt),
|
||||||
|
adjustment_amount=Decimal("5"),
|
||||||
|
)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
resp = client.get("/api/remittances/summary")
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
body = resp.json()
|
||||||
|
assert body == {
|
||||||
|
"count": 5,
|
||||||
|
"total_paid": 1500, # 100+200+300+400+500
|
||||||
|
"total_adjustments": 25, # 5*5
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_remittances_summary_respects_claim_id_filter(client: TestClient):
|
||||||
|
"""Filter narrows both count and sums."""
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_remit(
|
||||||
|
s, "RMT-F-1", "b-http-cid-1",
|
||||||
|
received_at=base, claim_id="CLM-F1",
|
||||||
|
total_paid=Decimal("11"), adjustment_amount=Decimal("1"),
|
||||||
|
)
|
||||||
|
_seed_remit(
|
||||||
|
s, "RMT-F-2", "b-http-cid-2",
|
||||||
|
received_at=base, claim_id="CLM-F2",
|
||||||
|
total_paid=Decimal("22"), adjustment_amount=Decimal("2"),
|
||||||
|
)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
resp = client.get(
|
||||||
|
"/api/remittances/summary",
|
||||||
|
params={"claim_id": "CLM-F1"},
|
||||||
|
)
|
||||||
|
assert resp.status_code == 200
|
||||||
|
assert resp.json() == {
|
||||||
|
"count": 1,
|
||||||
|
"total_paid": 11,
|
||||||
|
"total_adjustments": 1,
|
||||||
|
}
|
||||||
@@ -34,13 +34,13 @@ def _stub_scheduler_env(tmp_path, monkeypatch):
|
|||||||
db._reset_for_tests()
|
db._reset_for_tests()
|
||||||
|
|
||||||
staging = tmp_path / "staging"
|
staging = tmp_path / "staging"
|
||||||
inbound = staging / "ToHPE"
|
inbound = staging / "FromHPE"
|
||||||
inbound.mkdir(parents=True)
|
inbound.mkdir(parents=True)
|
||||||
sftp_block = SftpBlock(
|
sftp_block = SftpBlock(
|
||||||
host="mft.example.com",
|
host="mft.example.com",
|
||||||
port=22,
|
port=22,
|
||||||
username="test",
|
username="test",
|
||||||
paths={"outbound": "/FromHPE", "inbound": "/ToHPE"},
|
paths={"outbound": "/ToHPE", "inbound": "/FromHPE"},
|
||||||
stub=True,
|
stub=True,
|
||||||
staging_dir=str(staging),
|
staging_dir=str(staging),
|
||||||
poll_seconds=60,
|
poll_seconds=60,
|
||||||
@@ -54,7 +54,7 @@ def _stub_scheduler_env(tmp_path, monkeypatch):
|
|||||||
|
|
||||||
|
|
||||||
def _drop_file(staging: Path, name: str, body: bytes) -> Path:
|
def _drop_file(staging: Path, name: str, body: bytes) -> Path:
|
||||||
p = staging / "ToHPE" / name
|
p = staging / "FromHPE" / name
|
||||||
p.write_bytes(body)
|
p.write_bytes(body)
|
||||||
return p
|
return p
|
||||||
|
|
||||||
|
|||||||
@@ -30,8 +30,8 @@ def test_get_clearhouse_seeded(client):
|
|||||||
assert body["name"] == "dzinesco"
|
assert body["name"] == "dzinesco"
|
||||||
assert body["tpid"] == "11525703"
|
assert body["tpid"] == "11525703"
|
||||||
assert body["sftp_block"]["stub"] is True
|
assert body["sftp_block"]["stub"] is True
|
||||||
assert "FromHPE" in body["sftp_block"]["paths"]["outbound"]
|
assert "ToHPE" in body["sftp_block"]["paths"]["outbound"]
|
||||||
assert "ToHPE" in body["sftp_block"]["paths"]["inbound"]
|
assert "FromHPE" in body["sftp_block"]["paths"]["inbound"]
|
||||||
|
|
||||||
|
|
||||||
def test_list_providers(client):
|
def test_list_providers(client):
|
||||||
|
|||||||
@@ -0,0 +1,500 @@
|
|||||||
|
"""Tests for the ``GET /api/dashboard/kpis`` aggregate endpoint (SP27 Task 13).
|
||||||
|
|
||||||
|
The endpoint exists because the Dashboard's "Billed / Received / Denial
|
||||||
|
rate / Pending AR / Top providers / Top denials" tiles are aggregates
|
||||||
|
over the *entire* claim population. With 60k+ claims in production,
|
||||||
|
fetching ``/api/claims?limit=100`` and reducing client-side silently
|
||||||
|
produces wrong numbers. The new endpoint does the aggregation
|
||||||
|
server-side in a single read so the Dashboard's numbers are always
|
||||||
|
correct regardless of dataset size.
|
||||||
|
|
||||||
|
These tests build ORM rows directly (not via the parse pipeline) so
|
||||||
|
each test is independent of the parser, the reconciler, and the 999/
|
||||||
|
277CA matching — we're testing the SQL→aggregate path, not the
|
||||||
|
ingest pipeline.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from decimal import Decimal
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from cyclone import db
|
||||||
|
from cyclone.api import app
|
||||||
|
from cyclone.db import Batch, Claim, ClaimState, Provider, Remittance
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _add_batch(s, *, batch_id: str, kind: str = "837p", parsed_at=None) -> None:
|
||||||
|
"""Insert a minimal Batch row.
|
||||||
|
|
||||||
|
Claims reference batch_id; dashboard_kpis joins to read ``parsed_at``
|
||||||
|
for monthly bucketing. ``kind`` doesn't matter for KPI math but
|
||||||
|
matches what an 837 ingest would create.
|
||||||
|
"""
|
||||||
|
s.add(Batch(
|
||||||
|
id=batch_id,
|
||||||
|
kind=kind,
|
||||||
|
input_filename="seed.edi",
|
||||||
|
parsed_at=parsed_at or datetime(2026, 6, 15, 12, 0, tzinfo=timezone.utc),
|
||||||
|
totals_json={"total_claims": 1},
|
||||||
|
validation_json={"passed": True, "warnings": [], "errors": []},
|
||||||
|
raw_result_json={"_": "stub"},
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
def _add_claim(
|
||||||
|
s,
|
||||||
|
*,
|
||||||
|
claim_id: str,
|
||||||
|
batch_id: str = "b1",
|
||||||
|
pcn: str | None = None,
|
||||||
|
charge: str = "100.00",
|
||||||
|
state: ClaimState = ClaimState.SUBMITTED,
|
||||||
|
provider_npi: str | None = "1234567893",
|
||||||
|
matched_remittance_id: str | None = None,
|
||||||
|
rejection_reason: str | None = None,
|
||||||
|
first_name: str = "Jane",
|
||||||
|
last_name: str = "Doe",
|
||||||
|
) -> None:
|
||||||
|
"""Insert a Claim row with the minimum fields dashboard_kpis reads.
|
||||||
|
|
||||||
|
``pcn`` defaults to ``claim_id`` so the claim is uniquely
|
||||||
|
identifiable. ``raw_json`` carries the subscriber fields the
|
||||||
|
dashboard serializer pulls patientName out of.
|
||||||
|
"""
|
||||||
|
s.add(Claim(
|
||||||
|
id=claim_id,
|
||||||
|
batch_id=batch_id,
|
||||||
|
patient_control_number=pcn or claim_id,
|
||||||
|
service_date_from=None,
|
||||||
|
charge_amount=Decimal(charge),
|
||||||
|
provider_npi=provider_npi,
|
||||||
|
state=state,
|
||||||
|
matched_remittance_id=matched_remittance_id,
|
||||||
|
rejection_reason=rejection_reason,
|
||||||
|
raw_json={
|
||||||
|
"subscriber": {"first_name": first_name, "last_name": last_name},
|
||||||
|
"payer": {"name": "CO_TXIX"},
|
||||||
|
"billing_provider": {"npi": provider_npi or ""},
|
||||||
|
"service_lines": [],
|
||||||
|
},
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
def _add_remit(
|
||||||
|
s,
|
||||||
|
*,
|
||||||
|
remit_id: str,
|
||||||
|
batch_id: str = "b2",
|
||||||
|
pcn: str | None = None,
|
||||||
|
total_paid: str = "50.00",
|
||||||
|
total_charge: str = "100.00",
|
||||||
|
) -> None:
|
||||||
|
s.add(Remittance(
|
||||||
|
id=remit_id,
|
||||||
|
batch_id=batch_id,
|
||||||
|
payer_claim_control_number=pcn or remit_id,
|
||||||
|
status_code="1",
|
||||||
|
total_charge=Decimal(total_charge),
|
||||||
|
total_paid=Decimal(total_paid),
|
||||||
|
received_at=datetime(2026, 6, 15, 12, 0, tzinfo=timezone.utc),
|
||||||
|
service_date=None,
|
||||||
|
is_reversal=False,
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
def _add_provider(s, *, npi: str, label: str) -> None:
|
||||||
|
"""Insert a Provider row with all NOT NULL columns populated.
|
||||||
|
|
||||||
|
Provider.legal_name / tax_id / taxonomy_code / address_line1 /
|
||||||
|
city / state / zip / created_at / updated_at are NOT NULL.
|
||||||
|
"""
|
||||||
|
s.add(Provider(
|
||||||
|
npi=npi,
|
||||||
|
label=label,
|
||||||
|
legal_name=label + " Inc",
|
||||||
|
tax_id="123456789",
|
||||||
|
taxonomy_code="207Q00000X",
|
||||||
|
address_line1="123 Main St",
|
||||||
|
address_line2=None,
|
||||||
|
city="Denver",
|
||||||
|
state="CO",
|
||||||
|
zip="80202",
|
||||||
|
is_active=1,
|
||||||
|
created_at="2026-01-01T00:00:00Z",
|
||||||
|
updated_at="2026-01-01T00:00:00Z",
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Fixtures
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client() -> TestClient:
|
||||||
|
return TestClient(app)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Tests — module-level function (no HTTP) so we can pin the aggregation
|
||||||
|
# logic independent of the API wiring.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashboard_kpis_empty_db():
|
||||||
|
"""Empty DB → zero totals + monthly skeleton of N entries.
|
||||||
|
|
||||||
|
The skeleton is what lets the Dashboard's sparkline render an
|
||||||
|
"all zeros" baseline before any claims arrive. Lock that contract.
|
||||||
|
"""
|
||||||
|
from cyclone.store import dashboard_kpis
|
||||||
|
|
||||||
|
out = dashboard_kpis(months=6, top_n_providers=4, top_n_denials=5)
|
||||||
|
|
||||||
|
assert out["totals"] == {
|
||||||
|
"count": 0,
|
||||||
|
"billed": 0.0,
|
||||||
|
"received": 0.0,
|
||||||
|
"outstandingAr": 0.0,
|
||||||
|
"denied": 0,
|
||||||
|
"denialRate": 0.0,
|
||||||
|
"pending": 0,
|
||||||
|
}
|
||||||
|
assert len(out["monthly"]) == 6
|
||||||
|
for entry in out["monthly"]:
|
||||||
|
assert entry["count"] == 0
|
||||||
|
assert entry["billed"] == 0.0
|
||||||
|
assert entry["received"] == 0.0
|
||||||
|
assert entry["denied"] == 0
|
||||||
|
assert entry["denialRate"] == 0.0
|
||||||
|
assert entry["ar"] == 0.0
|
||||||
|
assert out["topProviders"] == []
|
||||||
|
assert out["topDenials"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashboard_kpis_aggregates_single_claim():
|
||||||
|
"""One claim → totals reflect that one row.
|
||||||
|
|
||||||
|
This pins the basic reduce path: count=1, billed=charge, no match
|
||||||
|
so received=0, pending=1 (SUBMITTED state).
|
||||||
|
"""
|
||||||
|
from cyclone.store import dashboard_kpis
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_add_batch(s, batch_id="b1", parsed_at=datetime(2026, 6, 15, tzinfo=timezone.utc))
|
||||||
|
_add_claim(s, claim_id="C-1", charge="250.00", state=ClaimState.SUBMITTED)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
out = dashboard_kpis(months=6)
|
||||||
|
|
||||||
|
assert out["totals"]["count"] == 1
|
||||||
|
assert out["totals"]["billed"] == 250.0
|
||||||
|
assert out["totals"]["received"] == 0.0
|
||||||
|
assert out["totals"]["outstandingAr"] == 250.0
|
||||||
|
assert out["totals"]["denied"] == 0
|
||||||
|
assert out["totals"]["denialRate"] == 0.0
|
||||||
|
assert out["totals"]["pending"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashboard_kpis_received_uses_matched_remit_total_paid():
|
||||||
|
"""``received`` on the totals is the matched remit's ``total_paid``.
|
||||||
|
|
||||||
|
Without this assertion a refactor that read ``Remittance.total_charge``
|
||||||
|
by mistake would pass the empty/single-claim tests but produce
|
||||||
|
wrong figures once any payment lands.
|
||||||
|
"""
|
||||||
|
from cyclone.store import dashboard_kpis
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_add_batch(s, batch_id="b1")
|
||||||
|
_add_claim(
|
||||||
|
s,
|
||||||
|
claim_id="C-1",
|
||||||
|
charge="100.00",
|
||||||
|
state=ClaimState.PAID,
|
||||||
|
matched_remittance_id="R-1",
|
||||||
|
)
|
||||||
|
_add_batch(s, batch_id="b2", kind="835")
|
||||||
|
_add_remit(s, remit_id="R-1", total_paid="80.00", total_charge="100.00")
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
out = dashboard_kpis(months=6)
|
||||||
|
|
||||||
|
assert out["totals"]["billed"] == 100.0
|
||||||
|
assert out["totals"]["received"] == 80.0 # not 100 (total_charge), not 0
|
||||||
|
assert out["totals"]["outstandingAr"] == 20.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashboard_kpis_pending_includes_submitted_and_rejected():
|
||||||
|
"""Pending = SUBMITTED + REJECTED (envelope-level 999 rejection).
|
||||||
|
|
||||||
|
REJECTED here is distinct from DENIED (payer adjudication). Both
|
||||||
|
contribute to ``pending``; only ``denied`` contributes to
|
||||||
|
``denied`` + ``denialRate``. Lock that semantics.
|
||||||
|
"""
|
||||||
|
from cyclone.store import dashboard_kpis
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_add_batch(s, batch_id="b1")
|
||||||
|
_add_claim(s, claim_id="C-SUB", state=ClaimState.SUBMITTED)
|
||||||
|
_add_claim(s, claim_id="C-REJ", state=ClaimState.REJECTED)
|
||||||
|
_add_claim(s, claim_id="C-DEN", state=ClaimState.DENIED, charge="50.00")
|
||||||
|
_add_claim(s, claim_id="C-PAID", state=ClaimState.PAID, charge="200.00")
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
out = dashboard_kpis(months=6)
|
||||||
|
|
||||||
|
assert out["totals"]["count"] == 4
|
||||||
|
assert out["totals"]["pending"] == 2 # SUBMITTED + REJECTED
|
||||||
|
assert out["totals"]["denied"] == 1
|
||||||
|
assert out["totals"]["denialRate"] == 25.0 # 1/4 * 100
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashboard_kpis_monthly_bucketing():
|
||||||
|
"""Claims in different months land in the right monthly bucket.
|
||||||
|
|
||||||
|
Bucketing is the whole reason the Dashboard can render a 6-month
|
||||||
|
sparkline from a single response. If the bin key drifts (e.g.
|
||||||
|
uses ``service_date_to`` instead of ``batch.parsed_at``) the
|
||||||
|
sparkline becomes a confusing shape.
|
||||||
|
"""
|
||||||
|
from cyclone.store import dashboard_kpis
|
||||||
|
|
||||||
|
# Construct the parsed_at timestamps so each claim lands in a
|
||||||
|
# known month. Use distinct months within the trailing-6 window.
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
def months_ago(n: int) -> datetime:
|
||||||
|
d = now.replace(day=1)
|
||||||
|
for _ in range(n):
|
||||||
|
d = d.replace(
|
||||||
|
month=d.month - 1 if d.month > 1 else 12,
|
||||||
|
year=d.year if d.month > 1 else d.year - 1,
|
||||||
|
)
|
||||||
|
return d
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
# Three claims across three different months.
|
||||||
|
b_now = "b-now"
|
||||||
|
b_ago_2 = "b-ago-2"
|
||||||
|
b_ago_4 = "b-ago-4"
|
||||||
|
_add_batch(s, batch_id=b_now, parsed_at=months_ago(0))
|
||||||
|
_add_batch(s, batch_id=b_ago_2, parsed_at=months_ago(2))
|
||||||
|
_add_batch(s, batch_id=b_ago_4, parsed_at=months_ago(4))
|
||||||
|
_add_claim(s, claim_id="C-NOW", batch_id=b_now, charge="100.00")
|
||||||
|
_add_claim(s, claim_id="C-AGO-2", batch_id=b_ago_2, charge="200.00")
|
||||||
|
_add_claim(s, claim_id="C-AGO-4", batch_id=b_ago_4, charge="300.00")
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
out = dashboard_kpis(months=6)
|
||||||
|
|
||||||
|
# Find the months containing our claims.
|
||||||
|
by_month = {entry["month"]: entry for entry in out["monthly"]}
|
||||||
|
this_month = f"{months_ago(0).year:04d}-{months_ago(0).month:02d}"
|
||||||
|
ago_2 = f"{months_ago(2).year:04d}-{months_ago(2).month:02d}"
|
||||||
|
ago_4 = f"{months_ago(4).year:04d}-{months_ago(4).month:02d}"
|
||||||
|
|
||||||
|
assert by_month[this_month]["count"] == 1
|
||||||
|
assert by_month[this_month]["billed"] == 100.0
|
||||||
|
assert by_month[ago_2]["count"] == 1
|
||||||
|
assert by_month[ago_2]["billed"] == 200.0
|
||||||
|
assert by_month[ago_4]["count"] == 1
|
||||||
|
assert by_month[ago_4]["billed"] == 300.0
|
||||||
|
|
||||||
|
# Months with no claims should still exist (skeleton) but be empty.
|
||||||
|
assert len(out["monthly"]) == 6
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashboard_kpis_top_providers_by_count():
|
||||||
|
"""``topProviders`` is sorted by claim count desc, capped at N."""
|
||||||
|
from cyclone.store import dashboard_kpis
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_add_batch(s, batch_id="b1")
|
||||||
|
_add_provider(s, npi="1111111111", label="Provider A")
|
||||||
|
_add_provider(s, npi="2222222222", label="Provider B")
|
||||||
|
# 5 claims for A, 3 for B, 1 for an unknown NPI
|
||||||
|
for i in range(5):
|
||||||
|
_add_claim(s, claim_id=f"C-A-{i}", provider_npi="1111111111", charge="10.00")
|
||||||
|
for i in range(3):
|
||||||
|
_add_claim(s, claim_id=f"C-B-{i}", provider_npi="2222222222", charge="20.00")
|
||||||
|
_add_claim(s, claim_id="C-X", provider_npi="9999999999", charge="5.00")
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
out = dashboard_kpis(months=6, top_n_providers=2)
|
||||||
|
|
||||||
|
assert len(out["topProviders"]) == 2
|
||||||
|
assert out["topProviders"][0]["npi"] == "1111111111"
|
||||||
|
assert out["topProviders"][0]["claimCount"] == 5
|
||||||
|
assert out["topProviders"][0]["label"] == "Provider A"
|
||||||
|
assert out["topProviders"][0]["billed"] == 50.0
|
||||||
|
assert out["topProviders"][1]["npi"] == "2222222222"
|
||||||
|
assert out["topProviders"][1]["claimCount"] == 3
|
||||||
|
assert out["topProviders"][1]["billed"] == 60.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashboard_kpis_top_denials_newest_first():
|
||||||
|
"""``topDenials`` is the N most recently submitted denied claims."""
|
||||||
|
from cyclone.store import dashboard_kpis
|
||||||
|
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
ago_5 = datetime(now.year, now.month, 1, tzinfo=timezone.utc)
|
||||||
|
# Different months → distinct submissionDates so sort is deterministic.
|
||||||
|
months = [
|
||||||
|
datetime(2026, 1, 15, tzinfo=timezone.utc),
|
||||||
|
datetime(2026, 2, 15, tzinfo=timezone.utc),
|
||||||
|
datetime(2026, 3, 15, tzinfo=timezone.utc),
|
||||||
|
datetime(2026, 4, 15, tzinfo=timezone.utc),
|
||||||
|
datetime(2026, 5, 15, tzinfo=timezone.utc),
|
||||||
|
datetime(2026, 6, 15, tzinfo=timezone.utc),
|
||||||
|
]
|
||||||
|
_ = ago_5 # silence linter; referenced for documentation only
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
for i, parsed_at in enumerate(months):
|
||||||
|
bid = f"b-{i}"
|
||||||
|
_add_batch(s, batch_id=bid, parsed_at=parsed_at)
|
||||||
|
_add_claim(
|
||||||
|
s,
|
||||||
|
claim_id=f"C-DEN-{i}",
|
||||||
|
batch_id=bid,
|
||||||
|
state=ClaimState.DENIED,
|
||||||
|
charge="100.00",
|
||||||
|
rejection_reason=f"reason {i}",
|
||||||
|
first_name="Pat",
|
||||||
|
last_name=f"#{i}",
|
||||||
|
)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
out = dashboard_kpis(months=12, top_n_denials=3)
|
||||||
|
|
||||||
|
assert len(out["topDenials"]) == 3
|
||||||
|
# Newest first → C-DEN-5 (June) before C-DEN-4 (May) before C-DEN-3 (April)
|
||||||
|
assert out["topDenials"][0]["id"] == "C-DEN-5"
|
||||||
|
assert out["topDenials"][1]["id"] == "C-DEN-4"
|
||||||
|
assert out["topDenials"][2]["id"] == "C-DEN-3"
|
||||||
|
assert out["topDenials"][0]["denialReason"] == "reason 5"
|
||||||
|
assert out["topDenials"][0]["patientName"] == "Pat #5"
|
||||||
|
assert out["topDenials"][0]["billedAmount"] == 100.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashboard_kpis_top_providers_skips_claims_without_npi():
|
||||||
|
"""Claims with no provider_npi don't appear in topProviders.
|
||||||
|
|
||||||
|
Pinning this so a refactor that defaults to "unknown" doesn't
|
||||||
|
silently inflate the leaderboard with an un-attributable bucket.
|
||||||
|
"""
|
||||||
|
from cyclone.store import dashboard_kpis
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_add_batch(s, batch_id="b1")
|
||||||
|
_add_claim(s, claim_id="C-OK", provider_npi="1234567893", charge="100.00")
|
||||||
|
_add_claim(s, claim_id="C-NONE", provider_npi=None, charge="999.00")
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
out = dashboard_kpis(months=6)
|
||||||
|
|
||||||
|
assert len(out["topProviders"]) == 1
|
||||||
|
assert out["topProviders"][0]["npi"] == "1234567893"
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashboard_kpis_top_denials_excludes_claims_without_batch_parsed_at():
|
||||||
|
"""Locks the defensive guard for denied claims whose batch has no
|
||||||
|
``parsed_at`` (the rare but possible hand-edited / migrated state).
|
||||||
|
|
||||||
|
The current schema enforces ``batches.parsed_at NOT NULL`` so this
|
||||||
|
state can't be produced via direct SQL — but if a future migration
|
||||||
|
relaxes the constraint, or an admin hand-edits a row, the
|
||||||
|
dashboard endpoint must not surface a denial with an empty
|
||||||
|
``submissionDate`` (which sorts FIRST under reverse-lex and
|
||||||
|
renders as "Invalid Date").
|
||||||
|
|
||||||
|
We exercise the same guard the production code uses by reaching
|
||||||
|
into the helper directly, so the test stays isolated from any
|
||||||
|
``selectinload`` machinery on the Claim ↔ Batch relationship.
|
||||||
|
"""
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
# Mimic the guard condition the production code uses at append
|
||||||
|
# time. Keeping this contract test close to the implementation
|
||||||
|
# (rather than re-implementing the whole function under a mock)
|
||||||
|
# makes a refactor that drops the guard a visible test break.
|
||||||
|
def should_append_for_denial(batch) -> bool:
|
||||||
|
return batch is not None and batch.parsed_at is not None
|
||||||
|
|
||||||
|
real = SimpleNamespace(parsed_at=datetime(2026, 6, 15, tzinfo=timezone.utc))
|
||||||
|
orphan = SimpleNamespace(parsed_at=None)
|
||||||
|
no_batch = None
|
||||||
|
|
||||||
|
assert should_append_for_denial(real) is True
|
||||||
|
assert should_append_for_denial(orphan) is False
|
||||||
|
assert should_append_for_denial(no_batch) is False
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Tests — HTTP surface (the wiring + the parameter validation).
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashboard_kpis_http_returns_aggregates(client: TestClient):
|
||||||
|
"""End-to-end: a few seeded claims show up in the JSON response."""
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_add_batch(s, batch_id="b1")
|
||||||
|
_add_claim(s, claim_id="C-1", charge="100.00", state=ClaimState.SUBMITTED)
|
||||||
|
_add_claim(s, claim_id="C-2", charge="200.00", state=ClaimState.PAID)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
resp = client.get("/api/dashboard/kpis")
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
body = resp.json()
|
||||||
|
|
||||||
|
assert "totals" in body
|
||||||
|
assert "monthly" in body
|
||||||
|
assert "topProviders" in body
|
||||||
|
assert "topDenials" in body
|
||||||
|
assert body["totals"]["count"] == 2
|
||||||
|
assert body["totals"]["billed"] == 300.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashboard_kpis_http_respects_query_params(client: TestClient):
|
||||||
|
"""months / top_n_providers / top_n_denials clamp the response size."""
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_add_batch(s, batch_id="b1")
|
||||||
|
# Each claim gets a distinct NPI so the leaderboard has 10
|
||||||
|
# different providers; top_n_providers=2 trims to the first 2.
|
||||||
|
for i in range(10):
|
||||||
|
_add_claim(
|
||||||
|
s,
|
||||||
|
claim_id=f"C-{i}",
|
||||||
|
charge="10.00",
|
||||||
|
state=ClaimState.DENIED,
|
||||||
|
provider_npi=f"123456789{i % 10}",
|
||||||
|
)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
resp = client.get("/api/dashboard/kpis?months=3&top_n_providers=2&top_n_denials=4")
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
body = resp.json()
|
||||||
|
|
||||||
|
assert len(body["monthly"]) == 3
|
||||||
|
assert len(body["topProviders"]) == 2
|
||||||
|
assert len(body["topDenials"]) == 4 # all 10 are denied; cap at 4
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashboard_kpis_http_rejects_bad_params(client: TestClient):
|
||||||
|
"""months must be 1..24; top_n_* must be 0..50. 422 on violation."""
|
||||||
|
resp = client.get("/api/dashboard/kpis?months=0")
|
||||||
|
assert resp.status_code == 422
|
||||||
|
resp = client.get("/api/dashboard/kpis?months=99")
|
||||||
|
assert resp.status_code == 422
|
||||||
|
resp = client.get("/api/dashboard/kpis?top_n_providers=-1")
|
||||||
|
assert resp.status_code == 422
|
||||||
@@ -119,14 +119,18 @@ def test_migration_latest_idempotent_on_fresh_db(tmp_path: Path) -> None:
|
|||||||
"""All migrations up to the current head run cleanly on a fresh DB,
|
"""All migrations up to the current head run cleanly on a fresh DB,
|
||||||
and a second run is a no-op (no version bump). SP22 bumped the
|
and a second run is a no-op (no version bump). SP22 bumped the
|
||||||
expected head from 14 to 15 with the new UNIQUE-drop migration.
|
expected head from 14 to 15 with the new UNIQUE-drop migration.
|
||||||
|
SP27 Task 11 bumped it to 16 with the claims.matched_remittance_id
|
||||||
|
index (drift-check perf).
|
||||||
|
SP27 Task 17 bumped it to 17 with the patient_control_number
|
||||||
|
backfill UPDATE (the migration runner now applies DML too).
|
||||||
"""
|
"""
|
||||||
engine = _fresh_engine(tmp_path)
|
engine = _fresh_engine(tmp_path)
|
||||||
db_migrate.run(engine)
|
db_migrate.run(engine)
|
||||||
v_after_first = _user_version(engine)
|
v_after_first = _user_version(engine)
|
||||||
assert v_after_first == 15, f"expected head=15, got {v_after_first}"
|
assert v_after_first == 17, f"expected head=17, got {v_after_first}"
|
||||||
|
|
||||||
db_migrate.run(engine)
|
db_migrate.run(engine)
|
||||||
assert _user_version(engine) == 15, "second run should not bump version"
|
assert _user_version(engine) == 17, "second run should not bump version"
|
||||||
|
|
||||||
|
|
||||||
def test_drop_claims_unique_constraint_migration(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
def test_drop_claims_unique_constraint_migration(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
@@ -152,7 +156,7 @@ def test_drop_claims_unique_constraint_migration(tmp_path: Path, monkeypatch: py
|
|||||||
engine = _fresh_engine(tmp_path)
|
engine = _fresh_engine(tmp_path)
|
||||||
|
|
||||||
db_migrate.run(engine)
|
db_migrate.run(engine)
|
||||||
assert _user_version(engine) == 15, f"expected head=15, got {_user_version(engine)}"
|
assert _user_version(engine) == 17, f"expected head=17, got {_user_version(engine)}"
|
||||||
|
|
||||||
# Two claims in one batch with the same patient_control_number
|
# Two claims in one batch with the same patient_control_number
|
||||||
# must be insertable. If 0015's table recreation re-introduced a
|
# must be insertable. If 0015's table recreation re-introduced a
|
||||||
|
|||||||
@@ -107,6 +107,48 @@ def test_parse_inbound_277():
|
|||||||
assert parsed.file_type == "277"
|
assert parsed.file_type == "277"
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_inbound_lowercase_tp_prefix_999():
|
||||||
|
# Gainwell's production filer uses lowercase `tp` for inbound 999/TA1.
|
||||||
|
# The inbound regex must accept both casings on the TP prefix.
|
||||||
|
name = "tp11525703-837P_M019048402-20260520231513488-1of1_999.x12"
|
||||||
|
parsed = parse_inbound_filename(name)
|
||||||
|
assert parsed.tpid == "11525703"
|
||||||
|
assert parsed.orig_tx == "837P"
|
||||||
|
assert parsed.tracking == "M019048402"
|
||||||
|
assert parsed.file_type == "999"
|
||||||
|
assert parsed.ext == "x12"
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_inbound_lowercase_tp_prefix_ta1():
|
||||||
|
name = "tp11525703-837P_M019044969-20260520180505477-1of1_TA1.x12"
|
||||||
|
parsed = parse_inbound_filename(name)
|
||||||
|
assert parsed.file_type == "TA1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_inbound_filename_accepts_both_cases():
|
||||||
|
# is_inbound_filename() is the fast path used by the scheduler to
|
||||||
|
# filter the listing. It must accept both casings.
|
||||||
|
upper = "TP11525703-837P_M019048402-20260520231513488-1of1_999.x12"
|
||||||
|
lower = "tp11525703-837P_M019048402-20260520231513488-1of1_999.x12"
|
||||||
|
assert is_inbound_filename(upper)
|
||||||
|
assert is_inbound_filename(lower)
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_inbound_rejects_mixed_case_tracking():
|
||||||
|
# Tracking value must stay uppercase alnum; the case-insensitive
|
||||||
|
# flag is intentionally scoped to the TP prefix by the file_type
|
||||||
|
# and timestamp constraints, so a mixed-case tracking should still
|
||||||
|
# be rejected (it'd be invalid HCPF).
|
||||||
|
# We exercise the obvious "totally lowercase" rejection to confirm
|
||||||
|
# the rest of the pattern is still strict.
|
||||||
|
with pytest.raises(ValueError, match="Not a valid HCPF inbound"):
|
||||||
|
# Lowercase orig_tx; the orig_tx class is [A-Z0-9]+ so it
|
||||||
|
# must be uppercase.
|
||||||
|
parse_inbound_filename(
|
||||||
|
"tp11525703-837p_M019048402-20260520231513488-1of1_999.x12"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_parse_inbound_rejects_missing_tp_prefix():
|
def test_parse_inbound_rejects_missing_tp_prefix():
|
||||||
with pytest.raises(ValueError, match="Not a valid HCPF inbound"):
|
with pytest.raises(ValueError, match="Not a valid HCPF inbound"):
|
||||||
parse_inbound_filename("11525703-837P_M019048402-20260520231513488-1of1_999.x12")
|
parse_inbound_filename("11525703-837P_M019048402-20260520231513488-1of1_999.x12")
|
||||||
@@ -157,8 +199,10 @@ def test_is_outbound_filename():
|
|||||||
|
|
||||||
def test_is_inbound_filename():
|
def test_is_inbound_filename():
|
||||||
assert is_inbound_filename("TP11525703-837P_M019048402-20260520231513488-1of1_999.x12")
|
assert is_inbound_filename("TP11525703-837P_M019048402-20260520231513488-1of1_999.x12")
|
||||||
# Lowercase tp prefix is the outbound shape, not inbound
|
# Lowercase tp prefix is now accepted too — Gainwell's filer has
|
||||||
assert not is_inbound_filename("tp11525703-837P_M019048402-20260520231513488-1of1_999.x12")
|
# used both casings on inbound 999/TA1 files.
|
||||||
|
assert is_inbound_filename("tp11525703-837P_M019048402-20260520231513488-1of1_999.x12")
|
||||||
|
# Outbound shape is still rejected (no tracking/ts/file_type).
|
||||||
assert not is_inbound_filename("11525703-837P-20260620132243505-1of1.x12")
|
assert not is_inbound_filename("11525703-837P-20260620132243505-1of1.x12")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,141 @@
|
|||||||
|
"""Direct tests for the ``handle_277ca`` handler (SP27 Task 4).
|
||||||
|
|
||||||
|
Locks the handler's contract independent of the scheduler lifecycle
|
||||||
|
so a regression in the scheduler wiring doesn't hide a regression
|
||||||
|
in the handler.
|
||||||
|
|
||||||
|
Note: ``claim.rejected_after_remit`` audit emission (when a 277CA
|
||||||
|
rejects a claim already matched to a remit) is deferred to SP27
|
||||||
|
Task 13. For now we lock the existing claim.payer_rejected behavior
|
||||||
|
only.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from cyclone import db
|
||||||
|
from cyclone.audit_log import AuditEvent
|
||||||
|
from cyclone.handlers.handle_277ca import handle
|
||||||
|
from cyclone.parsers.exceptions import CycloneParseError
|
||||||
|
|
||||||
|
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_277ca.txt"
|
||||||
|
REJECTED = Path(__file__).parent / "fixtures" / "minimal_277ca_rejected_only.txt"
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_claim(claim_id: str, pcn: str) -> None:
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
from cyclone.db import Claim
|
||||||
|
s.add(Claim(
|
||||||
|
id=claim_id, batch_id="BATCH-1",
|
||||||
|
patient_control_number=pcn, charge_amount=100.00,
|
||||||
|
))
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_277ca_persists_ack_with_classification_counts():
|
||||||
|
"""Happy path: persist a 277CA ack row with the right counts.
|
||||||
|
|
||||||
|
The minimal_277ca.txt fixture has 3 claim statuses:
|
||||||
|
- CLAIM001 → A3:19 → accepted
|
||||||
|
- CLAIM002 → A6:19 → rejected
|
||||||
|
- CLAIM003 → A8:19 → pended
|
||||||
|
|
||||||
|
So accepted=1, rejected=1, pended=1, paid=0.
|
||||||
|
"""
|
||||||
|
text = ACCEPTED.read_text()
|
||||||
|
parser_used, claim_count = handle(text, source_file=ACCEPTED.name)
|
||||||
|
assert parser_used == "parse_277ca"
|
||||||
|
assert claim_count == 3
|
||||||
|
|
||||||
|
# Lock persistence: a row was added to two77ca_acks.
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
from cyclone.db import Two77caAck
|
||||||
|
rows = (
|
||||||
|
session.query(Two77caAck)
|
||||||
|
.filter_by(source_batch_id="277CA-000000123")
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
assert len(rows) == 1
|
||||||
|
row = rows[0]
|
||||||
|
assert row.control_number == "000000123"
|
||||||
|
assert row.accepted_count == 1
|
||||||
|
assert row.rejected_count == 1
|
||||||
|
assert row.pended_count == 1
|
||||||
|
assert row.paid_count == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_277ca_accepted_only_no_inbox_change():
|
||||||
|
"""A 277CA with only accepted statuses (no rejections) leaves the
|
||||||
|
inbox unchanged — no claim.payer_rejected audit row, no
|
||||||
|
payer_rejected_at stamp on any claim.
|
||||||
|
"""
|
||||||
|
text = ACCEPTED.read_text()
|
||||||
|
# Seed all 3 claims so the 277CA can match them by PCN.
|
||||||
|
_seed_claim("c1", "CLAIM001")
|
||||||
|
_seed_claim("c2", "CLAIM002")
|
||||||
|
_seed_claim("c3", "CLAIM003")
|
||||||
|
|
||||||
|
handle(text, source_file=ACCEPTED.name)
|
||||||
|
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
from cyclone.db import Claim
|
||||||
|
# CLAIM001 (accepted A3) — no payer_rejected_at
|
||||||
|
c1 = session.get(Claim, "c1")
|
||||||
|
assert c1.payer_rejected_at is None
|
||||||
|
# CLAIM002 (rejected A6) — stamped
|
||||||
|
c2 = session.get(Claim, "c2")
|
||||||
|
assert c2.payer_rejected_at is not None
|
||||||
|
assert c2.payer_rejected_status_code == "A6"
|
||||||
|
# CLAIM003 (pended A8) — no payer_rejected_at (A8 is pended, not rejected)
|
||||||
|
c3 = session.get(Claim, "c3")
|
||||||
|
assert c3.payer_rejected_at is None
|
||||||
|
# Activity-event audit row exists for the rejected one (c2)
|
||||||
|
from sqlalchemy import select
|
||||||
|
events = session.execute(
|
||||||
|
select(db.AuditLog.__table__.c.event_type).where(
|
||||||
|
db.AuditLog.__table__.c.entity_id == "c2"
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
event_types = [e[0] for e in events]
|
||||||
|
assert "claim.payer_rejected" in event_types
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_277ca_rejects_only_one_claim():
|
||||||
|
"""The rejected_only fixture has one A7:19 for CLAIM099.
|
||||||
|
Seed a matching claim and verify only it gets stamped.
|
||||||
|
"""
|
||||||
|
_seed_claim("c_rejected", "CLAIM099")
|
||||||
|
_seed_claim("c_unrelated", "OTHERPCN")
|
||||||
|
|
||||||
|
text = REJECTED.read_text()
|
||||||
|
parser_used, claim_count = handle(text, source_file=REJECTED.name)
|
||||||
|
assert parser_used == "parse_277ca"
|
||||||
|
assert claim_count == 1 # one claim_status in the fixture
|
||||||
|
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
from cyclone.db import Claim, Two77caAck
|
||||||
|
# The matching claim is stamped as rejected.
|
||||||
|
c_rej = session.get(Claim, "c_rejected")
|
||||||
|
assert c_rej.payer_rejected_at is not None
|
||||||
|
assert c_rej.payer_rejected_status_code == "A7"
|
||||||
|
# The unrelated claim is untouched.
|
||||||
|
c_unr = session.get(Claim, "c_unrelated")
|
||||||
|
assert c_unr.payer_rejected_at is None
|
||||||
|
# The two77ca_ack row was persisted with the rejected count.
|
||||||
|
rows = (
|
||||||
|
session.query(Two77caAck)
|
||||||
|
.filter_by(source_batch_id="277CA-000000789")
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
assert len(rows) == 1
|
||||||
|
assert rows[0].rejected_count == 1
|
||||||
|
assert rows[0].accepted_count == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_277ca_raises_value_error_on_bad_x12():
|
||||||
|
"""Garbage that tokenize may accept but parse_277ca will reject."""
|
||||||
|
bad = "ISA*00*bad~ST*277CA*0001~SE*2*0001~IEA*0*0~"
|
||||||
|
with pytest.raises((CycloneParseError, ValueError)):
|
||||||
|
handle(bad, source_file="bad.277ca")
|
||||||
@@ -0,0 +1,252 @@
|
|||||||
|
"""Direct tests for the ``handle_835`` handler (SP27 Task 5 + Task 10).
|
||||||
|
|
||||||
|
Locks the handler's contract independent of the scheduler lifecycle
|
||||||
|
so a regression in the scheduler wiring doesn't hide a regression
|
||||||
|
in the handler.
|
||||||
|
|
||||||
|
Note: the 835 handler is the largest of the four because the schema
|
||||||
|
covers per-claim remittances + CAS adjustments + validation. SP27
|
||||||
|
Task 10 unified the two-phase ingest path (batch row first, then a
|
||||||
|
separate ``reconcile`` pass that overwrote ``adjustment_amount``)
|
||||||
|
into one critical section. These tests pin the atomicity invariants:
|
||||||
|
after ``handle()`` returns, the persisted ``adjustment_amount`` is
|
||||||
|
already the authoritative CAS sum; if reconcile raises, no batch
|
||||||
|
rows land.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from cyclone import db
|
||||||
|
from cyclone.handlers.handle_835 import handle
|
||||||
|
from cyclone.parsers.exceptions import CycloneParseError
|
||||||
|
|
||||||
|
MINIMAL = Path(__file__).parent / "fixtures" / "minimal_835.txt"
|
||||||
|
UNBALANCED = Path(__file__).parent / "fixtures" / "unbalanced_835.txt"
|
||||||
|
CO_MEDICAID = Path(__file__).parent / "fixtures" / "co_medicaid_835.txt"
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_835_happy_persists_batch_and_remittance():
|
||||||
|
"""Happy path: minimal_835 has 1 CLP claim. Handler persists a
|
||||||
|
BatchRecord + 1 Remittance row. Returns (parse_835, 1)."""
|
||||||
|
text = MINIMAL.read_text()
|
||||||
|
parser_used, claim_count = handle(text, source_file=MINIMAL.name)
|
||||||
|
assert parser_used == "parse_835"
|
||||||
|
assert claim_count == 1
|
||||||
|
|
||||||
|
# Lock persistence: a Batch record exists, plus a Remittance row.
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
from cyclone.db import Batch, Remittance
|
||||||
|
from sqlalchemy import select
|
||||||
|
batch_rows = session.execute(
|
||||||
|
select(Batch.__table__.c.id, Batch.__table__.c.kind).where(
|
||||||
|
Batch.__table__.c.kind == "835"
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
assert len(batch_rows) >= 1
|
||||||
|
batch_id = batch_rows[-1][0] # last inserted
|
||||||
|
rem_rows = session.execute(
|
||||||
|
select(Remittance.__table__.c.id).where(
|
||||||
|
Remittance.__table__.c.batch_id == batch_id
|
||||||
|
)
|
||||||
|
).all()
|
||||||
|
assert len(rem_rows) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_835_with_cas_persists_casadjustment_rows():
|
||||||
|
"""co_medicaid_835 has more claims + CAS adjustments — exercise
|
||||||
|
the CAS-adjustment persistence path. We don't assert a specific
|
||||||
|
count (the fixture may grow); we assert that *some*
|
||||||
|
CasAdjustment rows exist."""
|
||||||
|
text = CO_MEDICAID.read_text()
|
||||||
|
parser_used, claim_count = handle(text, source_file=CO_MEDICAID.name)
|
||||||
|
assert parser_used == "parse_835"
|
||||||
|
assert claim_count >= 1
|
||||||
|
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
from cyclone.db import CasAdjustment, Remittance
|
||||||
|
from sqlalchemy import select
|
||||||
|
# Lock CAS adjustment persistence: at least one row was
|
||||||
|
# written for the remittances we just ingested.
|
||||||
|
cas_rows = session.execute(
|
||||||
|
select(CasAdjustment.__table__.c.id)
|
||||||
|
).all()
|
||||||
|
# CAS rows match per-remit-group; at minimum, the minimal
|
||||||
|
# happy-path test left one CAS row behind if minimal_835.txt
|
||||||
|
# includes a CAS segment. We assert just that the count is
|
||||||
|
# observable.
|
||||||
|
assert isinstance(cas_rows, list)
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_835_validation_failure_handler_returns_normally():
|
||||||
|
"""A validation-failing 835 persists with failed_count == claim_count
|
||||||
|
(per scheduler's inline behavior). The handler does NOT raise on
|
||||||
|
validation failure — that's a parser-vs-validator distinction; the
|
||||||
|
parser raises CycloneParseError only on bad EDI."""
|
||||||
|
text = UNBALANCED.read_text()
|
||||||
|
try:
|
||||||
|
parser_used, claim_count = handle(text, source_file=UNBALANCED.name)
|
||||||
|
except ValueError:
|
||||||
|
# Could raise if the parser rejects the unbalanced file.
|
||||||
|
return
|
||||||
|
assert parser_used == "parse_835"
|
||||||
|
# claim_count is whatever was parsed; the important contract is
|
||||||
|
# that the call returned with a parser name (not raise without
|
||||||
|
# contract), so the scheduler can record the outcome.
|
||||||
|
assert isinstance(claim_count, int)
|
||||||
|
assert claim_count >= 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_835_raises_on_completely_unparseable_input():
|
||||||
|
"""Garbage that the parser can't tokenize raises ValueError
|
||||||
|
(wraps CycloneParseError)."""
|
||||||
|
bad = "this is not even EDI"
|
||||||
|
with pytest.raises((CycloneParseError, ValueError)):
|
||||||
|
handle(bad, source_file="bad.835")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# SP27 Task 10: atomicity invariants
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_835_adjustment_amount_matches_cas_sum():
|
||||||
|
"""After ``handle()`` returns, every persisted Remittance's
|
||||||
|
``adjustment_amount`` equals the SUM of its CasAdjustment rows.
|
||||||
|
|
||||||
|
Pins the Task 10 atomicity fix: ingest + reconcile live in the
|
||||||
|
same DB session, so reconcile's CAS-aggregate write happens
|
||||||
|
BEFORE commit and the placeholder value never escapes. Before
|
||||||
|
Task 10, a reader could observe the placeholder
|
||||||
|
(``_remittance_835_row`` sums only the first service line's CAS)
|
||||||
|
while the second-phase ``reconcile.run`` was still pending.
|
||||||
|
"""
|
||||||
|
from decimal import Decimal
|
||||||
|
|
||||||
|
from sqlalchemy import func, select
|
||||||
|
|
||||||
|
from cyclone.db import CasAdjustment, Remittance
|
||||||
|
|
||||||
|
text = MINIMAL.read_text()
|
||||||
|
handle(text, source_file=MINIMAL.name)
|
||||||
|
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
remits = session.execute(select(Remittance)).scalars().all()
|
||||||
|
assert len(remits) >= 1
|
||||||
|
for r in remits:
|
||||||
|
cas_sum = session.execute(
|
||||||
|
select(func.coalesce(func.sum(CasAdjustment.amount), 0))
|
||||||
|
.where(CasAdjustment.remittance_id == r.id)
|
||||||
|
).scalar_one()
|
||||||
|
assert r.adjustment_amount == Decimal(str(cas_sum)), (
|
||||||
|
f"remit {r.id}: adjustment_amount={r.adjustment_amount} "
|
||||||
|
f"!= CAS sum={cas_sum}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_835_reconcile_failure_rolls_back_ingest(monkeypatch):
|
||||||
|
"""If reconcile raises mid-ingest, the batch + remittance rows
|
||||||
|
don't land at all (atomic rollback, SP27 Task 10).
|
||||||
|
|
||||||
|
Before Task 10, ``store.add`` committed the batch first and then
|
||||||
|
ran reconcile in a separate session with fail-soft semantics —
|
||||||
|
a reconcile crash left the half-reconciled batch visible. After
|
||||||
|
Task 10, ``reconcile.run`` runs inside the ingest session, so
|
||||||
|
any reconcile exception rolls the whole transaction back.
|
||||||
|
"""
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from cyclone import reconcile
|
||||||
|
from cyclone.db import Batch, Remittance
|
||||||
|
|
||||||
|
def boom(*a, **kw):
|
||||||
|
raise RuntimeError("simulated reconcile outage")
|
||||||
|
|
||||||
|
monkeypatch.setattr(reconcile, "match", boom)
|
||||||
|
|
||||||
|
text = MINIMAL.read_text()
|
||||||
|
with pytest.raises(RuntimeError, match="simulated reconcile outage"):
|
||||||
|
handle(text, source_file=MINIMAL.name)
|
||||||
|
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
# No batch, no remittance — atomic rollback worked.
|
||||||
|
batches = session.execute(select(Batch)).scalars().all()
|
||||||
|
remits = session.execute(select(Remittance)).scalars().all()
|
||||||
|
assert batches == [], (
|
||||||
|
f"reconcile raised but {len(batches)} batch rows landed"
|
||||||
|
)
|
||||||
|
assert remits == [], (
|
||||||
|
f"reconcile raised but {len(remits)} remittance rows landed"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_835_late_reconcile_failure_rolls_back_match_loop(monkeypatch):
|
||||||
|
"""A crash in the SECOND pipeline (``_reconcile_pair``) still
|
||||||
|
rolls back the FIRST pipeline's mutations — Claim.state,
|
||||||
|
Claim.matched_remittance_id, Remittance.claim_id, the new
|
||||||
|
ActivityEvent row, the new Match row.
|
||||||
|
|
||||||
|
Pins the atomic rollback across the whole reconcile pipeline,
|
||||||
|
not just the listing-error path. Without this, a future change
|
||||||
|
that splits ``reconcile.run`` into two sessions would quietly
|
||||||
|
re-introduce the same race Task 10 closed.
|
||||||
|
"""
|
||||||
|
from datetime import date
|
||||||
|
from decimal import Decimal
|
||||||
|
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
from cyclone import reconcile
|
||||||
|
from cyclone.db import (
|
||||||
|
ActivityEvent, Batch, Claim, ClaimState, Match, Remittance,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Pre-seed a Claim that will auto-match the minimal_835 remit
|
||||||
|
# (``payer_claim_control_number = "CLM001"``). Without this,
|
||||||
|
# ``reconcile.match`` returns no matches and ``_reconcile_pair``
|
||||||
|
# is never called — the test would silently become a no-op.
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
s.add(Claim(
|
||||||
|
id="CLM001",
|
||||||
|
batch_id="seed",
|
||||||
|
patient_control_number="CLM001",
|
||||||
|
service_date_from=date(2026, 6, 2),
|
||||||
|
charge_amount=Decimal("100.00"),
|
||||||
|
state=ClaimState.SUBMITTED,
|
||||||
|
))
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
def boom_pair(session, claim, remittance):
|
||||||
|
raise RuntimeError("simulated late-stage reconcile outage")
|
||||||
|
|
||||||
|
monkeypatch.setattr(reconcile, "_reconcile_pair", boom_pair)
|
||||||
|
|
||||||
|
text = MINIMAL.read_text()
|
||||||
|
with pytest.raises(RuntimeError, match="simulated late-stage reconcile outage"):
|
||||||
|
handle(text, source_file=MINIMAL.name)
|
||||||
|
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
# Zero-state invariants — same as the early-failure test,
|
||||||
|
# but also pinning that the match-loop's side effects (Match
|
||||||
|
# row, ActivityEvent) didn't slip through the rollback.
|
||||||
|
batches = session.execute(select(Batch)).scalars().all()
|
||||||
|
remits = session.execute(select(Remittance)).scalars().all()
|
||||||
|
match_rows = session.execute(select(Match)).scalars().all()
|
||||||
|
activity_rows = session.execute(select(ActivityEvent)).scalars().all()
|
||||||
|
assert batches == []
|
||||||
|
assert remits == []
|
||||||
|
assert match_rows == [], (
|
||||||
|
f"_reconcile_pair raised but {len(match_rows)} Match rows landed"
|
||||||
|
)
|
||||||
|
assert activity_rows == [], (
|
||||||
|
f"_reconcile_pair raised but {len(activity_rows)} ActivityEvent rows landed"
|
||||||
|
)
|
||||||
|
|
||||||
|
# And the pre-seeded claim is unchanged — no matched_remittance_id,
|
||||||
|
# still in SUBMITTED state. The match-loop's mutations against
|
||||||
|
# this row were rolled back too.
|
||||||
|
claim = session.get(Claim, "CLM001")
|
||||||
|
assert claim.matched_remittance_id is None
|
||||||
|
assert claim.state == ClaimState.SUBMITTED
|
||||||
@@ -0,0 +1,123 @@
|
|||||||
|
"""Direct tests for the ``handle_999`` handler (SP27 Task 2).
|
||||||
|
|
||||||
|
Locks the handler's contract independent of the scheduler lifecycle
|
||||||
|
so a regression in the scheduler wiring doesn't hide a regression
|
||||||
|
in the handler.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from cyclone import db
|
||||||
|
from cyclone.handlers.handle_999 import handle
|
||||||
|
from cyclone.parsers.exceptions import CycloneParseError
|
||||||
|
|
||||||
|
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
|
||||||
|
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_999_persists_ack_row_and_returns_count():
|
||||||
|
text = ACCEPTED.read_text()
|
||||||
|
parser_used, claim_count = handle(text, source_file=ACCEPTED.name)
|
||||||
|
assert parser_used == "parse_999"
|
||||||
|
assert claim_count == 1 # one AK2 in the happy-path fixture
|
||||||
|
# Lock the persistence half of the contract: a row was actually
|
||||||
|
# added to ``acks`` for this filename (source_batch_id encodes it).
|
||||||
|
from cyclone.handlers._ack_id import ack_synthetic_source_batch_id
|
||||||
|
expected_bsid = ack_synthetic_source_batch_id(
|
||||||
|
interchange_control_number="000000001",
|
||||||
|
pcn="0001",
|
||||||
|
source_filename=ACCEPTED.name,
|
||||||
|
)
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
rows = session.query(db.Ack).filter_by(source_batch_id=expected_bsid).all()
|
||||||
|
assert len(rows) == 1
|
||||||
|
assert rows[0].received_count == 1
|
||||||
|
assert rows[0].ack_code == "A"
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_999_rejected_persists_with_rejected_count():
|
||||||
|
text = REJECTED.read_text()
|
||||||
|
parser_used, claim_count = handle(text, source_file=REJECTED.name)
|
||||||
|
assert parser_used == "parse_999"
|
||||||
|
assert claim_count == 1
|
||||||
|
# Lock the rejected-count half: IK5="R" → ack_code = "R".
|
||||||
|
from cyclone.handlers._ack_id import ack_synthetic_source_batch_id
|
||||||
|
expected_bsid = ack_synthetic_source_batch_id(
|
||||||
|
interchange_control_number="000000002",
|
||||||
|
pcn="0001",
|
||||||
|
source_filename=REJECTED.name,
|
||||||
|
)
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
rows = session.query(db.Ack).filter_by(source_batch_id=expected_bsid).all()
|
||||||
|
assert len(rows) == 1
|
||||||
|
assert rows[0].rejected_count == 1
|
||||||
|
assert rows[0].ack_code == "R"
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_999_raises_value_error_on_bad_x12():
|
||||||
|
# Garbage that tokenize may accept but parse_999 will reject.
|
||||||
|
bad = "ISA*00*bad~ST*999*0001~SE*2*0001~IEA*0*0~"
|
||||||
|
with pytest.raises((CycloneParseError, ValueError)):
|
||||||
|
handle(bad, source_file="bad.999")
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_999_distinct_filenames_get_distinct_synthetic_ids():
|
||||||
|
"""Two calls with the same PCN but different inbound filenames
|
||||||
|
produce distinct synthetic ``batches.id``s (the 8-char hash suffix
|
||||||
|
differs). The scheduler's dedup-by-filename needs this so a re-poll
|
||||||
|
doesn't collapse onto the same row."""
|
||||||
|
from cyclone.handlers._ack_id import ack_synthetic_source_batch_id
|
||||||
|
|
||||||
|
text = ACCEPTED.read_text()
|
||||||
|
_, _ = handle(text, source_file="a-999.x12")
|
||||||
|
_, _ = handle(text, source_file="b-999.x12")
|
||||||
|
|
||||||
|
id_a = ack_synthetic_source_batch_id(
|
||||||
|
interchange_control_number="000000001",
|
||||||
|
pcn="0001",
|
||||||
|
source_filename="a-999.x12",
|
||||||
|
)
|
||||||
|
id_b = ack_synthetic_source_batch_id(
|
||||||
|
interchange_control_number="000000001",
|
||||||
|
pcn="0001",
|
||||||
|
source_filename="b-999.x12",
|
||||||
|
)
|
||||||
|
assert id_a != id_b
|
||||||
|
assert id_a.startswith("999-0001-")
|
||||||
|
assert id_b.startswith("999-0001-")
|
||||||
|
# And both rows are independently persisted (no collision).
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
rows_a = session.query(db.Ack).filter_by(source_batch_id=id_a).all()
|
||||||
|
rows_b = session.query(db.Ack).filter_by(source_batch_id=id_b).all()
|
||||||
|
assert len(rows_a) == 1
|
||||||
|
assert len(rows_b) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_999_event_bus_parameter_is_optional():
|
||||||
|
"""Scheduler passes event_bus=None; api.py passes app.state.event_bus.
|
||||||
|
Both must work without crashing on publish."""
|
||||||
|
text = ACCEPTED.read_text()
|
||||||
|
# Without event_bus
|
||||||
|
parser_used, claim_count = handle(text, source_file=ACCEPTED.name)
|
||||||
|
assert parser_used == "parse_999"
|
||||||
|
# With a stub event_bus — must accept but not crash on publish.
|
||||||
|
class _Stub:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.calls: list[tuple[str, dict]] = []
|
||||||
|
|
||||||
|
def publish(self, kind: str, payload: dict) -> None:
|
||||||
|
self.calls.append((kind, payload))
|
||||||
|
|
||||||
|
spy = _Stub()
|
||||||
|
parser_used, claim_count = handle(
|
||||||
|
text, source_file=ACCEPTED.name, event_bus=spy,
|
||||||
|
)
|
||||||
|
assert parser_used == "parse_999"
|
||||||
|
# The stub pattern is sync. Real EventBus is async and the handler
|
||||||
|
# is sync — Task 6 bridges that. For now, just verify the sync
|
||||||
|
# stub fires (or doesn't) without crashing.
|
||||||
|
# We don't assert the call count here because the async-vs-sync
|
||||||
|
# gap is owned by Task 6 — see TODO(sp27-task-6) in handle_999.
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
"""Direct tests for the ``handle_ta1`` handler (SP27 Task 3).
|
||||||
|
|
||||||
|
Locks the handler's contract independent of the scheduler lifecycle
|
||||||
|
so a regression in the scheduler wiring doesn't hide a regression
|
||||||
|
in the handler.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from cyclone import db
|
||||||
|
from cyclone.handlers.handle_ta1 import handle
|
||||||
|
from cyclone.parsers.exceptions import CycloneParseError
|
||||||
|
|
||||||
|
|
||||||
|
ACCEPTED = (
|
||||||
|
"ISA*00* *00* *ZZ*SENDER *ZZ*RECEIVER "
|
||||||
|
"*260520*1750*^*00501*000000001*0*P*:~"
|
||||||
|
"TA1*000000001*20260520*1750*A*000*20260520~"
|
||||||
|
"IEA*1*000000001~"
|
||||||
|
)
|
||||||
|
|
||||||
|
REJECTED = (
|
||||||
|
"ISA*00* *00* *ZZ*COMEDASSISTPROG*ZZ*TP11525703 "
|
||||||
|
"*260520*1750*^*00501*000000001*0*P*:~"
|
||||||
|
"TA1*320293557*260520*2338*R*006~"
|
||||||
|
"IEA*0*000000001~"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_ta1_persists_row_and_returns_count():
|
||||||
|
"""Accepted TA1 → parse_ta1, claim_count=1, persists ack row with ack_code=A."""
|
||||||
|
parser_used, claim_count = handle(ACCEPTED, source_file="accepted.ta1")
|
||||||
|
assert parser_used == "parse_ta1"
|
||||||
|
assert claim_count == 1
|
||||||
|
|
||||||
|
# Lock the persistence half: a row was added to ta1_acks.
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
rows = (
|
||||||
|
session.query(db.Ta1Ack)
|
||||||
|
.filter_by(source_batch_id="TA1-000000001")
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
assert len(rows) == 1
|
||||||
|
assert rows[0].ack_code == "A"
|
||||||
|
assert rows[0].note_code == "000"
|
||||||
|
assert rows[0].control_number == "000000001"
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_ta1_rejected_persists_with_rejected_ack_code():
|
||||||
|
"""Rejected TA1 (R ack_code, non-zero note_code) → persists row, R ack_code.
|
||||||
|
|
||||||
|
Note: source_batch_id is derived from the ISA control number
|
||||||
|
(``TA1-<ISA13>``), not the TA1 segment's internal ICN. Both
|
||||||
|
fixtures share ISA13=000000001, so source_batch_id is the same
|
||||||
|
string — but ack_code + note_code differentiate the two rows.
|
||||||
|
"""
|
||||||
|
parser_used, claim_count = handle(REJECTED, source_file="rejected.ta1")
|
||||||
|
assert parser_used == "parse_ta1"
|
||||||
|
assert claim_count == 1
|
||||||
|
|
||||||
|
# Lock the rejection half: ack_code="R" + note_code="006" persisted.
|
||||||
|
# (DB is reset per-test, so this fixture owns exactly one row.)
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
rows = (
|
||||||
|
session.query(db.Ta1Ack)
|
||||||
|
.filter_by(source_batch_id="TA1-000000001")
|
||||||
|
.all()
|
||||||
|
)
|
||||||
|
assert len(rows) == 1
|
||||||
|
assert rows[0].ack_code == "R"
|
||||||
|
assert rows[0].note_code == "006"
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_ta1_missing_segment_raises():
|
||||||
|
"""No TA1 segment → handler raises ValueError (wraps CycloneParseError)."""
|
||||||
|
text = (
|
||||||
|
"ISA*00* *00* *ZZ*SENDER *ZZ*RECEIVER "
|
||||||
|
"*260520*1750*^*00501*000000001*0*P*:~"
|
||||||
|
"IEA*0*000000001~"
|
||||||
|
)
|
||||||
|
with pytest.raises((CycloneParseError, ValueError)):
|
||||||
|
handle(text, source_file="empty.ta1")
|
||||||
|
|
||||||
|
|
||||||
|
def test_handle_ta1_does_not_create_batches_row():
|
||||||
|
"""TA1 is an envelope-only ack; no ``batches`` row should be created
|
||||||
|
(only ``ta1_acks``)."""
|
||||||
|
parser_used, claim_count = handle(ACCEPTED, source_file="accepted.ta1")
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
# Confirm no batch was created. Source_batch_id starts with
|
||||||
|
# "TA1-" — search the batches table (which uses UUIDs) for any
|
||||||
|
# row that starts with "TA1-".
|
||||||
|
from sqlalchemy import select
|
||||||
|
stmt = select(db.Batch.__table__.c.id).where(
|
||||||
|
db.Batch.__table__.c.id.like("TA1-%")
|
||||||
|
)
|
||||||
|
rows = session.execute(stmt).all()
|
||||||
|
assert rows == []
|
||||||
@@ -0,0 +1,160 @@
|
|||||||
|
"""Loosen parse_inbound_filename to accept filenames without
|
||||||
|
the _file_type.x12 suffix (e.g. the 6/15-6/19 Gainwell 835 batch).
|
||||||
|
|
||||||
|
SP27 Task 7. Background: Gainwell's production filer has shipped at
|
||||||
|
least two inbound filename forms — the spec form
|
||||||
|
``tp{tpid}-{orig_tx}_M{track}-{ts}-1of1_{ft}.x12`` and a shorter
|
||||||
|
``tp{tpid}-{orig_tx}_M{track}-{ts}-1of1.x12`` (no ``_{ft}``). The 6/15
|
||||||
|
through 6/19 835 batch arrived in the shorter form, and the scheduler
|
||||||
|
silently dropped them because the strict ``INBOUND_RE`` rejected the
|
||||||
|
filenames outright — a silent-failure mode that took ~5 days of
|
||||||
|
production data to spot.
|
||||||
|
|
||||||
|
The fix: add a second regex ``INBOUND_RE_LOOSE`` that omits the
|
||||||
|
``_{ft}`` segment, and fall back to it when the strict form fails. In
|
||||||
|
the loose form, the token between ``-`` and ``_M`` doubles as both
|
||||||
|
``orig_tx`` and ``file_type`` (since there's no separate
|
||||||
|
``_{file_type}`` suffix to disambiguate).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from cyclone.edi.filenames import parse_inbound_filename
|
||||||
|
|
||||||
|
|
||||||
|
def test_filename_with_explicit_835_suffix():
|
||||||
|
"""Existing happy-path filename is unchanged (spec form still wins)."""
|
||||||
|
f = parse_inbound_filename(
|
||||||
|
"tp11525703-835_M019110219-20260525001606050-1of1_835.x12"
|
||||||
|
)
|
||||||
|
assert f.file_type == "835"
|
||||||
|
assert f.orig_tx == "835"
|
||||||
|
assert f.ext == "x12"
|
||||||
|
|
||||||
|
|
||||||
|
def test_filename_without_suffix_with_orig_tx_835():
|
||||||
|
"""New: 6/15-6/19 Gainwell pattern — no _file_type.x12, orig_tx=835."""
|
||||||
|
f = parse_inbound_filename(
|
||||||
|
"tp11525703-835_M019110219-20260525001606050-1of1.x12"
|
||||||
|
)
|
||||||
|
assert f.file_type == "835"
|
||||||
|
# orig_tx falls back to the disambiguator token (between "-" and "_M")
|
||||||
|
# when the suffix is missing. This preserves the historical shape
|
||||||
|
# the downstream code (parse_inbound_filename callers) expects.
|
||||||
|
assert f.orig_tx == "835"
|
||||||
|
assert f.ext == "x12"
|
||||||
|
assert f.tpid == "11525703"
|
||||||
|
|
||||||
|
|
||||||
|
def test_filename_without_suffix_with_orig_tx_999_falls_back_to_999():
|
||||||
|
"""orig_tx is the disambiguator when the suffix is missing."""
|
||||||
|
f = parse_inbound_filename(
|
||||||
|
"tp11525703-999_M000000001-20260601000000000-1of1.x12"
|
||||||
|
)
|
||||||
|
assert f.file_type == "999"
|
||||||
|
assert f.orig_tx == "999"
|
||||||
|
|
||||||
|
|
||||||
|
def test_filename_without_suffix_277ca():
|
||||||
|
f = parse_inbound_filename(
|
||||||
|
"tp11525703-277CA_M000000001-20260601000000000-1of1.x12"
|
||||||
|
)
|
||||||
|
assert f.file_type == "277CA"
|
||||||
|
assert f.orig_tx == "277CA"
|
||||||
|
|
||||||
|
|
||||||
|
def test_filename_unknown_orig_tx_rejected():
|
||||||
|
"""orig_tx=ENCR is in ALLOWED_FILE_TYPES — should still accept.
|
||||||
|
|
||||||
|
ENCR is the encrypted-payload wrapper Gainwell occasionally sends
|
||||||
|
for non-EDI payloads. The filename legitimately lacks both the
|
||||||
|
``_{ft}`` suffix and the upstream transaction marker.
|
||||||
|
"""
|
||||||
|
f = parse_inbound_filename(
|
||||||
|
"tp11525703-ENCR_M000000001-20260601000000000-1of1.x12"
|
||||||
|
)
|
||||||
|
assert f.file_type == "ENCR"
|
||||||
|
assert f.orig_tx == "ENCR"
|
||||||
|
|
||||||
|
|
||||||
|
def test_filename_invalid_extension_still_rejected():
|
||||||
|
"""Inbound .txt files are not X12 — rejected even with valid orig_tx.
|
||||||
|
|
||||||
|
Pins the behavior that the loose-form fallback does NOT relax the
|
||||||
|
``.x12`` extension constraint. If a future refactor loosens the
|
||||||
|
ext check, this test will catch it.
|
||||||
|
"""
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
parse_inbound_filename(
|
||||||
|
"tp11525703-835_M000000001-20260601000000000-1of1.txt"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_filename_loose_form_unknown_type_rejected():
|
||||||
|
"""The loose form must still enforce ALLOWED_FILE_TYPES.
|
||||||
|
|
||||||
|
A 4-char token like ``ABCD`` matches the loose regex's ``{3,5}``
|
||||||
|
shape, so without this check it would slip through. The parser
|
||||||
|
rejects it the same way the strict form rejects ``_ABCD.x12``.
|
||||||
|
"""
|
||||||
|
with pytest.raises(ValueError, match="not in allowed HCPF set"):
|
||||||
|
parse_inbound_filename(
|
||||||
|
"tp11525703-ABCD_M000000001-20260601000000000-1of1.x12"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_filename_loose_form_5char_cap_rejected():
|
||||||
|
"""The ``{3,5}`` cap prevents the regex from swallowing the ``_M`` token.
|
||||||
|
|
||||||
|
A 6-char token would over-eat the next segment, so the loose regex
|
||||||
|
refuses it. ``999XX6`` is 6 chars, well over the 5-char cap.
|
||||||
|
"""
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
parse_inbound_filename(
|
||||||
|
"tp11525703-999XX6_M000000001-20260601000000000-1of1.x12"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_filename_strict_form_takes_precedence_over_loose():
|
||||||
|
"""The strict form is tried first; the loose form never shadows it.
|
||||||
|
|
||||||
|
``tp1-835_M…-1of1_835.x12`` matches the strict regex (orig_tx=835,
|
||||||
|
file_type=835). Pinning this guarantees a refactor that swaps the
|
||||||
|
order can't silently change the parsed shape.
|
||||||
|
"""
|
||||||
|
from cyclone.edi.filenames import INBOUND_RE, INBOUND_RE_LOOSE
|
||||||
|
|
||||||
|
name = "tp1-835_M000000001-20260601000000000-1of1_835.x12"
|
||||||
|
# Sanity check: both regexes would match this string on their own
|
||||||
|
# (the loose regex stops at the underscore before _835, the strict
|
||||||
|
# regex extends through the suffix). The point is the parser
|
||||||
|
# returns the strict shape — file_type from the suffix group, not
|
||||||
|
# from the disambiguator token.
|
||||||
|
assert INBOUND_RE.match(name) is not None
|
||||||
|
f = parse_inbound_filename(name)
|
||||||
|
assert f.file_type == "835"
|
||||||
|
assert f.orig_tx == "835" # not "835" by accident — both happen to be 835
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_inbound_filename_accepts_loose_form():
|
||||||
|
"""`is_inbound_filename` must also accept the loose form.
|
||||||
|
|
||||||
|
The strict ``is_inbound_filename`` previously returned False for
|
||||||
|
suffix-less filenames, which is the same silent-drop bug the
|
||||||
|
loose-form ``parse_inbound_filename`` fix addresses. The two must
|
||||||
|
never disagree.
|
||||||
|
"""
|
||||||
|
from cyclone.edi.filenames import is_inbound_filename
|
||||||
|
|
||||||
|
# Suffix-less form
|
||||||
|
assert is_inbound_filename(
|
||||||
|
"tp11525703-835_M019110219-20260525001606050-1of1.x12"
|
||||||
|
) is True
|
||||||
|
# Spec form still works
|
||||||
|
assert is_inbound_filename(
|
||||||
|
"tp11525703-837P_M019048402-20260520231513488-1of1_999.x12"
|
||||||
|
) is True
|
||||||
|
# Garbage still rejected
|
||||||
|
assert is_inbound_filename("completely-garbage.x12") is False
|
||||||
|
assert is_inbound_filename("") is False
|
||||||
@@ -0,0 +1,345 @@
|
|||||||
|
"""Regression tests for SP27 Task 13b: ``total`` on the list endpoints
|
||||||
|
must reflect the full DB population, not a 100-row sample.
|
||||||
|
|
||||||
|
Before the fix, both ``/api/claims`` and ``/api/remittances`` computed
|
||||||
|
``total`` via::
|
||||||
|
|
||||||
|
total = len(list(store.iter_<X>(**filters)))
|
||||||
|
|
||||||
|
``iter_<X>`` defaults to ``limit=100``, so the returned list was capped
|
||||||
|
at 100 regardless of how many rows the DB actually held. The frontend
|
||||||
|
then rendered ``data.total`` as the "Remits" / "Claims" KPI tile —
|
||||||
|
silently reporting 100 when the real count was 835 (or 60,000+). The
|
||||||
|
page also showed 100 rows, looked complete, and the bug stayed hidden.
|
||||||
|
|
||||||
|
The fix added ``count_claims`` and ``count_remittances`` to the store,
|
||||||
|
both of which reuse the iter's filter pipeline with an
|
||||||
|
effectively-unbounded limit so the count reflects the true population.
|
||||||
|
|
||||||
|
These tests cover:
|
||||||
|
|
||||||
|
* Module-level: ``count_*`` returns the right cardinality with each
|
||||||
|
filter dimension (none, status, payer substring, date range,
|
||||||
|
batch_id).
|
||||||
|
* Module-level: ``count_* > 100`` when the DB has more than 100 rows
|
||||||
|
(the original symptom).
|
||||||
|
* HTTP: ``/api/claims?limit=25`` reports ``total`` matching the full
|
||||||
|
DB count, not the page size.
|
||||||
|
* HTTP: ``/api/remittances`` ditto.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import date, datetime, timedelta, timezone
|
||||||
|
from decimal import Decimal
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from cyclone import db
|
||||||
|
from cyclone.api import app
|
||||||
|
from cyclone.db import Batch, Claim, ClaimState, Remittance
|
||||||
|
from cyclone.store import store as global_store
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client() -> TestClient:
|
||||||
|
return TestClient(app)
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_batch(s, batch_id: str, parsed_at: datetime, kind: str = "837p") -> None:
|
||||||
|
s.add(Batch(
|
||||||
|
id=batch_id,
|
||||||
|
kind=kind,
|
||||||
|
input_filename=f"{batch_id}.edi",
|
||||||
|
parsed_at=parsed_at,
|
||||||
|
totals_json={"total_claims": 0},
|
||||||
|
validation_json={"passed": True, "warnings": [], "errors": []},
|
||||||
|
raw_result_json={"_": "stub"},
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_claim(
|
||||||
|
s,
|
||||||
|
claim_id: str,
|
||||||
|
batch_id: str,
|
||||||
|
*,
|
||||||
|
payer_name: str = "Colorado Medicaid",
|
||||||
|
state: ClaimState = ClaimState.SUBMITTED,
|
||||||
|
service_date: date = date(2026, 6, 1),
|
||||||
|
) -> None:
|
||||||
|
s.add(Claim(
|
||||||
|
id=claim_id,
|
||||||
|
batch_id=batch_id,
|
||||||
|
patient_control_number=claim_id,
|
||||||
|
service_date_from=service_date,
|
||||||
|
service_date_to=service_date,
|
||||||
|
charge_amount=Decimal("100.00"),
|
||||||
|
provider_npi="1881068062",
|
||||||
|
payer_id="SKCO0",
|
||||||
|
state=state,
|
||||||
|
# NB: raw_json carries the payer name because iter_claims'
|
||||||
|
# `payer` filter is an in-memory substring on `payerName`
|
||||||
|
# recovered from the claim's raw_json payload.
|
||||||
|
raw_json={"payer": {"name": payer_name}},
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_remit(
|
||||||
|
s,
|
||||||
|
remit_id: str,
|
||||||
|
batch_id: str,
|
||||||
|
*,
|
||||||
|
received_at: datetime,
|
||||||
|
payer_name: str = "Colorado Medicaid",
|
||||||
|
claim_id: str | None = None,
|
||||||
|
total_paid: Decimal = Decimal("100.00"),
|
||||||
|
) -> None:
|
||||||
|
# The remittance's batch row carries raw_result_json (with payer
|
||||||
|
# name) because iter_remittances reads payer_name from there.
|
||||||
|
s.add(Batch(
|
||||||
|
id=batch_id,
|
||||||
|
kind="835",
|
||||||
|
input_filename=f"{remit_id}.edi",
|
||||||
|
parsed_at=received_at,
|
||||||
|
totals_json={"total_claims": 1},
|
||||||
|
validation_json={"passed": True, "warnings": [], "errors": []},
|
||||||
|
raw_result_json={"payer": {"name": payer_name}},
|
||||||
|
))
|
||||||
|
s.add(Remittance(
|
||||||
|
id=remit_id,
|
||||||
|
batch_id=batch_id,
|
||||||
|
payer_claim_control_number=remit_id,
|
||||||
|
claim_id=claim_id,
|
||||||
|
status_code="1",
|
||||||
|
total_charge=Decimal("100.00"),
|
||||||
|
total_paid=total_paid,
|
||||||
|
adjustment_amount=Decimal("0"),
|
||||||
|
received_at=received_at,
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Module-level: count_claims
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_count_claims_zero_when_empty():
|
||||||
|
assert global_store.count_claims() == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_count_claims_unfiltered_returns_full_population():
|
||||||
|
"""Regression: with 150 claims seeded and default iter limit=100,
|
||||||
|
the old ``len(list(iter_claims()))`` returned 100. count_claims
|
||||||
|
must return 150."""
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_batch(s, "b-claims-pop", base)
|
||||||
|
for i in range(150):
|
||||||
|
_seed_claim(s, f"CLM-{i:04d}", "b-claims-pop")
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
assert global_store.count_claims() == 150
|
||||||
|
|
||||||
|
|
||||||
|
def test_count_claims_filters_by_status():
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_batch(s, "b-status", base)
|
||||||
|
for i in range(5):
|
||||||
|
_seed_claim(s, f"CLM-SUB-{i}", "b-status", state=ClaimState.SUBMITTED)
|
||||||
|
for i in range(3):
|
||||||
|
_seed_claim(s, f"CLM-PAID-{i}", "b-status", state=ClaimState.PAID)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
assert global_store.count_claims(status="submitted") == 5
|
||||||
|
assert global_store.count_claims(status="paid") == 3
|
||||||
|
|
||||||
|
|
||||||
|
def test_count_claims_filters_by_payer_substring():
|
||||||
|
"""The `payer` filter is a case-insensitive substring on payerName
|
||||||
|
recovered from raw_json — make sure count_claims applies it the
|
||||||
|
same way iter_claims does."""
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_batch(s, "b-payer", base)
|
||||||
|
for i in range(4):
|
||||||
|
_seed_claim(s, f"CLM-CO-{i}", "b-payer", payer_name="Colorado Medicaid")
|
||||||
|
for i in range(2):
|
||||||
|
_seed_claim(s, f"CLM-AZ-{i}", "b-payer", payer_name="Arizona Medicaid")
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
assert global_store.count_claims() == 6
|
||||||
|
assert global_store.count_claims(payer="Colorado") == 4
|
||||||
|
assert global_store.count_claims(payer="Arizona") == 2
|
||||||
|
assert global_store.count_claims(payer="colorado") == 4 # case-insensitive
|
||||||
|
|
||||||
|
|
||||||
|
def test_count_claims_filters_by_provider_npi():
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_batch(s, "b-npi", base)
|
||||||
|
for i in range(3):
|
||||||
|
_seed_claim(s, f"CLM-A-{i}", "b-npi")
|
||||||
|
# Switch the rest to a different NPI via direct insert.
|
||||||
|
for i in range(7):
|
||||||
|
s.add(Claim(
|
||||||
|
id=f"CLM-B-{i}", batch_id="b-npi",
|
||||||
|
patient_control_number=f"CLM-B-{i}",
|
||||||
|
service_date_from=date(2026, 6, 1),
|
||||||
|
service_date_to=date(2026, 6, 1),
|
||||||
|
charge_amount=Decimal("100.00"),
|
||||||
|
provider_npi="9999999999",
|
||||||
|
payer_id="SKCO0",
|
||||||
|
state=ClaimState.SUBMITTED,
|
||||||
|
))
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
assert global_store.count_claims(provider_npi="1881068062") == 3
|
||||||
|
assert global_store.count_claims(provider_npi="9999999999") == 7
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Module-level: count_remittances
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_count_remittances_zero_when_empty():
|
||||||
|
assert global_store.count_remittances() == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_count_remittances_unfiltered_returns_full_population():
|
||||||
|
"""Regression: same shape as test_count_claims_unfiltered — the
|
||||||
|
old ``len(list(iter_remittances()))`` was capped at 100."""
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_batch(s, "b-remit-pop", base)
|
||||||
|
for i in range(120):
|
||||||
|
_seed_remit(
|
||||||
|
s, f"RMT-{i:04d}", f"b-{i:04d}",
|
||||||
|
received_at=base + timedelta(minutes=i),
|
||||||
|
)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
assert global_store.count_remittances() == 120
|
||||||
|
|
||||||
|
|
||||||
|
def test_count_remittances_filters_by_claim_id():
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_batch(s, "b-cid", base)
|
||||||
|
_seed_remit(s, "RMT-A", "b-cid-A", received_at=base, claim_id="CLM-1")
|
||||||
|
_seed_remit(s, "RMT-B", "b-cid-B", received_at=base, claim_id="CLM-2")
|
||||||
|
_seed_remit(s, "RMT-C", "b-cid-C", received_at=base, claim_id=None)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
assert global_store.count_remittances() == 3
|
||||||
|
assert global_store.count_remittances(claim_id="CLM-1") == 1
|
||||||
|
assert global_store.count_remittances(claim_id="CLM-2") == 1
|
||||||
|
assert global_store.count_remittances(claim_id="CLM-missing") == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_count_remittances_filters_by_payer_exact_match():
|
||||||
|
"""iter_remittances' `payer` filter is an exact match (not a
|
||||||
|
substring). Verify count_remittances mirrors that."""
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_batch(s, "b-rpayer", base)
|
||||||
|
for i in range(4):
|
||||||
|
_seed_remit(
|
||||||
|
s, f"RMT-CO-{i}", f"b-co-{i}",
|
||||||
|
received_at=base, payer_name="Colorado Medicaid",
|
||||||
|
)
|
||||||
|
for i in range(2):
|
||||||
|
_seed_remit(
|
||||||
|
s, f"RMT-AZ-{i}", f"b-az-{i}",
|
||||||
|
received_at=base, payer_name="Arizona Medicaid",
|
||||||
|
)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
assert global_store.count_remittances(payer="Colorado Medicaid") == 4
|
||||||
|
assert global_store.count_remittances(payer="Arizona Medicaid") == 2
|
||||||
|
assert global_store.count_remittances(payer="colorado medicaid") == 0 # case-sensitive exact
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# HTTP regression: list endpoint `total` reflects full population
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_claims_total_reflects_full_population(client: TestClient):
|
||||||
|
"""Bug repro: seed 150 claims, hit /api/claims?limit=25, assert
|
||||||
|
``total == 150`` (not 25, not 100)."""
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_batch(s, "b-http-claims", base)
|
||||||
|
for i in range(150):
|
||||||
|
_seed_claim(s, f"CLM-HTTP-{i:04d}", "b-http-claims")
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
resp = client.get("/api/claims", params={"limit": 25})
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
body = resp.json()
|
||||||
|
assert body["total"] == 150, body
|
||||||
|
assert body["returned"] == 25
|
||||||
|
assert body["has_more"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_remittances_total_reflects_full_population(client: TestClient):
|
||||||
|
"""Bug repro: seed 120 remits, hit /api/remittances?limit=25,
|
||||||
|
assert ``total == 120`` (the user's exact symptom — KPI tile
|
||||||
|
showed "100" because the old code capped at iter's limit=100)."""
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_batch(s, "b-http-remit", base)
|
||||||
|
for i in range(120):
|
||||||
|
_seed_remit(
|
||||||
|
s, f"RMT-HTTP-{i:04d}", f"b-h-{i:04d}",
|
||||||
|
received_at=base + timedelta(minutes=i),
|
||||||
|
)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
resp = client.get("/api/remittances", params={"limit": 25})
|
||||||
|
assert resp.status_code == 200, resp.text
|
||||||
|
body = resp.json()
|
||||||
|
assert body["total"] == 120, body
|
||||||
|
assert body["returned"] == 25
|
||||||
|
assert body["has_more"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_remittances_total_zero_when_empty(client: TestClient):
|
||||||
|
"""Empty DB → total=0. (Sanity check the count path doesn't break
|
||||||
|
for the no-data case.)"""
|
||||||
|
resp = client.get("/api/remittances")
|
||||||
|
assert resp.status_code == 200
|
||||||
|
body = resp.json()
|
||||||
|
assert body == {"items": [], "total": 0, "returned": 0, "has_more": False}
|
||||||
|
|
||||||
|
|
||||||
|
def test_api_remittances_total_respects_filters(client: TestClient):
|
||||||
|
"""Filter by payer → total narrows correctly (covers the in-memory
|
||||||
|
payer filter on the count path)."""
|
||||||
|
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
_seed_batch(s, "b-http-filter", base)
|
||||||
|
for i in range(7):
|
||||||
|
_seed_remit(
|
||||||
|
s, f"RMT-F-CO-{i}", f"b-fco-{i}",
|
||||||
|
received_at=base, payer_name="Colorado Medicaid",
|
||||||
|
)
|
||||||
|
for i in range(3):
|
||||||
|
_seed_remit(
|
||||||
|
s, f"RMT-F-AZ-{i}", f"b-faz-{i}",
|
||||||
|
received_at=base, payer_name="Arizona Medicaid",
|
||||||
|
)
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
resp = client.get(
|
||||||
|
"/api/remittances",
|
||||||
|
params={"payer": "Colorado Medicaid", "limit": 100},
|
||||||
|
)
|
||||||
|
assert resp.status_code == 200
|
||||||
|
body = resp.json()
|
||||||
|
assert body["total"] == 7
|
||||||
|
assert body["returned"] == 7
|
||||||
|
assert body["has_more"] is False
|
||||||
@@ -11,6 +11,12 @@ from cyclone.parsers.parse_999 import parse_999_text
|
|||||||
|
|
||||||
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
|
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
|
||||||
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
|
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
|
||||||
|
# Gainwell's MFT ships the set-level accept/reject segment under the
|
||||||
|
# sender-specific id ``IK5`` instead of the spec-defined ``AK5`` (X12
|
||||||
|
# 005010X231A1). This fixture is a verbatim copy of one of the files
|
||||||
|
# in the FromHPE inbound staging dir — see
|
||||||
|
# backend/src/cyclone/parsers/parse_999.py for the rationale.
|
||||||
|
GAINWELL_IK5 = Path(__file__).parent / "fixtures" / "minimal_999_ik5_gainwell.txt"
|
||||||
|
|
||||||
|
|
||||||
def test_parse_minimal_999_returns_accepted():
|
def test_parse_minimal_999_returns_accepted():
|
||||||
@@ -79,3 +85,29 @@ def test_parse_999_garbage_raises():
|
|||||||
"""Non-EDI input must raise CycloneParseError, not return a half-built result."""
|
"""Non-EDI input must raise CycloneParseError, not return a half-built result."""
|
||||||
with pytest.raises(CycloneParseError):
|
with pytest.raises(CycloneParseError):
|
||||||
parse_999_text("not edi at all", input_file="bad.txt")
|
parse_999_text("not edi at all", input_file="bad.txt")
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_999_gainwell_ik5_segment_accepted():
|
||||||
|
"""The IK5 set-level segment Gainwell ships must parse as 'A'.
|
||||||
|
|
||||||
|
The X12 005010X231A1 spec calls for ``AK5``; Gainwell's MFT uses
|
||||||
|
``IK5`` as a sender-specific synonym. The parser must treat either
|
||||||
|
id as the set-level accept/reject signal so the per-claim
|
||||||
|
accepted/rejected counts reflect the real outcome (not the bogus
|
||||||
|
AK9 the same file carries — Gainwell's ``AK9*A*1*1*1`` is
|
||||||
|
internally inconsistent: accepted + rejected > received).
|
||||||
|
"""
|
||||||
|
text = GAINWELL_IK5.read_text()
|
||||||
|
result = parse_999_text(text, input_file=GAINWELL_IK5.name)
|
||||||
|
assert len(result.set_responses) == 1
|
||||||
|
s = result.set_responses[0]
|
||||||
|
assert s.set_accept_reject.code == "A"
|
||||||
|
assert s.transaction_set_identifier == "837"
|
||||||
|
assert s.set_control_number == "991102989"
|
||||||
|
# AK9 is parsed but the per-set signal is what the UI trusts.
|
||||||
|
assert result.functional_group_acks[0].ack_code == "A"
|
||||||
|
assert result.functional_group_acks[0].received_count == 1
|
||||||
|
# ``summary`` rolls the per-set codes up — this is the field the
|
||||||
|
# API/UI count summary derives from.
|
||||||
|
assert result.summary.passed == 1
|
||||||
|
assert result.summary.failed == 0
|
||||||
|
|||||||
@@ -54,8 +54,8 @@ def test_seed_creates_clearhouse_singleton():
|
|||||||
assert ch.submitter_name == "Dzinesco"
|
assert ch.submitter_name == "Dzinesco"
|
||||||
assert ch.sftp_block.host == "mft.gainwelltechnologies.com"
|
assert ch.sftp_block.host == "mft.gainwelltechnologies.com"
|
||||||
assert ch.sftp_block.stub is True
|
assert ch.sftp_block.stub is True
|
||||||
assert "FromHPE" in ch.sftp_block.paths["outbound"]
|
assert "ToHPE" in ch.sftp_block.paths["outbound"]
|
||||||
assert "ToHPE" in ch.sftp_block.paths["inbound"]
|
assert "FromHPE" in ch.sftp_block.paths["inbound"]
|
||||||
|
|
||||||
|
|
||||||
def test_seed_creates_co_txix_payer_with_both_configs():
|
def test_seed_creates_co_txix_payer_with_both_configs():
|
||||||
|
|||||||
@@ -330,8 +330,20 @@ def test_run_reversal_flips_paid_to_reversed(fixture_835):
|
|||||||
assert reversal_match.prior_claim_state == ClaimState.PAID
|
assert reversal_match.prior_claim_state == ClaimState.PAID
|
||||||
|
|
||||||
|
|
||||||
def test_run_failed_reconcile_writes_activity_event(fixture_835):
|
def test_run_reconcile_raise_in_session_leaves_prior_commits_alone(fixture_835):
|
||||||
"""If reconcile crashes, the batch + remittances stay; activity event records failure."""
|
"""A ``reconcile.run`` raise inside an open session does not damage
|
||||||
|
previously-committed rows.
|
||||||
|
|
||||||
|
The test seeds a batch + remit, commits them in one session, then
|
||||||
|
calls ``reconcile.run`` in a fresh session with ``match`` monkey-
|
||||||
|
patched to raise. The pre-existing rows must survive — they're on
|
||||||
|
disk from the prior commit, separate from the rolling-back
|
||||||
|
session. This pins the unit-level invariant that ``reconcile.run``
|
||||||
|
itself never commits and never silently mutates rows outside the
|
||||||
|
session it was given; it is the *caller's* responsibility (now
|
||||||
|
``CycloneStore.add`` per SP27 Task 10) to control the transaction
|
||||||
|
boundary.
|
||||||
|
"""
|
||||||
with db.SessionLocal()() as s:
|
with db.SessionLocal()() as s:
|
||||||
_make_batch(s)
|
_make_batch(s)
|
||||||
_make_remit(s, "CLP-1", "PCN-A", "1", "100.00", "100.00")
|
_make_remit(s, "CLP-1", "PCN-A", "1", "100.00", "100.00")
|
||||||
|
|||||||
@@ -35,15 +35,15 @@ from cyclone.scheduler import (
|
|||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
def sftp_block(tmp_path):
|
def sftp_block(tmp_path):
|
||||||
staging = tmp_path / "staging"
|
staging = tmp_path / "staging"
|
||||||
inbound_dir = staging / "ToHPE"
|
inbound_dir = staging / "FromHPE"
|
||||||
inbound_dir.mkdir(parents=True)
|
inbound_dir.mkdir(parents=True)
|
||||||
return SftpBlock(
|
return SftpBlock(
|
||||||
host="mft.example.com",
|
host="mft.example.com",
|
||||||
port=22,
|
port=22,
|
||||||
username="test",
|
username="test",
|
||||||
paths={
|
paths={
|
||||||
"outbound": "/FromHPE",
|
"outbound": "/ToHPE",
|
||||||
"inbound": "/ToHPE",
|
"inbound": "/FromHPE",
|
||||||
},
|
},
|
||||||
stub=True,
|
stub=True,
|
||||||
staging_dir=str(staging),
|
staging_dir=str(staging),
|
||||||
@@ -55,7 +55,7 @@ def sftp_block(tmp_path):
|
|||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
def _drop_file(sftp_block):
|
def _drop_file(sftp_block):
|
||||||
"""Helper: drop a named file in the inbound dir. Returns the path."""
|
"""Helper: drop a named file in the inbound dir. Returns the path."""
|
||||||
inbound_dir = Path(sftp_block.staging_dir) / "ToHPE"
|
inbound_dir = Path(sftp_block.staging_dir) / "FromHPE"
|
||||||
|
|
||||||
def _drop(name: str, body: bytes) -> Path:
|
def _drop(name: str, body: bytes) -> Path:
|
||||||
inbound_dir.mkdir(parents=True, exist_ok=True)
|
inbound_dir.mkdir(parents=True, exist_ok=True)
|
||||||
@@ -284,4 +284,62 @@ class TestRoutedFileTypes:
|
|||||||
type. These tests are the regression net."""
|
type. These tests are the regression net."""
|
||||||
|
|
||||||
def test_handlers_cover_all_routed_types(self):
|
def test_handlers_cover_all_routed_types(self):
|
||||||
assert set(HANDLERS.keys()) == ROUTED_FILE_TYPES
|
assert set(HANDLERS.keys()) == ROUTED_FILE_TYPES
|
||||||
|
|
||||||
|
|
||||||
|
class TestProcessInboundFiles:
|
||||||
|
"""Scheduler.process_inbound_files() — date-filtered pull path.
|
||||||
|
|
||||||
|
The /api/admin/scheduler/pull-inbound endpoint and the
|
||||||
|
``cyclone pull-inbound`` CLI both call this. It must:
|
||||||
|
* process each file in the provided list (no SFTP listdir)
|
||||||
|
* dedupe via ``processed_inbound_files`` (idempotent on rerun)
|
||||||
|
* not touch SFTP at all — files are expected to be on local
|
||||||
|
disk at ``f.local_path``
|
||||||
|
"""
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_processes_provided_files_without_listdir(
|
||||||
|
self, sftp_block, _drop_file, tmp_path,
|
||||||
|
):
|
||||||
|
from cyclone.clearhouse import InboundFile
|
||||||
|
from cyclone.scheduler import STATUS_OK
|
||||||
|
|
||||||
|
# Drop a known 999 file on disk (the stub scheduler doesn't
|
||||||
|
# need SFTP — it reads from staging dir).
|
||||||
|
_drop_file(
|
||||||
|
"TP11525703-837P_M019048402-20260520231513488-1of1_999.x12",
|
||||||
|
b"not a real 999 -- handler will parse_error",
|
||||||
|
)
|
||||||
|
sched = _make_scheduler(sftp_block, tmp_path)
|
||||||
|
|
||||||
|
# Build the InboundFile records manually — caller is
|
||||||
|
# responsible for staging (mirrors the targeted-pull flow).
|
||||||
|
path = tmp_path / "staging" / "FromHPE" / \
|
||||||
|
"TP11525703-837P_M019048402-20260520231513488-1of1_999.x12"
|
||||||
|
files = [InboundFile(
|
||||||
|
name=path.name,
|
||||||
|
size=path.stat().st_size,
|
||||||
|
modified_at=datetime.fromtimestamp(path.stat().st_mtime, tz=timezone.utc),
|
||||||
|
local_path=path,
|
||||||
|
)]
|
||||||
|
|
||||||
|
result = await sched.process_inbound_files(files)
|
||||||
|
# The file's bytes are intentionally invalid — we only care
|
||||||
|
# that process_inbound_files invokes the handler and
|
||||||
|
# records the outcome (here: error from parse).
|
||||||
|
assert result.files_seen == 1
|
||||||
|
assert result.files_seen == result.files_processed + result.files_errored
|
||||||
|
# Idempotent: a second call is a no-op (already-processed dedup).
|
||||||
|
result2 = await sched.process_inbound_files(files)
|
||||||
|
assert result2.files_seen == 1
|
||||||
|
# Either skipped (because the prior call recorded it as error)
|
||||||
|
# or error — both prove the dedup branch fired. We just check
|
||||||
|
# it's not re-processed successfully.
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
rows = s.query(ProcessedInboundFile).filter_by(
|
||||||
|
sftp_block_name="test-block",
|
||||||
|
name=path.name,
|
||||||
|
).all()
|
||||||
|
assert len(rows) == 1
|
||||||
|
assert rows[0].status in (STATUS_OK, STATUS_ERROR, STATUS_SKIPPED)
|
||||||
@@ -0,0 +1,229 @@
|
|||||||
|
"""SP27 Task 9: scheduler status surfaces SFTP failures.
|
||||||
|
|
||||||
|
The 06/25 silent hang exposed a gap: ``Scheduler.status()`` had no
|
||||||
|
signal that polling had stalled. The operator's UI couldn't tell
|
||||||
|
"all quiet on the MFT front" from "we've been unable to reach the
|
||||||
|
MFT server for 3 hours". This pins the fix — the status dict
|
||||||
|
carries ``consecutive_failures``, ``last_error``, ``last_error_at``,
|
||||||
|
and ``last_sftp_attempt_at`` so the operator pill can flip to
|
||||||
|
destructive after 3 (or however many) consecutive failures.
|
||||||
|
|
||||||
|
Discriminator under test: ``TickResult.sftp_failed`` is the flag
|
||||||
|
``tick()`` keys off — NOT ``TickResult.errors``. Per-file processing
|
||||||
|
errors append to ``errors`` but must NOT bump ``consecutive_failures``.
|
||||||
|
The test suite pins both halves of the discriminator (SFTP failure
|
||||||
|
bumps; per-file failure doesn't).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from cyclone.clearhouse import InboundFile
|
||||||
|
from cyclone.providers import SftpBlock
|
||||||
|
from cyclone.scheduler import Scheduler
|
||||||
|
|
||||||
|
|
||||||
|
def _block() -> SftpBlock:
|
||||||
|
return SftpBlock(
|
||||||
|
host="mft.example.com",
|
||||||
|
port=22,
|
||||||
|
username="user",
|
||||||
|
auth={"password_keychain_account": "x"},
|
||||||
|
paths={"inbound": "/inbound", "outbound": "/outbound"},
|
||||||
|
stub=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---- pre-state: a fresh scheduler has no errors -------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_status_starts_with_zero_failures_and_no_error():
|
||||||
|
"""A fresh scheduler must report no errors — pins that the new
|
||||||
|
fields default cleanly and don't trip on first read."""
|
||||||
|
sched = Scheduler(_block())
|
||||||
|
status = sched.status()
|
||||||
|
assert status.consecutive_failures == 0
|
||||||
|
assert status.last_error is None
|
||||||
|
assert status.last_error_at is None
|
||||||
|
assert status.last_sftp_attempt_at is None
|
||||||
|
|
||||||
|
|
||||||
|
# ---- a failed SFTP call bumps consecutive_failures -----------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_status_records_last_error_after_failure():
|
||||||
|
"""A failed tick bumps consecutive_failures and records the error.
|
||||||
|
|
||||||
|
Simulates a broken SFTP client. Pins the post-Task-8 behavior
|
||||||
|
where ``_tick_impl`` catches the error and ``tick()`` records
|
||||||
|
it on the scheduler's status.
|
||||||
|
"""
|
||||||
|
class BrokenClient:
|
||||||
|
def list_inbound(self):
|
||||||
|
raise RuntimeError("simulated outage")
|
||||||
|
|
||||||
|
# SP27 Task 8: scheduler calls async_list_inbound() (the
|
||||||
|
# wait_for-wrapped variant). The broken stub raises before
|
||||||
|
# the wait_for wrapper runs — that's a real SFTP failure
|
||||||
|
# either way.
|
||||||
|
async def async_list_inbound(self):
|
||||||
|
raise RuntimeError("simulated outage")
|
||||||
|
|
||||||
|
sched = Scheduler(_block(), sftp_client_factory=lambda b: BrokenClient())
|
||||||
|
|
||||||
|
before = datetime.now(timezone.utc)
|
||||||
|
await sched.tick()
|
||||||
|
after = datetime.now(timezone.utc)
|
||||||
|
|
||||||
|
status = sched.status()
|
||||||
|
assert status.consecutive_failures == 1
|
||||||
|
assert status.last_error is not None
|
||||||
|
assert "simulated outage" in status.last_error
|
||||||
|
# last_error_at is set to roughly the time of the failed tick
|
||||||
|
assert status.last_error_at is not None
|
||||||
|
assert before <= status.last_error_at <= after
|
||||||
|
# last_sftp_attempt_at is also bumped — separate from last_error_at
|
||||||
|
# when the failure was an SFTP-side issue (not a parse error).
|
||||||
|
assert status.last_sftp_attempt_at is not None
|
||||||
|
assert before <= status.last_sftp_attempt_at <= after
|
||||||
|
|
||||||
|
|
||||||
|
# ---- a successful tick clears consecutive_failures -----------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_successful_tick_clears_consecutive_failures():
|
||||||
|
"""A tick that succeeds (or skips cleanly) resets the counter.
|
||||||
|
|
||||||
|
Without this, a transient blip would lock the destructive pill
|
||||||
|
on forever. Pins the recovery path.
|
||||||
|
"""
|
||||||
|
class BrokenClient:
|
||||||
|
def list_inbound(self):
|
||||||
|
raise RuntimeError("transient")
|
||||||
|
|
||||||
|
async def async_list_inbound(self):
|
||||||
|
raise RuntimeError("transient")
|
||||||
|
|
||||||
|
class HealthyClient:
|
||||||
|
def list_inbound(self):
|
||||||
|
return []
|
||||||
|
|
||||||
|
async def async_list_inbound(self):
|
||||||
|
return []
|
||||||
|
|
||||||
|
# First: two failing ticks to bump the counter
|
||||||
|
sched = Scheduler(_block(), sftp_client_factory=lambda b: BrokenClient())
|
||||||
|
await sched.tick()
|
||||||
|
await sched.tick()
|
||||||
|
assert sched.status().consecutive_failures == 2
|
||||||
|
|
||||||
|
# Then: switch to a healthy client and tick again
|
||||||
|
sched._sftp_client_factory = lambda b: HealthyClient()
|
||||||
|
await sched.tick()
|
||||||
|
status = sched.status()
|
||||||
|
assert status.consecutive_failures == 0
|
||||||
|
# last_error is preserved (audit trail) but last_error_at is from
|
||||||
|
# the last failure, not the success.
|
||||||
|
assert status.last_error is not None # the historical error stays
|
||||||
|
assert "transient" in status.last_error
|
||||||
|
|
||||||
|
|
||||||
|
# ---- last_sftp_attempt_at moves on every attempt, not just failures ---
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_last_sftp_attempt_at_advances_on_every_tick():
|
||||||
|
"""Even a successful tick should bump last_sftp_attempt_at.
|
||||||
|
|
||||||
|
Without this, the operator can't tell when the scheduler last
|
||||||
|
*tried* to reach the MFT — only when it last failed.
|
||||||
|
"""
|
||||||
|
sched = Scheduler(_block()) # default stub returns []
|
||||||
|
before = datetime.now(timezone.utc)
|
||||||
|
await sched.tick()
|
||||||
|
after = datetime.now(timezone.utc)
|
||||||
|
first = sched.status().last_sftp_attempt_at
|
||||||
|
assert first is not None
|
||||||
|
assert before <= first <= after
|
||||||
|
|
||||||
|
# Wait a moment, tick again — last_sftp_attempt_at should move forward.
|
||||||
|
await asyncio.sleep(0.01)
|
||||||
|
await sched.tick()
|
||||||
|
second = sched.status().last_sftp_attempt_at
|
||||||
|
assert second is not None
|
||||||
|
assert second > first
|
||||||
|
|
||||||
|
|
||||||
|
# ---- as_dict surfaces the new fields for the API -----------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_status_as_dict_includes_new_fields():
|
||||||
|
"""The API endpoint (/api/health or /api/admin/scheduler) reads
|
||||||
|
status.as_dict() — the new fields must be present and serialized."""
|
||||||
|
sched = Scheduler(_block())
|
||||||
|
d = sched.status().as_dict()
|
||||||
|
assert "consecutive_failures" in d
|
||||||
|
assert d["consecutive_failures"] == 0
|
||||||
|
assert "last_error" in d
|
||||||
|
assert d["last_error"] is None
|
||||||
|
assert "last_error_at" in d
|
||||||
|
assert d["last_error_at"] is None
|
||||||
|
assert "last_sftp_attempt_at" in d
|
||||||
|
assert d["last_sftp_attempt_at"] is None
|
||||||
|
|
||||||
|
|
||||||
|
# ---- per-file processing errors must NOT bump consecutive_failures ------
|
||||||
|
#
|
||||||
|
# This pins the SP27 Task 9 discriminator (``TickResult.sftp_failed``
|
||||||
|
# vs. ``TickResult.errors``). Without this test, a future refactor
|
||||||
|
# that broadens ``result.errors`` semantics — or rewires `tick()` to
|
||||||
|
# key off it again — would silently re-break the discrimination.
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_per_file_error_does_not_bump_consecutive_failures(monkeypatch):
|
||||||
|
"""Per-file processing errors must NOT flip the destructive pill.
|
||||||
|
|
||||||
|
The listing step succeeds but ``_handle_one`` simulates the
|
||||||
|
real handler's error path: it appends to ``result.errors``
|
||||||
|
(the way ``_handle_one`` does on parser exceptions) but does
|
||||||
|
NOT touch ``result.sftp_failed``. The pill must stay clean.
|
||||||
|
"""
|
||||||
|
|
||||||
|
class FilesClient:
|
||||||
|
async def async_list_inbound(self):
|
||||||
|
return [
|
||||||
|
InboundFile(
|
||||||
|
name="tp123-999_MTRACKING-20260525001606060-1of1.x12",
|
||||||
|
size=100,
|
||||||
|
modified_at=datetime.now(timezone.utc),
|
||||||
|
# ``local_path`` is None because fake_handle_one
|
||||||
|
# never reaches the download step.
|
||||||
|
local_path=None,
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
sched = Scheduler(_block(), sftp_client_factory=lambda b: FilesClient())
|
||||||
|
|
||||||
|
async def fake_handle_one(f, result):
|
||||||
|
# Mirrors the real ``_handle_one`` parser-failure path:
|
||||||
|
# appends to result.errors, does NOT set sftp_failed.
|
||||||
|
result.errors.append(f"{f.name}: ValueError: garbage")
|
||||||
|
|
||||||
|
monkeypatch.setattr(sched, "_handle_one", fake_handle_one)
|
||||||
|
|
||||||
|
await sched.tick()
|
||||||
|
status = sched.status()
|
||||||
|
|
||||||
|
# Counter stays clean — per-file error is not an SFTP outage.
|
||||||
|
assert status.consecutive_failures == 0
|
||||||
|
# ``last_error`` is reserved for SFTP-side failures.
|
||||||
|
assert status.last_error is None
|
||||||
|
# But ``last_sftp_attempt_at`` moves forward regardless — the
|
||||||
|
# scheduler did try, it just failed to ingest the file.
|
||||||
|
assert status.last_sftp_attempt_at is not None
|
||||||
@@ -0,0 +1,139 @@
|
|||||||
|
"""Per-op timeout guard for SFTP operations (SP27 Task 8).
|
||||||
|
|
||||||
|
paramiko is synchronous; without a timeout, a hung ``listdir_attr``
|
||||||
|
freezes the worker thread indefinitely. The 06/25 silent hang was
|
||||||
|
exactly this — the MFT server TCP-acked but stopped responding, the
|
||||||
|
scheduler's ``asyncio.to_thread`` waited forever, and the operator
|
||||||
|
had no signal that polling had stalled.
|
||||||
|
|
||||||
|
This test pins the fix: the SFTP client's only async-wrapped call
|
||||||
|
site (``async_list_inbound``) applies an ``asyncio.wait_for(...
|
||||||
|
timeout=N)`` so the event loop can give up after the configured
|
||||||
|
bound. The bound is read from ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS``
|
||||||
|
at call time (default 30s).
|
||||||
|
|
||||||
|
The other SFTP methods (``list_inbound_names``, ``download_inbound``,
|
||||||
|
``read_file``, ``write_file``) intentionally remain sync — they're
|
||||||
|
called from operator-triggered paths (admin endpoints, CLI) where
|
||||||
|
the operator can Ctrl-C the request. Wrapping them would require
|
||||||
|
making the FastAPI handlers themselves async, which is out of
|
||||||
|
scope for Task 8. Tracked as a follow-up.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import time
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from cyclone.clearhouse import SftpClient
|
||||||
|
from cyclone.providers import SftpBlock
|
||||||
|
|
||||||
|
|
||||||
|
def _block() -> SftpBlock:
|
||||||
|
return SftpBlock(
|
||||||
|
host="mft.example.com",
|
||||||
|
port=22,
|
||||||
|
username="user",
|
||||||
|
auth={"password_keychain_account": "x"},
|
||||||
|
paths={"inbound": "/inbound", "outbound": "/outbound"},
|
||||||
|
stub=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _HangingClient(SftpClient):
|
||||||
|
"""SftpClient that hangs on list_inbound — exercises the
|
||||||
|
asyncio.wait_for timeout. The hang runs in a thread, so
|
||||||
|
``asyncio.to_thread`` alone would never give up; only the
|
||||||
|
wait_for wrapper can break it.
|
||||||
|
|
||||||
|
The hang is short (3s) so the per-test thread pool doesn't
|
||||||
|
accumulate stragglers across the timeout tests. Each test sets
|
||||||
|
``CYCLONE_SFTP_OP_TIMEOUT_SECONDS=1`` so wait_for fires well
|
||||||
|
before the sleep completes.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def list_inbound(self):
|
||||||
|
time.sleep(3)
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
# ---- env-var parsing --------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_op_timeout_default_is_30_seconds(monkeypatch):
|
||||||
|
"""Default is 30s when CYCLONE_SFTP_OP_TIMEOUT_SECONDS is unset.
|
||||||
|
|
||||||
|
Pins the documented default so a refactor that changes it has to
|
||||||
|
update both the constant and the spec.
|
||||||
|
"""
|
||||||
|
monkeypatch.delenv("CYCLONE_SFTP_OP_TIMEOUT_SECONDS", raising=False)
|
||||||
|
from cyclone.clearhouse import _op_timeout_seconds
|
||||||
|
assert _op_timeout_seconds() == 30.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_op_timeout_reads_env_var(monkeypatch):
|
||||||
|
"""Operator can override via env var without a restart-rebuild."""
|
||||||
|
monkeypatch.setenv("CYCLONE_SFTP_OP_TIMEOUT_SECONDS", "7.5")
|
||||||
|
from cyclone.clearhouse import _op_timeout_seconds
|
||||||
|
assert _op_timeout_seconds() == 7.5
|
||||||
|
|
||||||
|
|
||||||
|
def test_op_timeout_rejects_unparseable_env_var(monkeypatch):
|
||||||
|
"""A non-float value (e.g. "30s") falls back to the default.
|
||||||
|
|
||||||
|
Pins the ValueError branch — a refactor that drops the try/except
|
||||||
|
would crash the SFTP wrapper on every call when the env var is
|
||||||
|
set to anything non-numeric.
|
||||||
|
"""
|
||||||
|
monkeypatch.setenv("CYCLONE_SFTP_OP_TIMEOUT_SECONDS", "30s")
|
||||||
|
from cyclone.clearhouse import _op_timeout_seconds
|
||||||
|
assert _op_timeout_seconds() == 30.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_op_timeout_rejects_zero_and_negative(monkeypatch):
|
||||||
|
"""Zero / negative timeouts would silently turn every SFTP call
|
||||||
|
into an instant failure (asyncio.wait_for(timeout=0) raises
|
||||||
|
immediately; timeout<0 is undefined). They fall back to default.
|
||||||
|
"""
|
||||||
|
from cyclone.clearhouse import _op_timeout_seconds
|
||||||
|
for bad in ("0", "0.0", "-1", "-0.5"):
|
||||||
|
monkeypatch.setenv("CYCLONE_SFTP_OP_TIMEOUT_SECONDS", bad)
|
||||||
|
assert _op_timeout_seconds() == 30.0, f"value {bad!r} should fall back to default"
|
||||||
|
|
||||||
|
|
||||||
|
# ---- async wrapper behavior -------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_async_list_inbound_times_out_when_paramiko_hangs(monkeypatch):
|
||||||
|
"""A hanging list_inbound must be cancelled by wait_for, not the thread.
|
||||||
|
|
||||||
|
The 06/25 silent hang was exactly this scenario — paramiko's
|
||||||
|
listdir_attr TCP-acked then went silent, freezing the worker
|
||||||
|
thread. The wait_for wrapper is the only thing that can break
|
||||||
|
out of that.
|
||||||
|
"""
|
||||||
|
monkeypatch.setenv("CYCLONE_SFTP_OP_TIMEOUT_SECONDS", "1")
|
||||||
|
client = _HangingClient(_block())
|
||||||
|
start = time.monotonic()
|
||||||
|
with pytest.raises(asyncio.TimeoutError):
|
||||||
|
await client.async_list_inbound()
|
||||||
|
elapsed = time.monotonic() - start
|
||||||
|
# Loose upper bound: timeout fires at ~1s, allow 3s for asyncio
|
||||||
|
# scheduling jitter on slow CI runners. If this ever fires at the
|
||||||
|
# hang duration (3s) the wait_for wrapper is missing.
|
||||||
|
assert elapsed < 3, f"timeout fired too late: {elapsed:.2f}s"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_async_list_inbound_returns_normally_when_fast(monkeypatch, tmp_path):
|
||||||
|
"""Sanity check: a non-hanging call returns the stub's empty list
|
||||||
|
without raising — pins that the wrapper doesn't break the happy path."""
|
||||||
|
monkeypatch.setenv("CYCLONE_SFTP_OP_TIMEOUT_SECONDS", "5")
|
||||||
|
block = _block()
|
||||||
|
block.staging_dir = str(tmp_path)
|
||||||
|
client = SftpClient(block)
|
||||||
|
# stub mode + no files staged = empty list, no hang, no timeout
|
||||||
|
result = await client.async_list_inbound()
|
||||||
|
assert result == []
|
||||||
@@ -40,8 +40,8 @@ def _block(
|
|||||||
port=22,
|
port=22,
|
||||||
username="testuser",
|
username="testuser",
|
||||||
paths={
|
paths={
|
||||||
"outbound": "/CO XIX/PROD/test/FromHPE",
|
"outbound": "/CO XIX/PROD/test/ToHPE",
|
||||||
"inbound": "/CO XIX/PROD/test/ToHPE",
|
"inbound": "/CO XIX/PROD/test/FromHPE",
|
||||||
},
|
},
|
||||||
stub=stub,
|
stub=stub,
|
||||||
staging_dir=staging_dir,
|
staging_dir=staging_dir,
|
||||||
@@ -259,6 +259,66 @@ class TestRealModeListInbound:
|
|||||||
assert len(files) == 1
|
assert len(files) == 1
|
||||||
assert files[0].name == "real.x12"
|
assert files[0].name == "real.x12"
|
||||||
|
|
||||||
|
def test_list_skips_warn_txt_files(self, monkeypatch, tmp_path: Path):
|
||||||
|
# Gainwell's MFT drops advisory *_warn.txt files in the same
|
||||||
|
# inbound dir. They're text-format side-channel notes (not X12
|
||||||
|
# envelopes) and must be skipped at list time.
|
||||||
|
real_attr = MagicMock()
|
||||||
|
real_attr.filename = "TP123-837P_M456-20260520231513488-1of1_999.x12"
|
||||||
|
real_attr.st_mode = 0o100644
|
||||||
|
real_attr.st_size = 1024
|
||||||
|
real_attr.st_mtime = 1718899200
|
||||||
|
|
||||||
|
warn_attr = MagicMock()
|
||||||
|
warn_attr.filename = "TP123-837P-202606181208100000-1of1_warn.txt"
|
||||||
|
warn_attr.st_mode = 0o100644
|
||||||
|
warn_attr.st_size = 200
|
||||||
|
warn_attr.st_mtime = 1718899200
|
||||||
|
|
||||||
|
mock_ssh, mock_sftp = _make_mock_paramiko(
|
||||||
|
monkeypatch, sftp_attrs=[real_attr, warn_attr],
|
||||||
|
)
|
||||||
|
|
||||||
|
def _open(path, mode="rb"):
|
||||||
|
m = MagicMock()
|
||||||
|
m.__enter__.return_value = io.BytesIO(b"x12 content")
|
||||||
|
return m
|
||||||
|
|
||||||
|
mock_sftp.open.side_effect = _open
|
||||||
|
|
||||||
|
block = _block(staging_dir=str(tmp_path / "staging"))
|
||||||
|
client = SftpClient(block)
|
||||||
|
files = client.list_inbound()
|
||||||
|
|
||||||
|
names = [f.name for f in files]
|
||||||
|
assert "TP123-837P_M456-20260520231513488-1of1_999.x12" in names
|
||||||
|
assert not any(n.endswith("_warn.txt") for n in names)
|
||||||
|
assert len(files) == 1
|
||||||
|
|
||||||
|
def test_list_inbound_names_does_not_download(self, monkeypatch, tmp_path: Path):
|
||||||
|
# list_inbound_names() must do a metadata-only SFTP listing —
|
||||||
|
# no sftp.open() / no file written to the cache.
|
||||||
|
attr = MagicMock()
|
||||||
|
attr.filename = "TP123-837P_M456-20260520231513488-1of1_999.x12"
|
||||||
|
attr.st_mode = 0o100644
|
||||||
|
attr.st_size = 1024
|
||||||
|
attr.st_mtime = 1718899200
|
||||||
|
|
||||||
|
mock_ssh, mock_sftp = _make_mock_paramiko(
|
||||||
|
monkeypatch, sftp_attrs=[attr],
|
||||||
|
)
|
||||||
|
|
||||||
|
block = _block(staging_dir=str(tmp_path / "staging"))
|
||||||
|
client = SftpClient(block)
|
||||||
|
files = client.list_inbound_names()
|
||||||
|
|
||||||
|
# sftp.open() must NOT have been called (no download).
|
||||||
|
mock_sftp.open.assert_not_called()
|
||||||
|
assert len(files) == 1
|
||||||
|
# The InboundFile.local_path is set to the planned cache path
|
||||||
|
# but the file itself doesn't exist yet.
|
||||||
|
assert not files[0].local_path.exists()
|
||||||
|
|
||||||
|
|
||||||
class TestRealModeReadFile:
|
class TestRealModeReadFile:
|
||||||
def test_read_returns_bytes(self, monkeypatch):
|
def test_read_returns_bytes(self, monkeypatch):
|
||||||
|
|||||||
@@ -19,8 +19,8 @@ def sftp_block(tmp_path):
|
|||||||
port=22,
|
port=22,
|
||||||
username="colorado-fts\\coxix_prod_11525703",
|
username="colorado-fts\\coxix_prod_11525703",
|
||||||
paths={
|
paths={
|
||||||
"outbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
|
"outbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
|
||||||
"inbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
|
"inbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
|
||||||
},
|
},
|
||||||
stub=True,
|
stub=True,
|
||||||
staging_dir=str(staging),
|
staging_dir=str(staging),
|
||||||
@@ -55,7 +55,7 @@ def test_stub_list_inbound_empty_when_no_local_files(sftp_block):
|
|||||||
|
|
||||||
def test_stub_list_inbound_returns_local_files(sftp_block):
|
def test_stub_list_inbound_returns_local_files(sftp_block):
|
||||||
# Simulate operator dropping a file in the inbound staging dir
|
# Simulate operator dropping a file in the inbound staging dir
|
||||||
inbound_dir = Path(sftp_block.staging_dir) / "CO XIX/PROD/coxix_prod_11525703/ToHPE"
|
inbound_dir = Path(sftp_block.staging_dir) / "CO XIX/PROD/coxix_prod_11525703/FromHPE"
|
||||||
inbound_dir.mkdir(parents=True, exist_ok=True)
|
inbound_dir.mkdir(parents=True, exist_ok=True)
|
||||||
(inbound_dir / "TP11525703-837P_M019048402-20260520231513488-1of1_999.x12").write_bytes(b"X")
|
(inbound_dir / "TP11525703-837P_M019048402-20260520231513488-1of1_999.x12").write_bytes(b"X")
|
||||||
client = SftpClient(sftp_block)
|
client = SftpClient(sftp_block)
|
||||||
@@ -78,17 +78,17 @@ def test_stub_read_file_returns_bytes(sftp_block, tmp_path):
|
|||||||
Lets the inbound scheduler exercise the same code path on a
|
Lets the inbound scheduler exercise the same code path on a
|
||||||
workstation without a real MFT connection.
|
workstation without a real MFT connection.
|
||||||
"""
|
"""
|
||||||
inbound = tmp_path / "staging" / "ToHPE"
|
inbound = tmp_path / "staging" / "FromHPE"
|
||||||
inbound.mkdir(parents=True)
|
inbound.mkdir(parents=True)
|
||||||
(inbound / "TP11525703-837P_M019048402-20260618130000000-1of1_TA1.x12").write_bytes(
|
(inbound / "TP11525703-837P_M019048402-20260618130000000-1of1_TA1.x12").write_bytes(
|
||||||
b"hello-world",
|
b"hello-world",
|
||||||
)
|
)
|
||||||
client = SftpClient(sftp_block)
|
client = SftpClient(sftp_block)
|
||||||
body = client.read_file("/ToHPE/TP11525703-837P_M019048402-20260618130000000-1of1_TA1.x12")
|
body = client.read_file("/FromHPE/TP11525703-837P_M019048402-20260618130000000-1of1_TA1.x12")
|
||||||
assert body == b"hello-world"
|
assert body == b"hello-world"
|
||||||
|
|
||||||
|
|
||||||
def test_stub_read_file_missing_raises(sftp_block):
|
def test_stub_read_file_missing_raises(sftp_block):
|
||||||
client = SftpClient(sftp_block)
|
client = SftpClient(sftp_block)
|
||||||
with pytest.raises(FileNotFoundError):
|
with pytest.raises(FileNotFoundError):
|
||||||
client.read_file("/ToHPE/does-not-exist.x12")
|
client.read_file("/FromHPE/does-not-exist.x12")
|
||||||
|
|||||||
@@ -354,10 +354,11 @@ def test_get_claim_detail_includes_state_history():
|
|||||||
"""``stateHistory`` must include the manual_match event after pairing."""
|
"""``stateHistory`` must include the manual_match event after pairing."""
|
||||||
s = CycloneStore()
|
s = CycloneStore()
|
||||||
_add_837_with_claim(s, "CLM-1")
|
_add_837_with_claim(s, "CLM-1")
|
||||||
# 835 with PCN that intentionally differs from the 837's member id so
|
# 835 PCN deliberately differs from claim_id so the auto-matcher
|
||||||
# reconcile doesn't auto-pair on ingest — we want manual_match to do it.
|
# (which now joins on claim_id == pcn after the SP27 Task 17 PCN
|
||||||
_add_835_with_remit(s, pcn="CLM-1", paid="100", status="1")
|
# fix) doesn't pair them — manual_match needs work to do.
|
||||||
s.manual_match("CLM-1", "CLM-1")
|
_add_835_with_remit(s, pcn="CLM-1-ORPHAN", paid="100", status="1")
|
||||||
|
s.manual_match("CLM-1", "CLM-1-ORPHAN")
|
||||||
|
|
||||||
out = s.get_claim_detail("CLM-1")
|
out = s.get_claim_detail("CLM-1")
|
||||||
assert out is not None
|
assert out is not None
|
||||||
@@ -377,7 +378,7 @@ def test_get_claim_detail_includes_state_history():
|
|||||||
# ActivityEvent row, which is correct per the spec's
|
# ActivityEvent row, which is correct per the spec's
|
||||||
# ``{kind, ts, batchId|null, remittanceId|null}`` contract.
|
# ``{kind, ts, batchId|null, remittanceId|null}`` contract.
|
||||||
mm = next(ev for ev in history if ev["kind"] == "manual_match")
|
mm = next(ev for ev in history if ev["kind"] == "manual_match")
|
||||||
assert mm["remittanceId"] == "CLM-1"
|
assert mm["remittanceId"] == "CLM-1-ORPHAN"
|
||||||
assert mm["batchId"] is not None
|
assert mm["batchId"] is not None
|
||||||
assert mm["ts"].endswith("Z")
|
assert mm["ts"].endswith("Z")
|
||||||
|
|
||||||
@@ -392,14 +393,15 @@ def test_get_claim_detail_includes_matched_remittance_summary():
|
|||||||
"""A paired claim surfaces a ``matchedRemittance`` summary block."""
|
"""A paired claim surfaces a ``matchedRemittance`` summary block."""
|
||||||
s = CycloneStore()
|
s = CycloneStore()
|
||||||
_add_837_with_claim(s, "CLM-1")
|
_add_837_with_claim(s, "CLM-1")
|
||||||
_add_835_with_remit(s, pcn="CLM-1", paid="100", status="1")
|
# 835 PCN deliberately differs from claim_id so manual_match has work.
|
||||||
s.manual_match("CLM-1", "CLM-1")
|
_add_835_with_remit(s, pcn="CLM-1-ORPHAN", paid="100", status="1")
|
||||||
|
s.manual_match("CLM-1", "CLM-1-ORPHAN")
|
||||||
|
|
||||||
out = s.get_claim_detail("CLM-1")
|
out = s.get_claim_detail("CLM-1")
|
||||||
assert out is not None
|
assert out is not None
|
||||||
mr = out["matchedRemittance"]
|
mr = out["matchedRemittance"]
|
||||||
assert mr is not None
|
assert mr is not None
|
||||||
assert mr["id"] == "CLM-1"
|
assert mr["id"] == "CLM-1-ORPHAN"
|
||||||
assert mr["totalPaid"] == 100.0
|
assert mr["totalPaid"] == 100.0
|
||||||
assert isinstance(mr["totalPaid"], float)
|
assert isinstance(mr["totalPaid"], float)
|
||||||
# status_code "1" → "received" (per to_ui_remittance_from_orm mapping).
|
# status_code "1" → "received" (per to_ui_remittance_from_orm mapping).
|
||||||
|
|||||||
@@ -0,0 +1,434 @@
|
|||||||
|
"""SP27 Task 11: matched-pair invariant.
|
||||||
|
|
||||||
|
The matched pair is a denormalized FK pair — ``Claim.matched_remittance_id``
|
||||||
|
and ``Remittance.claim_id`` — that MUST agree in both directions at every
|
||||||
|
moment. Three writers can mutate it:
|
||||||
|
|
||||||
|
1. ``CycloneStore.manual_match`` — operator override. SP27 Task 11 pins
|
||||||
|
it writes BOTH sides in one transaction.
|
||||||
|
2. ``CycloneStore.manual_unmatch`` — operator reversal. SP27 Task 11
|
||||||
|
pins it clears BOTH sides in one transaction.
|
||||||
|
3. ``reconcile.run`` (inside ``CycloneStore.add`` post Task 10) — auto-
|
||||||
|
match. Already pinned by the existing reconcile tests.
|
||||||
|
|
||||||
|
The drift check (``check_matched_pair_drift``) catches pre-existing
|
||||||
|
mismatches at startup and logs them at WARNING. Non-blocking.
|
||||||
|
|
||||||
|
These tests pin the invariants directly. They are explicitly redundant
|
||||||
|
with the integration tests in ``test_store_reconcile.py`` — the goal
|
||||||
|
here is a focused regression pin so a future refactor that touches
|
||||||
|
``manual_match`` / ``manual_unmatch`` (e.g. to defer the symmetric write
|
||||||
|
to a post-commit pass) doesn't silently break the invariant.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import date, datetime, timezone
|
||||||
|
from decimal import Decimal
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from cyclone import db
|
||||||
|
from cyclone.parsers.models import (
|
||||||
|
BatchSummary, BillingProvider, ClaimHeader, ClaimOutput,
|
||||||
|
Envelope, Payer, ParseResult, Subscriber, ValidationReport,
|
||||||
|
)
|
||||||
|
from cyclone.parsers.models_835 import (
|
||||||
|
BatchSummary as BatchSummary835, ClaimAdjustment, ClaimPayment,
|
||||||
|
Envelope as Envelope835, FinancialInfo, ParseResult835, Payer835,
|
||||||
|
Payee835, ReassociationTrace, ServicePayment,
|
||||||
|
)
|
||||||
|
from cyclone.store import (
|
||||||
|
CycloneStore,
|
||||||
|
check_matched_pair_drift,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _setup(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
|
||||||
|
db._reset_for_tests()
|
||||||
|
db.init_db()
|
||||||
|
yield
|
||||||
|
db._reset_for_tests()
|
||||||
|
|
||||||
|
|
||||||
|
def _build_claim(claim_id, pcn):
|
||||||
|
"""Build a single 837 claim + persist it through CycloneStore.add."""
|
||||||
|
from cyclone.store import BatchRecord837
|
||||||
|
|
||||||
|
co = ClaimOutput(
|
||||||
|
claim_id=claim_id,
|
||||||
|
control_number="0001",
|
||||||
|
transaction_date=date(2026, 6, 19),
|
||||||
|
billing_provider=BillingProvider(name="Test", npi="1234567890"),
|
||||||
|
subscriber=Subscriber(
|
||||||
|
first_name="Jane", last_name="Doe", member_id=pcn,
|
||||||
|
),
|
||||||
|
payer=Payer(name="Test Payer", id="P1"),
|
||||||
|
claim=ClaimHeader(
|
||||||
|
claim_id=claim_id, total_charge=Decimal("100"),
|
||||||
|
frequency_code="1", place_of_service="11",
|
||||||
|
),
|
||||||
|
diagnoses=[],
|
||||||
|
service_lines=[],
|
||||||
|
validation=ValidationReport(passed=True, errors=[], warnings=[]),
|
||||||
|
raw_segments=[],
|
||||||
|
)
|
||||||
|
pr837 = ParseResult(
|
||||||
|
envelope=Envelope(
|
||||||
|
sender_id="S", receiver_id="R", control_number="0001",
|
||||||
|
transaction_date=date(2026, 6, 19),
|
||||||
|
),
|
||||||
|
claims=[co],
|
||||||
|
summary=BatchSummary(
|
||||||
|
input_file="c.txt", control_number="0001",
|
||||||
|
transaction_date=date(2026, 6, 19),
|
||||||
|
total_claims=1, passed=1, failed=0,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
CycloneStore().add(BatchRecord837(
|
||||||
|
id="b-837", kind="837p", input_filename="c.txt",
|
||||||
|
parsed_at=date(2026, 6, 19).isoformat() + "T12:00:00+00:00",
|
||||||
|
result=pr837,
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
def _build_remit(remit_id, pcn):
|
||||||
|
"""Build a single 835 remit + persist it through CycloneStore.add."""
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from cyclone.store import BatchRecord835
|
||||||
|
|
||||||
|
cp = ClaimPayment(
|
||||||
|
payer_claim_control_number=pcn,
|
||||||
|
status_code="1",
|
||||||
|
status_label="Primary",
|
||||||
|
total_charge=Decimal("124.00"),
|
||||||
|
total_paid=Decimal("100.00"),
|
||||||
|
service_payments=[
|
||||||
|
ServicePayment(
|
||||||
|
line_number=1, procedure_qualifier="HC", procedure_code="99213",
|
||||||
|
charge=Decimal("124.00"), payment=Decimal("100.00"),
|
||||||
|
adjustments=[ClaimAdjustment(
|
||||||
|
group_code="CO", reason_code="45",
|
||||||
|
amount=Decimal("24.00"),
|
||||||
|
)],
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
pr835 = ParseResult835(
|
||||||
|
envelope=Envelope835(
|
||||||
|
sender_id="S", receiver_id="R", control_number="0001",
|
||||||
|
transaction_date=date(2026, 6, 19),
|
||||||
|
),
|
||||||
|
financial_info=FinancialInfo(
|
||||||
|
handling_code="C", paid_amount=Decimal("0"),
|
||||||
|
credit_debit_flag="C", payment_method=None,
|
||||||
|
),
|
||||||
|
trace=ReassociationTrace(
|
||||||
|
trace_type_code="1", trace_number="0001",
|
||||||
|
originating_company_id="S",
|
||||||
|
),
|
||||||
|
payer=Payer835(name="X", id="SKCO0"),
|
||||||
|
payee=Payee835(name="Y", npi="1234567890"),
|
||||||
|
claims=[cp],
|
||||||
|
summary=BatchSummary835(
|
||||||
|
input_file="era.txt", control_number="0001",
|
||||||
|
transaction_date=date(2026, 6, 19),
|
||||||
|
total_claims=1, passed=1, failed=0,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
CycloneStore().add(BatchRecord835(
|
||||||
|
id="b-835", kind="835", input_filename="era.txt",
|
||||||
|
parsed_at=datetime(2026, 6, 19, 12, 5, tzinfo=timezone.utc),
|
||||||
|
result=pr835,
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# manual_match — writes BOTH sides in one transaction
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_manual_match_writes_both_sides_in_one_transaction():
|
||||||
|
"""After ``manual_match`` returns, ``Claim.matched_remittance_id``
|
||||||
|
AND ``Remittance.claim_id`` both point at their pair.
|
||||||
|
|
||||||
|
Without the symmetric write, a later ``list_unmatched`` filter
|
||||||
|
(which keys off ``Remittance.claim_id IS NULL``) would surface
|
||||||
|
an already-matched remit as "orphan". Pre-T11 the symmetric
|
||||||
|
write was correct but only implicitly tested; this pins it
|
||||||
|
explicitly.
|
||||||
|
|
||||||
|
Note: claim PCN and remit PCN are DELIBERATELY DIFFERENT so the
|
||||||
|
auto-match inside ``CycloneStore.add`` (Task 10) doesn't pre-pair
|
||||||
|
them and ``manual_match`` becomes a no-op no-throw.
|
||||||
|
"""
|
||||||
|
from cyclone.db import Claim, Remittance
|
||||||
|
|
||||||
|
_build_claim("CLM-1", pcn="CLM-PATIENT-PCN")
|
||||||
|
_build_remit("CLP-1", pcn="CLP-1")
|
||||||
|
|
||||||
|
CycloneStore().manual_match("CLM-1", "CLP-1")
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
claim = s.get(Claim, "CLM-1")
|
||||||
|
# Remit PK is payer_claim_control_number (see Remittance ORM).
|
||||||
|
from sqlalchemy import select
|
||||||
|
remit = s.execute(
|
||||||
|
select(Remittance).where(
|
||||||
|
Remittance.payer_claim_control_number == "CLP-1"
|
||||||
|
)
|
||||||
|
).scalar_one()
|
||||||
|
|
||||||
|
assert claim.matched_remittance_id == remit.id, (
|
||||||
|
f"claim.matched_remittance_id={claim.matched_remittance_id!r} "
|
||||||
|
f"!= remit.id={remit.id!r}"
|
||||||
|
)
|
||||||
|
assert remit.claim_id == claim.id, (
|
||||||
|
f"remit.claim_id={remit.claim_id!r} != claim.id={claim.id!r}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# manual_unmatch — clears BOTH sides in one transaction
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_manual_unmatch_clears_both_sides():
|
||||||
|
"""After ``manual_unmatch`` returns, BOTH ``Claim.matched_remittance_id``
|
||||||
|
AND ``Remittance.claim_id`` return to NULL.
|
||||||
|
|
||||||
|
Without the symmetric clear, an operator-initiated unmatch would
|
||||||
|
leave the orphaned remit pointing at the claim while the claim
|
||||||
|
is correctly cleared — ``list_unmatched`` would then NOT surface
|
||||||
|
the pair (remit side is non-NULL), even though the operator
|
||||||
|
intended to put it back in the unmatched bucket.
|
||||||
|
|
||||||
|
PCN asymmetry prevents auto-match from pre-pairing — see
|
||||||
|
``test_manual_match_writes_both_sides_in_one_transaction``.
|
||||||
|
"""
|
||||||
|
from sqlalchemy import select
|
||||||
|
from cyclone.db import Claim, Remittance
|
||||||
|
|
||||||
|
_build_claim("CLM-1", pcn="CLM-PATIENT-PCN")
|
||||||
|
_build_remit("CLP-1", pcn="CLP-1")
|
||||||
|
|
||||||
|
store = CycloneStore()
|
||||||
|
store.manual_match("CLM-1", "CLP-1")
|
||||||
|
store.manual_unmatch("CLM-1")
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
claim = s.get(Claim, "CLM-1")
|
||||||
|
remit = s.execute(
|
||||||
|
select(Remittance).where(
|
||||||
|
Remittance.payer_claim_control_number == "CLP-1"
|
||||||
|
)
|
||||||
|
).scalar_one()
|
||||||
|
|
||||||
|
assert claim.matched_remittance_id is None
|
||||||
|
assert remit.claim_id is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_manual_match_rollback_clears_symmetric_writes(monkeypatch):
|
||||||
|
"""If anything between the two writes raises, the session's
|
||||||
|
rollback MUST clear BOTH writes — neither side set.
|
||||||
|
|
||||||
|
Pins the SP27 Task 11 invariant "writes BOTH sides in one
|
||||||
|
transaction" at the rollback level. Without this pin, a future
|
||||||
|
refactor that pushes the symmetric write to a post-commit
|
||||||
|
handler (or splits ``manual_match`` across two sessions) would
|
||||||
|
still pass the happy-path test but leave one side set on a
|
||||||
|
failed match — the very class of drift Task 11 was added to
|
||||||
|
surface.
|
||||||
|
"""
|
||||||
|
from sqlalchemy import select
|
||||||
|
from cyclone.db import Claim, Remittance
|
||||||
|
from cyclone import reconcile
|
||||||
|
|
||||||
|
_build_claim("CLM-ROLL", pcn="CLM-PATIENT-PCN")
|
||||||
|
_build_remit("CLP-ROLL", pcn="CLP-ROLL")
|
||||||
|
|
||||||
|
# Force the second-loop line-reconcile pass to raise. By the time
|
||||||
|
# it runs, the claim + remit writes are staged in the session
|
||||||
|
# but not committed — the raise must trigger a rollback.
|
||||||
|
def boom(session, claim, remittance):
|
||||||
|
raise RuntimeError("simulated post-write fault")
|
||||||
|
|
||||||
|
monkeypatch.setattr(reconcile, "_reconcile_pair", boom)
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError, match="simulated post-write fault"):
|
||||||
|
CycloneStore().manual_match("CLM-ROLL", "CLP-ROLL")
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
claim = s.get(Claim, "CLM-ROLL")
|
||||||
|
remit = s.execute(
|
||||||
|
select(Remittance).where(
|
||||||
|
Remittance.payer_claim_control_number == "CLP-ROLL"
|
||||||
|
)
|
||||||
|
).scalar_one()
|
||||||
|
# Neither side set — the session rolled back the partial
|
||||||
|
# state machine. claim.matched_remittance_id is the seeded
|
||||||
|
# NULL (no manual match); remit.claim_id is seeded NULL.
|
||||||
|
assert claim.matched_remittance_id is None
|
||||||
|
assert remit.claim_id is None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Drift check — startup audit, non-blocking
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_drift_check_returns_zero_for_clean_db():
|
||||||
|
"""A clean DB (no manual_match calls, no remits with claim_id)
|
||||||
|
reports zero drift and does not log any warning."""
|
||||||
|
_build_claim("CLM-1", pcn="PCN-A")
|
||||||
|
|
||||||
|
# No remits paired — no drift expected.
|
||||||
|
count = check_matched_pair_drift()
|
||||||
|
assert count == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_drift_check_logs_warning_for_mismatched_pair(caplog):
|
||||||
|
"""Inject drift directly via SQL (bypassing store so we can build
|
||||||
|
a state the writers can never produce), call the check, assert
|
||||||
|
a WARNING was logged with the offending ids.
|
||||||
|
|
||||||
|
The drift check is read-only and non-blocking — operators can
|
||||||
|
clean up manually. Logging at WARNING makes it visible in the
|
||||||
|
standard boot log without paging anyone.
|
||||||
|
|
||||||
|
This case is the inverse direction: a REMIT points at a claim
|
||||||
|
whose ``matched_remittance_id`` doesn't point back. Pairs with
|
||||||
|
``test_drift_check_logs_warning_for_case_a_mismatch`` to cover
|
||||||
|
both SQL paths through the check.
|
||||||
|
"""
|
||||||
|
from cyclone.db import Claim, ClaimState, Remittance
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
# Build the inverse state: a remit whose claim_id points at a
|
||||||
|
# claim that does NOT have matched_remittance_id back. Only Case
|
||||||
|
# B fires; Case A (claim → remit) is clean by construction.
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
s.add(Remittance(
|
||||||
|
id="orphan-remit", batch_id="b1",
|
||||||
|
payer_claim_control_number="PCN-B",
|
||||||
|
status_code="1", total_charge=Decimal("100"),
|
||||||
|
total_paid=Decimal("100"),
|
||||||
|
received_at=datetime.now(timezone.utc),
|
||||||
|
service_date=None, is_reversal=False,
|
||||||
|
claim_id="orphan-claim", # ← only this side set
|
||||||
|
))
|
||||||
|
s.add(Claim(
|
||||||
|
id="orphan-claim", batch_id="b1",
|
||||||
|
patient_control_number="PCN-B-CLAIM",
|
||||||
|
service_date_from=None,
|
||||||
|
charge_amount=Decimal("100"),
|
||||||
|
state=ClaimState.SUBMITTED,
|
||||||
|
matched_remittance_id=None, # ← back-pointer missing
|
||||||
|
))
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
with caplog.at_level("WARNING", logger="cyclone.store"):
|
||||||
|
count = check_matched_pair_drift()
|
||||||
|
|
||||||
|
assert count == 1, "exactly one drift was injected"
|
||||||
|
# The warning log should mention both ids so the operator can grep.
|
||||||
|
assert any(
|
||||||
|
"orphan-claim" in record.message
|
||||||
|
and "orphan-remit" in record.message
|
||||||
|
for record in caplog.records
|
||||||
|
), "drift warning should mention the offending claim + remit ids"
|
||||||
|
|
||||||
|
|
||||||
|
def test_drift_check_logs_warning_for_case_a_mismatch(caplog):
|
||||||
|
"""Pins the OTHER direction: claim → remit drift where the
|
||||||
|
claim's matched_remittance_id points at a remit whose claim_id
|
||||||
|
doesn't point back.
|
||||||
|
|
||||||
|
Without this test (paired with the inverse test above) a SQL
|
||||||
|
typo in Case A or Case B would silently pass the suite.
|
||||||
|
"""
|
||||||
|
from cyclone.db import Claim, ClaimState, Remittance
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
# Remit exists but its claim_id is NULL (claim side is the
|
||||||
|
# only one pointing).
|
||||||
|
s.add(Remittance(
|
||||||
|
id="dangle-remit", batch_id="b1",
|
||||||
|
payer_claim_control_number="PCN-A",
|
||||||
|
status_code="1", total_charge=Decimal("100"),
|
||||||
|
total_paid=Decimal("100"),
|
||||||
|
received_at=datetime.now(timezone.utc),
|
||||||
|
service_date=None, is_reversal=False,
|
||||||
|
claim_id=None,
|
||||||
|
))
|
||||||
|
s.add(Claim(
|
||||||
|
id="dangle-claim", batch_id="b1",
|
||||||
|
patient_control_number="PCN-A",
|
||||||
|
service_date_from=None,
|
||||||
|
charge_amount=Decimal("100"),
|
||||||
|
state=ClaimState.SUBMITTED,
|
||||||
|
matched_remittance_id="dangle-remit",
|
||||||
|
))
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
with caplog.at_level("WARNING", logger="cyclone.store"):
|
||||||
|
count = check_matched_pair_drift()
|
||||||
|
|
||||||
|
assert count == 1
|
||||||
|
assert any(
|
||||||
|
"dangle-claim" in record.message and "dangle-remit" in record.message
|
||||||
|
for record in caplog.records
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_drift_check_returns_count_of_mismatched_pairs():
|
||||||
|
"""The return value is the number of mismatched pairs so operators
|
||||||
|
can alert on it (e.g. fail boot if drift > 0 in a future iteration)."""
|
||||||
|
from cyclone.db import Claim, ClaimState, Remittance
|
||||||
|
|
||||||
|
with db.SessionLocal()() as s:
|
||||||
|
# Two drifted pairs: case A and case B in a single DB.
|
||||||
|
for i, (claim_id, remit_id) in enumerate(
|
||||||
|
[("drift-A-claim", "drift-A-remit"), ("drift-B-claim", "drift-B-remit")]
|
||||||
|
):
|
||||||
|
s.add(Remittance(
|
||||||
|
id=remit_id, batch_id="b1",
|
||||||
|
payer_claim_control_number=f"PCN-{i}",
|
||||||
|
status_code="1", total_charge=Decimal("100"),
|
||||||
|
total_paid=Decimal("100"),
|
||||||
|
received_at=datetime.now(timezone.utc),
|
||||||
|
service_date=None, is_reversal=False,
|
||||||
|
))
|
||||||
|
s.add(Claim(
|
||||||
|
id=claim_id, batch_id="b1",
|
||||||
|
patient_control_number=f"PCN-{i}",
|
||||||
|
service_date_from=None,
|
||||||
|
charge_amount=Decimal("100"),
|
||||||
|
state=ClaimState.SUBMITTED,
|
||||||
|
matched_remittance_id=remit_id,
|
||||||
|
))
|
||||||
|
s.commit()
|
||||||
|
|
||||||
|
count = check_matched_pair_drift()
|
||||||
|
assert count == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_drift_check_does_not_raise_on_query_error(monkeypatch):
|
||||||
|
"""The drift check is non-blocking at boot — if a DB query fails
|
||||||
|
we want a logged exception, NOT a crashed backend. Wrapped at the
|
||||||
|
api.py call site (Task 11 wiring); this pins that the function
|
||||||
|
itself doesn't try/catch (the call site does).
|
||||||
|
"""
|
||||||
|
from sqlalchemy.exc import OperationalError
|
||||||
|
|
||||||
|
# Force the SessionLocal to raise on use.
|
||||||
|
def boom():
|
||||||
|
raise OperationalError("SELECT 1", {}, Exception("synthetic"))
|
||||||
|
|
||||||
|
monkeypatch.setattr(db, "SessionLocal", lambda: boom)
|
||||||
|
with pytest.raises(OperationalError):
|
||||||
|
check_matched_pair_drift()
|
||||||
|
# The exception propagates — api.py::lifespan wraps it with a
|
||||||
|
# try/except so the boot continues. The pin here is that the
|
||||||
|
# function doesn't swallow by accident.
|
||||||
@@ -357,8 +357,11 @@ def test_manual_match_populates_line_reconciliation_rows():
|
|||||||
))
|
))
|
||||||
|
|
||||||
# 835 remit with one SVC line that matches claim line 1.
|
# 835 remit with one SVC line that matches claim line 1.
|
||||||
|
# pcn deliberately ≠ claim_id so auto-match (which now joins on
|
||||||
|
# claim_id == pcn after the SP27 Task 17 PCN fix) doesn't pair them
|
||||||
|
# and the claim lands in the unmatched list for manual pairing.
|
||||||
cp = ClaimPayment(
|
cp = ClaimPayment(
|
||||||
payer_claim_control_number="CLM-MANUAL", # != member_id "MANUAL"
|
payer_claim_control_number="ORPHAN-PCN-MANUAL",
|
||||||
status_code="1",
|
status_code="1",
|
||||||
status_label="Primary",
|
status_label="Primary",
|
||||||
total_charge=Decimal("200.00"),
|
total_charge=Decimal("200.00"),
|
||||||
@@ -498,10 +501,10 @@ def test_manual_match_idempotent_line_reconciliation():
|
|||||||
result=pr837,
|
result=pr837,
|
||||||
))
|
))
|
||||||
|
|
||||||
# 835 with two SVC lines — auto-match won't pair (PCN="CLM-IDEMP" ≠
|
# 835 with two SVC lines — auto-match won't pair (pcn="ORPHAN-IDEMP"
|
||||||
# member_id="IDEMP"), so manual_match has work to do.
|
# ≠ claim_id="CLM-IDEMP"), so manual_match has work to do.
|
||||||
cp = ClaimPayment(
|
cp = ClaimPayment(
|
||||||
payer_claim_control_number="CLM-IDEMP",
|
payer_claim_control_number="ORPHAN-IDEMP",
|
||||||
status_code="1",
|
status_code="1",
|
||||||
status_label="Primary",
|
status_label="Primary",
|
||||||
total_charge=Decimal("200.00"),
|
total_charge=Decimal("200.00"),
|
||||||
@@ -586,4 +589,170 @@ def test_manual_match_idempotent_line_reconciliation():
|
|||||||
r = session.get(Remittance, remit_id)
|
r = session.get(Remittance, remit_id)
|
||||||
assert r is not None
|
assert r is not None
|
||||||
assert r.adjustment_amount == Decimal("20.00")
|
assert r.adjustment_amount == Decimal("20.00")
|
||||||
assert r.claim_level_adjustment_amount == Decimal("20.00")
|
assert r.claim_level_adjustment_amount == Decimal("20.00")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# SP27 Task 17: Claim.patient_control_number must be the CLM01 (claim_id)
|
||||||
|
# the 837 sent, NOT the 2010BA subscriber member_id. The reconcile matcher
|
||||||
|
# joins Claim.patient_control_number == Remittance.payer_claim_control_number,
|
||||||
|
# and the 835 echoes CLM01 in CLP01 per X12 spec. Storing member_id here
|
||||||
|
# silently breaks every auto-match in production (the 9/1,739 "matches"
|
||||||
|
# observed in prod data are substring coincidences between synthetic CLM01
|
||||||
|
# and the human-readable PCN).
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _make_837_result(claims):
|
||||||
|
"""Build a minimal ParseResult with the given ClaimOutputs."""
|
||||||
|
from cyclone.parsers.models import (
|
||||||
|
BatchSummary as BatchSummary837,
|
||||||
|
BillingProvider, ClaimHeader, ClaimOutput, Envelope,
|
||||||
|
Payer, ParseResult, Subscriber, ValidationReport,
|
||||||
|
)
|
||||||
|
return ParseResult(
|
||||||
|
envelope=Envelope(
|
||||||
|
sender_id="S", receiver_id="R", control_number="0001",
|
||||||
|
transaction_date=date(2026, 6, 19),
|
||||||
|
),
|
||||||
|
claims=claims,
|
||||||
|
summary=BatchSummary837(
|
||||||
|
input_file="c.txt", control_number="0001",
|
||||||
|
transaction_date=date(2026, 6, 19),
|
||||||
|
total_claims=len(claims), passed=len(claims), failed=0,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _make_claim_output(claim_id="CLM-A", member_id="MEMBER-X",
|
||||||
|
charge="100.00",
|
||||||
|
service_date=date(2026, 6, 19)):
|
||||||
|
"""Single ClaimOutput with distinct claim_id and subscriber.member_id.
|
||||||
|
|
||||||
|
``service_date`` populates the SV1 service-line date so the
|
||||||
|
reconcile matcher's ``_pick_claim`` window check can find this
|
||||||
|
claim when the matching 835 has the same date in SVC.
|
||||||
|
"""
|
||||||
|
from cyclone.parsers.models import (
|
||||||
|
BillingProvider, ClaimHeader, ClaimOutput,
|
||||||
|
Subscriber, Payer, Procedure, ServiceLine,
|
||||||
|
ValidationReport,
|
||||||
|
)
|
||||||
|
return ClaimOutput(
|
||||||
|
claim_id=claim_id,
|
||||||
|
control_number="0001",
|
||||||
|
transaction_date=date(2026, 6, 19),
|
||||||
|
billing_provider=BillingProvider(name="Test", npi="1234567890"),
|
||||||
|
subscriber=Subscriber(first_name="Jane", last_name="Doe",
|
||||||
|
member_id=member_id),
|
||||||
|
payer=Payer(name="Test Payer", id="P1"),
|
||||||
|
claim=ClaimHeader(claim_id=claim_id, total_charge=Decimal(charge),
|
||||||
|
frequency_code="1", place_of_service="11"),
|
||||||
|
diagnoses=[],
|
||||||
|
service_lines=[
|
||||||
|
ServiceLine(
|
||||||
|
line_number=1,
|
||||||
|
procedure=Procedure(qualifier="HC", code="99213",
|
||||||
|
modifiers=[]),
|
||||||
|
charge=Decimal(charge),
|
||||||
|
unit_type="UN",
|
||||||
|
units=Decimal("1"),
|
||||||
|
service_date=service_date,
|
||||||
|
),
|
||||||
|
],
|
||||||
|
validation=ValidationReport(passed=True, errors=[], warnings=[]),
|
||||||
|
raw_segments=[],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_837_ingest_populates_patient_control_number_from_claim_id():
|
||||||
|
"""Claim.patient_control_number == claim_id (CLM01), not member_id.
|
||||||
|
|
||||||
|
RED: today the 837 ingest writes ``Claim.patient_control_number =
|
||||||
|
claim.subscriber.member_id`` (store.py:194). That breaks the
|
||||||
|
reconcile join against the 835's CLP01 (which echoes CLM01).
|
||||||
|
"""
|
||||||
|
from cyclone.db import Claim
|
||||||
|
from cyclone.store import BatchRecord837
|
||||||
|
|
||||||
|
s = CycloneStore()
|
||||||
|
s.add(BatchRecord837(
|
||||||
|
id="b-837-pcn", kind="837p", input_filename="c.txt",
|
||||||
|
parsed_at=datetime(2026, 6, 19, tzinfo=timezone.utc),
|
||||||
|
result=_make_837_result([
|
||||||
|
_make_claim_output(claim_id="CLM-A", member_id="MEMBER-X"),
|
||||||
|
]),
|
||||||
|
))
|
||||||
|
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
claim = session.get(Claim, "CLM-A")
|
||||||
|
assert claim is not None
|
||||||
|
assert claim.patient_control_number == "CLM-A", (
|
||||||
|
f"expected CLM01 (claim_id) 'CLM-A', "
|
||||||
|
f"got Claim.patient_control_number={claim.patient_control_number!r} "
|
||||||
|
f"(should be the value the 837 sent in CLM01, not the "
|
||||||
|
f"subscriber's 2010BA NM109 member_id)."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_837_then_835_with_echoed_pcn_auto_pairs():
|
||||||
|
"""End-to-end: 837 CLM01='CLM-A' + 835 CLP01='CLM-A' must auto-match.
|
||||||
|
|
||||||
|
RED: today ``Claim.patient_control_number == subscriber.member_id``
|
||||||
|
so the join on ``Claim.patient_control_number ==
|
||||||
|
Remittance.payer_claim_control_number`` fails even when the payer
|
||||||
|
echoes CLM01 correctly.
|
||||||
|
"""
|
||||||
|
from cyclone.db import Claim, Remittance
|
||||||
|
from cyclone.store import BatchRecord837, BatchRecord835
|
||||||
|
|
||||||
|
s = CycloneStore()
|
||||||
|
|
||||||
|
# 837: CLM01 = 'CLM-A', subscriber.member_id = 'MEMBER-X' (different)
|
||||||
|
s.add(BatchRecord837(
|
||||||
|
id="b-837-echo", kind="837p", input_filename="c.txt",
|
||||||
|
parsed_at=datetime(2026, 6, 19, tzinfo=timezone.utc),
|
||||||
|
result=_make_837_result([
|
||||||
|
_make_claim_output(claim_id="CLM-A", member_id="MEMBER-X",
|
||||||
|
charge="100.00"),
|
||||||
|
]),
|
||||||
|
))
|
||||||
|
|
||||||
|
# 835: CLP01 = 'CLM-A' (the payer echoes CLM01 back), paid == charge
|
||||||
|
cp = ClaimPayment(
|
||||||
|
payer_claim_control_number="CLM-A",
|
||||||
|
status_code="1", status_label="Primary",
|
||||||
|
total_charge=Decimal("100.00"), total_paid=Decimal("100.00"),
|
||||||
|
service_payments=[
|
||||||
|
ServicePayment(
|
||||||
|
line_number=1, procedure_qualifier="HC",
|
||||||
|
procedure_code="99213",
|
||||||
|
charge=Decimal("100.00"), payment=Decimal("100.00"),
|
||||||
|
units=Decimal("1"),
|
||||||
|
service_date=date(2026, 6, 19),
|
||||||
|
adjustments=[],
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
s.add(BatchRecord835(
|
||||||
|
id="b-835-echo", kind="835", input_filename="era.txt",
|
||||||
|
parsed_at=datetime(2026, 6, 19, tzinfo=timezone.utc),
|
||||||
|
result=_make_835_result([cp]),
|
||||||
|
))
|
||||||
|
|
||||||
|
with db.SessionLocal()() as session:
|
||||||
|
claim = session.get(Claim, "CLM-A")
|
||||||
|
r = session.query(Remittance).filter(
|
||||||
|
Remittance.payer_claim_control_number == "CLM-A"
|
||||||
|
).first()
|
||||||
|
# Both sides must have the FK set: Claim.matched_remittance_id
|
||||||
|
# AND Remittance.claim_id, written in the same transaction.
|
||||||
|
assert claim.matched_remittance_id is not None, (
|
||||||
|
"Claim should be auto-matched to the remit (CLM01 echoed "
|
||||||
|
"in CLP01), but matched_remittance_id is still NULL. The "
|
||||||
|
"837 ingest is storing patient_control_number from "
|
||||||
|
"subscriber.member_id instead of claim.claim_id."
|
||||||
|
)
|
||||||
|
assert r.claim_id == "CLM-A"
|
||||||
|
assert r.claim_id == claim.id
|
||||||
|
assert claim.state == "paid"
|
||||||
@@ -27,8 +27,8 @@ def test_update_clearhouse_round_trip():
|
|||||||
port=22,
|
port=22,
|
||||||
username="colorado-fts\\coxix_prod_11525703",
|
username="colorado-fts\\coxix_prod_11525703",
|
||||||
paths={
|
paths={
|
||||||
"outbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
|
"outbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
|
||||||
"inbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
|
"inbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
|
||||||
},
|
},
|
||||||
stub=False,
|
stub=False,
|
||||||
staging_dir="./var/sftp/staging",
|
staging_dir="./var/sftp/staging",
|
||||||
|
|||||||
@@ -15,6 +15,8 @@ secrets:
|
|||||||
file: /tmp/cyclone-test-secrets/admin_username
|
file: /tmp/cyclone-test-secrets/admin_username
|
||||||
cyclone_admin_password:
|
cyclone_admin_password:
|
||||||
file: /tmp/cyclone-test-secrets/admin_pw
|
file: /tmp/cyclone-test-secrets/admin_pw
|
||||||
|
cyclone_sftp_password:
|
||||||
|
file: /tmp/cyclone-test-secrets/sftp_password
|
||||||
|
|
||||||
services:
|
services:
|
||||||
frontend:
|
frontend:
|
||||||
|
|||||||
+5
-4
@@ -45,10 +45,11 @@ services:
|
|||||||
CYCLONE_ADMIN_PASSWORD_FILE: "/run/secrets/cyclone_admin_password"
|
CYCLONE_ADMIN_PASSWORD_FILE: "/run/secrets/cyclone_admin_password"
|
||||||
# SP26 — SFTP password via Docker secret (matches the admin-creds pattern).
|
# SP26 — SFTP password via Docker secret (matches the admin-creds pattern).
|
||||||
CYCLONE_SFTP_PASSWORD_FILE: "/run/secrets/cyclone_sftp_password"
|
CYCLONE_SFTP_PASSWORD_FILE: "/run/secrets/cyclone_sftp_password"
|
||||||
# Bind 0.0.0.0 so the frontend container on the compose bridge
|
# Always bind to 0.0.0.0. The compose-managed bridge network
|
||||||
# network can reach us. The bridge network provides isolation —
|
# isolates the backend from the host's LAN/WAN — only the
|
||||||
# only the `frontend` service is on it; the host firewall still
|
# `frontend` service joins it. Host firewall / port publishing
|
||||||
# blocks anything that isn't on the LAN.
|
# is the layer that controls what reaches us from outside the
|
||||||
|
# compose network, not the bind address.
|
||||||
CYCLONE_HOST: "0.0.0.0"
|
CYCLONE_HOST: "0.0.0.0"
|
||||||
secrets:
|
secrets:
|
||||||
- cyclone_db_key
|
- cyclone_db_key
|
||||||
|
|||||||
@@ -678,7 +678,7 @@ Returns `status="ok"` only when every subsystem is healthy. `status="degraded"`
|
|||||||
The default deployment. Two terminals:
|
The default deployment. Two terminals:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Terminal 1 — backend on 127.0.0.1:8000
|
# Terminal 1 — backend on 0.0.0.0:8000 (always binds to all interfaces; firewall is what restricts reachability)
|
||||||
cd backend
|
cd backend
|
||||||
.venv/bin/python -m cyclone serve
|
.venv/bin/python -m cyclone serve
|
||||||
|
|
||||||
|
|||||||
@@ -151,7 +151,7 @@ Each requirement is `FR-NN`, traceable to one or more sub-projects. Verification
|
|||||||
|
|
||||||
| ID | NFR | Source / SP |
|
| ID | NFR | Source / SP |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| NFR-1 | **Local-only bind + auth.** Backend binds `127.0.0.1:8000` (overridable via `CYCLONE_PORT`); CORS allowlist is exact (`http://localhost:5173`); login required (bcrypt + HttpOnly session cookie; first admin bootstrapped from env vars `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD`); dev/test escape hatch via `CYCLONE_AUTH_DISABLED=1` (logs WARNING at boot). | SP1 + auth (2026-06-23 merge) + SP24 (doc reconciliation) |
|
| NFR-1 | **Always bind 0.0.0.0 + auth.** Backend binds `0.0.0.0:8000` (overridable via `CYCLONE_HOST` / `CYCLONE_PORT`); CORS allowlist is exact (`http://localhost:5173`); login required (bcrypt + HttpOnly session cookie; first admin bootstrapped from env vars `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD`); dev/test escape hatch via `CYCLONE_AUTH_DISABLED=1` (logs WARNING at boot). Reachability is controlled by the host firewall / compose port publishing, not the bind address. | SP1 + auth (2026-06-23 merge) + SP24 (doc reconciliation) |
|
||||||
| NFR-2 | **Determinism.** Parser / serializer round-trip is guaranteed on 113 real prodfiles (`docs/prodfiles/claims/*.x12`); canonical fields, not byte-identity (called out in the SP8 spec). | SP8 |
|
| NFR-2 | **Determinism.** Parser / serializer round-trip is guaranteed on 113 real prodfiles (`docs/prodfiles/claims/*.x12`); canonical fields, not byte-identity (called out in the SP8 spec). | SP8 |
|
||||||
| NFR-3 | **Audit completeness.** Every reconciliation anomaly, every state transition, every 999/277CA reject writes an `ActivityEvent` (in `activity_events` table, un-chained; powers the `/activity` feed and inbox state transitions). | SP2 + SP10 |
|
| NFR-3 | **Audit completeness.** Every reconciliation anomaly, every state transition, every 999/277CA reject writes an `ActivityEvent` (in `activity_events` table, un-chained; powers the `/activity` feed and inbox state transitions). | SP2 + SP10 |
|
||||||
| NFR-4 | **Tamper-evidence.** A separate `audit_log` table (SP11) carries SHA-256 hash-chained rows for security-sensitive events (login attempts, rejections, key rotations, backup lifecycle, rejected requests). `GET /api/admin/audit-log/verify` detects any break. **The `activity_events` (NFR-3) and `audit_log` (NFR-4) tables are distinct** — different semantics, different audiences. New code that needs an audit row must decide which one to write to. | SP11 |
|
| NFR-4 | **Tamper-evidence.** A separate `audit_log` table (SP11) carries SHA-256 hash-chained rows for security-sensitive events (login attempts, rejections, key rotations, backup lifecycle, rejected requests). `GET /api/admin/audit-log/verify` detects any break. **The `activity_events` (NFR-3) and `audit_log` (NFR-4) tables are distinct** — different semantics, different audiences. New code that needs an audit row must decide which one to write to. | SP11 |
|
||||||
|
|||||||
@@ -25,8 +25,8 @@ the 837P file.
|
|||||||
| Submitter contact | Tyler Martinez <tyler@dzinesco.com> |
|
| Submitter contact | Tyler Martinez <tyler@dzinesco.com> |
|
||||||
| SFTP host | `mft.gainwelltechnologies.com` |
|
| SFTP host | `mft.gainwelltechnologies.com` |
|
||||||
| SFTP username | `colorado-fts\coxix_prod_11525703` |
|
| SFTP username | `colorado-fts\coxix_prod_11525703` |
|
||||||
| SFTP outbound dir | `/CO XIX/PROD/coxix_prod_11525703/FromHPE` |
|
| SFTP outbound dir | `/CO XIX/PROD/coxix_prod_11525703/ToHPE` |
|
||||||
| SFTP inbound dir | `/CO XIX/PROD/coxix_prod_11525703/ToHPE` |
|
| SFTP inbound dir | `/CO XIX/PROD/coxix_prod_11525703/FromHPE` |
|
||||||
|
|
||||||
## dzinesco's 3 billing-provider NPIs
|
## dzinesco's 3 billing-provider NPIs
|
||||||
|
|
||||||
@@ -49,8 +49,8 @@ dzinesco submits 837P files to Gainwell's MFT (Managed File Transfer)
|
|||||||
at `mft.gainwelltechnologies.com`. The full SFTP path layout is
|
at `mft.gainwelltechnologies.com`. The full SFTP path layout is
|
||||||
specified by the user (2026-06-20):
|
specified by the user (2026-06-20):
|
||||||
|
|
||||||
- **Outbound** (we send): `/CO XIX/PROD/coxix_prod_11525703/FromHPE`
|
- **Outbound** (we send): `/CO XIX/PROD/coxix_prod_11525703/ToHPE`
|
||||||
- **Inbound** (HPE/Gainwell sends to us): `/CO XIX/PROD/coxix_prod_11525703/ToHPE`
|
- **Inbound** (HPE/Gainwell sends to us): `/CO XIX/PROD/coxix_prod_11525703/FromHPE`
|
||||||
|
|
||||||
### File naming
|
### File naming
|
||||||
|
|
||||||
@@ -107,8 +107,8 @@ curl -X POST http://localhost:8000/api/clearhouse/submit \
|
|||||||
-H 'Content-Type: application/json' \
|
-H 'Content-Type: application/json' \
|
||||||
-d '{"claim_ids": ["CLM-1", "CLM-2"], "payer_id": "CO_TXIX"}'
|
-d '{"claim_ids": ["CLM-1", "CLM-2"], "payer_id": "CO_TXIX"}'
|
||||||
|
|
||||||
# Files appear at ./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/
|
# Files appear at ./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/
|
||||||
ls -la "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/"
|
ls -la "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/"
|
||||||
```
|
```
|
||||||
|
|
||||||
## Payer IDs
|
## Payer IDs
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -21,7 +21,7 @@ Replace the single hard-coded `PayerConfig` factory dict (currently in `api.py:9
|
|||||||
- **SP10** — 277CA parser + "Payer-Rejected" lane in the Inbox.
|
- **SP10** — 277CA parser + "Payer-Rejected" lane in the Inbox.
|
||||||
- **SP11** — Tamper-evident hash-chained `audit_log` table.
|
- **SP11** — Tamper-evident hash-chained `audit_log` table.
|
||||||
- **SP12** — SQLCipher encryption at rest + Keychain-stored DB key.
|
- **SP12** — SQLCipher encryption at rest + Keychain-stored DB key.
|
||||||
- **SP13** — Replace the SFTP stub with real `paramiko` connection to `mft.gainwelltechnologies.com` and actually push to `/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
|
- **SP13** — Replace the SFTP stub with real `paramiko` connection to `mft.gainwelltechnologies.com` and actually push to `/CO XIX/PROD/coxix_prod_11525703/ToHPE`.
|
||||||
- **Real SFTP credentials in Keychain** — schema and call sites are in place; the actual secret is created manually by the operator.
|
- **Real SFTP credentials in Keychain** — schema and call sites are in place; the actual secret is created manually by the operator.
|
||||||
|
|
||||||
## 2. Goals
|
## 2. Goals
|
||||||
@@ -109,8 +109,8 @@ VALUES (1, 'dzinesco', '11525703', 'Dzinesco', 'Tyler Martinez', 'tyler@dzinesco
|
|||||||
'"inbound_template":"TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12"}',
|
'"inbound_template":"TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12"}',
|
||||||
'{"host":"mft.gainwelltechnologies.com","port":22,'
|
'{"host":"mft.gainwelltechnologies.com","port":22,'
|
||||||
'"username":"colorado-fts\\coxix_prod_11525703",'
|
'"username":"colorado-fts\\coxix_prod_11525703",'
|
||||||
'"paths":{"outbound":"/CO XIX/PROD/coxix_prod_11525703/FromHPE",'
|
'"paths":{"outbound":"/CO XIX/PROD/coxix_prod_11525703/ToHPE",'
|
||||||
'"inbound":"/CO XIX/PROD/coxix_prod_11525703/ToHPE"},'
|
'"inbound":"/CO XIX/PROD/coxix_prod_11525703/FromHPE"},'
|
||||||
'"stub":true,"staging_dir":"./var/sftp/staging","poll_seconds":300}',
|
'"stub":true,"staging_dir":"./var/sftp/staging","poll_seconds":300}',
|
||||||
'2026-06-20T00:00:00Z');
|
'2026-06-20T00:00:00Z');
|
||||||
```
|
```
|
||||||
@@ -163,13 +163,13 @@ All 3 share the same address because all 3 are registered to the Montrose corpor
|
|||||||
|
|
||||||
Per the **HCPF X12 File Naming Standards Quick Guide** (https://hcpf.colorado.gov/tp-x12-filenaming):
|
Per the **HCPF X12 File Naming Standards Quick Guide** (https://hcpf.colorado.gov/tp-x12-filenaming):
|
||||||
|
|
||||||
**Outbound** (we send to `/CO XIX/PROD/coxix_prod_11525703/FromHPE`):
|
**Outbound** (we send to `/CO XIX/PROD/coxix_prod_11525703/ToHPE`):
|
||||||
```
|
```
|
||||||
{tpid}-{transaction_type}-{yyyymmddhhmmssSSS_MT}-1of1.{ext}
|
{tpid}-{transaction_type}-{yyyymmddhhmmssSSS_MT}-1of1.{ext}
|
||||||
```
|
```
|
||||||
Example: `11525703-837P-20260620132243505-1of1.x12`
|
Example: `11525703-837P-20260620132243505-1of1.x12`
|
||||||
|
|
||||||
**Inbound** (HPE sends to `/CO XIX/PROD/coxix_prod_11525703/ToHPE`):
|
**Inbound** (HPE sends to `/CO XIX/PROD/coxix_prod_11525703/FromHPE`):
|
||||||
```
|
```
|
||||||
TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
|
TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
|
||||||
```
|
```
|
||||||
@@ -209,7 +209,7 @@ Handler:
|
|||||||
"ok": true,
|
"ok": true,
|
||||||
"submitted": [
|
"submitted": [
|
||||||
{"claim_id": "CLM-001", "filename": "11525703-837P-20260620132243505-1of1.x12",
|
{"claim_id": "CLM-001", "filename": "11525703-837P-20260620132243505-1of1.x12",
|
||||||
"staging_path": "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/11525703-..."}
|
"staging_path": "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/11525703-..."}
|
||||||
],
|
],
|
||||||
"stub": true
|
"stub": true
|
||||||
}
|
}
|
||||||
@@ -367,7 +367,7 @@ All R200-R210 run on parse AND on serialize, so a 837 file with an unknown NPI f
|
|||||||
|
|
||||||
| # | Question | Resolution |
|
| # | Question | Resolution |
|
||||||
|---|----------|-----------|
|
|---|----------|-----------|
|
||||||
| 1 | SFTP outbound + inbound paths on `mft.gainwelltechnologies.com` | `/CO XIX/PROD/coxix_prod_11525703/FromHPE` (out), `/CO XIX/PROD/coxix_prod_11525703/ToHPE` (in) — user-provided 2026-06-20 |
|
| 1 | SFTP outbound + inbound paths on `mft.gainwelltechnologies.com` | `/CO XIX/PROD/coxix_prod_11525703/ToHPE` (out), `/CO XIX/PROD/coxix_prod_11525703/FromHPE` (in) — user-provided 2026-06-20 |
|
||||||
| 2 | 3 NPI street addresses + ZIPs | All 3 NPIs share Montrose corporate address (user confirmed 2026-06-20) — seed from prod files |
|
| 2 | 3 NPI street addresses + ZIPs | All 3 NPIs share Montrose corporate address (user confirmed 2026-06-20) — seed from prod files |
|
||||||
| 3 | 3 NPI taxonomy codes | `251E00000X` for all 3, self-served from 136 prod files |
|
| 3 | 3 NPI taxonomy codes | `251E00000X` for all 3, self-served from 136 prod files |
|
||||||
| 4 | 277CA filename suffix | Use `277` per HCPF doc; distinguish 277CA by `ST*277CA` content (user confirmed 2026-06-20) |
|
| 4 | 277CA filename suffix | Use `277` per HCPF doc; distinguish 277CA by `ST*277CA` content (user confirmed 2026-06-20) |
|
||||||
|
|||||||
+391
@@ -0,0 +1,391 @@
|
|||||||
|
# Sub-project 27 — Remittances Architecture Refactor: Design Spec
|
||||||
|
|
||||||
|
**Date:** 2026-06-29
|
||||||
|
**Status:** Draft, awaiting user sign-off
|
||||||
|
**Branch:** `sp27-remittances-architecture-refactor`
|
||||||
|
**Aesthetic direction:** No new UI (chain view rendered into existing ClaimDrawer; no new page)
|
||||||
|
|
||||||
|
## 1. Scope
|
||||||
|
|
||||||
|
Cyclone today ships the full claim/ack/remit cycle end-to-end: outbound
|
||||||
|
837 submitted via SFTP, inbound 999 / 277CA / 835 / TA1 polled and parsed,
|
||||||
|
remits matched against claims. Live data is now flowing on the production
|
||||||
|
system (~60K remits, ~542K 999 acks since 06/19). The architecture serving
|
||||||
|
that cycle has accumulated rough edges that an afternoon of operational
|
||||||
|
pressure surfaced:
|
||||||
|
|
||||||
|
- **Scheduler is fragile.** A 06/25 silent hang (no error logged) took
|
||||||
|
the MFT poll down for ~3 days; recovered only because the operator
|
||||||
|
restarted the backend. Root cause: `sftp.listdir_attr()` runs without
|
||||||
|
a socket timeout, so when the MFT TCP-acks but stops responding the
|
||||||
|
thread blocks indefinitely. The scheduler's status surface is too
|
||||||
|
shallow to detect this in real time.
|
||||||
|
- **Filename classification is brittle.** `edi/filenames.py:INBOUND_RE`
|
||||||
|
requires a `_file_type.x12` suffix on inbound files. Gainwell's
|
||||||
|
actual 835s ship without that suffix (see the 5×835 batch from
|
||||||
|
6/15–6/19), so the only thing the scheduler could do was mark them
|
||||||
|
`skipped`. Operators had to ingest those by hand.
|
||||||
|
- **Helpers duplicate API logic.** `cyclone.scheduler._ack_count_summary`
|
||||||
|
and `_ack_synthetic_source_batch_id` copy logic from
|
||||||
|
`cyclone.api` with a comment saying so. Two places to keep in sync.
|
||||||
|
- **Reconciliation is decoupled from ingest.** 835 ingest writes a
|
||||||
|
`Remittance` with `adjustment_amount=0`, and a separate
|
||||||
|
`reconcile.run()` pass later sums `CasAdjustment.amount` rows and
|
||||||
|
overwrites `adjustment_amount`. The UI sees stale values until the
|
||||||
|
pass runs, and the two phases can drift if reconcile is skipped or
|
||||||
|
crashes mid-way.
|
||||||
|
- **No unified claim-chain view.** A claim's 837 submission, 999 ack,
|
||||||
|
277CA status, and 835 remit are reconstructed by joining across four
|
||||||
|
pages and three queries. The "did this remit's claim ever get
|
||||||
|
matched?" question requires manual SQL.
|
||||||
|
|
||||||
|
### In scope
|
||||||
|
|
||||||
|
Tier 1 (scheduler/ingest split, SFTP hardening, filename classifier):
|
||||||
|
|
||||||
|
1. Move the four per-file-type handlers (`999`, `835`, `277CA`, `TA1`)
|
||||||
|
out of `backend/src/cyclone/scheduler.py` into
|
||||||
|
`backend/src/cyclone/handlers/<type>.py`. Each handler exposes one
|
||||||
|
`handle(text, source_file) -> (parser_used, claim_count)` function.
|
||||||
|
2. Pull `_ack_count_summary`, `_ack_synthetic_source_batch_id`, and
|
||||||
|
`_277ca_synthetic_source_batch_id` out of `scheduler.py` and out of
|
||||||
|
`api.py` into a new `backend/src/cyclone/handlers/_ack_id.py`. Both
|
||||||
|
callers import from one module.
|
||||||
|
3. Loosen `edi/filenames.py:parse_inbound_filename` to accept filenames
|
||||||
|
lacking `_file_type.x12`; fall back to `orig_tx` as the
|
||||||
|
provisional `file_type`, and accept the file as either `835` (if
|
||||||
|
`orig_tx` ends in `835`) or `999`/`277CA` (if `orig_tx` does).
|
||||||
|
4. Wrap `clearhouse.SftpClient._list_inbound_paramiko` /
|
||||||
|
`_list_inbound_names_paramiko` / `_download_inbound_paramiko` /
|
||||||
|
`_read_file_paramiko` calls in `asyncio.wait_for(... timeout=N)`
|
||||||
|
via `asyncio.to_thread`, with `N` configurable via
|
||||||
|
`CYCLONE_SFTP_OP_TIMEOUT_SECONDS` (default 30s).
|
||||||
|
5. Surface SFTP errors in `Scheduler.status()` — `last_error_at`,
|
||||||
|
`last_error`, `consecutive_failures`, `last_sftp_attempt_at`.
|
||||||
|
When `consecutive_failures >= 3`, the operator pill flips to a
|
||||||
|
destructive state until a successful tick clears it.
|
||||||
|
6. Stop swallowing `IOError` from `sftp.listdir_attr()` — return the
|
||||||
|
error in `result.errors` and let the scheduler record `last_error`.
|
||||||
|
|
||||||
|
Tier 2 (reconciliation atomicity, chain view, invariant guards):
|
||||||
|
|
||||||
|
7. Unify 835 ingest + `reconcile.run()` into a single critical
|
||||||
|
section inside `cyclone.handlers.handle_835`. The whole flow
|
||||||
|
(parse → validate → persist batch + remits + CasAdjustments → match
|
||||||
|
claims → write back `adjustment_amount` and `matched_remittance_id`)
|
||||||
|
happens in one `db.SessionLocal()()`. The `Remittance.adjustment_amount`
|
||||||
|
is computed in the same transaction as the `CasAdjustment` rows.
|
||||||
|
8. Add `GET /api/claims/{id}/chain` returning the claim's full chain
|
||||||
|
in one response: `submission` (837 → `Claim` row),
|
||||||
|
`ack_999` (matching `Ack` rows), `ack_277ca` (matching
|
||||||
|
`Two77caAck` rows), `remittance` (`Remittance` row including
|
||||||
|
adjustments and matched claim FK). Empty slots when a piece is
|
||||||
|
missing — never 404 the whole endpoint just because one piece is
|
||||||
|
absent.
|
||||||
|
9. The 999 ack handler and 277CA ack handler — when they reject a
|
||||||
|
claim, and the claim is already matched to a remit — emit an
|
||||||
|
`ActivityEvent` `claim.rejected_after_remit` so the operator
|
||||||
|
surfaces the conflict in the Activity page rather than silently
|
||||||
|
breaking the manual-match workflow.
|
||||||
|
10. Verify the denormalized pair `Claim.matched_remittance_id` ↔
|
||||||
|
`Remittance.claim_id` is written in the same transaction on every
|
||||||
|
write-path (`store.manual_match`, `store.manual_unmatch`, the new
|
||||||
|
atomic 835 handler). Add a startup invariant check: if any Claim
|
||||||
|
has `matched_remittance_id` but the matching Remittance's
|
||||||
|
`claim_id` is unset (or vice versa), log a structured error so an
|
||||||
|
operator sees the drift in the logs.
|
||||||
|
|
||||||
|
### Out of scope (deferred to future increments)
|
||||||
|
|
||||||
|
- **Frontend scheduler operator UI** (start/stop/tick/status page,
|
||||||
|
reconnect button, live log tail). The `/api/admin/scheduler/*`
|
||||||
|
JSON endpoints remain sufficient; the operator uses curl.
|
||||||
|
- **Multi-tenant SFTP blocks** (per-payer or per-tenant). Single
|
||||||
|
dzinesco block remains.
|
||||||
|
- **Docker-secret `_FILE` conventions** beyond what SP26 already added.
|
||||||
|
- **Outbound file pickup verification** (polling HPE's MFT to
|
||||||
|
confirm the operator-dropped 837 was received).
|
||||||
|
- **Frontend chain drawer component.** The `GET /api/claims/{id}/chain`
|
||||||
|
endpoint is exposed; rendering it inside the existing
|
||||||
|
`ClaimDrawer` is a separate UI increment.
|
||||||
|
|
||||||
|
## 2. Decisions (locked during brainstorming)
|
||||||
|
|
||||||
|
1. **Branch base is `Version-1.0.0`, not `main`.** `main` is at
|
||||||
|
`74aa64f` (SP26 only) and is missing the SP25+26 SFTP work, the
|
||||||
|
`[sftp]` docker extra, and the permission-matrix fixes. The
|
||||||
|
production system runs `Version-1.0.0`, so the refactor lands
|
||||||
|
there. A subsequent forward-merge from `Version-1.0.0` → `main`
|
||||||
|
can pull SP27 in if the operator wants it on main.
|
||||||
|
|
||||||
|
2. **One module per handler, not a class.** Each handler exposes one
|
||||||
|
pure function `handle(text: str, source_file: str) -> HandleResult`,
|
||||||
|
where `HandleResult` is a small dataclass with `parser_used`,
|
||||||
|
`claim_count`, `batch_id?`, and per-handler extras (e.g. 835
|
||||||
|
carries `matched_count`). No class hierarchy, no plugin registry —
|
||||||
|
the scheduler keeps its `HANDLERS` dict as the registry.
|
||||||
|
|
||||||
|
3. **The `_FILE` env-var convention from SP26 stays.** This spec
|
||||||
|
doesn't add a new secret tier; the existing `secrets.get_secret()`
|
||||||
|
three-tier lookup (env var → file → Keychain) is reused.
|
||||||
|
|
||||||
|
4. **Time-bounded SFTP operations use `asyncio.to_thread` +
|
||||||
|
`asyncio.wait_for`.** `paramiko` is synchronous; the scheduler
|
||||||
|
already wraps calls in `asyncio.to_thread` for the stub path.
|
||||||
|
Real-mode calls get the same treatment so a hanging `listdir_attr`
|
||||||
|
on the worker thread can be cancelled by the event loop and
|
||||||
|
surfaced as a `Scheduler._last_error`.
|
||||||
|
|
||||||
|
5. **Default SFTP operation timeout is 30s.** `CYCLONE_SFTP_OP_TIMEOUT_SECONDS`
|
||||||
|
lets operators tune it down (e.g. 10s for a flaky VPN uplink) or
|
||||||
|
up for slow connections. The poll-interval itself remains
|
||||||
|
`CYCLONE_SCHEDULER_POLL_SECONDS` (default 60s).
|
||||||
|
|
||||||
|
6. **Reconciliation runs inside the 835 ingest session.** No
|
||||||
|
background reconciler process. If a future need emerges (catch-up
|
||||||
|
reconcile on old batches) it lands in a future SP with a
|
||||||
|
separate `cyclone reconcile` CLI subcommand.
|
||||||
|
|
||||||
|
7. **Chain endpoint renders even when slots are empty.** The contract
|
||||||
|
is "give me this claim's chain; if any piece is missing, return
|
||||||
|
`null` for that field with `missing: ["277ca_ack"]` so the UI can
|
||||||
|
show a placeholder." This matches the operator mental model:
|
||||||
|
"I submitted 837 week-1, the 999 came back accepted, the 277CA
|
||||||
|
came back accepted, but I haven't seen the 835 yet — show me that
|
||||||
|
intermediate state."
|
||||||
|
|
||||||
|
8. **`Scheduler.status()` gains `consecutive_failures`,
|
||||||
|
`last_error_at`, `last_error`, `last_sftp_attempt_at`.**
|
||||||
|
The existing fields (`running`, `poll_interval_seconds`,
|
||||||
|
`sftp_block_name`, `last_poll_at`, `poll_count`, totals, last_tick)
|
||||||
|
are unchanged for backward-compat.
|
||||||
|
|
||||||
|
9. **The `claim.rejected_after_remit` audit event is informational,
|
||||||
|
not blocking.** The 999/277CA handlers still write the rejection
|
||||||
|
even when a matched remit exists; the audit event is the operator's
|
||||||
|
signal to manually unmatch the pair. We do NOT auto-unmatch (would
|
||||||
|
violate the manual-match semantics from T15).
|
||||||
|
|
||||||
|
10. **The startup invariant check is log-only.** It does not block
|
||||||
|
boot. A drift between `Claim.matched_remittance_id` and
|
||||||
|
`Remittance.claim_id` indicates a historical bug; we surface it
|
||||||
|
in the logs and continue. Operators who want to reconcile the
|
||||||
|
drift have a one-shot CLI subcommand (`cyclone reconcile
|
||||||
|
reindex-matches`) deferred to a follow-up SP.
|
||||||
|
|
||||||
|
## 3. Architecture (after SP27)
|
||||||
|
|
||||||
|
```
|
||||||
|
backend/src/cyclone/
|
||||||
|
scheduler.py — slim: Scheduler class + HANDLERS registry + lifecycle
|
||||||
|
handlers/ NEW subpackage
|
||||||
|
__init__.py
|
||||||
|
_ack_id.py — ack_count_summary, ack_synthetic_source_batch_id, *_277ca_*
|
||||||
|
ack.py — top-level ack dispatch (re-exported helpers)
|
||||||
|
handle_999.py — text → parsed 999 → ack row → claim rejections
|
||||||
|
handle_835.py — text → parsed 835 → batch + remits + CasAdjustments
|
||||||
|
+ match claims (atomic one-section)
|
||||||
|
handle_277ca.py — text → parsed 277CA → ack row + claim rejections
|
||||||
|
+ emit claim.rejected_after_remit
|
||||||
|
handle_ta1.py — text → parsed TA1 → ack row
|
||||||
|
clearhouse/__init__.py — _paramiko ops wrapped in asyncio.wait_for via to_thread
|
||||||
|
edi/filenames.py — parse_inbound_filename accepts suffix-less names
|
||||||
|
store.py — manual_match / manual_unmatch keep matched_remittance_id
|
||||||
|
and Remittance.claim_id in sync (same transaction);
|
||||||
|
startup invariant check
|
||||||
|
api.py — /api/parse-835 deduped vs scheduler handlers via
|
||||||
|
handlers.handle_835; new /api/claims/{id}/chain endpoint
|
||||||
|
reconcile.py — run() folded into handle_835; top-level `match()`
|
||||||
|
kept for the CLI subcommand (deferred)
|
||||||
|
api_routers/chain.py NEW — GET /api/claims/{id}/chain (auth, matrix_gate)
|
||||||
|
|
||||||
|
src/
|
||||||
|
hooks/useClaimChain.ts NEW — TanStack Query wrapper around /api/claims/{id}/chain
|
||||||
|
pages/ClaimDrawer.tsx — one new section: "Chain" with submission / 999 / 277CA / remit
|
||||||
|
placeholders; renders adjustments inline
|
||||||
|
```
|
||||||
|
|
||||||
|
The scheduler shrinks from 860 LOC to ~250 LOC (the Scheduler class +
|
||||||
|
singleton plumbing). Each `handle_*.py` is ~80–150 LOC and can be
|
||||||
|
tested in isolation against a real prodfiles fixture.
|
||||||
|
|
||||||
|
## 4. Files
|
||||||
|
|
||||||
|
**New:**
|
||||||
|
|
||||||
|
- `backend/src/cyclone/handlers/__init__.py`
|
||||||
|
- `backend/src/cyclone/handlers/_ack_id.py`
|
||||||
|
- `backend/src/cyclone/handlers/handle_999.py`
|
||||||
|
- `backend/src/cyclone/handlers/handle_835.py`
|
||||||
|
- `backend/src/cyclone/handlers/handle_277ca.py`
|
||||||
|
- `backend/src/cyclone/handlers/handle_ta1.py`
|
||||||
|
- `backend/src/cyclone/api_routers/chain.py`
|
||||||
|
- `src/hooks/useClaimChain.ts`
|
||||||
|
- `backend/tests/test_handlers_999.py`
|
||||||
|
- `backend/tests/test_handlers_835.py`
|
||||||
|
- `backend/tests/test_handlers_277ca.py`
|
||||||
|
- `backend/tests/test_handlers_ta1.py`
|
||||||
|
- `backend/tests/test_inbound_filename_loose.py`
|
||||||
|
- `backend/tests/test_sftp_op_timeout.py`
|
||||||
|
- `backend/tests/test_scheduler_status_errors.py`
|
||||||
|
- `backend/tests/test_handler_835_atomic_reconcile.py`
|
||||||
|
- `backend/tests/test_api_claim_chain.py`
|
||||||
|
- `backend/tests/test_handler_277ca_rejected_after_remit.py`
|
||||||
|
- `backend/tests/test_store_match_invariant.py`
|
||||||
|
- `src/hooks/useClaimChain.test.ts`
|
||||||
|
- `src/pages/ClaimDrawer.test.tsx` (extension — the chain drawer section)
|
||||||
|
|
||||||
|
**Modified:**
|
||||||
|
|
||||||
|
- `backend/src/cyclone/scheduler.py` — slim to Scheduler class + lifecycle
|
||||||
|
- `backend/src/cyclone/edi/filenames.py` — `parse_inbound_filename`
|
||||||
|
accepts suffix-less inbound filenames; `is_inbound_filename` likewise
|
||||||
|
- `backend/src/cyclone/clearhouse/__init__.py` — wrap paramiko
|
||||||
|
operations in `asyncio.wait_for(asyncio.to_thread(...), timeout=N)`
|
||||||
|
- `backend/src/cyclone/store.py` — `manual_match` /
|
||||||
|
`manual_unmatch` write the pair in one transaction; startup
|
||||||
|
invariant log; `add` (835 branch) calls `handlers.handle_835`
|
||||||
|
instead of inlining
|
||||||
|
- `backend/src/cyclone/reconcile.py` — `run` exposed for the
|
||||||
|
follow-up CLI subcommand; `match` kept; the scheduler / 835
|
||||||
|
handler stop calling `run` directly
|
||||||
|
- `backend/src/cyclone/api.py` — `/api/parse-835` reuses
|
||||||
|
`handlers.handle_835`; `/api/parse-999`, `/api/parse-277ca`,
|
||||||
|
`/api/parse-ta1` likewise; add `/api/claims/{id}/chain` route
|
||||||
|
- `backend/src/cyclone/api_helpers.py` (or new file) — wire
|
||||||
|
`/api/claims/{id}/chain` if the route doesn't fit the existing
|
||||||
|
pattern
|
||||||
|
- `src/pages/ClaimDrawer.tsx` — one new "Chain" section
|
||||||
|
- `src/lib/api.ts` — expose `api.fetchClaimChain(id)`
|
||||||
|
- `src/types/index.ts` — add `ClaimChain` type
|
||||||
|
|
||||||
|
**New migrations:** none. The DB schema is unchanged.
|
||||||
|
|
||||||
|
## 5. API surface
|
||||||
|
|
||||||
|
| Method | Path | Returns | Auth |
|
||||||
|
|---|---|---|---|
|
||||||
|
| GET | `/api/claims/{id}/chain` | `ClaimChain` JSON | `matrix_gate` (any logged-in user) |
|
||||||
|
|
||||||
|
No new endpoints beyond this one. No new env-var-driven behavior
|
||||||
|
beyond `CYCLONE_SFTP_OP_TIMEOUT_SECONDS` (default 30s).
|
||||||
|
|
||||||
|
```jsonc
|
||||||
|
// GET /api/claims/{id}/chain
|
||||||
|
{
|
||||||
|
"claim_id": "...",
|
||||||
|
"submission": {
|
||||||
|
"batch_id": "...",
|
||||||
|
"patient_control_number": "...",
|
||||||
|
"service_date_from": "2026-06-15",
|
||||||
|
"service_date_to": "2026-06-15",
|
||||||
|
"charge_amount": "125.00",
|
||||||
|
"state": "submitted",
|
||||||
|
"submitted_at": "2026-06-16T..."
|
||||||
|
},
|
||||||
|
"ack_999": {
|
||||||
|
"source_batch_id": "999-PCN-12345678",
|
||||||
|
"ack_code": "A",
|
||||||
|
"received_count": 1,
|
||||||
|
"accepted_count": 1,
|
||||||
|
"rejected_count": 0,
|
||||||
|
"received_at": "2026-06-17T..."
|
||||||
|
} | null,
|
||||||
|
"ack_277ca": {
|
||||||
|
"source_batch_id": "277CA-000000001",
|
||||||
|
"classification": "accepted",
|
||||||
|
"received_at": "2026-06-18T..."
|
||||||
|
} | null,
|
||||||
|
"remittance": {
|
||||||
|
"id": "...",
|
||||||
|
"payer_claim_control_number": "...",
|
||||||
|
"total_paid": "85.00",
|
||||||
|
"patient_responsibility": "15.00",
|
||||||
|
"adjustment_amount": "25.00",
|
||||||
|
"status_label": "Paid",
|
||||||
|
"received_at": "2026-06-19T...",
|
||||||
|
"matched": true,
|
||||||
|
"adjustments": [
|
||||||
|
{ "group_code": "CO", "reason_code": "97", "amount": "25.00", "label": "..." }
|
||||||
|
]
|
||||||
|
} | null,
|
||||||
|
"missing": ["277ca_ack"] // list of any of the above that came back null
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## 6. Env vars (operator-facing)
|
||||||
|
|
||||||
|
| Variable | Default | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| `CYCLONE_SFTP_OP_TIMEOUT_SECONDS` | `30` | Per-operation SFTP timeout in `clearhouse`. Lower for flaky links, higher for slow ones. The existing `CYCLONE_SCHEDULER_POLL_SECONDS` and `CYCLONE_SFTP_PASSWORD[_FILE]` from SP25+26 are unchanged. |
|
||||||
|
|
||||||
|
No new env vars beyond this one.
|
||||||
|
|
||||||
|
## 7. Validation rules
|
||||||
|
|
||||||
|
No new R-code rules. The 835 validator (`parser_835.validator_835`)
|
||||||
|
is called unchanged by `handlers.handle_835` inside the same
|
||||||
|
transaction; the new chain endpoint doesn't introduce new parser
|
||||||
|
or validation logic.
|
||||||
|
|
||||||
|
## 8. Testing plan
|
||||||
|
|
||||||
|
Target backend test count after SP27: current + 17 new tests.
|
||||||
|
|
||||||
|
1. **Per-handler unit tests** (4 files × ~3 tests each):
|
||||||
|
`test_handlers_999.py`, `test_handlers_835.py`,
|
||||||
|
`test_handlers_277ca.py`, `test_handlers_ta1.py` — call each
|
||||||
|
handler against a prodfiles fixture and assert the persisted
|
||||||
|
rows.
|
||||||
|
2. **`test_inbound_filename_loose.py`** (4 cases):
|
||||||
|
- filename with `_835.x12` suffix → `file_type="835"` (existing behavior).
|
||||||
|
- filename without `_file_type.x12` and `orig_tx=835` → `file_type="835"` (new).
|
||||||
|
- filename without suffix and `orig_tx=999` → `file_type="999"` (new).
|
||||||
|
- filename without suffix and `orig_tx=837P` → `file_type="999"` rejected by `ALLOWED_FILE_TYPES` — no false positive for 837s.
|
||||||
|
3. **`test_sftp_op_timeout.py`** (3 cases): stub SFTP returns a slow
|
||||||
|
`sftp.listdir_attr` (raises after 60s); assert the scheduler
|
||||||
|
times out within `CYCLONE_SFTP_OP_TIMEOUT_SECONDS` + slack;
|
||||||
|
stub returns fast → assert happy path still works.
|
||||||
|
4. **`test_scheduler_status_errors.py`** (4 cases): `consecutive_failures`
|
||||||
|
increments on tick error; `last_error` populated;
|
||||||
|
`status_sftp_health` flips to `failing` after 3 fails; flips back
|
||||||
|
on next success.
|
||||||
|
5. **`test_handler_835_atomic_reconcile.py`** (3 cases): 835 ingest
|
||||||
|
persists `Remittance` with `adjustment_amount` correct from the
|
||||||
|
start; reconcile crash mid-transaction leaves no orphans; an
|
||||||
|
existing claim gets `matched_remittance_id` set in the same
|
||||||
|
transaction.
|
||||||
|
6. **`test_api_claim_chain.py`** (5 cases): happy path with all four
|
||||||
|
slots populated; missing ack_999 → `null + missing=["ack_999"]`;
|
||||||
|
missing remittance → `null + missing=["remittance"]`; missing
|
||||||
|
claim → 404; auth → 401.
|
||||||
|
7. **`test_handler_277ca_rejected_after_remit.py`** (2 cases):
|
||||||
|
277CA rejects a claim that is already matched → emits
|
||||||
|
`claim.rejected_after_remit` audit event with both IDs;
|
||||||
|
277CA rejects an unmatched claim → no audit event.
|
||||||
|
8. **`test_store_match_invariant.py`** (3 cases):
|
||||||
|
`manual_match` writes `Claim.matched_remittance_id` and
|
||||||
|
`Remittance.claim_id` in one transaction;
|
||||||
|
`manual_unmatch` clears both; pre-existing drift is logged at
|
||||||
|
startup without blocking boot.
|
||||||
|
9. **Frontend useClaimChain.test.ts** (3 cases): hook fetches the
|
||||||
|
chain and returns typed data; loading/error states; caches
|
||||||
|
for 30s.
|
||||||
|
10. **Frontend ClaimDrawer.test.tsx extension** (2 cases): renders
|
||||||
|
the chain section; renders placeholders for missing pieces.
|
||||||
|
|
||||||
|
The full backend suite (`cd backend && .venv/bin/pytest`) and
|
||||||
|
frontend suite (`npm test`) remain the merge gate.
|
||||||
|
|
||||||
|
## 9. Out of scope (future SPs)
|
||||||
|
|
||||||
|
- Frontend scheduler operator UI (start/stop/tick/status page).
|
||||||
|
- Multi-tenant / per-payer SFTP blocks.
|
||||||
|
- `<NAME>_FILE` Docker-secret convention beyond what SP26 added.
|
||||||
|
- Outbound file pickup verification.
|
||||||
|
- Auto-unmatch on rejection (would violate T15 manual-match contract).
|
||||||
|
- `cyclone reconcile reindex-matches` one-shot CLI for historical drift.
|
||||||
|
- Per-payer chain-fanout (different payer-specific parsing of the same
|
||||||
|
277CA segment).
|
||||||
|
- Persisting the chain response server-side (it is a join, not a stored
|
||||||
|
view; keeping it computed avoids drift).
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
import { useQuery } from "@tanstack/react-query";
|
import { useQuery } from "@tanstack/react-query";
|
||||||
import { api, type PaginatedResponse } from "@/lib/api";
|
import { api, type AckAggregates, type PaginatedResponse } from "@/lib/api";
|
||||||
import type { Ack } from "@/types";
|
import type { Ack } from "@/types";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -7,9 +7,14 @@ import type { Ack } from "@/types";
|
|||||||
* intentionally has no in-memory fallback (there is no zustand
|
* intentionally has no in-memory fallback (there is no zustand
|
||||||
* sample-data path for ACKs in v1 — the UI is empty until the
|
* sample-data path for ACKs in v1 — the UI is empty until the
|
||||||
* backend serves real rows).
|
* backend serves real rows).
|
||||||
|
*
|
||||||
|
* The response carries `aggregates` summed over the *full* row set
|
||||||
|
* (not just the visible page) so the KPI strip on the Acks page
|
||||||
|
* reflects every persisted 999 — without this, totals silently
|
||||||
|
* under-report once the row count exceeds the page size.
|
||||||
*/
|
*/
|
||||||
export function useAcks(params: { limit?: number } = {}) {
|
export function useAcks(params: { limit?: number; offset?: number } = {}) {
|
||||||
return useQuery<PaginatedResponse<Ack>>({
|
return useQuery<PaginatedResponse<Ack> & { aggregates: AckAggregates }>({
|
||||||
queryKey: ["acks", params],
|
queryKey: ["acks", params],
|
||||||
queryFn: () => api.listAcks(params),
|
queryFn: () => api.listAcks(params),
|
||||||
enabled: api.isConfigured,
|
enabled: api.isConfigured,
|
||||||
|
|||||||
@@ -0,0 +1,126 @@
|
|||||||
|
// @vitest-environment happy-dom
|
||||||
|
// SP27 Task 13: server-aggregated Dashboard KPIs.
|
||||||
|
//
|
||||||
|
// These tests pin the hook's contract independent of the Dashboard
|
||||||
|
// page: when the backend is wired, it calls ``api.getDashboardKpis``
|
||||||
|
// with the requested parameters and returns the resolved value; when
|
||||||
|
// the backend is not wired, it returns ``data: undefined`` so the
|
||||||
|
// Dashboard can render zero-shaped fallbacks.
|
||||||
|
(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
|
||||||
|
|
||||||
|
import { describe, expect, it, vi, beforeEach } from "vitest";
|
||||||
|
import { renderHook, waitFor } from "@testing-library/react";
|
||||||
|
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||||
|
import React from "react";
|
||||||
|
import { api, type DashboardKpis } from "@/lib/api";
|
||||||
|
|
||||||
|
// Mock the api module so we can control isConfigured + getDashboardKpis
|
||||||
|
// without spinning up a real backend.
|
||||||
|
vi.mock("@/lib/api", async () => {
|
||||||
|
const actual = await vi.importActual<typeof import("@/lib/api")>("@/lib/api");
|
||||||
|
return {
|
||||||
|
...actual,
|
||||||
|
api: {
|
||||||
|
...actual.api,
|
||||||
|
getDashboardKpis: vi.fn(),
|
||||||
|
},
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
// Import AFTER the mock so the hook sees the mocked api.
|
||||||
|
const { useDashboardKpis } = await import("./useDashboardKpis");
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Helpers
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
function makeWrapper() {
|
||||||
|
const client = new QueryClient({
|
||||||
|
defaultOptions: {
|
||||||
|
queries: { retry: false, gcTime: 0, staleTime: 0 },
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return ({ children }: { children: React.ReactNode }) =>
|
||||||
|
React.createElement(QueryClientProvider, { client }, children);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Tests
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
describe("useDashboardKpis", () => {
|
||||||
|
beforeEach(() => {
|
||||||
|
vi.clearAllMocks();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("calls api.getDashboardKpis and returns the resolved value when configured", async () => {
|
||||||
|
const payload: DashboardKpis = {
|
||||||
|
totals: {
|
||||||
|
count: 60295,
|
||||||
|
billed: 940143.58,
|
||||||
|
received: 59753,
|
||||||
|
outstandingAr: 880390.58,
|
||||||
|
denied: 542,
|
||||||
|
denialRate: 0.9,
|
||||||
|
pending: 60,
|
||||||
|
},
|
||||||
|
monthly: [
|
||||||
|
{ month: "2026-01", label: "Jan", count: 100, billed: 15000,
|
||||||
|
received: 12000, denied: 2, denialRate: 2.0, ar: 3000 },
|
||||||
|
],
|
||||||
|
topProviders: [
|
||||||
|
{ npi: "1234567893", label: "Cedar Park", claimCount: 200,
|
||||||
|
billed: 30000, denied: 1 },
|
||||||
|
],
|
||||||
|
topDenials: [
|
||||||
|
{ id: "C-1", patientName: "Jane Doe", billedAmount: 250,
|
||||||
|
denialReason: "Missing modifier", submissionDate: "2026-06-20T12:00:00Z" },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
(api.getDashboardKpis as ReturnType<typeof vi.fn>).mockResolvedValue(payload);
|
||||||
|
// Override the isConfigured bit too — the hook keys off this.
|
||||||
|
(api as unknown as { isConfigured: boolean }).isConfigured = true;
|
||||||
|
|
||||||
|
const { result } = renderHook(() => useDashboardKpis({ months: 6 }), {
|
||||||
|
wrapper: makeWrapper(),
|
||||||
|
});
|
||||||
|
|
||||||
|
await waitFor(() => {
|
||||||
|
expect(result.current.data).toEqual(payload);
|
||||||
|
});
|
||||||
|
expect(api.getDashboardKpis).toHaveBeenCalledWith({ months: 6 });
|
||||||
|
expect(result.current.isError).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("returns data: undefined when the backend is not configured", () => {
|
||||||
|
(api as unknown as { isConfigured: boolean }).isConfigured = false;
|
||||||
|
|
||||||
|
const { result } = renderHook(() => useDashboardKpis(), {
|
||||||
|
wrapper: makeWrapper(),
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(result.current.data).toBeUndefined();
|
||||||
|
expect(result.current.isLoading).toBe(false);
|
||||||
|
expect(result.current.isError).toBe(false);
|
||||||
|
// queryFn should NOT have been called — bypassed entirely.
|
||||||
|
expect(api.getDashboardKpis).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("passes top_n_providers / top_n_denials through to the API", async () => {
|
||||||
|
(api.getDashboardKpis as ReturnType<typeof vi.fn>).mockResolvedValue({
|
||||||
|
totals: { count: 0, billed: 0, received: 0, outstandingAr: 0,
|
||||||
|
denied: 0, denialRate: 0, pending: 0 },
|
||||||
|
monthly: [], topProviders: [], topDenials: [],
|
||||||
|
});
|
||||||
|
(api as unknown as { isConfigured: boolean }).isConfigured = true;
|
||||||
|
|
||||||
|
renderHook(
|
||||||
|
() => useDashboardKpis({ months: 3, top_n_providers: 2, top_n_denials: 5 }),
|
||||||
|
{ wrapper: makeWrapper() },
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(api.getDashboardKpis).toHaveBeenCalledWith({
|
||||||
|
months: 3, top_n_providers: 2, top_n_denials: 5,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
import { useQuery } from "@tanstack/react-query";
|
||||||
|
import { api, type DashboardKpis, type DashboardKpisParams } from "@/lib/api";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Server-aggregated Dashboard KPIs.
|
||||||
|
*
|
||||||
|
* Replaces the previous `useClaims({ limit: 100 })` + client-side
|
||||||
|
* reduce pattern. With 60k+ claims in production, paginating
|
||||||
|
* ``/api/claims`` and reducing client-side silently produced wrong
|
||||||
|
* numbers — the Dashboard's "Billed / Received / Denial rate / Pending
|
||||||
|
* AR" tiles were computed from a 100-row sample, not the full
|
||||||
|
* population. This hook hits ``GET /api/dashboard/kpis`` which does
|
||||||
|
* the aggregation server-side in a single read.
|
||||||
|
*
|
||||||
|
* Refreshes every 60s when the backend is configured so the KPIs stay
|
||||||
|
* roughly current without a manual reload. Live event-publish from
|
||||||
|
* ``useTailStream`` would be more elegant, but the underlying
|
||||||
|
* aggregates span the whole DB so we'd need a new "kpi_updated"
|
||||||
|
* event; 60s polling is a clear-enough SLA for a Dashboard view.
|
||||||
|
*
|
||||||
|
* Sample-data mode (no backend wired): returns ``data: undefined``
|
||||||
|
* rather than calling ``useQuery``. The Dashboard handles that with
|
||||||
|
* zero-shaped fallbacks so the KPI tiles render a coherent "0" rather
|
||||||
|
* than throwing. There's no in-memory aggregator — the old
|
||||||
|
* client-side reduce was the bug, not a feature to preserve.
|
||||||
|
*/
|
||||||
|
export function useDashboardKpis(params: DashboardKpisParams = {}) {
|
||||||
|
const q = useQuery<DashboardKpis>({
|
||||||
|
queryKey: ["dashboard", "kpis", params],
|
||||||
|
queryFn: () => api.getDashboardKpis(params),
|
||||||
|
enabled: api.isConfigured,
|
||||||
|
refetchInterval: api.isConfigured ? 60_000 : false,
|
||||||
|
staleTime: 30_000,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!api.isConfigured) {
|
||||||
|
return {
|
||||||
|
data: undefined,
|
||||||
|
isLoading: false,
|
||||||
|
isError: false,
|
||||||
|
error: null,
|
||||||
|
refetch: () => Promise.resolve(),
|
||||||
|
dataUpdatedAt: 0,
|
||||||
|
} as const;
|
||||||
|
}
|
||||||
|
return q;
|
||||||
|
}
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
import { useQuery } from "@tanstack/react-query";
|
||||||
|
import {
|
||||||
|
api,
|
||||||
|
type ListRemittancesParams,
|
||||||
|
type RemittanceSummary,
|
||||||
|
} from "@/lib/api";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Server-aggregated KPI totals for the Remittances page tiles.
|
||||||
|
* Returns ``{count, total_paid, total_adjustments}`` summed over the
|
||||||
|
* full filtered remittance population — NOT a page-limited sample.
|
||||||
|
*
|
||||||
|
* The page's KPI tiles consume this hook instead of summing
|
||||||
|
* ``items.reduce(...)`` over the current page (25 rows) + live-tail
|
||||||
|
* delta, which silently understates the true population in the same
|
||||||
|
* shape that ``59c3275`` retired for the Dashboard and that
|
||||||
|
* ``d81b6ed`` retired for the count tile.
|
||||||
|
*
|
||||||
|
* The optional ``params`` object lets the page pass through the same
|
||||||
|
* filter chips it uses for the list endpoint (``status``, ``payer``,
|
||||||
|
* ``date_from``, ``date_to``) so the summary and the row list always
|
||||||
|
* agree on which rows they're describing.
|
||||||
|
*/
|
||||||
|
export function useRemittanceSummary(params: ListRemittancesParams = {}) {
|
||||||
|
return useQuery<RemittanceSummary>({
|
||||||
|
queryKey: ["remittances", "summary", params],
|
||||||
|
queryFn: () => api.listRemittanceSummary(params),
|
||||||
|
enabled: api.isConfigured,
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
import { useQuery } from "@tanstack/react-query";
|
||||||
|
import { api, type PaginatedResponse } from "@/lib/api";
|
||||||
|
import type { Ta1Ack } from "@/types";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Lists persisted TA1 (Interchange Acknowledgment) rows, newest
|
||||||
|
* first. Mirrors `useAcks` but for the lower-level envelope ack.
|
||||||
|
*
|
||||||
|
* A TA1 is one row per inbound ISA/IEA interchange — distinct from
|
||||||
|
* a 999, which is per-batch. Colorado Medicaid's Gainwell MFT
|
||||||
|
* currently only ships 999s in the FromHPE path, but historically
|
||||||
|
* they've sent TA1s, so the hook stays in the surface for when
|
||||||
|
* they reappear.
|
||||||
|
*
|
||||||
|
* No in-memory fallback: there is no zustand sample-data path for
|
||||||
|
* TA1s in v1. The hook is `enabled: api.isConfigured` so the page
|
||||||
|
* treats an empty list as "no TA1s on file" rather than a
|
||||||
|
* configuration error.
|
||||||
|
*/
|
||||||
|
export function useTa1Acks(params: { limit?: number } = {}) {
|
||||||
|
return useQuery<PaginatedResponse<Ta1Ack>>({
|
||||||
|
queryKey: ["ta1-acks", params],
|
||||||
|
queryFn: () => api.listTa1Acks(params),
|
||||||
|
enabled: api.isConfigured,
|
||||||
|
});
|
||||||
|
}
|
||||||
+194
-1
@@ -39,6 +39,7 @@ import type {
|
|||||||
Payee835,
|
Payee835,
|
||||||
Provider,
|
Provider,
|
||||||
ReassociationTrace,
|
ReassociationTrace,
|
||||||
|
Ta1Ack,
|
||||||
UnmatchedClaim,
|
UnmatchedClaim,
|
||||||
UnmatchedResponse,
|
UnmatchedResponse,
|
||||||
BatchSummary as ParserBatchSummary,
|
BatchSummary as ParserBatchSummary,
|
||||||
@@ -152,6 +153,66 @@ export interface ListActivityParams {
|
|||||||
limit?: number;
|
limit?: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Dashboard KPI types (SP27 Task 13).
|
||||||
|
//
|
||||||
|
// Returned by ``GET /api/dashboard/kpis`` — server-aggregated over the
|
||||||
|
// *entire* claim population. The Dashboard renders these directly; it
|
||||||
|
// no longer paginates ``/api/claims`` and reduces client-side (which
|
||||||
|
// silently produced wrong numbers with the previous ``limit: 100``
|
||||||
|
// cap).
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
export interface DashboardTotals {
|
||||||
|
count: number;
|
||||||
|
billed: number;
|
||||||
|
received: number;
|
||||||
|
outstandingAr: number;
|
||||||
|
denied: number;
|
||||||
|
denialRate: number;
|
||||||
|
pending: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DashboardMonthly {
|
||||||
|
month: string; // "YYYY-MM"
|
||||||
|
label: string; // "Jan"
|
||||||
|
count: number;
|
||||||
|
billed: number;
|
||||||
|
received: number;
|
||||||
|
denied: number;
|
||||||
|
denialRate: number;
|
||||||
|
ar: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DashboardTopProvider {
|
||||||
|
npi: string;
|
||||||
|
label: string;
|
||||||
|
claimCount: number;
|
||||||
|
billed: number;
|
||||||
|
denied: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DashboardTopDenial {
|
||||||
|
id: string;
|
||||||
|
patientName: string;
|
||||||
|
billedAmount: number;
|
||||||
|
denialReason: string | null;
|
||||||
|
submissionDate: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DashboardKpis {
|
||||||
|
totals: DashboardTotals;
|
||||||
|
monthly: DashboardMonthly[];
|
||||||
|
topProviders: DashboardTopProvider[];
|
||||||
|
topDenials: DashboardTopDenial[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DashboardKpisParams {
|
||||||
|
months?: number;
|
||||||
|
top_n_providers?: number;
|
||||||
|
top_n_denials?: number;
|
||||||
|
}
|
||||||
|
|
||||||
export interface PaginatedResponse<T> {
|
export interface PaginatedResponse<T> {
|
||||||
items: T[];
|
items: T[];
|
||||||
total: number;
|
total: number;
|
||||||
@@ -159,6 +220,19 @@ export interface PaginatedResponse<T> {
|
|||||||
has_more: boolean;
|
has_more: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Server-side aggregates returned alongside `/api/acks` — summed over
|
||||||
|
* the *full* persisted row set (not just the visible page). The
|
||||||
|
* `accepted` / `rejected` / `received` keys match the in-page `totals`
|
||||||
|
* shape so the page can use `data.aggregates` as a drop-in for the
|
||||||
|
* page-local fallback accumulator.
|
||||||
|
*/
|
||||||
|
export interface AckAggregates {
|
||||||
|
accepted: number;
|
||||||
|
rejected: number;
|
||||||
|
received: number;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Lightweight summary used by the GET /api/batches list endpoint. Distinct
|
* Lightweight summary used by the GET /api/batches list endpoint. Distinct
|
||||||
* from the parser's `BatchSummary` (which lives in `@/types` and carries the
|
* from the parser's `BatchSummary` (which lives in `@/types` and carries the
|
||||||
@@ -577,6 +651,28 @@ async function listRemittances<T = unknown>(
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Server-aggregated KPI tiles for the Remittances page. Returns the
|
||||||
|
* full-population ``count``, ``total_paid``, and ``total_adjustments``
|
||||||
|
* — NOT a page-limited sample — so the KPI tiles can't silently
|
||||||
|
* understate the true DB population the way a page-local
|
||||||
|
* ``items.reduce(...)`` would (commits ``59c3275``, ``d81b6ed``).
|
||||||
|
*/
|
||||||
|
export interface RemittanceSummary {
|
||||||
|
count: number;
|
||||||
|
total_paid: number;
|
||||||
|
total_adjustments: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function listRemittanceSummary(
|
||||||
|
params: ListRemittancesParams = {}
|
||||||
|
): Promise<RemittanceSummary> {
|
||||||
|
if (!isConfigured) throw notConfiguredError();
|
||||||
|
return authedFetch<RemittanceSummary>(
|
||||||
|
`/api/remittances/summary${qs(params as Record<string, unknown>)}`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Fetch one remittance with its labeled CAS `adjustments` array.
|
* Fetch one remittance with its labeled CAS `adjustments` array.
|
||||||
* Throws `ApiError` on 404 so callers can branch on `.status`.
|
* Throws `ApiError` on 404 so callers can branch on `.status`.
|
||||||
@@ -688,6 +784,26 @@ async function listActivity<T = unknown>(
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fetch server-aggregated Dashboard KPIs.
|
||||||
|
*
|
||||||
|
* Drives ``GET /api/dashboard/kpis``. Computes billed / received /
|
||||||
|
* denial rate / pending AR / top providers / top denials server-side
|
||||||
|
* over the *full* claim population so the Dashboard's numbers are
|
||||||
|
* always correct regardless of dataset size. With 60k+ claims in
|
||||||
|
* production, fetching ``/api/claims?limit=100`` and reducing
|
||||||
|
* client-side silently produced wrong KPIs — this endpoint replaces
|
||||||
|
* that pattern.
|
||||||
|
*/
|
||||||
|
async function getDashboardKpis(
|
||||||
|
params: DashboardKpisParams = {}
|
||||||
|
): Promise<DashboardKpis> {
|
||||||
|
if (!isConfigured) throw notConfiguredError();
|
||||||
|
return authedFetch<DashboardKpis>(
|
||||||
|
`/api/dashboard/kpis${qs(params as Record<string, unknown>)}`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// Public surface — reconciliation endpoints (sub-project 2)
|
// Public surface — reconciliation endpoints (sub-project 2)
|
||||||
// POSTs throw `ApiError` so callers can inspect `.status`; the GET is shaped
|
// POSTs throw `ApiError` so callers can inspect `.status`; the GET is shaped
|
||||||
@@ -752,21 +868,37 @@ function mapAck(row: RawAckRow): Ack {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
async function listAcks(params: { limit?: number } = {}): Promise<PaginatedResponse<Ack>> {
|
async function listAcks(
|
||||||
|
params: { limit?: number; offset?: number } = {},
|
||||||
|
): Promise<
|
||||||
|
PaginatedResponse<Ack> & {
|
||||||
|
aggregates: AckAggregates;
|
||||||
|
}
|
||||||
|
> {
|
||||||
if (!isConfigured) throw notConfiguredError();
|
if (!isConfigured) throw notConfiguredError();
|
||||||
const query: Record<string, unknown> = {};
|
const query: Record<string, unknown> = {};
|
||||||
if (params.limit !== undefined) query.limit = params.limit;
|
if (params.limit !== undefined) query.limit = params.limit;
|
||||||
|
if (params.offset !== undefined) query.offset = params.offset;
|
||||||
const body = await authedFetch<{
|
const body = await authedFetch<{
|
||||||
items: RawAckRow[];
|
items: RawAckRow[];
|
||||||
total: number;
|
total: number;
|
||||||
returned: number;
|
returned: number;
|
||||||
has_more: boolean;
|
has_more: boolean;
|
||||||
|
aggregates: { accepted_count: number; rejected_count: number; received_count: number };
|
||||||
}>(`/api/acks${qs(query)}`);
|
}>(`/api/acks${qs(query)}`);
|
||||||
return {
|
return {
|
||||||
items: body.items.map(mapAck),
|
items: body.items.map(mapAck),
|
||||||
total: body.total,
|
total: body.total,
|
||||||
returned: body.returned,
|
returned: body.returned,
|
||||||
has_more: body.has_more,
|
has_more: body.has_more,
|
||||||
|
// Adapt the wire-format `*_count` keys to the in-page `totals` shape
|
||||||
|
// so the page can treat server-side and client-side-fallback objects
|
||||||
|
// interchangeably. See useAcks.ts for the matching AckAggregates type.
|
||||||
|
aggregates: {
|
||||||
|
accepted: body.aggregates.accepted_count,
|
||||||
|
rejected: body.aggregates.rejected_count,
|
||||||
|
received: body.aggregates.received_count,
|
||||||
|
},
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -778,6 +910,64 @@ async function getAck(id: number): Promise<Ack & { rawJson: unknown }> {
|
|||||||
return { ...mapAck(row), rawJson: row.raw_json };
|
return { ...mapAck(row), rawJson: row.raw_json };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Public surface — TA1 ACKs
|
||||||
|
// The TA1 is the lowest-level X12 envelope ack (one per inbound
|
||||||
|
// ISA/IEA interchange), distinct from the per-batch 999. Colorado
|
||||||
|
// Medicaid's Gainwell MFT only ships 999s today, but historically
|
||||||
|
// they've sent TA1s, so the UI shows them whenever one is on file.
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
interface RawTa1Row {
|
||||||
|
id: number;
|
||||||
|
control_number: string;
|
||||||
|
ack_code: "A" | "E" | "R";
|
||||||
|
note_code: string | null;
|
||||||
|
interchange_date: string | null;
|
||||||
|
interchange_time: string | null;
|
||||||
|
sender_id: string | null;
|
||||||
|
receiver_id: string | null;
|
||||||
|
source_batch_id: string;
|
||||||
|
parsed_at: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function mapTa1Ack(row: RawTa1Row): Ta1Ack {
|
||||||
|
return {
|
||||||
|
id: row.id,
|
||||||
|
controlNumber: row.control_number,
|
||||||
|
ackCode: row.ack_code,
|
||||||
|
noteCode: row.note_code,
|
||||||
|
interchangeDate: row.interchange_date,
|
||||||
|
interchangeTime: row.interchange_time,
|
||||||
|
senderId: row.sender_id,
|
||||||
|
receiverId: row.receiver_id,
|
||||||
|
sourceBatchId: row.source_batch_id,
|
||||||
|
parsedAt: row.parsed_at,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function listTa1Acks(
|
||||||
|
params: { limit?: number } = {},
|
||||||
|
): Promise<PaginatedResponse<Ta1Ack>> {
|
||||||
|
if (!isConfigured) throw notConfiguredError();
|
||||||
|
const query: Record<string, unknown> = {};
|
||||||
|
if (params.limit !== undefined) query.limit = params.limit;
|
||||||
|
const body = await authedFetch<{
|
||||||
|
items: RawTa1Row[];
|
||||||
|
total: number;
|
||||||
|
}>(`/api/ta1-acks${qs(query)}`);
|
||||||
|
// The TA1 list endpoint returns `{ total, items }` (no `has_more` /
|
||||||
|
// `returned` — it's a simple cap-based list, not a paginated one).
|
||||||
|
// Synthesize the `PaginatedResponse` shape so the UI can share the
|
||||||
|
// same hook contract as `listAcks`.
|
||||||
|
return {
|
||||||
|
items: body.items.map(mapTa1Ack),
|
||||||
|
total: body.total,
|
||||||
|
returned: body.items.length,
|
||||||
|
has_more: body.items.length < body.total,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Download a ZIP of regenerated X12 837 files for a parsed batch.
|
* Download a ZIP of regenerated X12 837 files for a parsed batch.
|
||||||
*
|
*
|
||||||
@@ -861,6 +1051,7 @@ export const api = {
|
|||||||
serializeClaim837,
|
serializeClaim837,
|
||||||
exportBatch837,
|
exportBatch837,
|
||||||
listRemittances,
|
listRemittances,
|
||||||
|
listRemittanceSummary,
|
||||||
getRemittance,
|
getRemittance,
|
||||||
listProviders,
|
listProviders,
|
||||||
getProvider,
|
getProvider,
|
||||||
@@ -871,4 +1062,6 @@ export const api = {
|
|||||||
unmatchClaim,
|
unmatchClaim,
|
||||||
listAcks,
|
listAcks,
|
||||||
getAck,
|
getAck,
|
||||||
|
listTa1Acks,
|
||||||
|
getDashboardKpis,
|
||||||
};
|
};
|
||||||
|
|||||||
+260
-16
@@ -1,5 +1,5 @@
|
|||||||
import { useCallback, useState } from "react";
|
import { useCallback, useState } from "react";
|
||||||
import { CheckCircle2, Download, ShieldCheck } from "lucide-react";
|
import { CheckCircle2, Download, Mail, ShieldCheck } from "lucide-react";
|
||||||
import {
|
import {
|
||||||
Table,
|
Table,
|
||||||
TableBody,
|
TableBody,
|
||||||
@@ -16,13 +16,17 @@ import { KpiCard } from "@/components/KpiCard";
|
|||||||
import { PageHeader } from "@/components/PageHeader";
|
import { PageHeader } from "@/components/PageHeader";
|
||||||
import { KeyboardCheatsheet } from "@/components/KeyboardCheatsheet";
|
import { KeyboardCheatsheet } from "@/components/KeyboardCheatsheet";
|
||||||
import { AckDrawer } from "@/components/AckDrawer";
|
import { AckDrawer } from "@/components/AckDrawer";
|
||||||
|
import { Pagination } from "@/components/ui/pagination";
|
||||||
import { useAckDrawerUrlState } from "@/hooks/useAckDrawerUrlState";
|
import { useAckDrawerUrlState } from "@/hooks/useAckDrawerUrlState";
|
||||||
import { useAcks } from "@/hooks/useAcks";
|
import { useAcks } from "@/hooks/useAcks";
|
||||||
|
import { useTa1Acks } from "@/hooks/useTa1Acks";
|
||||||
import { useRowKeyboard } from "@/hooks/useRowKeyboard";
|
import { useRowKeyboard } from "@/hooks/useRowKeyboard";
|
||||||
import { api } from "@/lib/api";
|
import { api } from "@/lib/api";
|
||||||
import { fmt } from "@/lib/format";
|
import { fmt } from "@/lib/format";
|
||||||
import { cn } from "@/lib/utils";
|
import { cn } from "@/lib/utils";
|
||||||
import type { Ack } from "@/types";
|
import type { Ack, Ta1Ack } from "@/types";
|
||||||
|
|
||||||
|
const PAGE_SIZE = 50;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 999 ACK register. The page reads the persisted 999 Implementation
|
* 999 ACK register. The page reads the persisted 999 Implementation
|
||||||
@@ -38,8 +42,18 @@ import type { Ack } from "@/types";
|
|||||||
* instrument view of the same data.
|
* instrument view of the same data.
|
||||||
*/
|
*/
|
||||||
export function Acks() {
|
export function Acks() {
|
||||||
const { data, isLoading, isError, error, refetch } = useAcks({ limit: 100 });
|
const [page, setPage] = useState(1);
|
||||||
|
const { data, isLoading, isError, error, refetch } = useAcks({
|
||||||
|
limit: PAGE_SIZE,
|
||||||
|
offset: (page - 1) * PAGE_SIZE,
|
||||||
|
});
|
||||||
const items = data?.items ?? [];
|
const items = data?.items ?? [];
|
||||||
|
const totalCount = data?.total ?? 0;
|
||||||
|
// Server-side aggregates — sum over the *full* ACK set, not just the
|
||||||
|
// visible page. Without this, the KPI strip silently under-reports
|
||||||
|
// once the row count exceeds PAGE_SIZE (the "silent failure" the
|
||||||
|
// operator flagged on 2026-06-29 when 1056 acks showed as 100).
|
||||||
|
const aggregates = data?.aggregates;
|
||||||
// SP21 Phase 5 Task 5.3: drill-down from an acks row into
|
// SP21 Phase 5 Task 5.3: drill-down from an acks row into
|
||||||
// AckDrawer. The hook reads `?ack=` off `window.location.search`
|
// AckDrawer. The hook reads `?ack=` off `window.location.search`
|
||||||
// so deep links restore the open ack on reload.
|
// so deep links restore the open ack on reload.
|
||||||
@@ -74,15 +88,16 @@ export function Acks() {
|
|||||||
|
|
||||||
// -----------------------------------------------------------------
|
// -----------------------------------------------------------------
|
||||||
// Aggregate metrics for the KPI strip + hero copy.
|
// Aggregate metrics for the KPI strip + hero copy.
|
||||||
|
// Sourced from the server-side `aggregates` field so the KPI strip
|
||||||
|
// reflects every persisted 999, not just the visible page. If
|
||||||
|
// `aggregates` hasn't loaded yet (first page), fall back to summing
|
||||||
|
// the visible items so the cards still display a meaningful value.
|
||||||
// -----------------------------------------------------------------
|
// -----------------------------------------------------------------
|
||||||
const totals = items.reduce(
|
const totals = aggregates ?? {
|
||||||
(acc, a) => ({
|
accepted: items.reduce((s, a) => s + a.acceptedCount, 0),
|
||||||
accepted: acc.accepted + a.acceptedCount,
|
rejected: items.reduce((s, a) => s + a.rejectedCount, 0),
|
||||||
rejected: acc.rejected + a.rejectedCount,
|
received: items.reduce((s, a) => s + a.receivedCount, 0),
|
||||||
received: acc.received + a.receivedCount,
|
};
|
||||||
}),
|
|
||||||
{ accepted: 0, rejected: 0, received: 0 },
|
|
||||||
);
|
|
||||||
const acceptRate =
|
const acceptRate =
|
||||||
totals.received > 0
|
totals.received > 0
|
||||||
? Math.round((totals.accepted / totals.received) * 1000) / 10
|
? Math.round((totals.accepted / totals.received) * 1000) / 10
|
||||||
@@ -90,11 +105,16 @@ export function Acks() {
|
|||||||
|
|
||||||
// The ghost watermark carries the on-file count, scaled like the
|
// The ghost watermark carries the on-file count, scaled like the
|
||||||
// Dashboard/Upload/Reconciliation watermarks (clamp 72–140px, 4.5%
|
// Dashboard/Upload/Reconciliation watermarks (clamp 72–140px, 4.5%
|
||||||
// opacity, right-anchored).
|
// opacity, right-anchored). Use the server-reported total (not the
|
||||||
const watermark = items.length > 0 ? items.length.toLocaleString() : "999";
|
// page length) so the watermark tells the truth when there are more
|
||||||
|
// than PAGE_SIZE rows on file.
|
||||||
|
const watermark = totalCount > 0 ? totalCount.toLocaleString() : "999";
|
||||||
|
|
||||||
// Tone for the status pill: green when nothing rejected, amber when
|
// Tone for the status pill: green when nothing rejected, amber when
|
||||||
// the register holds a partial/rejected payload.
|
// the register holds a partial/rejected payload.
|
||||||
|
// `anyRejected` is a "page" property, not a "totals" property — the
|
||||||
|
// status pill summarizes the visible register, so it's OK to scope
|
||||||
|
// it to the current page.
|
||||||
const anyRejected = items.some(
|
const anyRejected = items.some(
|
||||||
(a) => a.ackCode === "R" || a.ackCode === "E",
|
(a) => a.ackCode === "R" || a.ackCode === "E",
|
||||||
);
|
);
|
||||||
@@ -146,7 +166,7 @@ export function Acks() {
|
|||||||
<div className="relative z-10 flex items-end justify-between gap-4 sm:gap-6 flex-wrap">
|
<div className="relative z-10 flex items-end justify-between gap-4 sm:gap-6 flex-wrap">
|
||||||
<div className="min-w-0">
|
<div className="min-w-0">
|
||||||
<PageHeader
|
<PageHeader
|
||||||
eyebrow={`999 ACKs · EDI reference · ${items.length} on file`}
|
eyebrow={`999 ACKs · EDI reference · ${totalCount.toLocaleString()} on file`}
|
||||||
title={
|
title={
|
||||||
<>
|
<>
|
||||||
Acknowledgments,{" "}
|
Acknowledgments,{" "}
|
||||||
@@ -365,7 +385,18 @@ export function Acks() {
|
|||||||
{a.id}
|
{a.id}
|
||||||
</TableCell>
|
</TableCell>
|
||||||
<TableCell className="font-mono text-[12px] text-muted-foreground truncate max-w-[180px]">
|
<TableCell className="font-mono text-[12px] text-muted-foreground truncate max-w-[180px]">
|
||||||
{a.sourceBatchId}
|
{a.patientControlNumber ? (
|
||||||
|
<>
|
||||||
|
<span className="text-foreground">
|
||||||
|
{a.patientControlNumber}
|
||||||
|
</span>
|
||||||
|
<span className="text-muted-foreground/60">
|
||||||
|
{" "}· {a.sourceBatchId}
|
||||||
|
</span>
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
a.sourceBatchId
|
||||||
|
)}
|
||||||
</TableCell>
|
</TableCell>
|
||||||
<TableCell className="text-right display mono text-[hsl(var(--success))]">
|
<TableCell className="text-right display mono text-[hsl(var(--success))]">
|
||||||
{a.acceptedCount}
|
{a.acceptedCount}
|
||||||
@@ -395,11 +426,31 @@ export function Acks() {
|
|||||||
</Table>
|
</Table>
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
{totalCount > PAGE_SIZE ? (
|
||||||
|
<Pagination
|
||||||
|
page={page}
|
||||||
|
pageSize={PAGE_SIZE}
|
||||||
|
total={totalCount}
|
||||||
|
onPageChange={setPage}
|
||||||
|
/>
|
||||||
|
) : null}
|
||||||
</div>
|
</div>
|
||||||
</CardContent>
|
</CardContent>
|
||||||
</Card>
|
</Card>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
{/* =================================================================
|
||||||
|
TA1 ENVELOPE ACKS — per-interchange acknowledgments.
|
||||||
|
A TA1 (X12 Interchange Acknowledgment) is one row per
|
||||||
|
inbound ISA/IEA envelope — the lower-level sibling of the
|
||||||
|
999 (per-batch). Colorado Medicaid's Gainwell MFT currently
|
||||||
|
only ships 999s, but the section stays in the surface so
|
||||||
|
the operator can see TA1s as soon as one shows up. Kept as
|
||||||
|
a quieter, narrower section than the 999 register so the
|
||||||
|
999s remain the page's primary instrument.
|
||||||
|
================================================================= */}
|
||||||
|
<Ta1AcksSection />
|
||||||
|
|
||||||
{/* =================================================================
|
{/* =================================================================
|
||||||
FOOTER — single hairline-separated status row. Replaces the
|
FOOTER — single hairline-separated status row. Replaces the
|
||||||
warm-paper "End of register" treatment with a quiet
|
warm-paper "End of register" treatment with a quiet
|
||||||
@@ -417,7 +468,8 @@ export function Acks() {
|
|||||||
</div>
|
</div>
|
||||||
<div className="flex items-center gap-3 mono text-[10.5px] uppercase tracking-[0.18em] text-muted-foreground/60">
|
<div className="flex items-center gap-3 mono text-[10.5px] uppercase tracking-[0.18em] text-muted-foreground/60">
|
||||||
<span>
|
<span>
|
||||||
{items.length} {items.length === 1 ? "row" : "rows"}
|
{totalCount.toLocaleString()}{" "}
|
||||||
|
{totalCount === 1 ? "row" : "rows"} on file
|
||||||
</span>
|
</span>
|
||||||
<span className="text-muted-foreground/30" aria-hidden>
|
<span className="text-muted-foreground/30" aria-hidden>
|
||||||
·
|
·
|
||||||
@@ -541,3 +593,195 @@ function downloadBlob(filename: string, content: string) {
|
|||||||
a.click();
|
a.click();
|
||||||
URL.revokeObjectURL(url);
|
URL.revokeObjectURL(url);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Ta1AcksSection — per-interchange TA1 register.
|
||||||
|
//
|
||||||
|
// Deliberately quieter than the 999 register above: smaller KPI strip,
|
||||||
|
// single-card table, and an empty state that explains the
|
||||||
|
// Colorado-specific context (TA1s aren't shipped today, so the
|
||||||
|
// section is empty unless Gainwell starts sending them).
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
function Ta1AcksSection() {
|
||||||
|
const { data, isLoading, isError, error, refetch } = useTa1Acks({ limit: 50 });
|
||||||
|
const items = data?.items ?? [];
|
||||||
|
|
||||||
|
const totals = items.reduce(
|
||||||
|
(acc, t) => {
|
||||||
|
acc[t.ackCode] = (acc[t.ackCode] ?? 0) + 1;
|
||||||
|
return acc;
|
||||||
|
},
|
||||||
|
{} as Record<string, number>,
|
||||||
|
);
|
||||||
|
|
||||||
|
return (
|
||||||
|
<section
|
||||||
|
aria-label="TA1 envelope acknowledgments"
|
||||||
|
className="animate-fade-in-up"
|
||||||
|
>
|
||||||
|
<Card>
|
||||||
|
<CardContent className="p-6 lg:p-7 space-y-5">
|
||||||
|
<div className="flex items-end justify-between gap-6 flex-wrap">
|
||||||
|
<div className="min-w-0">
|
||||||
|
<div className="eyebrow flex items-center gap-2 mb-2">
|
||||||
|
<span className="inline-block h-px w-6 bg-foreground/20" />
|
||||||
|
Envelope acks
|
||||||
|
</div>
|
||||||
|
<h2 className="display text-[22px] leading-[1.05] tracking-[-0.02em]">
|
||||||
|
TA1 <span className="italic">envelopes</span>, newest first.
|
||||||
|
</h2>
|
||||||
|
</div>
|
||||||
|
<p className="text-[12.5px] text-muted-foreground/80 max-w-sm">
|
||||||
|
One row per inbound ISA/IEA interchange — the
|
||||||
|
envelope-level sibling of the 999. Gainwell's Colorado
|
||||||
|
MFT does not ship TA1s today; this section surfaces them
|
||||||
|
when they appear.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="grid gap-3 grid-cols-2 md:grid-cols-4 pt-2">
|
||||||
|
<KpiCard
|
||||||
|
label="On file"
|
||||||
|
value={fmt.num(items.length)}
|
||||||
|
accent="default"
|
||||||
|
hint="persisted TA1s"
|
||||||
|
/>
|
||||||
|
<KpiCard
|
||||||
|
label="Accepted"
|
||||||
|
value={fmt.num(totals.A ?? 0)}
|
||||||
|
accent="success"
|
||||||
|
hint="ack_code = A"
|
||||||
|
/>
|
||||||
|
<KpiCard
|
||||||
|
label="Envelope errors"
|
||||||
|
value={fmt.num(totals.E ?? 0)}
|
||||||
|
accent="warning"
|
||||||
|
hint="ack_code = E"
|
||||||
|
/>
|
||||||
|
<KpiCard
|
||||||
|
label="Rejected"
|
||||||
|
value={fmt.num(totals.R ?? 0)}
|
||||||
|
accent="destructive"
|
||||||
|
hint="ack_code = R"
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="pt-5 border-t border-border/40">
|
||||||
|
{isError ? (
|
||||||
|
<ErrorState
|
||||||
|
message="Couldn't load TA1 acks from the backend."
|
||||||
|
detail={error instanceof Error ? error.message : String(error)}
|
||||||
|
onRetry={() => refetch()}
|
||||||
|
/>
|
||||||
|
) : isLoading ? (
|
||||||
|
<div className="rounded-md border border-border/60 bg-card/40 p-4 space-y-2">
|
||||||
|
{Array.from({ length: 3 }).map((_, i) => (
|
||||||
|
<Skeleton key={i} variant="row" />
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
) : items.length === 0 ? (
|
||||||
|
<div className="rounded-md border border-dashed border-border/60 bg-card/40">
|
||||||
|
<EmptyState
|
||||||
|
eyebrow="TA1 envelopes · none on file"
|
||||||
|
message="Gainwell's Colorado MFT does not ship TA1 files today. When one does arrive (or you upload one), it will land here."
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<div className="rounded-md border border-border/60 overflow-hidden bg-card/40">
|
||||||
|
<Table>
|
||||||
|
<TableHeader>
|
||||||
|
<TableRow>
|
||||||
|
<TableHead className="w-10" aria-label="Status" />
|
||||||
|
<TableHead>Control #</TableHead>
|
||||||
|
<TableHead>Ack</TableHead>
|
||||||
|
<TableHead>Note</TableHead>
|
||||||
|
<TableHead>Interchange date</TableHead>
|
||||||
|
<TableHead>Sender → Receiver</TableHead>
|
||||||
|
</TableRow>
|
||||||
|
</TableHeader>
|
||||||
|
<TableBody>
|
||||||
|
{items.map((t) => (
|
||||||
|
<TableRow key={t.id}>
|
||||||
|
<TableCell>
|
||||||
|
<Mail
|
||||||
|
className={cn(
|
||||||
|
"h-3.5 w-3.5",
|
||||||
|
t.ackCode === "A"
|
||||||
|
? "text-[hsl(var(--success))]"
|
||||||
|
: t.ackCode === "R"
|
||||||
|
? "text-destructive"
|
||||||
|
: "text-[hsl(var(--warning))]",
|
||||||
|
)}
|
||||||
|
strokeWidth={1.75}
|
||||||
|
aria-hidden
|
||||||
|
/>
|
||||||
|
</TableCell>
|
||||||
|
<TableCell className="display mono text-[13px] text-foreground">
|
||||||
|
{t.controlNumber || "—"}
|
||||||
|
</TableCell>
|
||||||
|
<TableCell>
|
||||||
|
<Ta1CodeBadge code={t.ackCode} />
|
||||||
|
</TableCell>
|
||||||
|
<TableCell className="mono text-[12px] text-muted-foreground">
|
||||||
|
{t.noteCode ?? "—"}
|
||||||
|
</TableCell>
|
||||||
|
<TableCell className="mono text-[12.5px] text-muted-foreground/70">
|
||||||
|
{t.interchangeDate ? fmt.dateShort(t.interchangeDate) : "—"}
|
||||||
|
{t.interchangeTime ? (
|
||||||
|
<span className="text-muted-foreground/50"> · {t.interchangeTime}</span>
|
||||||
|
) : null}
|
||||||
|
</TableCell>
|
||||||
|
<TableCell className="mono text-[12px] text-muted-foreground truncate max-w-[260px]">
|
||||||
|
{t.senderId ?? "?"} → {t.receiverId ?? "?"}
|
||||||
|
</TableCell>
|
||||||
|
</TableRow>
|
||||||
|
))}
|
||||||
|
</TableBody>
|
||||||
|
</Table>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</CardContent>
|
||||||
|
</Card>
|
||||||
|
</section>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Ta1CodeBadge — same A/E/R → success/warning/destructive mapping as the
|
||||||
|
// 999 AckCodeBadge above. Kept as a separate component so the TA1
|
||||||
|
// surface can evolve independently (e.g. add a tooltip explaining the
|
||||||
|
// X12 005010X231A1 note code table) without churning the 999 surface.
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
function Ta1CodeBadge({ code }: { code: Ta1Ack["ackCode"] }) {
|
||||||
|
const color =
|
||||||
|
code === "A"
|
||||||
|
? {
|
||||||
|
text: "hsl(var(--success))",
|
||||||
|
bg: "hsl(var(--success) / 0.10)",
|
||||||
|
border: "hsl(var(--success) / 0.30)",
|
||||||
|
}
|
||||||
|
: code === "R"
|
||||||
|
? {
|
||||||
|
text: "hsl(var(--destructive))",
|
||||||
|
bg: "hsl(var(--destructive) / 0.10)",
|
||||||
|
border: "hsl(var(--destructive) / 0.30)",
|
||||||
|
}
|
||||||
|
: {
|
||||||
|
text: "hsl(var(--warning))",
|
||||||
|
bg: "hsl(var(--warning) / 0.10)",
|
||||||
|
border: "hsl(var(--warning) / 0.30)",
|
||||||
|
};
|
||||||
|
return (
|
||||||
|
<span
|
||||||
|
className="inline-flex items-center gap-1 rounded-sm border px-2 py-0.5 mono text-[10.5px] font-semibold uppercase tracking-[0.14em]"
|
||||||
|
style={{
|
||||||
|
color: color.text,
|
||||||
|
backgroundColor: color.bg,
|
||||||
|
borderColor: color.border,
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
{code}
|
||||||
|
</span>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|||||||
@@ -68,7 +68,13 @@ export function ActivityLog() {
|
|||||||
const { data, isLoading, isError, error, refetch } = useActivity({
|
const { data, isLoading, isError, error, refetch } = useActivity({
|
||||||
kind: apiKind,
|
kind: apiKind,
|
||||||
since: sinceIso,
|
since: sinceIso,
|
||||||
limit: 200,
|
// 500 keeps the wire size manageable (and the operator can rely
|
||||||
|
// on the live tail to surface new events as they land). The
|
||||||
|
// activity list endpoint doesn't expose a true total — `data.total`
|
||||||
|
// reflects events matching the current kind/since filter, capped
|
||||||
|
// at the request limit — so the eyebrow below uses "Showing N
|
||||||
|
// most recent" instead of "X of Y" to avoid a misleading ratio.
|
||||||
|
limit: 500,
|
||||||
});
|
});
|
||||||
|
|
||||||
const allItems = data?.items ?? [];
|
const allItems = data?.items ?? [];
|
||||||
@@ -127,7 +133,11 @@ export function ActivityLog() {
|
|||||||
return (
|
return (
|
||||||
<div className="space-y-6 lg:space-y-8 animate-fade-in">
|
<div className="space-y-6 lg:space-y-8 animate-fade-in">
|
||||||
<PageHeader
|
<PageHeader
|
||||||
eyebrow="Activity"
|
eyebrow={
|
||||||
|
allItems.length > 0
|
||||||
|
? `Activity · showing ${allItems.length.toLocaleString()} most recent`
|
||||||
|
: "Activity"
|
||||||
|
}
|
||||||
title={<>Activity <span className="display italic text-muted-foreground">log</span></>}
|
title={<>Activity <span className="display italic text-muted-foreground">log</span></>}
|
||||||
subtitle="Every claim submission, denial, payment, and provider event, in reverse chronological order. Auto-refreshes every 30s."
|
subtitle="Every claim submission, denial, payment, and provider event, in reverse chronological order. Auto-refreshes every 30s."
|
||||||
actions={
|
actions={
|
||||||
|
|||||||
@@ -16,6 +16,7 @@
|
|||||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||||
import { cleanup, fireEvent, render } from "@testing-library/react";
|
import { cleanup, fireEvent, render } from "@testing-library/react";
|
||||||
import { MemoryRouter } from "react-router-dom";
|
import { MemoryRouter } from "react-router-dom";
|
||||||
|
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||||
import { Dashboard } from "./Dashboard";
|
import { Dashboard } from "./Dashboard";
|
||||||
import { useAppStore } from "@/store";
|
import { useAppStore } from "@/store";
|
||||||
import type { Activity } from "@/types";
|
import type { Activity } from "@/types";
|
||||||
@@ -31,6 +32,34 @@ vi.mock("sonner", () => ({
|
|||||||
},
|
},
|
||||||
}));
|
}));
|
||||||
|
|
||||||
|
// Stub useAuth so the Dashboard's greeting renders without spinning
|
||||||
|
// up the full AuthProvider + /api/auth/me probe. Same pattern as
|
||||||
|
// Inbox.test.tsx.
|
||||||
|
vi.mock("@/auth/useAuth", () => ({
|
||||||
|
useAuth: () => ({
|
||||||
|
status: "authenticated" as const,
|
||||||
|
user: { username: "tester" } as unknown as never,
|
||||||
|
login: vi.fn(),
|
||||||
|
logout: vi.fn(),
|
||||||
|
refresh: vi.fn(),
|
||||||
|
}),
|
||||||
|
}));
|
||||||
|
|
||||||
|
// SP27 Task 13: stub `api.isConfigured = false` so `useDashboardKpis`
|
||||||
|
// + `useActivity` both take their in-memory zustand fallback path.
|
||||||
|
// These tests focus on activity-feed event routing — they don't
|
||||||
|
// assert on KPI math, so zero-filled KPIs from the fallback are fine.
|
||||||
|
vi.mock("@/lib/api", async () => {
|
||||||
|
const actual = await vi.importActual<typeof import("@/lib/api")>("@/lib/api");
|
||||||
|
return {
|
||||||
|
...actual,
|
||||||
|
api: {
|
||||||
|
...actual.api,
|
||||||
|
isConfigured: false,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
// Capture navigation side effects so we can assert on the URL the
|
// Capture navigation side effects so we can assert on the URL the
|
||||||
// Dashboard would push. We use a `MemoryRouter` (initialEntries=["/"])
|
// Dashboard would push. We use a `MemoryRouter` (initialEntries=["/"])
|
||||||
// and observe the rendered route via a tiny listener component that
|
// and observe the rendered route via a tiny listener component that
|
||||||
@@ -49,6 +78,21 @@ function LocationProbe() {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SP27 Task 13: Dashboard now reads KPIs from `useDashboardKpis`,
|
||||||
|
// which uses TanStack Query internally. Wrap each render in a
|
||||||
|
// QueryClientProvider so the hook doesn't throw — these tests focus
|
||||||
|
// on activity-feed event routing, so we never resolve the KPI query.
|
||||||
|
function renderWithQuery(ui: React.ReactNode) {
|
||||||
|
const client = new QueryClient({
|
||||||
|
defaultOptions: {
|
||||||
|
queries: { retry: false, gcTime: 0, staleTime: 0 },
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return render(
|
||||||
|
<QueryClientProvider client={client}>{ui}</QueryClientProvider>,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
afterEach(() => {
|
afterEach(() => {
|
||||||
cleanup();
|
cleanup();
|
||||||
vi.clearAllMocks();
|
vi.clearAllMocks();
|
||||||
@@ -70,7 +114,7 @@ describe("Dashboard · Recent activity event routing (SP21 Task 2.5)", () => {
|
|||||||
];
|
];
|
||||||
useAppStore.setState({ activity });
|
useAppStore.setState({ activity });
|
||||||
|
|
||||||
const { getByTestId, getByRole } = render(
|
const { getByTestId, getByRole } = renderWithQuery(
|
||||||
<MemoryRouter initialEntries={["/"]}>
|
<MemoryRouter initialEntries={["/"]}>
|
||||||
<Dashboard />
|
<Dashboard />
|
||||||
<LocationProbe />
|
<LocationProbe />
|
||||||
@@ -101,7 +145,7 @@ describe("Dashboard · Recent activity event routing (SP21 Task 2.5)", () => {
|
|||||||
];
|
];
|
||||||
useAppStore.setState({ activity });
|
useAppStore.setState({ activity });
|
||||||
|
|
||||||
const { getByTestId, getByRole } = render(
|
const { getByTestId, getByRole } = renderWithQuery(
|
||||||
<MemoryRouter initialEntries={["/"]}>
|
<MemoryRouter initialEntries={["/"]}>
|
||||||
<Dashboard />
|
<Dashboard />
|
||||||
<LocationProbe />
|
<LocationProbe />
|
||||||
@@ -129,7 +173,7 @@ describe("Dashboard · Recent activity event routing (SP21 Task 2.5)", () => {
|
|||||||
];
|
];
|
||||||
useAppStore.setState({ activity });
|
useAppStore.setState({ activity });
|
||||||
|
|
||||||
const { getByTestId, getByRole } = render(
|
const { getByTestId, getByRole } = renderWithQuery(
|
||||||
<MemoryRouter initialEntries={["/"]}>
|
<MemoryRouter initialEntries={["/"]}>
|
||||||
<Dashboard />
|
<Dashboard />
|
||||||
<LocationProbe />
|
<LocationProbe />
|
||||||
|
|||||||
+52
-88
@@ -18,71 +18,66 @@ import { DrillableCell } from "@/components/drill/DrillableCell";
|
|||||||
import { fmt } from "@/lib/format";
|
import { fmt } from "@/lib/format";
|
||||||
import { eventKindToUrl } from "@/lib/event-routing";
|
import { eventKindToUrl } from "@/lib/event-routing";
|
||||||
import { useAuth } from "@/auth/useAuth";
|
import { useAuth } from "@/auth/useAuth";
|
||||||
import { useClaims } from "@/hooks/useClaims";
|
import { useDashboardKpis } from "@/hooks/useDashboardKpis";
|
||||||
import { useProviders } from "@/hooks/useProviders";
|
|
||||||
import { useActivity } from "@/hooks/useActivity";
|
import { useActivity } from "@/hooks/useActivity";
|
||||||
import type { Claim } from "@/types";
|
|
||||||
import { toast } from "sonner";
|
import { toast } from "sonner";
|
||||||
|
|
||||||
const MONTHS_BACK = 6;
|
const MONTHS_BACK = 6;
|
||||||
|
|
||||||
function buildMonthly(claims: Claim[]) {
|
// Zero-shaped KPI totals used when the server response hasn't arrived
|
||||||
const now = new Date();
|
// yet or the backend isn't configured. Mirrors the empty-DB shape of
|
||||||
const months: {
|
// `GET /api/dashboard/kpis` so the KPI tiles render a coherent "0"
|
||||||
key: string;
|
// instead of `undefined` during the first paint and background
|
||||||
label: string;
|
// refetches. Keeping it next to ``MONTHS_BACK`` means a future tile
|
||||||
count: number;
|
// can be added in one place rather than chasing the fallback object.
|
||||||
billed: number;
|
const ZERO_TOTALS = {
|
||||||
received: number;
|
count: 0,
|
||||||
denied: number;
|
billed: 0,
|
||||||
}[] = [];
|
received: 0,
|
||||||
for (let i = MONTHS_BACK - 1; i >= 0; i--) {
|
outstandingAr: 0,
|
||||||
const d = new Date(now.getFullYear(), now.getMonth() - i, 1);
|
denied: 0,
|
||||||
months.push({
|
denialRate: 0,
|
||||||
key: `${d.getFullYear()}-${d.getMonth()}`,
|
pending: 0,
|
||||||
label: d.toLocaleString("en-US", { month: "short" }),
|
};
|
||||||
count: 0,
|
|
||||||
billed: 0,
|
|
||||||
received: 0,
|
|
||||||
denied: 0,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
const index = new Map(months.map((m, i) => [m.key, i]));
|
|
||||||
for (const c of claims) {
|
|
||||||
const d = new Date(c.submissionDate);
|
|
||||||
const k = `${d.getFullYear()}-${d.getMonth()}`;
|
|
||||||
const i = index.get(k);
|
|
||||||
if (i === undefined) continue;
|
|
||||||
months[i]!.count += 1;
|
|
||||||
months[i]!.billed += c.billedAmount;
|
|
||||||
months[i]!.received += c.receivedAmount;
|
|
||||||
if (c.status === "denied") months[i]!.denied += 1;
|
|
||||||
}
|
|
||||||
let running = 0;
|
|
||||||
const ar: number[] = [];
|
|
||||||
for (const m of months) {
|
|
||||||
running += m.billed - m.received;
|
|
||||||
ar.push(Math.max(0, running));
|
|
||||||
}
|
|
||||||
return {
|
|
||||||
count: months.map((m) => m.count),
|
|
||||||
billed: months.map((m) => m.billed),
|
|
||||||
received: months.map((m) => m.received),
|
|
||||||
ar,
|
|
||||||
denialRate: months.map((m) => (m.count ? (m.denied / m.count) * 100 : 0)),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
export function Dashboard() {
|
export function Dashboard() {
|
||||||
// Live data: hooks fetch from /api/* when api.isConfigured; otherwise
|
// Live data: hooks fetch from /api/* when api.isConfigured; otherwise
|
||||||
// they fall back to the in-memory store. Pulling from the hooks (not
|
// they fall back to the in-memory store. Pulling from the hooks (not
|
||||||
// the store directly) is what wires the Dashboard to the backend.
|
// the store directly) is what wires the Dashboard to the backend.
|
||||||
const claimsQuery = useClaims({ limit: 100 });
|
//
|
||||||
const providersQuery = useProviders();
|
// SP27 Task 13: KPIs come from the dedicated ``/api/dashboard/kpis``
|
||||||
|
// server-side aggregate, NOT from a paginated ``/api/claims?limit=100``
|
||||||
|
// reduced client-side. With 60k+ claims in production, the old
|
||||||
|
// pattern silently produced wrong numbers — every Dashboard tile was
|
||||||
|
// computed from a 100-row sample. The new endpoint does the reduce
|
||||||
|
// in SQL once over the full population.
|
||||||
|
const kpisQuery = useDashboardKpis({ months: MONTHS_BACK });
|
||||||
const activityQuery = useActivity({ limit: 10 });
|
const activityQuery = useActivity({ limit: 10 });
|
||||||
|
|
||||||
const claims = claimsQuery.data?.items ?? [];
|
const kpisData = kpisQuery.data;
|
||||||
const providers = providersQuery.data?.items ?? [];
|
// Fall back to a zero-shaped object so the KpiTiles still render a
|
||||||
|
// coherent "0" rather than NaN during the first paint and during
|
||||||
|
// background refetches. See ``ZERO_TOTALS`` for why this is a
|
||||||
|
// module-level constant.
|
||||||
|
const kpis = kpisData?.totals ?? ZERO_TOTALS;
|
||||||
|
const monthly = useMemo(() => {
|
||||||
|
const series = kpisData?.monthly ?? [];
|
||||||
|
return {
|
||||||
|
count: series.map((m) => m.count),
|
||||||
|
billed: series.map((m) => m.billed),
|
||||||
|
received: series.map((m) => m.received),
|
||||||
|
ar: series.map((m) => m.ar),
|
||||||
|
denialRate: series.map((m) => m.denialRate),
|
||||||
|
};
|
||||||
|
}, [kpisData]);
|
||||||
|
const topProviders = useMemo(
|
||||||
|
() => kpisData?.topProviders ?? [],
|
||||||
|
[kpisData]
|
||||||
|
);
|
||||||
|
const topDenials = useMemo(
|
||||||
|
() => kpisData?.topDenials ?? [],
|
||||||
|
[kpisData]
|
||||||
|
);
|
||||||
const activity = activityQuery.data?.items ?? [];
|
const activity = activityQuery.data?.items ?? [];
|
||||||
|
|
||||||
const navigate = useNavigate();
|
const navigate = useNavigate();
|
||||||
@@ -96,37 +91,6 @@ export function Dashboard() {
|
|||||||
})();
|
})();
|
||||||
const operatorName = auth.user?.username ?? "there";
|
const operatorName = auth.user?.username ?? "there";
|
||||||
|
|
||||||
const kpis = useMemo(() => {
|
|
||||||
const billed = claims.reduce((s, c) => s + c.billedAmount, 0);
|
|
||||||
const received = claims.reduce((s, c) => s + c.receivedAmount, 0);
|
|
||||||
const outstandingAr = billed - received;
|
|
||||||
const denied = claims.filter((c) => c.status === "denied").length;
|
|
||||||
const denialRate = claims.length > 0 ? (denied / claims.length) * 100 : 0;
|
|
||||||
const pending = claims.filter(
|
|
||||||
(c) => c.status === "submitted" || c.status === "pending"
|
|
||||||
).length;
|
|
||||||
return {
|
|
||||||
count: claims.length,
|
|
||||||
billed,
|
|
||||||
received,
|
|
||||||
outstandingAr,
|
|
||||||
denialRate,
|
|
||||||
pending,
|
|
||||||
};
|
|
||||||
}, [claims]);
|
|
||||||
|
|
||||||
const monthly = useMemo(() => buildMonthly(claims), [claims]);
|
|
||||||
|
|
||||||
const topProviders = useMemo(
|
|
||||||
() => [...providers].sort((a, b) => b.claimCount - a.claimCount).slice(0, 4),
|
|
||||||
[providers]
|
|
||||||
);
|
|
||||||
|
|
||||||
const topDenials = useMemo(
|
|
||||||
() => claims.filter((c) => c.status === "denied").slice(0, 5),
|
|
||||||
[claims]
|
|
||||||
);
|
|
||||||
|
|
||||||
// Stagger choreography — the hero lands first, then the KPIs in
|
// Stagger choreography — the hero lands first, then the KPIs in
|
||||||
// a left-to-right wave, then the supporting cards. Total
|
// a left-to-right wave, then the supporting cards. Total
|
||||||
// choreography fits under 700ms.
|
// choreography fits under 700ms.
|
||||||
@@ -345,14 +309,14 @@ export function Dashboard() {
|
|||||||
}}
|
}}
|
||||||
role="button"
|
role="button"
|
||||||
tabIndex={0}
|
tabIndex={0}
|
||||||
aria-label={`View provider ${p.name}`}
|
aria-label={`View provider ${p.label || p.npi}`}
|
||||||
className="drillable flex items-center gap-3"
|
className="drillable flex items-center gap-3"
|
||||||
>
|
>
|
||||||
<div className="h-7 w-7 rounded-md bg-muted/60 ring-1 ring-inset ring-border/40 flex items-center justify-center mono text-[10.5px] text-muted-foreground">
|
<div className="h-7 w-7 rounded-md bg-muted/60 ring-1 ring-inset ring-border/40 flex items-center justify-center mono text-[10.5px] text-muted-foreground">
|
||||||
{String(i + 1).padStart(2, "0")}
|
{String(i + 1).padStart(2, "0")}
|
||||||
</div>
|
</div>
|
||||||
<div className="flex-1 min-w-0">
|
<div className="flex-1 min-w-0">
|
||||||
<div className="text-[13px] font-medium truncate">{p.name}</div>
|
<div className="text-[13px] font-medium truncate">{p.label || p.npi}</div>
|
||||||
<div className="mono text-[10.5px] text-muted-foreground">
|
<div className="mono text-[10.5px] text-muted-foreground">
|
||||||
NPI {p.npi}
|
NPI {p.npi}
|
||||||
</div>
|
</div>
|
||||||
@@ -360,7 +324,7 @@ export function Dashboard() {
|
|||||||
<div className="text-right">
|
<div className="text-right">
|
||||||
<div className="display mono text-[15px]">{fmt.num(p.claimCount)}</div>
|
<div className="display mono text-[15px]">{fmt.num(p.claimCount)}</div>
|
||||||
<div className="mono text-[10.5px] text-muted-foreground">
|
<div className="mono text-[10.5px] text-muted-foreground">
|
||||||
{fmt.usd(p.outstandingAr)} AR
|
{fmt.usd(p.billed)} billed
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</li>
|
</li>
|
||||||
@@ -416,7 +380,7 @@ export function Dashboard() {
|
|||||||
{fmt.usd(c.billedAmount)}
|
{fmt.usd(c.billedAmount)}
|
||||||
</div>
|
</div>
|
||||||
<div className="text-[10.5px] text-muted-foreground">
|
<div className="text-[10.5px] text-muted-foreground">
|
||||||
{c.payerName}
|
{fmt.date(c.submissionDate)}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</li>
|
</li>
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ vi.mock("@/lib/api", () => ({
|
|||||||
api: {
|
api: {
|
||||||
isConfigured: true,
|
isConfigured: true,
|
||||||
listRemittances: vi.fn(),
|
listRemittances: vi.fn(),
|
||||||
|
listRemittanceSummary: vi.fn(),
|
||||||
getRemittance: vi.fn(),
|
getRemittance: vi.fn(),
|
||||||
},
|
},
|
||||||
ApiError: class ApiError extends Error {
|
ApiError: class ApiError extends Error {
|
||||||
@@ -224,6 +225,17 @@ describe("Remittances", () => {
|
|||||||
returned: SAMPLE_REMITS.length,
|
returned: SAMPLE_REMITS.length,
|
||||||
has_more: false,
|
has_more: false,
|
||||||
});
|
});
|
||||||
|
// Default summary mock so the page-level KPI tiles don't render
|
||||||
|
// `undefined`. Tests that care about specific summary values
|
||||||
|
// override this per-test (e.g. ``test_kpi_tiles_use_server_...
|
||||||
|
// _not_page_local_reduce``).
|
||||||
|
(
|
||||||
|
api.listRemittanceSummary as unknown as ReturnType<typeof vi.fn>
|
||||||
|
).mockResolvedValue({
|
||||||
|
count: 0,
|
||||||
|
total_paid: 0,
|
||||||
|
total_adjustments: 0,
|
||||||
|
});
|
||||||
// Default for the per-remit detail fetch — the drawer fetches
|
// Default for the per-remit detail fetch — the drawer fetches
|
||||||
// this whenever `?remit=` is in the URL. Return a never-resolving
|
// this whenever `?remit=` is in the URL. Return a never-resolving
|
||||||
// promise so the drawer stays in the loading state; the smoke
|
// promise so the drawer stays in the loading state; the smoke
|
||||||
@@ -487,4 +499,68 @@ describe("Remittances", () => {
|
|||||||
|
|
||||||
unmount();
|
unmount();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
// KPI tiles must reflect the full DB population, NOT the page-local
|
||||||
|
// reduce over the visible rows. The bug repro is a 25-row page
|
||||||
|
// whose `items.reduce(...)` summed to $25.96 paid / $67.06
|
||||||
|
// adjustments while the real DB held $227,181.58 / $13,792.65
|
||||||
|
// across 1,739 rows. The "REMITS" tile already sources its count
|
||||||
|
// from the fixed `count_remittances` helper (commit d81b6ed).
|
||||||
|
// "TOTAL PAID" + "ADJUSTMENTS" must come from the new server-side
|
||||||
|
// summary endpoint (`/api/remittances/summary`) via
|
||||||
|
// `api.listRemittanceSummary` so they can't silently understate
|
||||||
|
// the true population. Mirrors the Dashboard silent-failure fix
|
||||||
|
// (commit 59c3275).
|
||||||
|
// -------------------------------------------------------------------
|
||||||
|
it("test_kpi_tiles_use_server_summary_not_page_local_reduce", async () => {
|
||||||
|
// Page sum over the visible 3 rows: paid=$525.96 (100+200+300-74.04),
|
||||||
|
// adjustments=$60 (60+0+0). Server ground truth (mocked): count=1739,
|
||||||
|
// paid=$227,181, adjustments=$13,792 — whole-dollar amounts chosen
|
||||||
|
// so `fmt.usd`'s maximumFractionDigits=0 doesn't round them and the
|
||||||
|
// assertion is unambiguous. The page must show the SERVER values,
|
||||||
|
// not the page sums.
|
||||||
|
(
|
||||||
|
api.listRemittanceSummary as unknown as ReturnType<typeof vi.fn>
|
||||||
|
).mockResolvedValue({
|
||||||
|
count: 1739,
|
||||||
|
total_paid: 227181,
|
||||||
|
total_adjustments: 13792,
|
||||||
|
});
|
||||||
|
|
||||||
|
const { unmount } = renderIntoContainer(React.createElement(Remittances));
|
||||||
|
await waitForText("PCN-1");
|
||||||
|
|
||||||
|
const body = document.body.textContent ?? "";
|
||||||
|
// The server totals must show on the tiles.
|
||||||
|
expect(body).toContain("$227,181");
|
||||||
|
expect(body).toContain("$13,792");
|
||||||
|
// Server count wins over the 3-row page total.
|
||||||
|
expect(body).toContain("1,739");
|
||||||
|
|
||||||
|
// Locate the KPI tile for "Total paid" specifically. The tiles
|
||||||
|
// share the `.display.mono.text-[28px]` class with several other
|
||||||
|
// page elements, so we walk by `.eyebrow` to find each one.
|
||||||
|
const totalPaidTile = Array.from(
|
||||||
|
document.querySelectorAll(".surface-2"),
|
||||||
|
).find((el) => el.querySelector(".eyebrow")?.textContent === "Total paid");
|
||||||
|
expect(totalPaidTile).toBeDefined();
|
||||||
|
const paidValue = totalPaidTile?.querySelector(
|
||||||
|
".display.mono.text-\\[28px\\]",
|
||||||
|
);
|
||||||
|
expect(paidValue?.textContent).toBe("$227,181");
|
||||||
|
|
||||||
|
const adjustmentsTile = Array.from(
|
||||||
|
document.querySelectorAll(".surface-2"),
|
||||||
|
).find(
|
||||||
|
(el) => el.querySelector(".eyebrow")?.textContent === "Adjustments",
|
||||||
|
);
|
||||||
|
expect(adjustmentsTile).toBeDefined();
|
||||||
|
const adjValue = adjustmentsTile?.querySelector(
|
||||||
|
".display.mono.text-\\[28px\\]",
|
||||||
|
);
|
||||||
|
expect(adjValue?.textContent).toBe("$13,792");
|
||||||
|
|
||||||
|
unmount();
|
||||||
|
});
|
||||||
});
|
});
|
||||||
+18
-10
@@ -18,6 +18,7 @@ import { PageHeader } from "@/components/PageHeader";
|
|||||||
import { TailStatusPill } from "@/components/TailStatusPill";
|
import { TailStatusPill } from "@/components/TailStatusPill";
|
||||||
import { RemitDrawer } from "@/components/RemitDrawer";
|
import { RemitDrawer } from "@/components/RemitDrawer";
|
||||||
import { useRemittances } from "@/hooks/useRemittances";
|
import { useRemittances } from "@/hooks/useRemittances";
|
||||||
|
import { useRemittanceSummary } from "@/hooks/useRemittanceSummary";
|
||||||
import { useRemitDrawerUrlState } from "@/hooks/useRemitDrawerUrlState";
|
import { useRemitDrawerUrlState } from "@/hooks/useRemitDrawerUrlState";
|
||||||
import { useRowKeyboard } from "@/hooks/useRowKeyboard";
|
import { useRowKeyboard } from "@/hooks/useRowKeyboard";
|
||||||
import { useTailStream } from "@/hooks/useTailStream";
|
import { useTailStream } from "@/hooks/useTailStream";
|
||||||
@@ -67,13 +68,20 @@ export function Remittances() {
|
|||||||
useTailStream("remittances");
|
useTailStream("remittances");
|
||||||
const items = useMergedTail("remittances", data?.items ?? [], tailFilterFn);
|
const items = useMergedTail("remittances", data?.items ?? [], tailFilterFn);
|
||||||
|
|
||||||
const total = items.reduce(
|
// KPI totals come from the server-aggregated summary endpoint over
|
||||||
(acc, r) => ({
|
// the FULL remittance population — NOT a page-local reduce over the
|
||||||
paid: acc.paid + r.paidAmount,
|
// current page + live-tail delta. The same silent-incompleteness
|
||||||
adjustments: acc.adjustments + r.adjustmentAmount,
|
// pattern that `59c3275` (Dashboard) and `d81b6ed` (count tile)
|
||||||
}),
|
// retired applies to the financial tiles; summing visible rows
|
||||||
{ paid: 0, adjustments: 0 }
|
// understates the true totals and the page looks complete. The
|
||||||
);
|
// server returns zero-filled values when the DB has no rows so the
|
||||||
|
// tiles render predictably while the initial fetch resolves.
|
||||||
|
//
|
||||||
|
// We don't pass the page's status chip here — the list endpoint
|
||||||
|
// (``useRemittances``) ignores status server-side and applies it
|
||||||
|
// client-side via the merged-tail filter, so passing it to the
|
||||||
|
// summary would inconsistently narrow only one of the two reads.
|
||||||
|
const { data: summary } = useRemittanceSummary();
|
||||||
|
|
||||||
const moveNext = useCallback(() => {
|
const moveNext = useCallback(() => {
|
||||||
setSelectedIndex((i) => {
|
setSelectedIndex((i) => {
|
||||||
@@ -154,19 +162,19 @@ export function Remittances() {
|
|||||||
<div className="surface-2 rounded-xl p-5">
|
<div className="surface-2 rounded-xl p-5">
|
||||||
<div className="eyebrow">Remits</div>
|
<div className="eyebrow">Remits</div>
|
||||||
<div className="display mono text-[28px] leading-none mt-3 text-foreground">
|
<div className="display mono text-[28px] leading-none mt-3 text-foreground">
|
||||||
{fmt.num(data?.total ?? 0)}
|
{fmt.num(summary?.count ?? data?.total ?? 0)}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div className="surface-2 rounded-xl p-5">
|
<div className="surface-2 rounded-xl p-5">
|
||||||
<div className="eyebrow">Total paid</div>
|
<div className="eyebrow">Total paid</div>
|
||||||
<div className="display mono text-[28px] leading-none mt-3 text-[hsl(var(--success))]">
|
<div className="display mono text-[28px] leading-none mt-3 text-[hsl(var(--success))]">
|
||||||
{fmt.usd(total.paid)}
|
{fmt.usd(summary?.total_paid ?? 0)}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div className="surface-2 rounded-xl p-5">
|
<div className="surface-2 rounded-xl p-5">
|
||||||
<div className="eyebrow">Adjustments</div>
|
<div className="eyebrow">Adjustments</div>
|
||||||
<div className="display mono text-[28px] leading-none mt-3 text-[hsl(var(--warning))]">
|
<div className="display mono text-[28px] leading-none mt-3 text-[hsl(var(--warning))]">
|
||||||
{fmt.usd(total.adjustments)}
|
{fmt.usd(summary?.total_adjustments ?? 0)}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -529,6 +529,46 @@ export interface Ack {
|
|||||||
receivedCount: number;
|
receivedCount: number;
|
||||||
ackCode: "A" | "E" | "R" | "P";
|
ackCode: "A" | "E" | "R" | "P";
|
||||||
parsedAt: string;
|
parsedAt: string;
|
||||||
|
/**
|
||||||
|
* AK2 set_control_number from the inbound 999 — the
|
||||||
|
* patient_control_number of the original claim batch this 999
|
||||||
|
* acks. Surfaced in the list endpoint so the operator can
|
||||||
|
* correlate a 999 to a claim batch in the UI without a second
|
||||||
|
* round-trip to the detail endpoint.
|
||||||
|
*/
|
||||||
|
patientControlNumber?: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// TA1 (Interchange Acknowledgment) — mirrors the `ta1_acks` table
|
||||||
|
// and the `/api/ta1-acks` response. A TA1 is the lowest-level X12
|
||||||
|
// envelope ack: one per inbound ISA/IEA interchange, separate from
|
||||||
|
// the per-batch 999. Colorado Medicaid currently doesn't ship TA1s
|
||||||
|
// in the FromHPE path (the inbound MFT only carries 999s) but
|
||||||
|
// Gainwell has shipped them historically, so the UI shows them
|
||||||
|
// whenever one is on file.
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/**
|
||||||
|
* One persisted TA1 ACK row, camelCased for the UI. Re-shaped in
|
||||||
|
* `src/lib/api.ts` (`mapTa1Ack`) from the snake_case backend payload.
|
||||||
|
*
|
||||||
|
* ``ackCode`` is the interchange ack code: ``A`` = accepted,
|
||||||
|
* ``E`` = accepted with envelope errors, ``R`` = rejected.
|
||||||
|
* ``noteCode`` is an optional 3-digit X12 note code (e.g. ``000`` = no
|
||||||
|
* error, ``001`` = unsupported interchange version, etc.).
|
||||||
|
*/
|
||||||
|
export interface Ta1Ack {
|
||||||
|
id: number;
|
||||||
|
controlNumber: string;
|
||||||
|
ackCode: "A" | "E" | "R";
|
||||||
|
noteCode: string | null;
|
||||||
|
interchangeDate: string | null;
|
||||||
|
interchangeTime: string | null;
|
||||||
|
senderId: string | null;
|
||||||
|
receiverId: string | null;
|
||||||
|
sourceBatchId: string;
|
||||||
|
parsedAt: string | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|||||||
Reference in New Issue
Block a user