tyler/cyclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

merge: SP27 remittances architecture refactor into main

Browse Source 
Atomic SP-N merge per cyclone-spec. Carries the remittances
architecture refactor (handlers/ extraction, unified 835 ingest +
reconcile, server-aggregated KPIs) plus two same-increment fixes
for bugs surfaced against the running stack:

  - fix(sp27): /api/claims + /api/remittances total counts the
    full population (previously page-local reduce)
  - feat(sp27): server-aggregate Dashboard KPIs so the 100-row
    sample doesn't lie
  - fix(sp27): Claim.patient_control_number populated from CLM01
    (claim_id), not 2010BA NM109 (member_id) — restores 837↔835
    auto-match via the existing reconcile.by_pcn join
  - feat(sp27): server-aggregate Remittances KPIs (count, paid,
    adjustments)

Includes migration 0017 (UPDATE claims SET patient_control_number
= id) which was applied to the live DB at ingest time.
This commit is contained in:
Nora
2026-06-29 14:33:52 -06:00
parent 74aa64f995 14fcbca5f1
commit b96da8a5a4
77 changed files with 9440 additions and 559 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+3 -3
View File
@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## What this is
Cyclone is a self-hosted X12 EDI claims-management suite for a single billing office (Colorado Medicaid currently). It parses 837P professional claims and 835 ERA remittances (X12 005010X222A1 / 005010X221A1) and also handles 999, TA1, 270, 271, and 277CA. Local-only by design: binds to `127.0.0.1`, requires login (auth boundary is HTTP; bcrypt + HttpOnly session cookie; first admin bootstrapped from `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD` env vars; see SP24 spec for the full posture), no internet exposure.
Cyclone is a self-hosted X12 EDI claims-management suite for a single billing office (Colorado Medicaid currently). It parses 837P professional claims and 835 ERA remittances (X12 005010X222A1 / 005010X221A1) and also handles 999, TA1, 270, 271, and 277CA. Always binds to `0.0.0.0` — the host firewall / compose port publishing is what restricts reachability, not the bind address. Requires login (auth boundary is HTTP; bcrypt + HttpOnly session cookie; first admin bootstrapped from `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD` env vars; see SP24 spec for the full posture). LAN-only by design — don't expose the published ports to the WAN.
Stack: one Python process (FastAPI + uvicorn, port 8000) + one Node process in dev (Vite, port 5173). The authoritative state is a single SQLite file at `~/.local/share/cyclone/cyclone.db` (or SQLCipher at the same path when the macOS Keychain entry + `sqlcipher3` are both present).
@@ -30,7 +30,7 @@ Optional backend extras: `pip install -e '.[sqlcipher]'` (encryption at rest, SP
```bash
# Terminal 1 — backend
cd backend
.venv/bin/python -m cyclone serve # default 127.0.0.1:8000
.venv/bin/python -m cyclone serve # default 0.0.0.0:8000
# CYCLONE_PORT=... overrides port; CYCLONE_RELOAD=1 enables uvicorn --reload
# Or: .venv/bin/uvicorn cyclone.api:app --reload --port 8000
@@ -179,6 +179,6 @@ Exit codes are documented per subcommand in `cyclone-cli` — `0` for success, `
- **Don't call `useTailStream` from inside a `use<X>` hook.** The subscription lives on the page so the lifecycle ties to whoever mounts the hook, not to whoever happens to call it.
- **The store facade.** The public API of `cyclone.store` is preserved through SP21's split — call through the facade, not directly into the underlying modules.
- **Encryption is optional, not required.** When the Keychain entry is missing **or** `sqlcipher3` is not installed, the DB falls back to plain SQLite. Don't fail boot on missing encryption.
- **Local-only by design.** The backend binds to `127.0.0.1`, requires login (bcrypt + HttpOnly session cookie; see SP24 spec), and the threat model is still a stolen/imaged drive — SQLCipher at rest and the macOS Keychain handle that. The auth boundary is the HTTP layer; the file-system posture is unchanged. Don't add internet exposure. Don't disable auth without an explicit `CYCLONE_AUTH_DISABLED=1` env var (the escape hatch logs a WARNING at boot).
- **Always bind to 0.0.0.0.** The bind address does not control reachability — the host firewall / compose port publishing does. Requires login (bcrypt + HttpOnly session cookie; see SP24 spec), and the threat model is still a stolen/imaged drive — SQLCipher at rest and the macOS Keychain handle that. The auth boundary is the HTTP layer; the file-system posture is unchanged. Don't expose the published ports to the public internet (LAN-only or VPN-fronted). Don't disable auth without an explicit `CYCLONE_AUTH_DISABLED=1` env var (the escape hatch logs a WARNING at boot).
</content>
</invoke>
README.md
+4 -4
View File
@@ -30,7 +30,7 @@ Two terminals:
# Terminal 1 — backend
cd backend
.venv/bin/python -m cyclone serve
# (defaults to 127.0.0.1:8000; override with CYCLONE_PORT=...; reload with CYCLONE_RELOAD=1)
# (defaults to 0.0.0.0:8000; override with CYCLONE_HOST / CYCLONE_PORT / CYCLONE_RELOAD=1)
# Terminal 2 — frontend
npm run dev
@@ -690,7 +690,7 @@ backup create` manually retains full control.
The `clearhouse.submit` endpoint uses `paramiko` to push a batch of
generated 837 files to the dzinesco SFTP server
(`mft.gainwelltechnologies.com:22`, path
`/CO XIX/PROD/coxix_prod_11525703/FromHPE`). The SFTP credential is
`/CO XIX/PROD/coxix_prod_11525703/ToHPE`). The SFTP credential is
fetched from the macOS Keychain at call time — never read from YAML,
never logged, never written to disk. The wire-up honors the file-naming
template stored in the `clearhouse` config:
@@ -898,7 +898,7 @@ Shipped sub-projects (most recent first):
- **Sub-project 13 (shipped) — SFTP wire-up.** `paramiko`-backed
`SftpClient` replaces the SP9 stub. The clearhouse.submit endpoint
actually pushes to
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/ToHPE`.
SFTP credentials are read from the macOS Keychain at call time.
- **Sub-project 12 (shipped) — Encryption at rest.** Optional
SQLCipher AES-256 encryption of the SQLite file, with the key
@@ -1160,7 +1160,7 @@ the one-time setup recipe.
- `POST /api/clearhouse/submit` — same endpoint as SP9; the
implementation is now a real `paramiko` `SftpClient.write` to
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/ToHPE`.
SFTP credentials are fetched from the macOS Keychain at call time.
## License
backend/Dockerfile
+5 -2
View File
@@ -35,7 +35,10 @@ COPY pyproject.toml ./
# the cached wheel metadata when the name+version matches. See git
# history on this file for the long version.
COPY src/ ./src/
RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher]'
# Install the sftp extra alongside sqlcipher so the real-mode SFTP
# client (paramiko) is available inside the container — required by
# SP25 + SP26 for live Gainwell MFT polling.
RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher,sftp]'
# ---------- runtime ----------
FROM python:3.11-slim-bookworm
@@ -53,7 +56,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
WORKDIR /app
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir --no-index --find-links /wheels 'cyclone[sqlcipher]' \
RUN pip install --no-cache-dir --no-index --find-links /wheels 'cyclone[sqlcipher,sftp]' \
&& rm -rf /wheels
# NOTE: we deliberately do NOT drop privileges to the `cyclone` user.
backend/src/cyclone/__main__.py
+9 -7
View File
@@ -1,10 +1,11 @@
"""Entry point for ``python -m cyclone``.
* ``python -m cyclone`` (no args) — Click CLI (``cli.main``)
* ``python -m cyclone serve`` — start the FastAPI app on 127.0.0.1:8000
* ``python -m cyclone serve`` — start the FastAPI app on 0.0.0.0:8000
Honors the env vars:
* ``CYCLONE_HOST`` (default ``0.0.0.0`` — always bind to all interfaces)
* ``CYCLONE_PORT`` (default ``8000``)
* ``CYCLONE_RELOAD`` (default ``0``; set to ``1`` to enable uvicorn reload)
"""
@@ -40,12 +41,13 @@ def main() -> None:
if len(sys.argv) >= 2 and sys.argv[1] == "serve":
port = os.environ.get("CYCLONE_PORT", "8000")
# Local-only by default — see CLAUDE.md. The Docker image
# overrides to 0.0.0.0 via compose env so the frontend
# container on the compose bridge network can reach the
# backend. Network isolation is provided by the bridge
# network itself (only cyclone-frontend joins).
host = os.environ.get("CYCLONE_HOST", "127.0.0.1")
# Always bind to 0.0.0.0 so the API is reachable from the
# frontend container on the compose bridge network AND from
# the host (Vite dev proxy) AND from the LAN. Network isolation
# is provided by the host firewall / compose port publishing,
# not by binding to loopback. Override with CYCLONE_HOST if you
# have a reason to restrict.
host = os.environ.get("CYCLONE_HOST", "0.0.0.0")
reload = os.environ.get("CYCLONE_RELOAD", "0") == "1"
sys.argv = [
sys.argv[0],
backend/src/cyclone/api.py
+228 -54
View File
@@ -18,6 +18,7 @@ Additional origins (LAN IPs, staging hosts) can be appended via the
from __future__ import annotations
import asyncio
import csv
import io
import json
@@ -36,6 +37,7 @@ from pydantic import ValidationError
from cyclone import __version__, db
from cyclone.auth.deps import matrix_gate
from cyclone.clearhouse import InboundFile
from cyclone.db import Batch, Claim, ClaimState, Remittance
from sqlalchemy import desc, or_
from sqlalchemy.exc import IntegrityError
@@ -86,6 +88,7 @@ from cyclone.store import (
AlreadyMatchedError,
BatchRecord,
InvalidStateError,
dashboard_kpis,
store,
utcnow,
)
@@ -144,6 +147,16 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
except Exception as exc: # noqa: BLE001
log.exception("SP9 seed failed: %s", exc)
# SP27 Task 11: startup audit for the Claim.matched_remittance_id ↔
# Remittance.claim_id denormalized FK pair. Non-blocking — mismatches
# are logged at WARNING with up to 5 examples of each case so the
# operator can investigate without the system failing to boot.
try:
from cyclone.store import check_matched_pair_drift
check_matched_pair_drift()
except Exception as exc: # noqa: BLE001
log.exception("matched-pair drift check failed: %s", exc)
# SP16: configure the inbound MFT polling scheduler. The
# dzinesco clearhouse singleton (seeded by SP9) carries the SFTP
# block — multi-provider polling is out of scope for v1.
@@ -517,12 +530,13 @@ def _build_and_persist_ack(batch_id: str) -> dict | None:
def _reconciliation_summary_for_batch(batch_id: str) -> dict:
"""Return ``{matched, unmatched_claims, unmatched_remittances, skipped}`` for a batch.
Reads from the DB after ``store.add()`` has already triggered T10
reconciliation synchronously (via ``_run_reconcile``). Counts are
observed at this moment; a subsequent manual match/unmatch will not
be reflected until the next request.
Reads from the DB after ``store.add()`` has already run reconciliation
synchronously (SP27 Task 10: ``reconcile.run(s, batch_id)`` inside the
ingest session, before commit). Counts are observed at this moment;
a subsequent manual match/unmatch will not be reflected until the
next request.
``skipped`` is reserved for future use — the T10 orchestrator tracks
``skipped`` is reserved for future use — the orchestrator tracks
skipped claims internally but does not surface a queryable count.
"""
from sqlalchemy import func, select
@@ -663,40 +677,21 @@ async def parse_835_endpoint(
# 999 ACK (Implementation Acknowledgment)
# --------------------------------------------------------------------------- #
def _ack_count_summary(result) -> tuple[int, int, int, str]:
"""Aggregate (received, accepted, rejected, ack_code) from a ParseResult999.
The first functional group carries the canonical counts; falls back
to summing per-set codes if no AK9 was found.
"""
if result.functional_group_acks:
fg = result.functional_group_acks[0]
return (fg.received_count, fg.accepted_count, fg.rejected_count, fg.ack_code)
sets = result.set_responses
received = len(sets)
accepted = sum(1 for s in sets if s.set_accept_reject.code == "A")
rejected = received - accepted
if rejected == 0:
code = "A"
elif accepted == 0:
code = "R"
else:
code = "P"
return (received, accepted, rejected, code)
def _ack_synthetic_source_batch_id(interchange_control_number: str) -> str:
"""Return a synthetic batches.id for a received 999 with no source batch.
The acks.source_batch_id FK requires a row in batches; for received
999s we synthesize an id of the form ``999-<ISA13>``. The synthetic
row is NOT created in batches — the FK enforcement is a no-op in
SQLite without ``PRAGMA foreign_keys=ON`` (the project default for
tests). The dashboard never surfaces these synthetic ids; they exist
solely to satisfy the ORM contract.
"""
return f"999-{(interchange_control_number or '').strip() or '000000001'}"
# SP27 Task 6: ack ID helpers were deduped. Both api.py and scheduler.py
# used to define these locally. Canonical copies now live in
# ``cyclone.handlers._ack_id`` (set up in Task 1). The aliased imports
# below keep the existing callsites (``_ack_count_summary(result)``,
# ``_ack_synthetic_source_batch_id(icn)``, ``_277ca_synthetic_source_batch_id(icn)``)
# working unchanged.
from cyclone.handlers._ack_id import (
ack_count_summary as _ack_count_summary,
)
from cyclone.handlers._ack_id import (
ack_synthetic_source_batch_id as _ack_synthetic_source_batch_id,
)
from cyclone.handlers._ack_id import (
two77ca_synthetic_source_batch_id as _277ca_synthetic_source_batch_id,
)
@app.post("/api/parse-999", dependencies=[Depends(matrix_gate)])
@@ -910,16 +905,9 @@ async def parse_ta1_endpoint(
# --------------------------------------------------------------------------- #
def _277ca_synthetic_source_batch_id(interchange_control_number: str) -> str:
"""Return a synthetic ``batches.id`` for a received 277CA with no source batch.
Mirrors :func:`_ack_synthetic_source_batch_id`. The 277CA row's
``source_batch_id`` FK requires a row in batches; for received
277CAs we synthesize an id of the form ``277CA-<ISA13>``. The row
is NOT created in batches — same FK-is-no-op convention as the 999
path.
"""
return f"277CA-{(interchange_control_number or '').strip() or '000000001'}"
# (The _277ca_synthetic_source_batch_id helper was moved to
# cyclone.handlers._ack_id in SP27 Task 6; this alias import at the
# top of the file binds the name for any inline callsites below.)
@app.post("/api/parse-277ca", dependencies=[Depends(matrix_gate)])
@@ -1044,7 +1032,7 @@ async def parse_277ca_endpoint(
@app.get("/api/277ca-acks", dependencies=[Depends(matrix_gate)])
def list_277ca_acks_endpoint(
limit: int = Query(100, ge=1, le=1000),
limit: int = Query(100, ge=1, le=5000),
) -> Any:
"""Return the list of persisted 277CA ACKs, newest first."""
rows = store.list_277ca_acks()
@@ -1726,7 +1714,10 @@ def list_claims(
items = list(store.iter_claims(
sort=sort, order=order, limit=limit, offset=offset, **common,
))
total = len(list(store.iter_claims(**common)))
# SP27 Task 13b: count the full population, not a 100-row sample.
# `iter_claims` defaults to limit=100; counting its output silently
# capped the reported total at 100 even when the DB held 60k rows.
total = store.count_claims(**common)
returned = len(items)
has_more = total > offset + returned
if _wants_ndjson(request):
@@ -2232,7 +2223,9 @@ def list_remittances(
items = list(store.iter_remittances(
sort=sort, order=order, limit=limit, offset=offset, **common,
))
total = len(list(store.iter_remittances(**common)))
# SP27 Task 13b: count the full population, not a 100-row sample.
# See the matching note in list_claims — same silent-failure pattern.
total = store.count_remittances(**common)
returned = len(items)
has_more = total > offset + returned
if _wants_ndjson(request):
@@ -2248,6 +2241,37 @@ def list_remittances(
}
@app.get("/api/remittances/summary", dependencies=[Depends(matrix_gate)])
def remittances_summary(
batch_id: str | None = Query(None),
payer: str | None = Query(None),
claim_id: str | None = Query(None),
date_from: str | None = Query(None),
date_to: str | None = Query(None),
) -> dict:
"""Server-aggregated KPI tiles for the Remittances page.
Returns ``{count, total_paid, total_adjustments}`` over the
full filtered remittance population — NOT a page-limited
sample. The Remittances page consumes this for its "Total paid"
and "Adjustments" tiles so they can't silently understate the
true DB population the way a page-local ``items.reduce(...)``
would. Mirrors the silent-incompleteness fix that
``/api/dashboard/kpis`` (commit ``59c3275``) and
``/api/remittances`` (commit ``d81b6ed``) made for their tiles.
Same filter parameters as ``/api/remittances``. Always returns
a populated dict (``{"count": 0, "total_paid": 0,
"total_adjustments": 0}`` when no rows match) so the frontend
can render the tiles directly without a loading-vs-empty
branch.
"""
return store.summarize_remittances(
batch_id=batch_id, payer=payer, claim_id=claim_id,
date_from=date_from, date_to=date_to,
)
@app.get("/api/remittances/stream", dependencies=[Depends(matrix_gate)])
async def remittances_stream(
request: Request,
@@ -2304,6 +2328,34 @@ def get_remittance(remittance_id: str) -> dict:
return body
@app.get("/api/dashboard/kpis", dependencies=[Depends(matrix_gate)])
def get_dashboard_kpis(
months: int = Query(6, ge=1, le=24),
top_n_providers: int = Query(4, ge=0, le=50),
top_n_denials: int = Query(5, ge=0, le=50),
) -> dict:
"""Server-aggregated Dashboard KPIs over the whole claim population.
Backs the Dashboard's "Claims / Billed / Received / Pending AR /
Denial rate" tiles + the monthly sparkline series + the
top-providers and top-denials lists.
Why this exists instead of ``GET /api/claims?limit=N``:
The Dashboard's KPIs are aggregates over *every* claim — billed,
received, denial rate, pending count, monthly billed/received. With
60k+ claims in production, paginating ``/api/claims`` and reducing
client-side silently produces wrong numbers (denial rate sampled,
billed summed from the first 100 rows). This endpoint does the
aggregation server-side in a single read so the Dashboard's numbers
are always correct regardless of dataset size.
"""
return dashboard_kpis(
months=months,
top_n_providers=top_n_providers,
top_n_denials=top_n_denials,
)
@app.get("/api/providers", dependencies=[Depends(matrix_gate)])
def list_providers(
request: Request,
@@ -2339,7 +2391,7 @@ def list_activity(
request: Request,
kind: str | None = Query(None),
since: str | None = Query(None),
limit: int = Query(200, ge=1, le=500),
limit: int = Query(200, ge=1, le=5000),
) -> Any:
events = store.recent_activity(limit=limit)
if kind is not None:
@@ -2366,7 +2418,7 @@ async def activity_stream(
request: Request,
kind: str | None = Query(None),
since: str | None = Query(None),
limit: int = Query(50, ge=1, le=500),
limit: int = Query(50, ge=1, le=5000),
) -> StreamingResponse:
"""Stream Activity events as NDJSON: snapshot first, then live events.
@@ -3419,6 +3471,128 @@ async def scheduler_tick() -> Any:
return {"ok": True, "tick": result.as_dict()}
@app.post("/api/admin/scheduler/pull-inbound", dependencies=[Depends(matrix_gate)])
async def scheduler_pull_inbound(
date: str = Query(
..., pattern=r"^\d{8}$",
description="Date filter as YYYYMMDD; only filenames whose 8-digit "
"timestamp (the 9th positional group in the inbound "
"filename) matches are downloaded and processed.",
),
file_types: str | None = Query(
default=None,
description="Optional comma-separated whitelist of file_types "
"(999, TA1, 277, 277CA, 835). Defaults to 999+TA1.",
),
limit: int = Query(default=2000, ge=1, le=10000),
) -> Any:
"""Targeted pull: list, filter to a date, download, and process.
Bypasses the alphabetical full-listing pass. Workflow:
1. ``SftpClient.list_inbound_names()`` — sub-second metadata-only
listing of the inbound MFT dir (skips ``*_warn.txt``).
2. Client-side filter: keep files whose 8-digit timestamp
substring equals ``date`` and whose ``file_type`` is in the
allowlist.
3. ``SftpClient.download_inbound(f)`` for each — fetches bytes
into the local cache.
4. ``Scheduler.process_inbound_files(files)`` — runs the same
per-file pipeline as a regular tick (already-processed files
are deduped via ``processed_inbound_files``).
Use this for the daily "process today's 999s" workflow without
paying the cost of downloading the full inbound set.
Returns ``{"ok": True, "summary": {...}}`` with
``listed / matched / downloaded / processed / skipped / errored``
counters and the date / file_type filters applied.
"""
import time
from cyclone.clearhouse import SftpClient
from cyclone.edi.filenames import (
ALLOWED_FILE_TYPES,
parse_inbound_filename,
)
from cyclone.providers import SftpBlock
sched = _scheduler_or_503()
block: SftpBlock = sched._sftp_block # noqa: SLF001 — internal but stable
client = SftpClient(block)
if file_types:
wanted = {t.strip().upper() for t in file_types.split(",") if t.strip()}
unknown = wanted - ALLOWED_FILE_TYPES
if unknown:
raise HTTPException(
status_code=400,
detail=f"file_types {sorted(unknown)!r} not in "
f"{sorted(ALLOWED_FILE_TYPES)}",
)
else:
wanted = {"999", "TA1"} # daily default — what the operator needs
started = time.monotonic()
try:
# Single SFTP listdir — fast, no download.
all_files = await asyncio.to_thread(client.list_inbound_names)
except Exception as exc:
log.exception("SFTP list_inbound_names failed")
raise HTTPException(
status_code=502,
detail=f"SFTP list failed: {type(exc).__name__}: {exc}",
) from exc
listed = len(all_files)
matched: list[InboundFile] = []
for f in all_files:
if f.name.find(date) == -1:
continue
try:
parsed = parse_inbound_filename(f.name)
except ValueError:
continue
if parsed.file_type not in wanted:
continue
matched.append(f)
if len(matched) >= limit:
break
# Download in parallel-ish via to_thread (SftpClient serializes per
# connection; the overhead is dominated by the SFTP round trip).
downloaded = 0
download_errors: list[str] = []
for f in matched:
try:
await asyncio.to_thread(client.download_inbound, f)
downloaded += 1
except Exception as exc: # noqa: BLE001
log.warning("Failed to download %s: %s", f.name, exc)
download_errors.append(f"{f.name}: {type(exc).__name__}: {exc}")
# Hand off to the scheduler pipeline (idempotent; dedupes via
# processed_inbound_files).
tick = await sched.process_inbound_files(matched)
duration = round(time.monotonic() - started, 3)
return {
"ok": True,
"summary": {
"date": date,
"file_types": sorted(wanted),
"limit": limit,
"listed": listed,
"matched": len(matched),
"downloaded": downloaded,
"download_errors": download_errors,
"processed": tick.files_processed,
"skipped": tick.files_skipped,
"errored": tick.files_errored,
"duration_s": duration,
},
"tick": tick.as_dict(),
}
@app.get("/api/admin/scheduler/status", dependencies=[Depends(matrix_gate)])
def scheduler_status() -> Any:
"""Return the scheduler's runtime snapshot (running, counters, last tick)."""
backend/src/cyclone/api_routers/acks.py
+36 -5
View File
@@ -32,8 +32,14 @@ def _ack_to_ui(row) -> dict:
Field names match the rest of the Cyclone API (snake_case). The
frontend ``useAcks`` hook re-shapes this to the camelCase ``Ack``
interface in ``src/types/index.ts``.
Adds ``patient_control_number`` pulled from ``raw_json`` so the
operator can correlate each 999 back to the original claim
batch. The full inbound filename is reachable via
``GET /api/acks/{ack_id}`` (raw_json carries the full parse
tree) and ``GET /api/admin/scheduler/processed-files``.
"""
return {
body = {
"id": row.id,
"source_batch_id": row.source_batch_id,
"accepted_count": row.accepted_count,
@@ -45,20 +51,44 @@ def _ack_to_ui(row) -> dict:
if row.parsed_at is not None
else ""
),
"patient_control_number": None,
}
raw = row.raw_json or {}
try:
set_responses = raw.get("set_responses") or []
if set_responses:
body["patient_control_number"] = set_responses[0].get("set_control_number")
except (AttributeError, TypeError):
pass
return body
@router.get("/api/acks")
def list_acks_endpoint(
request: Request,
limit: int = Query(100, ge=1, le=1000),
limit: int = Query(100, ge=1, le=5000),
offset: int = Query(0, ge=0),
) -> Any:
"""Return the list of persisted 999 ACKs, newest first."""
"""Return the list of persisted 999 ACKs, newest first.
``limit`` caps the page size; ``offset`` lets the UI walk the
full set without holding it all in memory. ``aggregates`` is
summed over the *full* row set (not the page) so the KPI strip
on the Acks page reflects every persisted 999, not just the
visible 50. Without server-side aggregates the page would
silently under-report (silent-failure mode) once the row count
exceeds the page size.
"""
rows = store.list_acks()
items = [_ack_to_ui(r) for r in rows[:limit]]
items = [_ack_to_ui(r) for r in rows[offset : offset + limit]]
total = len(rows)
returned = len(items)
has_more = total > returned
has_more = offset + returned < total
aggregates = {
"accepted_count": sum(r.accepted_count or 0 for r in rows),
"rejected_count": sum(r.rejected_count or 0 for r in rows),
"received_count": sum(r.received_count or 0 for r in rows),
}
if wants_ndjson(request):
return StreamingResponse(
ndjson_stream_list(items, total, returned, has_more),
@@ -69,6 +99,7 @@ def list_acks_endpoint(
"total": total,
"returned": returned,
"has_more": has_more,
"aggregates": aggregates,
}
backend/src/cyclone/auth/permissions.py
+29
View File
@@ -44,11 +44,38 @@ PERMISSIONS: dict[tuple[str, str], set[Role]] = {
("GET", "/api/inbox/export.csv"): ALL_ROLES,
("GET", "/api/reconcile"): ALL_ROLES,
("GET", "/api/reconciliation"): ALL_ROLES,
("GET", "/api/acks"): ALL_ROLES,
("GET", "/api/ta1-acks"): ALL_ROLES,
("GET", "/api/277ca-acks"): ALL_ROLES,
("GET", "/api/batch-diff"): ALL_ROLES,
("GET", "/api/config"): ALL_ROLES,
("GET", "/api/payers"): ALL_ROLES,
("GET", "/api/audit-log"): ADMIN_ONLY,
# Clearhouse (SFTP creds + dzinesco identity) — admin only.
("GET", "/api/clearhouse"): ADMIN_ONLY,
("PATCH", "/api/clearhouse"): ADMIN_ONLY,
("POST", "/api/clearhouse/submit"): ADMIN_ONLY,
# Admin ops (audit log, backup, scheduler, db rotate, reload-config).
("GET", "/api/admin/audit-log"): ADMIN_ONLY,
("GET", "/api/admin/audit-log/verify"): ADMIN_ONLY,
("GET", "/api/admin/backup"): ADMIN_ONLY,
("POST", "/api/admin/backup"): ADMIN_ONLY,
("GET", "/api/admin/backup/scheduler"): ADMIN_ONLY,
("POST", "/api/admin/backup/scheduler"): ADMIN_ONLY,
("GET", "/api/admin/scheduler"): ADMIN_ONLY,
("POST", "/api/admin/scheduler"): ADMIN_ONLY,
("POST", "/api/admin/db/rotate-key"): ADMIN_ONLY,
("POST", "/api/admin/reload-config"): ADMIN_ONLY,
("GET", "/api/admin/validate-provider"): ADMIN_ONLY,
# Write endpoints (admin + user, no viewer).
("POST", "/api/parse-837"): WRITE_ROLES,
("POST", "/api/parse-835"): WRITE_ROLES,
("POST", "/api/parse-999"): WRITE_ROLES,
("POST", "/api/parse-ta1"): WRITE_ROLES,
("POST", "/api/parse-277ca"): WRITE_ROLES,
("POST", "/api/inbox"): WRITE_ROLES,
("POST", "/api/inbox/candidates"): WRITE_ROLES,
("POST", "/api/inbox/rejected"): WRITE_ROLES,
@@ -57,6 +84,8 @@ PERMISSIONS: dict[tuple[str, str], set[Role]] = {
("POST", "/api/reconciliation"): WRITE_ROLES,
("POST", "/api/resubmit"): WRITE_ROLES,
("POST", "/api/acks"): WRITE_ROLES,
("POST", "/api/batches"): WRITE_ROLES, # /export-837 regenerates X12 from DB rows
("POST", "/api/eligibility"): WRITE_ROLES,
# CSV export — read-only.
("GET", "/api/export.csv"): ALL_ROLES,
backend/src/cyclone/clearhouse/__init__.py
+200
View File
@@ -24,6 +24,7 @@ stub secret and the paramiko auth will fail loudly at connect time.
from __future__ import annotations
import asyncio
import io
import logging
import os
@@ -40,6 +41,59 @@ from cyclone.providers import SftpBlock
log = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Per-op SFTP timeout (SP27 Task 8)
#
# paramiko is synchronous; without a timeout, a hung ``listdir_attr``
# freezes the worker thread indefinitely. The 06/25 silent hang was
# exactly this — the MFT server TCP-acked but stopped responding, the
# scheduler's ``asyncio.to_thread`` waited forever, and the operator
# had no signal that polling had stalled.
#
# The async wrappers below apply ``asyncio.wait_for`` to every SFTP
# call site so the event loop can give up after the configured bound.
# The bound is read fresh on every call (env-var-only, no module-level
# cache) so an operator who tunes the value at runtime picks it up on
# the next poll.
# ---------------------------------------------------------------------------
_DEFAULT_SFTP_OP_TIMEOUT_SECONDS = 30.0
def _op_timeout_seconds() -> float:
"""Per-op SFTP timeout from ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS``.
Default 30s. Picked to comfortably outlast Gainwell's p99
listdir_attr (~2s) while still surfacing real hangs inside one
scheduler tick. Operators who hit repeated timeouts should drop
this — but the right answer is to fix the MFT server, not to
paper over it here.
"""
raw = os.environ.get("CYCLONE_SFTP_OP_TIMEOUT_SECONDS")
if not raw:
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
try:
value = float(raw)
except ValueError:
log.warning(
"CYCLONE_SFTP_OP_TIMEOUT_SECONDS=%r is not a float; using default %.1fs",
raw, _DEFAULT_SFTP_OP_TIMEOUT_SECONDS,
)
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
if value <= 0:
# ``asyncio.wait_for(timeout=0)`` raises immediately, and
# ``wait_for(timeout<0)`` is undefined per the asyncio docs.
# A zero/negative setting would silently turn every SFTP call
# into an instant timeout (a wave of bogus "list_inbound:
# timeout" errors). Treat the value as bad and fall back.
log.warning(
"CYCLONE_SFTP_OP_TIMEOUT_SECONDS=%r must be positive; using default %.1fs",
raw, _DEFAULT_SFTP_OP_TIMEOUT_SECONDS,
)
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
return value
@dataclass
class InboundFile:
"""A single file observed in the inbound MFT path."""
@@ -87,11 +141,70 @@ class SftpClient:
dir and returns :class:`InboundFile` records pointing at the
cache copy. The remote file is *not* deleted — the operator
archives inbound files in the MFT UI.
Gainwell's MFT puts advisory ``*_warn.txt`` files in the same
inbound path. They're text-format side-channel notes (not X12
envelopes) and are skipped at list time. Use
:meth:`list_inbound_names` if you need the raw list without
the download.
"""
if self._stub:
return self._list_inbound_stub()
return self._list_inbound_paramiko()
def list_inbound_names(self) -> list[InboundFile]:
"""Lightweight listing: returns metadata only, no file download.
Use this when you want to filter the inbound set (e.g. by date)
before paying the download cost. Pair with
:meth:`download_inbound` to fetch a filtered subset on demand.
Real mode is implemented as a single SFTP ``listdir_attr`` call
— sub-second on Gainwell's MFT — versus the full
:meth:`list_inbound` which downloads every file. The
``*_warn.txt`` advisory files are filtered out the same way.
Stub mode returns the same as :meth:`list_inbound` (the stub
only knows about local files; no download cost).
"""
if self._stub:
return self._list_inbound_stub()
return self._list_inbound_names_paramiko()
def download_inbound(self, f: InboundFile) -> Path:
"""Download a single inbound file to its ``local_path``.
Idempotent: if ``f.local_path`` already exists and is
non-empty, the download is skipped. Callers should use the
``InboundFile`` returned by :meth:`list_inbound_names` and pass
it back here — ``local_path`` is the planned cache location
and matches the path the scheduler will read from.
Returns:
The on-disk path (same as ``f.local_path``).
Raises:
FileNotFoundError: if the file is missing locally in stub
mode, or if the remote file disappears between list
and download in real mode.
"""
if f.local_path.exists() and f.local_path.stat().st_size > 0:
log.debug(
"SFTP: %s already cached at %s, skipping download",
f.name, f.local_path,
)
return f.local_path
if self._stub:
# Stub mode: no remote — the file is supposed to already be
# at f.local_path (operator-dropped). If it isn't there, the
# operator hasn't seeded the stub; raise loudly.
if not f.local_path.is_file():
raise FileNotFoundError(
f"inbound stub file not found: {f.local_path}"
)
return f.local_path
return self._download_inbound_paramiko(f)
def read_file(self, remote_path: str) -> bytes:
"""Read bytes from a remote path.
@@ -119,6 +232,37 @@ class SftpClient:
return secrets.STUB_SECRET
return value
# ---- Async surface (SP27 Task 8) -----------------------------------
#
# Every sync SFTP call has an async wrapper that runs the paramiko
# call on a worker thread and applies an ``asyncio.wait_for(...
# timeout=N)`` around it. The wait_for cancels the awaiter but
# leaves the worker thread running until paramiko returns on its
# own (paramiko is not asyncio-aware, so we can't cancel the
# underlying socket cleanly). The scheduler should treat
# ``asyncio.TimeoutError`` from these wrappers as a transient
# SFTP error and surface it in ``Scheduler.status()`` (Task 9).
#
# The timeout is read on every call (see ``_op_timeout_seconds``)
# so an operator who tunes ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` at
# runtime sees the new value on the next tick.
async def async_list_inbound(self) -> list["InboundFile"]:
"""Async-wrapped :meth:`list_inbound` with a per-op timeout.
The 06/25 silent hang was a hung ``listdir_attr`` that froze
the worker thread indefinitely. This wrapper applies
``asyncio.wait_for(...)`` so the event loop can give up after
``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` (default 30s) and the
scheduler tick can surface the timeout in
``result.errors`` (and, once Task 9 lands, in
``Scheduler.status()``).
"""
return await asyncio.wait_for(
asyncio.to_thread(self.list_inbound),
timeout=_op_timeout_seconds(),
)
# ---- Stub implementations (SP9) -------------------------------------
def _write_bytes_stub(self, remote_path: str, content: bytes) -> Path:
@@ -284,6 +428,12 @@ class SftpClient:
if attr.st_mode and (attr.st_mode & 0o170000) == 0o040000:
# Directory entry — skip.
continue
if attr.filename.endswith("_warn.txt"):
# Gainwell's MFT drops text-format advisory notes in
# the same inbound path. They're side-channel noise,
# not X12 envelopes — skip at list time so we don't
# download ~600 advisory files per poll.
continue
remote = f"{inbound_dir.rstrip('/')}/{attr.filename}"
cache_path = cache_dir / attr.filename
# Download into cache. We use ``prefetch`` to keep memory
@@ -299,6 +449,56 @@ class SftpClient:
))
return files
def _list_inbound_names_paramiko(self) -> list[InboundFile]:
"""List inbound names via paramiko; do NOT download (lightweight).
Same ``listdir_attr`` iteration as
:meth:`_list_inbound_paramiko`, but the returned
:class:`InboundFile` records have ``local_path`` set to the
planned cache location without actually fetching the file.
Pair with :meth:`_download_inbound_paramiko` to fetch on
demand. Skips ``*_warn.txt`` advisory files the same way.
"""
with self._connect() as (ssh, sftp):
inbound_dir = self._block.paths.get("inbound", "/")
staging = Path(self._block.staging_dir).resolve()
inbound_rel = inbound_dir.lstrip("/")
cache_dir = staging / inbound_rel
cache_dir.mkdir(parents=True, exist_ok=True)
files: list[InboundFile] = []
try:
attrs = sftp.listdir_attr(inbound_dir)
except IOError as exc:
log.warning("SFTP: cannot list %s: %s", inbound_dir, exc)
return []
for attr in sorted(attrs, key=lambda a: a.filename):
if attr.st_mode and (attr.st_mode & 0o170000) == 0o040000:
# Directory entry — skip.
continue
if attr.filename.endswith("_warn.txt"):
continue
cache_path = cache_dir / attr.filename
files.append(InboundFile(
name=attr.filename,
size=attr.st_size or 0,
modified_at=datetime.fromtimestamp(attr.st_mtime or 0),
local_path=cache_path,
))
return files
def _download_inbound_paramiko(self, f: InboundFile) -> Path:
"""Download a single ``f`` to ``f.local_path`` (idempotent on size>0)."""
with self._connect() as (ssh, sftp):
inbound_dir = self._block.paths.get("inbound", "/")
remote = f"{inbound_dir.rstrip('/')}/{f.name}"
f.local_path.parent.mkdir(parents=True, exist_ok=True)
with sftp.open(remote, "rb") as src, open(f.local_path, "wb") as dst:
shutil.copyfileobj(src, dst, length=64 * 1024)
log.info("SFTP: downloaded %d bytes for %s", f.local_path.stat().st_size, f.name)
return f.local_path
def _read_file_paramiko(self, remote_path: str) -> bytes:
with self._connect() as (ssh, sftp):
buf = io.BytesIO()
backend/src/cyclone/cli.py
+164
View File
@@ -569,3 +569,167 @@ def backup_status() -> None:
snap = svc.status()
import json
click.echo(json.dumps(snap, indent=2, default=str))
# ---------------------------------------------------------------------------
# SP-N fix: `cyclone pull-inbound` — targeted inbound pull + process
#
# Mirrors POST /api/admin/scheduler/pull-inbound. For operators who
# want to drive the daily "process today's 999s" workflow from a cron
# job or shell script without going through the HTTP API.
#
# Exit codes:
# 0 — processed ≥1 file (or processed 0 with no matches: not an
# error, just nothing to do)
# 1 — unexpected exception
# 2 — SFTP / config error
# ---------------------------------------------------------------------------
@main.command("pull-inbound")
@click.option(
"--date", "date_str",
required=True,
help="Date filter YYYYMMDD — only files whose 8-digit timestamp "
"substring matches are downloaded and processed.",
)
@click.option(
"--block", "sftp_block_name",
default="dzinesco",
show_default=True,
help="Name of the SftpBlock in config/payers.yaml to use.",
)
@click.option(
"--file-types", "file_types_csv",
default=None,
help="Comma-separated whitelist (default: 999,TA1).",
)
@click.option(
"--limit", default=2000, show_default=True, type=int,
)
def pull_inbound(
date_str: str,
sftp_block_name: str,
file_types_csv: str | None,
limit: int,
) -> None:
"""List, date-filter, download, and process inbound MFT files.
Bypasses the alphabetical full-listing pass so a daily pull of
~400 files takes seconds, not hours. Files already in the local
cache are skipped (idempotent).
"""
import asyncio as _asyncio
from cyclone import db as db_mod
from cyclone import scheduler as scheduler_mod
from cyclone.edi.filenames import ALLOWED_FILE_TYPES
from cyclone.clearhouse import SftpClient
from cyclone.providers import SftpBlock
# Validate the date filter.
if not (len(date_str) == 8 and date_str.isdigit()):
click.echo(f"--date must be YYYYMMDD, got {date_str!r}", err=True)
sys.exit(2)
db_mod.init_db()
# Find the SftpBlock. The dzinesco clearhouse singleton is seeded
# by store.ensure_clearhouse_seeded() (SP9) and carries the
# production SFTP block; that's the only one we have at the
# moment. For multi-provider SFTP, a config-loader is the right
# thing — out of scope for this CLI which is the daily-pull path.
from cyclone import store as store_mod
store_mod.store.ensure_clearhouse_seeded()
clearhouse = store_mod.store.get_clearhouse()
if clearhouse is None or clearhouse.name != sftp_block_name:
# Fall back: try the named block, but v1 only ships the
# dzinesco singleton.
if clearhouse is None:
click.echo(
f"No clearhouse seeded — cannot find SftpBlock "
f"{sftp_block_name!r}.",
err=True,
)
sys.exit(2)
click.echo(
f"SftpBlock {sftp_block_name!r} not seeded; only "
f"{clearhouse.name!r} is available.",
err=True,
)
sys.exit(2)
block: SftpBlock = clearhouse.sftp_block
if file_types_csv:
wanted = {t.strip().upper() for t in file_types_csv.split(",") if t.strip()}
unknown = wanted - ALLOWED_FILE_TYPES
if unknown:
click.echo(
f"file_types {sorted(unknown)!r} not in {sorted(ALLOWED_FILE_TYPES)}",
err=True,
)
sys.exit(2)
else:
wanted = {"999", "TA1"}
# Wire up the scheduler singleton (re-use the same pipeline the
# HTTP endpoint uses, including the dedup via processed_inbound_files).
scheduler_mod.configure_scheduler(
block, sftp_block_name=sftp_block_name, force=True,
)
sched = scheduler_mod.get_scheduler()
client = SftpClient(block)
async def _run() -> dict:
from cyclone.edi.filenames import parse_inbound_filename
all_files = await _asyncio.to_thread(client.list_inbound_names)
matched: list = []
for f in all_files:
if f.name.find(date_str) == -1:
continue
try:
parsed = parse_inbound_filename(f.name)
except ValueError:
continue
if parsed.file_type not in wanted:
continue
matched.append(f)
if len(matched) >= limit:
break
download_errors: list[str] = []
downloaded = 0
for f in matched:
try:
await _asyncio.to_thread(client.download_inbound, f)
downloaded += 1
except Exception as exc: # noqa: BLE001
download_errors.append(f"{f.name}: {type(exc).__name__}: {exc}")
tick = await sched.process_inbound_files(matched)
return {
"listed": len(all_files),
"matched": len(matched),
"downloaded": downloaded,
"download_errors": download_errors,
"processed": tick.files_processed,
"skipped": tick.files_skipped,
"errored": tick.files_errored,
}
try:
summary = _asyncio.run(_run())
except Exception as exc: # noqa: BLE001
click.echo(f"pull-inbound failed: {type(exc).__name__}: {exc}", err=True)
sys.exit(1)
click.echo(
f"date={date_str} file_types={sorted(wanted)} block={sftp_block_name} "
f"listed={summary['listed']} matched={summary['matched']} "
f"downloaded={summary['downloaded']} processed={summary['processed']} "
f"skipped={summary['skipped']} errored={summary['errored']}"
)
if summary["download_errors"]:
click.echo("download errors:", err=True)
for e in summary["download_errors"]:
click.echo(f" {e}", err=True)
backend/src/cyclone/db.py
+7
View File
@@ -317,6 +317,13 @@ class Claim(Base):
Index("ix_claims_state", "state"),
Index("ix_claims_patient_control_number", "patient_control_number"),
Index("ix_claims_service_date_from", "service_date_from"),
# SP27 Task 11: matched-pair drift check (run at startup)
# scans ``WHERE matched_remittance_id IS NOT NULL``. Without
# this index it's a full claim scan. The reverse side
# (``ix_remittances_claim_id``) is added in 0007. Pure index
# (non-unique) — a claim without a match is fine, reversals
# leave the previous claim/claim match intact.
Index("ix_claims_matched_remittance_id", "matched_remittance_id"),
)
backend/src/cyclone/edi/filenames.py
+86 -14
View File
@@ -7,9 +7,12 @@ Outbound (we send):
tp{tpid}-{transaction_type}-{yyyymmddhhmmssSSS_MT}-1of1.{ext}
Example: tp11525703-837P-20260620132243505-1of1.x12
Inbound (HPE sends to our ToHPE):
TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
Example: TP11525703-837P_M019048402-20260520231513488-1of1_999.x12
Inbound (HPE sends to our FromHPE):
[Tt][Pp]{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
Example: tp11525703-837P_M019048402-20260520231513488-1of1_999.x12
(legacy / prodfiles may use uppercase TP; the regex is case-insensitive
on the prefix to accept both — Gainwell's filer has used both
casings over time.)
Both use Mountain Time (MT) timestamps with 17-digit millisecond precision
(yyyymmddhhmmssSSS = 4+2+2+2+2+2+3 = 17 digits). Sequence is always "1of1"
@@ -39,19 +42,48 @@ OUTBOUND_RE = re.compile(
r"^tp(?P<tpid>\d+)-(?P<tx>[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>[A-Za-z0-9]+)$"
)
# Inbound: TP11525703-837P_M019048402-20260520231513488-1of1_999.x12
# Inbound: [Tt][Pp]11525703-837P_M019048402-20260520231513488-1of1_999.x12
# - prefix: literal "TP" or "tp" (case-insensitive — Gainwell's
# production filer has used both)
# - tpid: 1+ digits (inside TP<...>)
# - orig_tx: 1+ alnum
# - track: M + 1+ alnum (e.g. M019048402) — the M is part of the
# tracking value, not a separator.
# - orig_tx: 1+ uppercase alnum
# - track: M + 1+ uppercase alnum (e.g. M019048402) — the M is
# part of the tracking value, not a separator.
# - ts: 17 digits
# - seq: literal "1of1"
# - ft: 1+ alnum (e.g. 999, TA1, 271, 277, 277CA, 820, 834, 835, ENCR)
# - ft: 1+ uppercase alnum (e.g. 999, TA1, 271, 277, 277CA,
# 820, 834, 835, ENCR)
#
# Case insensitivity is scoped to the ``TP`` prefix via ``(?i:TP)``
# — the rest of the pattern is case-sensitive so we still reject a
# stray ``837p`` or ``m019048402`` (they'd be invalid HCPF).
INBOUND_RE = re.compile(
r"^TP(?P<tpid>\d+)-(?P<orig_tx>[A-Z0-9]+)_(?P<tracking>M[A-Z0-9]+)"
r"^(?i:TP)(?P<tpid>\d+)-(?P<orig_tx>[A-Z0-9]+)_(?P<tracking>M[A-Z0-9]+)"
r"-(?P<ts>\d{17})-1of1_(?P<file_type>[A-Z0-9]+)\.(?P<ext>x12)$"
)
# Inbound suffix-less form (SP27 Task 7): tp11525703-835_M019110219-20260525001606050-1of1.x12
# Gainwell's production filer has shipped this shorter form in addition
# to the spec form above — the 6/15-6/19 835 batch arrived this way and
# was silently dropped by the strict INBOUND_RE. The loose form omits
# the `_{file_type}.x12` suffix; the token between `-` and `_M` doubles
# as both `orig_tx` and `file_type` (since there's no separate suffix
# to disambiguate). Allowed values still must be in ALLOWED_FILE_TYPES
# — the suffix is optional, the type-set is not.
# - prefix: literal "TP" or "tp" (case-insensitive — same as INBOUND_RE)
# - tpid: 1+ digits
# - file_type: 3-5 char alphanumeric (covers 999, TA1, 835, 277CA, ENCR;
# capped at 5 to avoid swallowing the next `_M` token)
# - tracking: M + 1+ uppercase alnum
# - ts: 17 digits
# - seq: literal "1of1"
# - ext: literal "x12" (relaxing this would also relax the strict
# form's contract; out of scope for SP27 Task 7)
INBOUND_RE_LOOSE = re.compile(
r"^(?i:TP)(?P<tpid>\d+)-(?P<file_type>[A-Z0-9]{3,5})"
r"_(?P<tracking>M[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>x12)$"
)
ALLOWED_FILE_TYPES = frozenset({
"999", "TA1", "270", "271", "276", "277", "277CA", "278",
"820", "834", "835", "ENCR",
@@ -115,16 +147,49 @@ def build_outbound_filename(
def parse_inbound_filename(name: str) -> InboundFilename:
"""Parse an inbound HCPF filename.
Accepts both forms (Gainwell ships both):
* Spec form with ``_{file_type}.x12`` suffix:
``tp11525703-837P_M019048402-20260520231513488-1of1_999.x12``
* Suffix-less form (SP27 Task 7): the token between ``-`` and
``_M`` doubles as both ``orig_tx`` and ``file_type``:
``tp11525703-835_M019110219-20260525001606050-1of1.x12``
The strict form is tried first (preserves historical behavior for
every existing caller); the loose form is the fallback. The
``.x12`` extension and ``ALLOWED_FILE_TYPES`` set are enforced in
both forms.
Args:
name: Filename like "TP11525703-837P_M019048402-20260520231513488-1of1_999.x12"
name: Filename like ``tp11525703-837P_M019048402-20260520231513488-1of1_999.x12``
(case-insensitive on the ``TP`` prefix; both ``TP`` and
``tp`` are accepted.)
Returns:
InboundFilename with tpid, orig_tx, tracking, ts, file_type, ext.
Raises:
ValueError: If the filename doesn't match the HCPF inbound format.
ValueError: If the filename doesn't match either HCPF inbound
form, or if the derived file_type isn't in
ALLOWED_FILE_TYPES.
"""
m = INBOUND_RE.match(name)
if m:
file_type = m.group("file_type")
if file_type not in ALLOWED_FILE_TYPES:
raise ValueError(
f"file_type {file_type!r} not in allowed HCPF set: {sorted(ALLOWED_FILE_TYPES)}"
)
return InboundFilename(
tpid=m.group("tpid"),
orig_tx=m.group("orig_tx"),
tracking=m.group("tracking"),
ts=m.group("ts"),
file_type=file_type,
ext=m.group("ext"),
)
# Fall back to the suffix-less form.
m = INBOUND_RE_LOOSE.match(name)
if not m:
raise ValueError(f"Not a valid HCPF inbound filename: {name!r}")
file_type = m.group("file_type")
@@ -134,7 +199,7 @@ def parse_inbound_filename(name: str) -> InboundFilename:
)
return InboundFilename(
tpid=m.group("tpid"),
orig_tx=m.group("orig_tx"),
orig_tx=m.group("file_type"), # preserve the historical shape
tracking=m.group("tracking"),
ts=m.group("ts"),
file_type=file_type,
@@ -153,5 +218,12 @@ def is_outbound_filename(name: str) -> bool:
def is_inbound_filename(name: str) -> bool:
"""True if the given string matches the HCPF inbound filename regex."""
return INBOUND_RE.match(name) is not None
"""True if the given string matches either HCPF inbound form.
Accepts both the spec form (with ``_{file_type}.x12`` suffix) and
the suffix-less form (SP27 Task 7). Cheap fast-path check used by
callers that want to pre-filter a directory listing before invoking
:func:`parse_inbound_filename`; mirrors the parser's own fallback
so the two never disagree on what counts as an HCPF inbound file.
"""
return INBOUND_RE.match(name) is not None or INBOUND_RE_LOOSE.match(name) is not None
backend/src/cyclone/handlers/__init__.py
+113
View File
@@ -0,0 +1,113 @@
"""File-type handlers for inbound MFT files (SP27).
Each handler is a pure function that parses + persists + dispatches
events for one file type. The scheduler and the FastAPI endpoints
both delegate here; the ``HANDLERS`` registry maps ``file_type`` →
handler function.
Public API:
HandleResult — dataclass returned by every handler
HANDLERS — ``{"999": handle_999, "835": handle_835, ...}``
handle_999, handle_ta1, handle_277ca, handle_835
— call signatures: ``handle(text: str, source_file: str) -> HandleResult``
The handlers own their own DB session lifecycle. They emit pubsub
events via the optional ``event_bus`` parameter (the FastAPI endpoint
injects ``app.state.event_bus``; the scheduler passes ``None``).
They never raise on per-segment problems; per-segment issues are
logged and folded into the result. Whole-document failures
(missing ISA, bad encoding) surface as ``CycloneParseError``, which
the caller catches and records as ``STATUS_ERROR``.
"""
from __future__ import annotations
import importlib
import logging
from ._ack_id import (
ack_count_summary,
ack_synthetic_source_batch_id,
two77ca_synthetic_source_batch_id,
)
from .handle_result import HandleResult
log = logging.getLogger(__name__)
# Module-level HANDLERS dict populated lazily once handler modules
# ship. Keys are file_type strings, values are the ``handle``
# callable for that type.
HANDLERS: dict[str, object] = {}
# Candidate handlers, in registration order. Each tuple is
# (module-path, file_type). The 277/277CA mapping is added explicitly
# after registration when the 277CA handler is present.
_CANDIDATES: list[tuple[str, str]] = [
("cyclone.handlers.handle_999", "999"),
("cyclone.handlers.handle_ta1", "TA1"),
("cyclone.handlers.handle_277ca", "277CA"),
("cyclone.handlers.handle_835", "835"),
]
def register_handlers() -> None:
"""Populate ``HANDLERS`` from the per-type handler modules.
Tolerates missing or broken handler modules so the package can be
imported incrementally as each handler ships (Tasks 2-5), and so
a partial-modification error in one handler module can't break
scheduler / API import. Safe to call multiple times.
"""
if HANDLERS:
return
for mod_path, file_type in _CANDIDATES:
try:
mod = importlib.import_module(mod_path)
fn = getattr(mod, "handle")
except Exception as exc: # noqa: BLE001 — best-effort registry
log.debug(
"handler %s unavailable: %s",
mod_path, exc,
)
continue
HANDLERS[file_type] = fn
# The 277 filename maps to the same 277CA handler.
if file_type == "277CA":
HANDLERS["277"] = fn
register_handlers()
# Re-export handler functions on this package so callers can use the
# flat import (``from cyclone.handlers import handle_999``) once each
# module ships. Set after registration so we know what's present.
def _reexport_handlers() -> None:
"""Re-export each handler's ``handle`` fn as ``handle_<type>``.
No-op for absent handlers. Re-run on each import so freshly-
installed handler modules (e.g. Tasks 2-5 commits) are visible
after ``register_handlers()`` without a process restart.
"""
for file_type, fn in list(HANDLERS.items()):
if file_type in ("277",): # alias of 277CA; don't re-export twice
continue
globals()[f"handle_{file_type.lower()}"] = fn
_reexport_handlers()
__all__ = [
"HANDLERS",
"HandleResult",
"register_handlers",
"ack_count_summary",
"ack_synthetic_source_batch_id",
"two77ca_synthetic_source_batch_id",
# handler_* names are added at module load via _reexport_handlers
]
# Populate __all__ with the present handler symbols.
for _h in ("handle_999", "handle_ta1", "handle_277ca", "handle_835"):
if _h in globals():
__all__.append(_h) # noqa: PYI056
backend/src/cyclone/handlers/_ack_id.py
+87
View File
@@ -0,0 +1,87 @@
"""Ack ID helpers shared between the scheduler and the FastAPI
endpoints (SP27 Task 1).
Lifted from ``scheduler.py:_ack_count_summary`` +
``_ack_synthetic_source_batch_id`` + ``_277ca_synthetic_source_batch_id`` —
all three had inline copies in both ``scheduler.py`` and ``api.py``
prior to this SP. Both callers now import from this module.
Helpers
-------
``ack_count_summary(result)``
Aggregate ``(received, accepted, rejected, ack_code)`` from a
parsed ``ParseResult999``, trusting set-level IK5 over the
functional-group AK9. Gainwell's MFT ships AK9 segments that
contradict the per-set IK5 (e.g. ``AK9*A*1*1*1`` with
``IK5*A``), so trusting AK9's rejected count over-reports.
See scheduler commit ``6507a8c`` for the operational context.
``ack_synthetic_source_batch_id(icn, *, pcn, source_filename)``
Build a unique-per-file ``batches.id`` for a 999 that ships
without its own source batch. Falls back to a hash suffix of
the filename so daily pulls don't all collapse onto the same
ICN. Gainwell's MFT ships every 999 with the default ICN
``000000001`` (per the scheduler docstring).
``two77ca_synthetic_source_batch_id(icn)``
Same idea for a 277CA without its own source batch.
"""
from __future__ import annotations
import hashlib
from typing import Any
def ack_count_summary(result: Any) -> tuple[int, int, int, str]:
"""Aggregate ``(received, accepted, rejected, ack_code)`` from
a ``ParseResult999``.
Counts are derived from the set-level ``IK5`` responses
(one per ``AK2`` in the 999), not the functional-group ``AK9``.
Gainwell's MFT ships contradictory AK9 segments; the per-set
IK5 is the authoritative per-claim accept/reject signal.
"""
sets = result.set_responses
received = len(sets)
accepted = sum(1 for s in sets if s.set_accept_reject.code == "A")
rejected = received - accepted
if rejected == 0:
code = "A"
elif accepted == 0:
code = "R"
else:
code = "P"
return (received, accepted, rejected, code)
def ack_synthetic_source_batch_id(
interchange_control_number: str,
*,
pcn: str | None = None,
source_filename: str | None = None,
) -> str:
"""Synthetic ``batches.id`` for a received 999 with no source batch.
Precedence for the human-readable part of the id (column is
``VARCHAR(32)``):
1. ``999-{pcn}-{hash8}`` if ``AK2`` set_control_number is
present (the common case). 4 + 9 + 1 + 8 = 22 chars max.
2. ``999-{icn}-{hash8}`` if no AK2 (envelope-only 999).
3. ``999-{hash12}`` if no filename either (shouldn't happen
in production).
"""
short_hash = ""
if source_filename:
short_hash = hashlib.sha1(source_filename.encode("utf-8")).hexdigest()[:8]
if pcn and pcn.strip():
return f"999-{pcn.strip()}-{short_hash}"
icn = (interchange_control_number or "").strip() or "000000001"
if short_hash:
return f"999-{icn}-{short_hash}"
return f"999-{short_hash or icn}"
def two77ca_synthetic_source_batch_id(interchange_control_number: str) -> str:
"""Synthetic ``batches.id`` for a received 277CA with no source batch."""
return f"277CA-{(interchange_control_number or '').strip() or '000000001'}"
backend/src/cyclone/handlers/handle_277ca.py
+122
View File
@@ -0,0 +1,122 @@
"""Handle a 277CA Claim Acknowledgment file (SP27 Task 4).
Lifted verbatim from ``scheduler.py:_handle_277ca``. The handler owns
its own DB session, dispatches to ``parse_277ca_text``, persists the
277CA ack row, applies 277CA rejections to matched claims via
``inbox_state.apply_277ca_rejections``, and emits
``claim.payer_rejected`` audit events for each newly-stamped claim.
The actor tag (``"277ca-parser-scheduler"``) is preserved so the
audit log keeps tracing back to the same source after extraction.
Both the FastAPI endpoint and the scheduler path (re)use this module
— the API migration drops the inline copy in Task 6.
``claim.rejected_after_remit`` audit emission (when a 277CA rejects
a claim that already has ``matched_remittance_id`` set) is deferred
to SP27 Task 13. Today the handler only emits ``claim.payer_rejected``.
"""
from __future__ import annotations
import json
import logging
from typing import Optional
from cyclone import db
from cyclone.audit_log import AuditEvent, append_event
from cyclone.handlers._ack_id import two77ca_synthetic_source_batch_id
from cyclone.inbox_state_277ca import apply_277ca_rejections
from cyclone.parsers.exceptions import CycloneParseError
from cyclone.parsers.parse_277ca import parse_277ca_text
from cyclone.store import store as cycl_store
log = logging.getLogger(__name__)
def handle(
text: str,
source_file: str,
*,
event_bus: Optional[object] = None,
) -> tuple[str, int]:
"""Parse a 277CA, persist ack + stamp payer-rejected claims.
Args:
text: Raw 277CA document bytes (decoded).
source_file: Filename the 277CA came from.
event_bus: Optional pubsub handle. Reserved for the
FastAPI-endpoint migration in Task 6 — the scheduler
doesn't pass one.
Returns:
``(parser_used, claim_count)`` tuple where ``claim_count`` is
the number of STC statuses in the file (one per claim).
Raises:
ValueError: on parser-level failure (wraps CycloneParseError).
"""
try:
result = parse_277ca_text(text, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"277CA parse error: {exc}") from exc
icn = result.envelope.control_number
synthetic_id = two77ca_synthetic_source_batch_id(icn)
accepted = sum(
1 for s in result.claim_statuses if s.classification == "accepted"
)
paid = sum(
1 for s in result.claim_statuses if s.classification == "paid"
)
rejected = sum(
1 for s in result.claim_statuses if s.classification == "rejected"
)
pended = sum(
1 for s in result.claim_statuses if s.classification == "pended"
)
with db.SessionLocal()() as session:
row = cycl_store.add_277ca_ack(
source_batch_id=synthetic_id,
control_number=icn,
accepted_count=accepted,
rejected_count=rejected,
paid_count=paid,
pended_count=pended,
raw_json=json.loads(result.model_dump_json()),
)
def _lookup(pcn: str):
return (
session.query(db.Claim)
.filter(db.Claim.patient_control_number == pcn)
.first()
)
apply_result = apply_277ca_rejections(
session, result, claim_lookup=_lookup, two77ca_id=row.id,
)
if apply_result.matched:
for cid in apply_result.matched:
append_event(session, AuditEvent(
event_type="claim.payer_rejected",
entity_type="claim",
entity_id=cid,
payload={"source_batch_id": synthetic_id, "277ca_id": row.id},
actor="277ca-parser-scheduler",
))
session.commit()
# TODO(sp27-task-6): bridge async EventBus.publish → sync caller
# (same gap as handle_999 + handle_ta1; Task 6 will fix when the
# FastAPI endpoints migrate to call these handlers).
if event_bus is not None:
publish = getattr(event_bus, "publish", None)
if callable(publish):
try:
publish("ack_received", {
"source_batch_id": synthetic_id,
"ack_code": "277CA",
"kind": "277CA",
})
except Exception as exc: # noqa: BLE001
log.warning("event_bus publish failed: %s", exc)
return ("parse_277ca", len(result.claim_statuses))
backend/src/cyclone/handlers/handle_835.py
+121
View File
@@ -0,0 +1,121 @@
"""Handle an 835 ERA Remittance file (SP27 Task 5).
Lifted verbatim from ``scheduler.py:_handle_835``. The handler owns
its own state, dispatches to ``parse_835`` (raises
``CycloneParseError`` on bad EDI), runs the per-payer 835 validator,
stamps the validation report into ``result.summary``, and persists
the BatchRecord via ``cycl_store.add``.
The 835 handler is the largest of the four because the schema
covers per-claim remittances + CAS adjustments + validation.
``PAYER_FACTORIES_835`` lives in ``cyclone.api`` (it was always
intended to live in ``cyclone.payers`` — a TODO pre-dating SP27 —
but moving it is out of Task 5 scope). We import it lazily inside
``handle`` to avoid a module-load-time cycle; the cyclic risk is
acceptable because api.py also imports scheduler lazily (inside its
lifespan handler).
Two-phase ingest closed in SP27 Task 10: the placeholder
``adjustment_amount`` is overwritten by reconcile in the same DB
session as the insert (before commit), so a reader never sees a
half-reconciled Remittance row. If reconcile raises, the whole
ingest rolls back and the scheduler records the per-file error.
``event_bus`` is best-effort and follows the same TODO(sp27-task-6)
sync/async gap as handle_999 / handle_ta1 / handle_277ca.
"""
from __future__ import annotations
import json
import logging
import uuid
from datetime import datetime, timezone
from typing import Optional
from cyclone.parsers.exceptions import CycloneParseError
from cyclone.parsers.parse_835 import parse as parse_835
from cyclone.parsers.validator_835 import validate as validate_835
from cyclone.store import BatchRecord
from cyclone.store import store as cycl_store
log = logging.getLogger(__name__)
def handle(
text: str,
source_file: str,
*,
event_bus: Optional[object] = None,
) -> tuple[str, int]:
"""Parse an 835, run validation, persist batch + remittances.
Args:
text: Raw 835 document bytes (decoded).
source_file: Filename the 835 came from.
event_bus: Optional pubsub handle. Reserved for the
FastAPI-endpoint migration in Task 6 — the scheduler
doesn't pass one.
Returns:
``(parser_used, claim_count)`` tuple. ``claim_count`` is the
number of CLP claims parsed; the BatchRecord's summary's
``passed`` / ``failed`` fields are derived from the
validator's ``report.passed`` flag.
Raises:
ValueError: on parser-level failure (wraps CycloneParseError).
Validation failures don't raise — they're stamped into
``summary.passed = 0``.
"""
# TODO(sp27-pre-t5): move PAYER_FACTORIES_835 out of api.py into
# ``cyclone.payers`` to remove the lazy cyclic import below. The
# import works today because api.py also imports scheduler lazily.
from cyclone.api import PAYER_FACTORIES_835
config = PAYER_FACTORIES_835["co_medicaid_835"]()
try:
result = parse_835(text, config, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"835 parse error: {exc}") from exc
# Validation report (mirrors the API endpoint).
report = validate_835(result, config)
n = len(result.claims)
if report.passed:
passed, failed, failed_claim_ids = n, 0, []
else:
passed, failed, failed_claim_ids = 0, n, [
c.payer_claim_control_number for c in result.claims
]
result = result.model_copy(update={
"validation": report,
"summary": result.summary.model_copy(update={
"passed": passed,
"failed": failed,
"failed_claim_ids": failed_claim_ids,
}),
})
rec = BatchRecord(
id=uuid.uuid4().hex,
kind="835",
input_filename=source_file,
parsed_at=datetime.now(timezone.utc),
result=result,
)
cycl_store.add(rec)
# TODO(sp27-task-6): bridge async EventBus.publish → sync caller.
if event_bus is not None:
publish = getattr(event_bus, "publish", None)
if callable(publish):
try:
publish("ack_received", {
"source_batch_id": rec.id,
"ack_code": "835",
"kind": "835",
})
except Exception as exc: # noqa: BLE001
log.warning("event_bus publish failed: %s", exc)
return ("parse_835", len(result.claims))
backend/src/cyclone/handlers/handle_999.py
+129
View File
@@ -0,0 +1,129 @@
"""Handle a 999 Implementation Acknowledgment file (SP27 Task 2).
Lifted verbatim from ``scheduler.py:_handle_999``. The handler owns
its own DB session, dispatches to ``parse_999_text``, applies 999
rejections to any matched claims via ``inbox_state.apply_999_rejections``,
persists the ack row, and returns a ``(parser_used, claim_count)``
tuple.
The actor tag (``"999-parser-scheduler"``) is preserved so the audit
log keeps tracing back to the same source after extraction. Both
the FastAPI endpoint and the scheduler path (re)use this module —
see Task 6 for the API migration that drops the inline copy.
``claim_count`` mirrors ``parsed.received`` — the count of AK2
(set-level) responses in the 999, which is the authoritative
per-claim accept/reject signal (see ``_ack_id`` for why AK9 is
ignored).
"""
from __future__ import annotations
import json
import logging
from typing import Optional
from cyclone import db
from cyclone.audit_log import AuditEvent, append_event
from cyclone.handlers._ack_id import (
ack_count_summary,
ack_synthetic_source_batch_id,
)
from cyclone.inbox_state import apply_999_rejections
from cyclone.parsers.exceptions import CycloneParseError
from cyclone.parsers.parse_999 import parse_999_text
from cyclone.store import store as cycl_store
log = logging.getLogger(__name__)
def handle(
text: str,
source_file: str,
*,
event_bus: Optional[object] = None,
) -> tuple[str, int]:
"""Parse a 999, apply rejections, persist ack row.
Args:
text: Raw 999 document bytes (decoded).
source_file: Filename the 999 came from. Used to derive a
unique synthetic ``batches.id`` (see ``_ack_id``).
event_bus: Optional pubsub handle. When provided, an
``ack_received`` event is best-effort published. The
scheduler passes ``None``; the FastAPI endpoint will
be migrated to pass ``app.state.event_bus`` in Task 6.
Returns:
``(parser_used, claim_count)`` tuple. Matches the scheduler
``_download_and_parse`` destructure; the HandleResult
migration happens in Task 7.
Raises:
ValueError: on parser-level failure (wraps CycloneParseError).
"""
try:
result = parse_999_text(text, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"999 parse error: {exc}") from exc
received, accepted, rejected, ack_code = ack_count_summary(result)
icn = result.envelope.control_number
pcn = (
result.set_responses[0].set_control_number
if result.set_responses else None
)
synthetic_id = ack_synthetic_source_batch_id(
icn, pcn=pcn, source_filename=source_file,
)
with db.SessionLocal()() as session:
def _lookup(pcn: str):
return (
session.query(db.Claim)
.filter_by(patient_control_number=pcn)
.first()
)
rejection_result = apply_999_rejections(
session, result, claim_lookup=_lookup,
)
if rejection_result.matched:
for cid in rejection_result.matched:
append_event(session, AuditEvent(
event_type="claim.rejected",
entity_type="claim",
entity_id=cid,
payload={"source_batch_id": synthetic_id},
actor="999-parser-scheduler",
))
row = cycl_store.add_ack(
source_batch_id=synthetic_id,
accepted_count=accepted,
rejected_count=rejected,
received_count=received,
ack_code=ack_code,
raw_json=json.loads(result.model_dump_json()),
)
session.commit()
# Best-effort pubsub publish. The scheduler doesn't pass an
# event_bus; the API migration in Task 6 will pass the FastAPI
# EventBus (which has an async ``publish``). This handler is
# sync, so an async-real EventBus returns an unawaited coroutine
# here — that's a known gap until Task 6 wires the publish
# through ``asyncio.run_coroutine_threadsafe`` or makes
# ``handle`` async. See TODO(sp27-task-6) below.
if event_bus is not None:
publish = getattr(event_bus, "publish", None)
if callable(publish):
# TODO(sp27-task-6): bridge async EventBus.publish → sync
# caller (see comment above).
try:
publish("ack_received", {
"source_batch_id": synthetic_id,
"ack_code": ack_code,
"kind": "999",
})
except Exception as exc: # noqa: BLE001
log.warning("event_bus publish failed: %s", exc)
return ("parse_999", received)
backend/src/cyclone/handlers/handle_result.py
+37
View File
@@ -0,0 +1,37 @@
"""Shared ``HandleResult`` dataclass for handlers (SP27).
Every per-file-type handler returns the same shape so the
scheduler's ``_handle_one`` and the FastAPI parse endpoints can
process them uniformly.
"""
from __future__ import annotations
from dataclasses import dataclass
@dataclass
class HandleResult:
"""Outcome of one handler invocation.
Attributes:
parser_used: The parser name written to
``processed_inbound_files.parser_used`` (e.g. ``"parse_999"``)
and surfaced in the UI.
claim_count: The number of claim rows persisted (or batch
records, depending on the file type). For 999/TA1 this
is the receipt count; for 835 this is the per-claim
remittance count; for 277CA this is the per-claim
status count.
batch_id: The persisted batch id (when the handler creates a
row in ``batches``). ``None`` for handlers that persist
into per-file-type ack tables (999/TA1/277CA) rather
than into the unified ``batches`` table.
matched_count: For 835, the number of remits that were
matched to an existing claim by ``reconcile.match``.
Zero for other handlers.
"""
parser_used: str
claim_count: int
batch_id: str | None = None
matched_count: int = 0
backend/src/cyclone/handlers/handle_ta1.py
+89
View File
@@ -0,0 +1,89 @@
"""Handle a TA1 Interchange Acknowledgment file (SP27 Task 3).
Lifted verbatim from ``scheduler.py:_handle_ta1``. The handler owns
its own DB session, dispatches to ``parse_ta1_text``, persists the
interchange ack row in ``ta1_acks``, and returns a
``(parser_used, claim_count)`` tuple.
Unlike the 999 handler, TA1 is envelope-only — there is one TA1 per
ISA/IEA interchange, no set-level (AK2) or claim-level matching.
The claim_count is always 1 (one TA1 ack row per file).
The actor tag is implicit (no audit event here — TA1 doesn't tag
anything in the activity log; it's an infrastructure-level ack).
"""
from __future__ import annotations
import json
import logging
from typing import Optional
from cyclone import db
from cyclone.parsers.exceptions import CycloneParseError
from cyclone.parsers.parse_ta1 import parse_ta1_text
from cyclone.store import store as cycl_store
log = logging.getLogger(__name__)
def handle(
text: str,
source_file: str,
*,
event_bus: Optional[object] = None,
) -> tuple[str, int]:
"""Parse a TA1, persist the interchange ack row.
Args:
text: Raw TA1 document bytes (decoded).
source_file: Filename the TA1 came from. Used for audit
attribution; the ``batches.id`` for TA1 rows is derived
internally from the parsed envelope's control number
(``TA1-{ICN}``).
event_bus: Optional pubsub handle. Reserved for the
FastAPI-endpoint migration in Task 6 — the scheduler
doesn't pass one.
Returns:
``(parser_used, claim_count)`` tuple. TA1 always returns
``claim_count=1`` (one TA1 ack row per interchange file).
Raises:
ValueError: on parser-level failure (wraps CycloneParseError).
"""
try:
result = parse_ta1_text(text, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"TA1 parse error: {exc}") from exc
with db.SessionLocal()() as session:
cycl_store.add_ta1_ack(
source_batch_id=result.source_batch_id,
control_number=result.ta1.control_number,
interchange_date=result.ta1.interchange_date,
interchange_time=result.ta1.interchange_time,
ack_code=result.ta1.ack_code,
note_code=result.ta1.note_code,
ack_generated_date=result.ta1.ack_generated_date,
sender_id=result.envelope.sender_id,
receiver_id=result.envelope.receiver_id,
raw_json=json.loads(result.model_dump_json()),
)
session.commit()
# TODO(sp27-task-6): bridge async EventBus.publish → sync caller
# (same gap as handle_999; both fixed when the FastAPI endpoints
# migrate to call these handlers instead of inline code).
if event_bus is not None:
publish = getattr(event_bus, "publish", None)
if callable(publish):
try:
publish("ack_received", {
"source_batch_id": result.source_batch_id,
"ack_code": result.ta1.ack_code,
"kind": "TA1",
})
except Exception as exc: # noqa: BLE001
log.warning("event_bus publish failed: %s", exc)
return ("parse_ta1", 1)
backend/src/cyclone/migrations/0016_claims_matched_remittance_id_index.sql
+15
View File
@@ -0,0 +1,15 @@
-- version: 16
-- Add the missing index on claims.matched_remittance_id.
--
-- SP27 Task 11 added ``check_matched_pair_drift`` at startup, which
-- scans ``WHERE Claim.matched_remittance_id IS NOT NULL``. Without an
-- index this is a full-table scan; becomes a noticeable boot
-- latency cost past ~10k claims. The companion index on the
-- reverse side (``remittances.claim_id``) was added in 0007.
--
-- A plain index is enough — neither side is unique (reversals
-- re-reference the original PCN, and a claim without a match is
-- fine).
CREATE INDEX ix_claims_matched_remittance_id
ON claims(matched_remittance_id);
backend/src/cyclone/migrations/0017_backfill_claim_patient_control_number.sql
+28
View File
@@ -0,0 +1,28 @@
-- version: 17
-- Backfill claims.patient_control_number = claims.id.
--
-- SP27 Task 17 fixed the 837 ingest in store.py:_claim_837_row so
-- ``Claim.patient_control_number`` is populated from
-- ``claim.claim_id`` (CLM01) instead of ``claim.subscriber.member_id``
-- (the 2010BA NM109). The reconcile matcher joins on
-- ``Claim.patient_control_number == Remittance.payer_claim_control_number``
-- and the 835 echoes CLM01 in CLP01 per X12 spec, so the wrong field
-- silently broke every auto-match in production.
--
-- For rows written BEFORE the fix, the stored value is the member_id
-- (e.g. "W953474") which never matches any remit's CLP01. This
-- migration backfills those rows by aligning
-- ``patient_control_number`` with the row's own ``id`` (= CLM01).
-- After this, every claim row's PCN is consistent with the value the
-- 837 actually sent, and any newly-ingested 835 whose CLP01 echoes
-- that CLM01 will auto-pair.
--
-- Idempotent: only touches rows where the stored PCN doesn't already
-- match the row's id, so re-running on already-fixed rows is a no-op.
--
-- Reversal safety: the migration does NOT clear matched_remittance_id,
-- so any pre-existing manual_match pairs stay intact.
UPDATE claims
SET patient_control_number = id
WHERE patient_control_number IS DISTINCT FROM id;
backend/src/cyclone/parsers/parse_999.py
+19 -2
View File
@@ -8,6 +8,12 @@ Single-pass walker over the tokenized segment list:
- AK3 (Segment Context) + AK4 (Element Context) — optional per-segment errors
- AK5 (Transaction Set Response Status) — per-set accept/reject
- AK9 (Functional Group Response Status) — per-group counts + ack code
- IK5 — a non-standard synonym for ``AK5`` that Gainwell's MFT ships
in place of the spec-defined ``AK5``. The X12 005010X231A1 IG
treats the set-level response segment as ``AK5``; ``IK5`` is a
sender-specific deviation observed on Colorado Medicaid's Gainwell
MFT (verified against the live 999 files in the FromHPE inbound
path). We accept either.
- SE / GE / IEA
Errors at the file level raise :class:`CycloneParseError`. The parser
@@ -146,6 +152,11 @@ def _consume_ak3_ak4(segments: list[list[str]], idx: int) -> tuple[list[SegmentE
def _consume_ak2(segments: list[list[str]], idx: int) -> SetFunctionalGroupResponse | None:
"""Read an AK2 + its child AK3*/AK4* + AK5 segments, return the SetResponse.
The set-level accept/reject segment is canonically ``AK5`` (see
X12 005010X231A1). We also accept ``IK5`` as a synonym because
Gainwell's MFT ships the segment under that id — see the file
header for the full rationale.
Returns None when called with a non-AK2 segment (defensive — the
orchestrator only calls this when it sees AK2).
"""
@@ -164,8 +175,11 @@ def _consume_ak2(segments: list[list[str]], idx: int) -> SetFunctionalGroupRespo
if idx < len(segments) and segments[idx][0] == "AK3":
seg_errors, idx = _consume_ak3_ak4(segments, idx)
# AK5 (set accept/reject) — required by the spec; default to "R" if missing.
# Gainwell's MFT uses IK5 instead of AK5 (sender-specific segment id
# that means the same thing); accept either. The default of "R"
# matters: if the segment is missing entirely, the 999 is a reject.
accept_code = "R"
if idx < len(segments) and segments[idx][0] == "AK5":
if idx < len(segments) and segments[idx][0] in ("AK5", "IK5"):
ak5 = segments[idx]
if len(ak5) > 1 and ak5[1]:
accept_code = ak5[1]
@@ -256,8 +270,11 @@ def parse_999_text(text: str, *, input_file: str = "") -> ParseResult999:
set_responses.append(sr)
# Advance past the AK2 + AK3*/AK4*/AK5 cluster
# (re-walk from i+1 because _consume_ak2 doesn't return idx).
# ``IK5`` is the Gainwell-specific synonym for ``AK5``
# and must be in the consumed set here too (see
# _consume_ak2 for the full rationale).
i += 1
while i < len(segments) and segments[i][0] in {"AK3", "AK4", "AK5"}:
while i < len(segments) and segments[i][0] in {"AK3", "AK4", "AK5", "IK5"}:
i += 1
else:
i += 1
backend/src/cyclone/reconcile.py
+9 -4
View File
@@ -224,10 +224,15 @@ class ReconcileResult:
def run(session, batch_id: str) -> ReconcileResult:
"""Orchestrate reconciliation for an 835 batch.
Expects Batch + Remittance + CasAdjustment rows already persisted
(this is called from store.add() AFTER commit). Loads the new
remittances + all currently-unmatched Claims, calls match(), then
applies each match's intent inside the caller's session.
Expects Batch + Remittance + CasAdjustment rows already added to
``session`` (not yet committed). SP27 Task 10 calls this from
inside ``CycloneStore.add``'s ingest session, BEFORE
``s.commit()`` — so the whole 835 ingest (rows + reconcile) is
a single transaction. If anything here raises, the ingest rolls
back; no half-reconciled batch ever lands.
Loads the new remittances + all currently-unmatched Claims, calls
match(), then applies each match's intent inside the caller's session.
Returns counts. Does NOT commit; caller controls transaction.
Raises on failure (caller catches and writes activity event).
backend/src/cyclone/scheduler.py
+186 -228
View File
@@ -55,7 +55,10 @@ from cyclone.audit_log import AuditEvent, append_event
from cyclone.clearhouse import InboundFile, SftpClient
from cyclone.db import ProcessedInboundFile
from cyclone.edi.filenames import parse_inbound_filename
from cyclone.inbox_state import apply_999_rejections
from cyclone.handlers import handle_277ca as _handle_277ca
from cyclone.handlers import handle_835 as _handle_835
from cyclone.handlers import handle_999 as _handle_999
from cyclone.handlers import handle_ta1 as _handle_ta1
from cyclone.inbox_state_277ca import apply_277ca_rejections
from cyclone.providers import SftpBlock
@@ -92,6 +95,12 @@ class TickResult:
files_skipped: int = 0
files_errored: int = 0
errors: list[str] = field(default_factory=list)
# SP27 Task 9: ``sftp_failed`` is the discriminator that separates
# SFTP-side failures (the listing step) from per-file processing
# errors. ``tick()`` keys off this flag — NOT ``result.errors`` —
# so a single malformed 999 cannot flip the operator's destructive
# pill. Set in ``_tick_impl``'s listing-error branches only.
sftp_failed: bool = False
def as_dict(self) -> dict[str, Any]:
return {
@@ -104,12 +113,22 @@ class TickResult:
"files_skipped": self.files_skipped,
"files_errored": self.files_errored,
"errors": list(self.errors),
"sftp_failed": self.sftp_failed,
}
@dataclass
class SchedulerStatus:
"""Snapshot of the scheduler's runtime state."""
"""Snapshot of the scheduler's runtime state.
SP27 Task 9 added the four SFTP-error fields
(``consecutive_failures``, ``last_error``, ``last_error_at``,
``last_sftp_attempt_at``) so the operator's UI can tell "all
quiet on the MFT front" from "we've been unable to reach the
MFT server for 3 hours". The previous 06/25 incident went
unnoticed because the status dict had no signal for a hung
SFTP poll.
"""
running: bool
poll_interval_seconds: int
@@ -120,6 +139,13 @@ class SchedulerStatus:
total_skipped: int
total_errored: int
last_tick: Optional[TickResult] = None
# SP27 Task 9: SFTP-error surfacing. The UI flips to a destructive
# pill when ``consecutive_failures >= 3`` (``CYCLONE_SCHEDULER_FAIL_THRESHOLD``
# in the operator pill config; not enforced server-side).
consecutive_failures: int = 0
last_error: Optional[str] = None
last_error_at: Optional[datetime] = None
last_sftp_attempt_at: Optional[datetime] = None
def as_dict(self) -> dict[str, Any]:
return {
@@ -134,6 +160,15 @@ class SchedulerStatus:
"total_skipped": self.total_skipped,
"total_errored": self.total_errored,
"last_tick": self.last_tick.as_dict() if self.last_tick else None,
"consecutive_failures": self.consecutive_failures,
"last_error": self.last_error,
"last_error_at": (
self.last_error_at.isoformat() if self.last_error_at else None
),
"last_sftp_attempt_at": (
self.last_sftp_attempt_at.isoformat()
if self.last_sftp_attempt_at else None
),
}
@@ -141,179 +176,17 @@ class SchedulerStatus:
# Per-file-type handlers. Each returns (parser_name, claim_count) and
# persists its own DB rows. The scheduler records the outcome.
# ---------------------------------------------------------------------------
#
# 999 (Task 2), TA1 (Task 3), 277CA (Task 4), and 835 (Task 5) now
# live in ``cyclone.handlers``. The HANDLERS dict literal below
# references the alias imports so the wiring stays identical.
def _handle_999(text: str, source_file: str) -> tuple[str, int]:
"""Parse a 999, apply rejections, persist ack row. Returns (parser, count)."""
from cyclone.parsers.parse_999 import parse_999_text
from cyclone.parsers.exceptions import CycloneParseError
try:
result = parse_999_text(text, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"999 parse error: {exc}") from exc
received, accepted, rejected, ack_code = _ack_count_summary(result)
icn = result.envelope.control_number
synthetic_id = _ack_synthetic_source_batch_id(icn)
with db.SessionLocal()() as session:
def _lookup(pcn: str):
return (
session.query(db.Claim)
.filter_by(patient_control_number=pcn)
.first()
)
rejection_result = apply_999_rejections(
session, result, claim_lookup=_lookup,
)
if rejection_result.matched:
for cid in rejection_result.matched:
append_event(session, AuditEvent(
event_type="claim.rejected",
entity_type="claim",
entity_id=cid,
payload={"source_batch_id": synthetic_id},
actor="999-parser-scheduler",
))
row = cycl_store.add_ack(
source_batch_id=synthetic_id,
accepted_count=accepted,
rejected_count=rejected,
received_count=received,
ack_code=ack_code,
raw_json=json.loads(result.model_dump_json()),
)
session.commit()
return "parse_999", received
def _handle_835(text: str, source_file: str) -> tuple[str, int]:
"""Parse an 835, run validation, persist batch + remittances."""
import uuid
from cyclone.parsers.parse_835 import parse as parse_835
from cyclone.parsers.exceptions import CycloneParseError
from cyclone.parsers.validator_835 import validate as validate_835
from cyclone.payers import PAYER_FACTORIES_835
from cyclone.store import BatchRecord
config = PAYER_FACTORIES_835["co_medicaid_835"]()
try:
result = parse_835(text, config, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"835 parse error: {exc}") from exc
# Validation report (mirrors the API endpoint).
report = validate_835(result, config)
n = len(result.claims)
if report.passed:
passed, failed, failed_claim_ids = n, 0, []
else:
passed, failed, failed_claim_ids = 0, n, [
c.payer_claim_control_number for c in result.claims
]
result = result.model_copy(update={
"validation": report,
"summary": result.summary.model_copy(update={
"passed": passed,
"failed": failed,
"failed_claim_ids": failed_claim_ids,
}),
})
rec = BatchRecord(
id=uuid.uuid4().hex,
kind="835",
input_filename=source_file,
parsed_at=datetime.now(timezone.utc),
result=result,
)
cycl_store.add(rec)
return "parse_835", len(result.claims)
def _handle_277ca(text: str, source_file: str) -> tuple[str, int]:
"""Parse a 277CA, persist ack + stamp payer-rejected claims."""
from cyclone.parsers.parse_277ca import parse_277ca_text
from cyclone.parsers.exceptions import CycloneParseError
try:
result = parse_277ca_text(text, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"277CA parse error: {exc}") from exc
icn = result.envelope.control_number
synthetic_id = _277ca_synthetic_source_batch_id(icn)
accepted = sum(
1 for s in result.claim_statuses if s.classification == "accepted"
)
paid = sum(
1 for s in result.claim_statuses if s.classification == "paid"
)
rejected = sum(
1 for s in result.claim_statuses if s.classification == "rejected"
)
pended = sum(
1 for s in result.claim_statuses if s.classification == "pended"
)
with db.SessionLocal()() as session:
row = cycl_store.add_277ca_ack(
source_batch_id=synthetic_id,
control_number=icn,
accepted_count=accepted,
rejected_count=rejected,
paid_count=paid,
pended_count=pended,
raw_json=json.loads(result.model_dump_json()),
)
def _lookup(pcn: str):
return (
session.query(db.Claim)
.filter(db.Claim.patient_control_number == pcn)
.first()
)
apply_result = apply_277ca_rejections(
session, result, claim_lookup=_lookup, two77ca_id=row.id,
)
if apply_result.matched:
for cid in apply_result.matched:
append_event(session, AuditEvent(
event_type="claim.payer_rejected",
entity_type="claim",
entity_id=cid,
payload={"source_batch_id": synthetic_id, "277ca_id": row.id},
actor="277ca-parser-scheduler",
))
session.commit()
return "parse_277ca", len(result.claim_statuses)
def _handle_ta1(text: str, source_file: str) -> tuple[str, int]:
"""Parse a TA1, persist the interchange ack row."""
from cyclone.parsers.parse_ta1 import parse_ta1_text
from cyclone.parsers.exceptions import CycloneParseError
try:
result = parse_ta1_text(text, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"TA1 parse error: {exc}") from exc
with db.SessionLocal()() as session:
cycl_store.add_ta1_ack(
source_batch_id=result.source_batch_id,
control_number=result.ta1.control_number,
interchange_date=result.ta1.interchange_date,
interchange_time=result.ta1.interchange_time,
ack_code=result.ta1.ack_code,
note_code=result.ta1.note_code,
ack_generated_date=result.ta1.ack_generated_date,
sender_id=result.envelope.sender_id,
receiver_id=result.envelope.receiver_id,
raw_json=json.loads(result.model_dump_json()),
)
session.commit()
return "parse_ta1", 1
# The 277CA handler has moved to ``cyclone.handlers.handle_277ca``
# (SP27 Task 4); the inline def was deleted. The HANDLERS dict literal
# below still references ``_handle_277ca`` because the import-alias
# line at the top of this module binds that name to the new function.
# Only the 835 handler stays inline (Task 5).
# Map file_type → handler. Mirrors ROUTED_FILE_TYPES.
@@ -328,43 +201,15 @@ HANDLERS: dict[str, Callable[[str, str], tuple[str, int]]] = {
# ---------------------------------------------------------------------------
# Light copies of helpers the API endpoints use, so the scheduler can
# run without depending on the FastAPI module.
# ---------------------------------------------------------------------------
def _ack_count_summary(result: Any) -> tuple[int, int, int, str]:
"""Return (received, accepted, rejected, ack_code) for a 999.
Mirrors the logic in ``cyclone.api._ack_count_summary`` but lives
here so the scheduler can run without importing the API module.
"""
if result.functional_group_acks:
fg = result.functional_group_acks[0]
return (
fg.received_count, fg.accepted_count,
fg.rejected_count, fg.ack_code,
)
sets = result.set_responses
received = len(sets)
accepted = sum(1 for s in sets if s.set_accept_reject.code == "A")
rejected = received - accepted
if rejected == 0:
code = "A"
elif accepted == 0:
code = "R"
else:
code = "P"
return (received, accepted, rejected, code)
def _ack_synthetic_source_batch_id(interchange_control_number: str) -> str:
"""Synthetic batches.id for a received 999 with no source batch."""
return f"999-{(interchange_control_number or '').strip() or '000000001'}"
def _277ca_synthetic_source_batch_id(interchange_control_number: str) -> str:
"""Synthetic batches.id for a received 277CA with no source batch."""
return f"277CA-{(interchange_control_number or '').strip() or '000000001'}"
# Run without depending on the FastAPI module.
#
# Note: ``_277ca_synthetic_source_batch_id`` lived here as a
# scheduler-local duplicate of
# ``cyclone.handlers._ack_id.two77ca_synthetic_source_batch_id`` until
# SP27 Task 4 landed; the inline def has been deleted because
# ``handle_277ca`` now imports the canonical copy. The historical
# helper was the only remaining inline def in this section — the
# surviving inline handler is ``_handle_835`` (lifts in Task 5).
# ---------------------------------------------------------------------------
@@ -418,6 +263,40 @@ class Scheduler:
# ticks stack up; the next tick fires only after the previous
# one finishes).
self._tick_in_progress = False
# SP27 Task 9: SFTP-error surfacing. ``_consecutive_failures``
# counts back-to-back tick failures (cleared on the next
# successful tick). The other three fields are the most
# recent failure for the operator's pill — preserved across
# successes so the audit trail doesn't disappear the moment
# the MFT server recovers.
self._consecutive_failures = 0
self._last_error: Optional[str] = None
self._last_error_at: Optional[datetime] = None
self._last_sftp_attempt_at: Optional[datetime] = None
def _record_sftp_outcome(
self, *, success: bool, error: Optional[str] = None,
) -> None:
"""Update the SFTP-error state after a tick.
``success=True`` resets the consecutive-failure counter; the
last-error fields are preserved as an audit trail (the
operator's UI can still surface "last failure: 2 hours ago"
after a recovery).
``success=False`` bumps the counter and records the error
message + timestamp. ``last_sftp_attempt_at`` is bumped in
both cases so the operator can tell when the scheduler last
*tried* to reach the MFT (not just when it last failed).
"""
now = datetime.now(timezone.utc)
self._last_sftp_attempt_at = now
if success:
self._consecutive_failures = 0
return
self._consecutive_failures += 1
self._last_error = error
self._last_error_at = now
# ---- Public API -------------------------------------------------------
@@ -465,6 +344,10 @@ class Scheduler:
total_skipped=self._total_skipped,
total_errored=self._total_errored,
last_tick=self._last_tick,
consecutive_failures=self._consecutive_failures,
last_error=self._last_error,
last_error_at=self._last_error_at,
last_sftp_attempt_at=self._last_sftp_attempt_at,
)
def is_running(self) -> bool:
@@ -478,6 +361,13 @@ class Scheduler:
SFTP server from a stampede when the operator hits
``/api/admin/scheduler/tick`` while a scheduled tick is
already running.
SP27 Task 9: the tick outcome is also recorded on the
scheduler's SFTP-error state via
:meth:`_record_sftp_outcome`. The discriminator is
``TickResult.sftp_failed`` (set by ``_tick_impl``'s
listing-error branches only), not ``TickResult.errors``
which is also populated by per-file processing errors.
"""
while self._tick_in_progress:
await asyncio.sleep(0.05)
@@ -490,6 +380,63 @@ class Scheduler:
self._total_processed += result.files_processed
self._total_skipped += result.files_skipped
self._total_errored += result.files_errored
# SFTP-error surfacing: discriminator is ``result.sftp_failed``,
# NOT ``result.errors``. ``sftp_failed`` is set by
# ``_tick_impl``'s listing-error branches (timeout, connect
# refused). Per-file processing errors append to
# ``result.errors`` but do NOT set ``sftp_failed`` — a
# single malformed 999 in a 5-file batch must not flip
# the operator's pill to destructive.
if result.sftp_failed:
self._record_sftp_outcome(
success=False,
error="; ".join(result.errors),
)
else:
self._record_sftp_outcome(success=True)
return result
finally:
self._tick_in_progress = False
async def process_inbound_files(
self, files: list[InboundFile],
) -> TickResult:
"""Run the per-file processing pipeline over a pre-fetched list.
Unlike :meth:`tick`, this does **not** call SFTP — the caller
is expected to have already downloaded (or arranged the local
copy of) each ``InboundFile.local_path``. Used by the
``/api/admin/scheduler/pull-inbound`` admin endpoint and the
``cyclone pull-inbound`` CLI command to process a date-filtered
subset of the inbound MFT path without paying the cost of a
full poll.
Honors ``_stop_event`` between files. Concurrent calls are
coalesced the same way :meth:`tick` does, so a slow handler
can't be stampeded by a parallel admin invocation.
SP27 Task 9: deliberately does NOT update the SFTP-error
state (no listing, no SFTP). The consecutive-failure counter
is the signal for scheduled SFTP polling; operator-initiated
batches shouldn't be able to flip it.
"""
while self._tick_in_progress:
await asyncio.sleep(0.05)
self._tick_in_progress = True
try:
started = datetime.now(timezone.utc)
result = TickResult(started_at=started, files_seen=len(files))
for f in files:
if self._stop_event.is_set():
break
await self._handle_one(f, result)
result.finished_at = datetime.now(timezone.utc)
self._last_tick = result
self._last_poll_at = result.finished_at
self._poll_count += 1
self._total_processed += result.files_processed
self._total_skipped += result.files_skipped
self._total_errored += result.files_errored
return result
finally:
self._tick_in_progress = False
@@ -522,11 +469,26 @@ class Scheduler:
started = datetime.now(timezone.utc)
result = TickResult(started_at=started)
# SP27 Task 8: use the async-wrapped SFTP client so a hung
# ``listdir_attr`` (the 06/25 silent-hang failure mode) is
# bounded by ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` instead of
# waiting on paramiko forever. ``asyncio.TimeoutError`` is
# handled explicitly so the tick surfaces a clear error in
# ``Scheduler.status()`` (Task 9) rather than a generic
# ``Exception`` catch-all.
client = self._sftp_client_factory(self._sftp_block)
try:
files = await asyncio.to_thread(self._list_inbound)
files = await client.async_list_inbound()
except asyncio.TimeoutError:
log.exception("SFTP list_inbound timed out")
result.errors.append("list_inbound: timeout")
result.sftp_failed = True
result.finished_at = datetime.now(timezone.utc)
return result
except Exception as exc: # noqa: BLE001
log.exception("SFTP list_inbound failed")
result.errors.append(f"list_inbound: {exc}")
result.sftp_failed = True
result.finished_at = datetime.now(timezone.utc)
return result
@@ -539,11 +501,6 @@ class Scheduler:
result.finished_at = datetime.now(timezone.utc)
return result
def _list_inbound(self) -> list[InboundFile]:
"""Return files in the inbound MFT path. Runs on a thread."""
client = self._sftp_client_factory(self._sftp_block)
return client.list_inbound()
async def _handle_one(self, f: InboundFile, result: TickResult) -> None:
"""Process one inbound file: skip-if-seen, classify, parse, record."""
if await self._already_processed(f.name):
@@ -650,20 +607,21 @@ class Scheduler:
def _download_and_parse(
self, f: InboundFile, file_type: str,
) -> tuple[Path, str, int]:
"""Download from MFT, run the right handler. Returns (path, parser, count).
"""Run the right handler on one inbound file. Returns (path, parser, count).
Stub mode: ``f.local_path`` already points at the staged file
(set by ``SftpClient._list_inbound_stub``). Real mode: the
remote name is ``f.name`` and we round-trip through paramiko.
Both stub and real modes read from ``f.local_path`` — the
inbound file is already on disk:
* Stub mode: ``_list_inbound_stub`` points ``local_path`` at
the operator-dropped staging file.
* Real mode: ``_list_inbound_paramiko`` downloads each
``listdir_attr`` entry into the local cache as part of the
listing pass. Re-reading from the MFT would require
``SftpClient.read_file`` with a full remote path, which the
scheduler was passing just ``f.name`` for (i.e. a bare
filename at the SFTP root, not the inbound dir). Use the
cached bytes instead.
"""
if self._sftp_block.stub:
# In stub mode the InboundFile already has a local_path;
# reading the staged bytes directly avoids the stub's
# remote-path semantics (which expect a full inbound path).
content = f.local_path.read_bytes()
else:
client = self._sftp_client_factory(self._sftp_block)
content = client.read_file(f.name)
content = f.local_path.read_bytes()
text = content.decode("utf-8")
handler = HANDLERS[file_type]
parser_used, claim_count = handler(text, f.name)
backend/src/cyclone/store.py
+525 -38
View File
@@ -191,7 +191,14 @@ def _claim_837_row(claim: ClaimOutput, batch_id: str) -> Claim:
return Claim(
id=claim.claim_id,
batch_id=batch_id,
patient_control_number=claim.subscriber.member_id or "",
# SP27 Task 17: Claim.patient_control_number must hold the CLM01
# claim_submittr's_identifier the 837 sent — that's the value the
# 835 echoes in CLP01, which the reconcile matcher joins on
# (reconcile.py:by_pcn), and what 999 / 277CA ACK lookups also
# use to cross-reference the original claim. Storing
# subscriber.member_id here (the 2010BA NM109) silently broke
# every auto-match in production.
patient_control_number=claim.claim_id or "",
service_date_from=d_from,
service_date_to=d_to,
charge_amount=Decimal(claim.claim.total_charge or 0),
@@ -851,6 +858,15 @@ def _date_in_bounds(
return True
# Effectively-unbounded iter_* limit, used by count_claims /
# count_remittances so they can reuse the iter's filter pipeline
# (incl. the in-memory ``payer`` substring + ``date_from/to`` checks)
# without being silently capped at the iter's default ``limit=100``.
# 2**31 - 1 is the largest signed 32-bit int — far above any realistic
# X12 batch population.
_ITER_UNBOUNDED = 2**31 - 1
# ---------------------------------------------------------------------------
# Backward-compat shim: tests called ``_batches.clear()`` on the in-memory
# store. The DB-backed store doesn't have an in-memory list, so we expose
@@ -878,6 +894,110 @@ class _BatchesShim:
s.commit()
def check_matched_pair_drift() -> int:
"""Audit the ``Claim.matched_remittance_id`` ↔ ``Remittance.claim_id``
FK pair at startup. Non-blocking (SP27 Task 11).
The matched pair is a denormalized FK pair maintained transactionally
by ``manual_match``, ``manual_unmatch``, and ``reconcile.run``. A
pre-existing mismatch (e.g. a row written before a state migration
that added one column but not the other) would otherwise stay
invisible until the next operator pair attempt fails confusingly.
This check logs the count + up to N examples so operators can
investigate without booting the system.
Returns the number of drifted rows (0 means clean). Does not
raise; bootstrap continues even if drift is detected.
Count semantics: this returns *drifted rows*, not *drifted pairs*.
A single broken pair (``Claim.matched_remittance_id = X`` AND
``Remittance.claim_id = Y != nil`` with neither pointing back)
can produce TWO drifted rows — one in case A (the claim) and
one in case B (the remit). In practice drift is almost always
asymmetric (one side NULL), so count == count(pairs); for the
fully-symmetric minority, divide by ~2 when alerting. Real drift
should be fixed by repairing the writer path, not by counting.
Cases:
A. Claim ``matched_remittance_id = X`` but the paired remit X's
``claim_id`` is either NULL or doesn't point back to the claim.
B. Remit ``claim_id = A`` but the paired claim A's
``matched_remittance_id`` is either NULL or doesn't point back.
"""
import logging
from sqlalchemy import select
log = logging.getLogger(__name__)
with db.SessionLocal()() as s:
# Case A: claim says X is paired, but X.claim_id doesn't point back.
case_a = list(
s.execute(
select(
Claim.id.label("claim_id"),
Claim.matched_remittance_id.label("claimed_remit_id"),
Remittance.claim_id.label("remit_points_to"),
)
.outerjoin(
Remittance,
Remittance.id == Claim.matched_remittance_id,
)
.where(Claim.matched_remittance_id.is_not(None))
.where(
(Remittance.claim_id.is_(None))
| (Remittance.claim_id != Claim.id)
)
).all()
)
# Case B: remit says A is paired, but A.matched_remittance_id
# doesn't point back.
case_b = list(
s.execute(
select(
Remittance.id.label("remit_id"),
Remittance.claim_id.label("claimed_claim_id"),
Claim.matched_remittance_id.label("claim_points_to"),
)
.outerjoin(
Claim,
Claim.id == Remittance.claim_id,
)
.where(Remittance.claim_id.is_not(None))
.where(
(Claim.matched_remittance_id.is_(None))
| (Claim.matched_remittance_id != Remittance.id)
)
).all()
)
total = len(case_a) + len(case_b)
if total == 0:
log.info("matched-pair drift check: 0 mismatches (clean)")
return 0
log.warning(
"matched-pair drift check: %d mismatched pair(s) (showing up to 5 "
"of each). Investigate via SELECT against claim / remittance; "
"manual re-pair via /api/claims/{id}/manual-match will repair.",
total,
)
for r in case_a[:5]:
log.warning(
" case A: claim %s -> remit %s, but remit.claim_id=%r",
r.claim_id,
r.claimed_remit_id,
r.remit_points_to,
)
for r in case_b[:5]:
log.warning(
" case B: remit %s -> claim %s, but claim.matched_remittance_id=%r",
r.remit_id,
r.claimed_claim_id,
r.claim_points_to,
)
return total
# ---------------------------------------------------------------------------
# CycloneStore: the SQLAlchemy-backed facade.
# ---------------------------------------------------------------------------
@@ -912,9 +1032,12 @@ class CycloneStore:
For 835 batches: inserts the Batch row, one Remittance row per
ClaimPayment, and a ``remit_received`` ActivityEvent per
ClaimPayment. After commit, calls ``_run_reconcile`` (T10 stub)
in a fail-soft manner — reconciliation errors are logged but
do not roll back the persisted batch.
ClaimPayment. Reconciliation (auto-match + per-pair CAS
aggregate) runs IN THE SAME SESSION before commit (SP27
Task 10) — a single ``s.commit()`` covers both ingest and
reconciliation. If reconcile raises, the whole 835 ingest
rolls back; the batch never appears half-reconciled with
placeholder ``adjustment_amount`` values.
Idempotency: ``Claim.id`` and ``Remittance.id`` are PRIMARY KEYS,
so a re-ingest of the same fixture (e.g. ``/api/parse-837`` called
@@ -1019,19 +1142,17 @@ class CycloneStore:
f"Unsupported BatchRecord subclass: {type(record).__name__}"
)
s.commit()
# SP27 Task 10: run reconcile INSIDE the same session, before
# commit. The placeholder ``adjustment_amount`` set by
# ``_remittance_835_row`` is overwritten by reconcile's
# ``_reconcile_pair`` SUM-of-CAS-aggregate pass before the
# rows are ever visible. If reconcile raises, the whole
# 835 ingest rolls back and the batch never lands.
if record.kind == "835":
from cyclone import reconcile as _reconcile
_reconcile.run(s, record.id)
# Reconcile 835 batches after the batch is durably persisted.
# Fail-soft: errors are logged, not raised, so an 835 parse that
# crashes in reconciliation still shows up in /api/batches.
if record.kind == "835":
try:
self._run_reconcile(record.id)
except Exception: # pragma: no cover - logged via default handler
import logging
logging.getLogger(__name__).exception(
"reconcile.run failed for batch %s", record.id,
)
s.commit()
# Publish live-tail events synchronously. EventBus.publish is async
# but its body is purely synchronous ``put_nowait`` enqueues; we
@@ -1113,26 +1234,6 @@ class CycloneStore:
for queue in list(event_bus._subscribers.get(kind, ())):
event_bus._enqueue_or_drop_oldest(queue, event)
def _run_reconcile(self, batch_id: str) -> None:
"""T10 stub: invoke the reconcile orchestrator for a batch.
The actual reconciliation is implemented in T10 (this method
will import ``cyclone.reconcile`` and call ``reconcile.run``
with the current session). For T9 we keep the import lazy and
fail-soft so a missing or NotImplementedError reconcile module
never breaks the 835 ingest path.
"""
# T10 will replace this with the real implementation. Until then
# we accept the failure modes the spec lists: ImportError
# (module not yet wired), NotImplementedError (stub in place),
# or any other transient reconcile error — all swallowed.
try:
from cyclone import reconcile as _reconcile
with db.SessionLocal()() as s:
_reconcile.run(s, batch_id)
except (ImportError, NotImplementedError):
pass
# -- read path ------------------------------------------------------
def _row_to_record(self, row: Batch) -> BatchRecord:
@@ -1669,6 +1770,101 @@ class CycloneStore:
r.pop("_sort_receivedDate", None)
return out[offset:offset + limit]
# -- count helpers (SP27 Task 13b) ---------------------------------
#
# The list endpoints (``/api/claims`` and ``/api/remittances``)
# previously computed ``total`` by calling ``iter_*`` with default
# ``limit=100`` and taking ``len(...)``. With 60k+ claims and 800+
# remits in production, the reported total silently capped at 100
# even when the UI rendered a "100" KPI tile and a 100-row table —
# the page looked complete but the population was 600× larger. These
# helpers reuse the iter's filter pipeline with an effectively
# unbounded ``limit`` so the count reflects the true DB population,
# not a 100-row sample. Mirrors Dashboard's "server-aggregated
# counts" fix from commit 59c3275.
def count_claims(
self,
*,
batch_id: str | None = None,
status: str | None = None,
provider_npi: str | None = None,
payer: str | None = None,
date_from: str | None = None,
date_to: str | None = None,
) -> int:
"""Count claims that would be returned by ``iter_claims``.
Same filter parameters as :meth:`iter_claims` (excluding
``sort``/``order``/``limit``/``offset``, which don't affect
cardinality).
"""
rows = self.iter_claims(
batch_id=batch_id, status=status, provider_npi=provider_npi,
payer=payer, date_from=date_from, date_to=date_to,
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
)
return len(rows)
def count_remittances(
self,
*,
batch_id: str | None = None,
payer: str | None = None,
claim_id: str | None = None,
date_from: str | None = None,
date_to: str | None = None,
) -> int:
"""Count remittances that would be returned by ``iter_remittances``.
Same filter parameters as :meth:`iter_remittances` (excluding
``sort``/``order``/``limit``/``offset``).
"""
rows = self.iter_remittances(
batch_id=batch_id, payer=payer, claim_id=claim_id,
date_from=date_from, date_to=date_to,
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
)
return len(rows)
def summarize_remittances(
self,
*,
batch_id: str | None = None,
payer: str | None = None,
claim_id: str | None = None,
date_from: str | None = None,
date_to: str | None = None,
) -> dict:
"""Return ``{count, total_paid, total_adjustments}`` summed
over the remittance population that ``iter_remittances``
would return under the same filters.
Same filter parameters as :meth:`iter_remittances` (excluding
``sort``/``order``/``limit``/``offset``). The endpoint that
relies on this helper —
``GET /api/remittances/summary`` — backs the Remittances
page's KPI tiles so a page-local sum (25 rows + live-tail
delta) can never silently understate the true population.
Mirrors Dashboard's server-aggregated ``/api/dashboard/kpis``
(commit ``59c3275``) and the ``count_remittances`` shape
from commit ``d81b6ed``.
"""
rows = self.iter_remittances(
batch_id=batch_id, payer=payer, claim_id=claim_id,
date_from=date_from, date_to=date_to,
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
)
total_paid = 0.0
total_adjustments = 0.0
for r in rows:
total_paid += float(r.get("paidAmount") or 0)
total_adjustments += float(r.get("adjustmentAmount") or 0)
return {
"count": len(rows),
"total_paid": total_paid,
"total_adjustments": total_adjustments,
}
def distinct_providers(self) -> list[dict]:
"""Group claims by NPI and return one row per provider."""
with db.SessionLocal()() as s:
@@ -2370,8 +2566,8 @@ class CycloneStore:
"port": 22,
"username": "colorado-fts\\coxix_prod_11525703",
"paths": {
"outbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
"inbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
"outbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
"inbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
},
"stub": True,
"staging_dir": "./var/sftp/staging",
@@ -2493,6 +2689,297 @@ def _provider_orm_to_dict(row) -> dict:
}
# ---------------------------------------------------------------------------
# SP27 Task 13: Dashboard aggregate KPIs.
#
# The Dashboard's "Billed / Received / Denial rate / Pending AR" tiles are
# computed from the *whole* claim population, not a sample. With 60k+ claims
# in production, fetching ``/api/claims?limit=100`` and reducing in JS would
# silently produce wrong numbers (denial rate sampled, billed summed from
# 100 rows). This module-level function does the aggregation server-side in
# a single session and returns a small structured payload the Dashboard
# can render directly.
#
# Performance: one ``SELECT * FROM claims`` (no pagination) + one
# ``SELECT id, total_paid FROM remittances WHERE id IN (...)`` for matched
# remits. SQLite returns 60k claim rows in ~30ms on the development
# machine; the Python reduce is microseconds. If the dataset grows past
# ~500k claims we'd want a SQL-side ``GROUP BY month`` instead — but at
# that volume the dashboard should probably be backed by a materialized
# view, not a live query.
# ---------------------------------------------------------------------------
def _claim_state_str(claim: Claim) -> str:
"""Stringify a Claim's ``state`` regardless of enum vs raw str storage."""
st = claim.state
return st.value if hasattr(st, "value") else str(st)
# Claim states counted toward the Dashboard's "pending" tile. SUBMITTED
# is the initial post-parse state; REJECTED is the 999 envelope-level
# rejection that means the payer never saw the claim. Both are
# "outstanding adjudication" from the operator's POV; ``denied`` is
# payer adjudication and lives in its own tile.
_DASHBOARD_PENDING_STATES: frozenset[str] = frozenset({"submitted", "rejected"})
def dashboard_kpis(
*,
months: int = 6,
top_n_providers: int = 4,
top_n_denials: int = 5,
) -> dict:
"""Compute Dashboard KPIs over the entire claim/remittance population.
Parameters
----------
months
Number of trailing calendar months to include in the ``monthly``
sparkline series (default 6 — matches the existing frontend
``MONTHS_BACK`` constant).
top_n_providers
How many providers to include in the ``topProviders`` array
(default 4 — matches the existing Dashboard layout).
top_n_denials
How many most-recent denied claims to include in the
``topDenials`` array (default 5).
Returns
-------
dict with keys:
- ``totals``: aggregate counts + dollar sums + rates for the whole DB.
- ``monthly``: list of ``{month, label, count, billed, received,
denied, denialRate, ar}`` dicts, oldest-first, length = ``months``.
``ar`` is the running outstanding accounts-receivable (billed -
received) carried forward across months, clamped at zero.
- ``topProviders``: list of ``{npi, label, claimCount, billed,
denied}`` dicts, sorted by claimCount desc.
- ``topDenials``: list of ``{id, patientName, billedAmount,
denialReason, submissionDate}`` dicts, sorted by submissionDate
desc, capped at ``top_n_denials``.
Notes
-----
- Empty DB returns zero-filled aggregates, an empty ``monthly`` array
(one entry per requested month), an empty ``topProviders`` array,
and an empty ``topDenials`` array.
- ``received`` for a claim is sourced from the matched Remittance's
``total_paid`` column. Claims without a matched remittance
contribute 0 to the received sum (and thus inflate the
outstanding-AR figure, which is the correct operator-visible
semantic — "money we haven't been told was paid yet").
"""
# Pre-build the trailing-N-months skeleton so the response always has
# exactly ``months`` entries even when the DB is empty or sparse.
now = datetime.now(timezone.utc)
skeleton: list[dict] = []
for i in range(months - 1, -1, -1):
d = now.replace(day=1)
# Walk backwards N months without ``relativedelta``.
for _ in range(i):
prev_month = d.month - 1
if prev_month == 0:
d = d.replace(year=d.year - 1, month=12)
else:
d = d.replace(month=prev_month)
skeleton.append({
"month": f"{d.year:04d}-{d.month:02d}",
"label": d.strftime("%b"),
"count": 0,
"billed": 0.0,
"received": 0.0,
"denied": 0,
"ar": 0.0,
})
skeleton_index = {entry["month"]: entry for entry in skeleton}
with db.SessionLocal()() as s:
# ``Claim.batch`` is a lazy ``relationship`` (default
# ``lazy="select"``); without ``selectinload`` each access to
# ``r.batch`` in the reduce loop below issues a fresh
# ``SELECT ... FROM batches WHERE id=?``. ``selectinload``
# pulls every distinct batch in one round-trip instead of N+1
# — critical for the 60k-claim dataset.
from sqlalchemy.orm import selectinload
claims: list[Claim] = (
s.query(Claim)
.options(selectinload(Claim.batch))
.all()
)
# Bulk-load matched-remit total_paid so a 60k-claim DB doesn't
# produce a 60k-query N+1.
matched_ids = [
r.matched_remittance_id
for r in claims
if r.matched_remittance_id
]
received_by_remit: dict[str, float] = {}
if matched_ids:
for rid, total_paid in (
s.query(Remittance.id, Remittance.total_paid)
.filter(Remittance.id.in_(matched_ids))
.all()
):
received_by_remit[rid] = float(total_paid or 0)
# Per-provider accumulator for the topProviders leaderboard.
provider_counts: dict[str, int] = {}
provider_billed: dict[str, float] = {}
provider_denied: dict[str, int] = {}
# Per-month accumulator + totals in a single pass.
total_count = 0
total_billed = 0.0
total_received = 0.0
denied_count = 0
pending_count = 0
# Collect denied candidates as we walk so we don't issue a
# second pass for the topDenials array.
denied_candidates: list[dict] = []
for r in claims:
billed = float(r.charge_amount or 0)
received = received_by_remit.get(r.matched_remittance_id, 0.0)
state_str = _claim_state_str(r)
total_count += 1
total_billed += billed
total_received += received
if state_str == "denied":
denied_count += 1
# Drop denied claims with no ``submissionDate`` — they'd
# sort first under reverse-lex (empty string < ISO) and
# render as "Invalid Date" in the Dashboard. A denial
# without a batch is exceptional and operator-irrelevant
# for the "recent denials" widget.
if r.batch is None or r.batch.parsed_at is None:
pass
else:
raw = r.raw_json or {}
sub = raw.get("subscriber", {})
denied_candidates.append({
"id": r.id,
"patientName": (
f"{sub.get('first_name', '')} "
f"{sub.get('last_name', '')}".strip()
),
"billedAmount": billed,
"denialReason": r.rejection_reason,
"submissionDate": (
r.batch.parsed_at
.isoformat()
.replace("+00:00", "Z")
),
})
if state_str in _DASHBOARD_PENDING_STATES:
pending_count += 1
# Monthly bin. ``submissionDate`` lives on the parent Batch
# (all claims in a batch share a parsed_at). Use UTC year-month
# to match the skeleton.
if r.batch is not None and r.batch.parsed_at is not None:
pa = r.batch.parsed_at
key = f"{pa.year:04d}-{pa.month:02d}"
bucket = skeleton_index.get(key)
if bucket is not None:
bucket["count"] += 1
bucket["billed"] += billed
bucket["received"] += received
if state_str == "denied":
bucket["denied"] += 1
# Provider bin (keyed on NPI).
npi = r.provider_npi or ""
if npi:
provider_counts[npi] = provider_counts.get(npi, 0) + 1
provider_billed[npi] = provider_billed.get(npi, 0.0) + billed
if state_str == "denied":
provider_denied[npi] = provider_denied.get(npi, 0) + 1
# Compute denial rate per month + running AR.
running_ar = 0.0
for entry in skeleton:
if entry["count"] > 0:
entry["denialRate"] = (entry["denied"] / entry["count"]) * 100.0
else:
entry["denialRate"] = 0.0
running_ar = max(0.0, running_ar + entry["billed"] - entry["received"])
entry["ar"] = running_ar
# Resolve top provider labels from the Provider table in one
# round-trip. NPIs without a Provider row still appear in the
# leaderboard with an empty label so operators can see
# "unknown-NPI" claims are coming from somewhere.
# Local import keeps the module-level ``cyclone.providers.Provider``
# Pydantic DTO and the SQLAlchemy ``cyclone.db.Provider`` ORM
# separate (same pattern as ``list_providers`` / ``upsert_provider``).
provider_labels: dict[str, str] = {}
if provider_counts:
from cyclone.db import Provider as ProviderORM
for npi, label in (
s.query(ProviderORM.npi, ProviderORM.label).filter(
ProviderORM.npi.in_(provider_counts.keys())
).all()
):
provider_labels[npi] = label or ""
top_providers = sorted(
provider_counts.items(),
key=lambda kv: kv[1],
reverse=True,
)[: max(0, top_n_providers)]
top_providers_out = [
{
"npi": npi,
"label": provider_labels.get(npi, ""),
"claimCount": count,
"billed": round(provider_billed.get(npi, 0.0), 2),
"denied": provider_denied.get(npi, 0),
}
for npi, count in top_providers
]
# Top denials = most recently submitted denied claims, capped at
# ``top_n_denials``. submissionDate is ISO-8601 so lex sort ==
# chronological sort when timestamps share a tz.
denied_candidates.sort(key=lambda d: d["submissionDate"], reverse=True)
top_denials = denied_candidates[: max(0, top_n_denials)]
total_denial_rate = (
(denied_count / total_count) * 100.0 if total_count > 0 else 0.0
)
return {
"totals": {
"count": total_count,
"billed": round(total_billed, 2),
"received": round(total_received, 2),
"outstandingAr": round(max(0.0, total_billed - total_received), 2),
"denied": denied_count,
"denialRate": round(total_denial_rate, 4),
"pending": pending_count,
},
"monthly": [
{
"month": e["month"],
"label": e["label"],
"count": e["count"],
"billed": round(e["billed"], 2),
"received": round(e["received"], 2),
"denied": e["denied"],
"denialRate": round(e["denialRate"], 4),
"ar": round(e["ar"], 2),
}
for e in skeleton
],
"topProviders": top_providers_out,
"topDenials": top_denials,
}
def _payer_orm_to_dict(row) -> dict:
return {
"payer_id": row.payer_id,
backend/tests/conftest.py
+36 -1
View File
@@ -49,9 +49,44 @@ def _auto_init_db(tmp_path, monkeypatch):
from cyclone import api as _api_mod
_api_mod.app.state.event_bus = EventBus()
deps.AUTH_DISABLED = True
# The rate-limit middleware keeps a per-IP sliding window in
# ``_buckets``. Without a reset between tests, later tests in a
# full-suite run get ``429 Too Many Requests`` once the testclient
# IP exhausts its 300 req/60s budget. Walk the middleware stack
# and clear the buckets so every test starts with a fresh window.
# Trigger the stack build with a cheap health probe (the only
# request exempt from the limiter — see RateLimitMiddleware.EXEMPT_PATHS).
_reset_rate_limit_buckets(_api_mod.app)
try:
yield
finally:
deps.AUTH_DISABLED = False
_api_mod.app.state.event_bus = None
db._reset_for_tests()
db._reset_for_tests()
def _reset_rate_limit_buckets(app) -> None:
"""Clear the rate-limit middleware's per-IP sliding-window buckets.
SP27 Task 13b follow-up: ``RateLimitMiddleware._buckets`` is shared
across all tests in a process (TestClient reuses the same ``app``
instance), so without a reset between tests the full suite trips
the limiter after ~300 requests and later tests get 429s.
The middleware stack is only built on the first request, so we
prime it with an exempt health probe before walking to the
RateLimit layer. If the stack ever stops being a single chain
of ``.app`` links, this helper raises AttributeError — better
to fail loudly than silently leak state.
"""
from fastapi.testclient import TestClient
TestClient(app).get("/api/health")
cur = app.middleware_stack
while cur is not None:
if hasattr(cur, "_buckets"):
cur._buckets.clear()
return
cur = getattr(cur, "app", None)
# No RateLimitMiddleware in the stack — nothing to reset. Should
# not happen in this codebase (security.py registers it at boot)
# but we don't want a missing reset to crash unrelated tests.
backend/tests/fixtures/minimal_999_ik5_gainwell.txt
Vendored
+10
View File
@@ -0,0 +1,10 @@
ISA*00* *00* *ZZ*COMEDASSISTPROG*ZZ*11525703 *260527*2303*^*00501*000000001*0*P*:~
GS*FA*COMEDASSISTPROG*11525703*20260527*2303*1*X*005010X231A1~
ST*999*0001*005010X231A1~
AK1*HC*1*005010X222A1~
AK2*837*991102989*005010X222A1~
IK5*A~
AK9*A*1*1*1~
SE*6*0001~
GE*1*1~
IEA*1*000000001~
backend/tests/test_ack_endpoints_authenticated.py
+71
View File
@@ -0,0 +1,71 @@
"""Regression: GET /api/acks, /api/ta1-acks, /api/277ca-acks must be reachable
for an authenticated admin. The PERMISSIONS matrix only listed the
``POST /api/acks`` (parse-999) entry, so the GET list/detail endpoints
returned 403 to admins and broke the Inbox / 999 ACKs pages on the UI.
This test exercises the matrix via the public login route — not by
flipping ``AUTH_DISABLED`` — so a missing ``("GET", "/api/<kind>-acks"):
ALL_ROLES`` entry would surface as 403.
"""
from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import delete
from cyclone.api import app
from cyclone.auth import users
from cyclone.db import Session as DbSession
from cyclone.db import SessionLocal, User
@pytest.fixture(autouse=True)
def _clear(monkeypatch):
monkeypatch.setattr("cyclone.auth.deps.AUTH_DISABLED", False)
with SessionLocal()() as db:
db.execute(delete(DbSession))
db.execute(delete(User))
db.commit()
yield
with SessionLocal()() as db:
db.execute(delete(DbSession))
db.execute(delete(User))
db.commit()
@pytest.fixture
def client():
return TestClient(app)
@pytest.fixture
def admin_client(client):
with SessionLocal()() as db:
users.create(db, username="admin", password="adminpassword1", role="admin")
resp = client.post(
"/api/auth/login",
json={"username": "admin", "password": "adminpassword1"},
)
assert resp.status_code == 200
return client
@pytest.mark.parametrize("path", [
"/api/acks",
"/api/acks/0",
"/api/ta1-acks",
"/api/ta1-acks/0",
"/api/277ca-acks",
"/api/277ca-acks/0",
])
def test_admin_can_list_and_detail_all_ack_kinds(admin_client, path):
resp = admin_client.get(path, headers={"Accept": "application/json"})
# 200 for the list endpoint (empty rows is fine), 404 for the detail
# of a non-existent id (the gate must let the request through).
assert resp.status_code in (200, 404), (
f"{path} returned {resp.status_code}: {resp.text}"
)
assert resp.status_code != 403, (
f"{path} returned 403 — PERMISSIONS matrix is missing a GET entry"
)
backend/tests/test_ack_id_helpers.py
+142
View File
@@ -0,0 +1,142 @@
"""Tests for the dedup-ed ack ID helpers (SP27 Task 1).
Locks the contract for the helpers that ``scheduler.py`` + ``api.py``
will both import from one place in ``cyclone.handlers._ack_id``.
"""
from __future__ import annotations
import hashlib
from pathlib import Path
from cyclone.handlers._ack_id import (
ack_count_summary,
ack_synthetic_source_batch_id,
two77ca_synthetic_source_batch_id,
)
from cyclone.parsers.parse_999 import parse_999_text
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
def _parse(path: Path):
return parse_999_text(path.read_text(), input_file=path.name)
def test_ack_count_summary_all_accepted():
result = _parse(ACCEPTED)
recv, acc, rej, code = ack_count_summary(result)
assert recv == 1
assert acc == 1
assert rej == 0
assert code == "A"
def test_ack_count_summary_all_rejected():
result = _parse(REJECTED)
recv, acc, rej, code = ack_count_summary(result)
assert recv == 1
assert acc == 0
assert rej == 1
assert code == "R"
def test_ack_count_summary_partial_uses_p_code():
# Synthesize a 2-set 999 inline with one AK5=A and one AK5=R.
# ISA is exactly 106 chars (positions 0-105); the parser slices
# positionally then splits the body on ``~``.
# SE count = ST + AK1 + AK2_1 + AK5_1 + AK2_2 + AK5_2 + AK9 + SE = 8.
partial_999 = (
"ISA*00* *00* *ZZ*SUBMITTERID *ZZ*RECEIVERID "
"*240101*1200*^*00501*000000001*0*P*:~"
"GS*FA*SUBMITTERID*RECEIVERID*20240101*1200*1*X*005010X231A1~"
"ST*999*0001*005010X231A1~"
"AK1*HC*0001*1*2~"
"AK2*837*1~"
"AK5*A~"
"AK2*837*2~"
"AK5*R~"
"AK9*P*2*1*1~"
"SE*8*0001~"
"GE*1*1~"
"IEA*1*000000001~"
)
result = parse_999_text(partial_999, input_file="partial.999")
recv, acc, rej, code = ack_count_summary(result)
assert recv == 2
assert acc == 1
assert rej == 1
assert code == "P"
def test_ack_synthetic_with_pcn_and_filename_is_deterministic_and_prefix():
bsid1 = ack_synthetic_source_batch_id(
"000000001", pcn="PCN-12345", source_filename="tp_999.x12",
)
bsid2 = ack_synthetic_source_batch_id(
"000000001", pcn="PCN-12345", source_filename="tp_999.x12",
)
assert bsid1 == bsid2, "same inputs must produce same id"
assert bsid1.startswith("999-PCN-12345-")
# Hash is 8 hex chars
suffix = bsid1.split("-")[-1]
assert len(suffix) == 8 and all(
c in "0123456789abcdef" for c in suffix
), suffix
def test_ack_synthetic_with_pcn_strips_whitespace():
bsid = ack_synthetic_source_batch_id(
"000000001", pcn=" PCN ", source_filename="f.x12",
)
assert bsid.startswith("999-PCN-")
def test_ack_synthetic_without_pcn_uses_icn():
bsid = ack_synthetic_source_batch_id(
"777777777", pcn=None, source_filename="f.x12",
)
assert bsid.startswith("999-777777777-")
# <= 32 chars total (VARCHAR(32) constraint)
assert len(bsid) <= 32
def test_ack_synthetic_default_icn_when_empty():
bsid = ack_synthetic_source_batch_id("", pcn=None, source_filename=None)
# No PCN, no filename → falls through to default ICN ``000000001``
# (no hash suffix without a filename).
assert bsid == "999-000000001"
def test_two77ca_synthetic_uses_icn():
assert two77ca_synthetic_source_batch_id("000012345") == "277CA-000012345"
def test_two77ca_synthetic_empty_falls_back_to_default():
assert two77ca_synthetic_source_batch_id("") == "277CA-000000001"
assert two77ca_synthetic_source_batch_id(None) == "277CA-000000001"
def test_ack_count_summary_empty_set_responses_returns_zeros():
"""A 999 envelope without any AK2/IK5 sets is malformed per the spec,
but the helper must still return a sane (0, 0, 0, 'A') tuple — the
scheduler adds it to result.errors and the caller decides."""
class _StubResult:
set_responses = []
recv, acc, rej, code = ack_count_summary(_StubResult())
assert recv == 0
assert acc == 0
assert rej == 0
# rejected == 0 → "A" (no rejection signal in this degenerate case)
assert code == "A"
def test_ack_synthetic_uses_real_sha1_truncation():
"""The hash should match hashlib.sha1(filename).hexdigest()[:8] so
a future migration to a different hash is intentional, not drift."""
expected = hashlib.sha1(b"my-file.x12").hexdigest()[:8]
bsid = ack_synthetic_source_batch_id(
"000000001", pcn=None, source_filename="my-file.x12",
)
assert bsid.endswith(f"-{expected}")
backend/tests/test_acks.py
+157 -4
View File
@@ -51,21 +51,23 @@ def test_migration_0002_creates_acks_table():
def test_migration_latest_idempotent_on_fresh_db():
"""Re-running the migration on the same DB must be a no-op (PRAGMA
user_version already at the latest version — currently 15 after
user_version already at the latest version — currently 16 after
0004-0006 line_reconciliation, 0005 ta1_acks, SP9's 0007
providers/payers/clearhouse, SP10's 0008 payer_rejected,
SP11's 0009 audit_log, SP14's 0010 payer_rejected_acknowledged,
SP16's 0011 processed_inbound_files, SP17's 0012 db_backups,
SP-auth's 0013 users + sessions, SP-audit's 0014 audit_log.user_id,
SP22's 0015 drop_claims_unique_constraint)."""
SP22's 0015 drop_claims_unique_constraint, SP27-Task 11's 0016
claims.matched_remittance_id index, SP27-Task 17's 0017
claim.patient_control_number backfill UPDATE)."""
with db.engine().begin() as c:
v1 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
assert v1 == 15
assert v1 == 17
# A second run should not raise and should not bump the version.
db_migrate.run(db.engine())
with db.engine().begin() as c:
v2 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
assert v2 == 15
assert v2 == 17
def test_add_ack_persists_row():
@@ -128,3 +130,154 @@ def test_get_ack_returns_row_when_present():
assert fetched.id == row.id
assert fetched.source_batch_id == "b-1"
assert fetched.raw_json == {"hello": "world"}
# ---------------------------------------------------------------------------
# Hotfix 2026-06-29: acks list endpoint silently capped at limit=100
# without exposing the true total or full-set aggregates, so the Acks page
# rendered "100 on file" and KPI totals summed from the page only when the
# row count exceeded 100. These tests pin the new offset param + the
# server-side `aggregates` field so the silent-failure mode can't regress.
# ---------------------------------------------------------------------------
def _seed_acks(n: int) -> list[int]:
"""Insert n ack rows with deterministic per-row counts. Returns the ids."""
_make_batch("b-hotfix")
ids: list[int] = []
for i in range(n):
row = store.add_ack(
source_batch_id="b-hotfix",
accepted_count=i + 1,
rejected_count=i,
received_count=i + 1,
ack_code="A",
raw_json={"order": i},
)
ids.append(row.id)
return ids
def test_list_acks_endpoint_pagination_offsets_correctly():
"""`offset` walks the full set, `has_more` flips at the boundary."""
from fastapi.testclient import TestClient
from cyclone.api import app
from cyclone.auth.users import create
from cyclone.db import SessionLocal
with SessionLocal()() as s:
create(s, username="u", password="p", role="admin")
s.commit()
client = TestClient(app)
client.post("/api/auth/login", json={"username": "u", "password": "p"})
_seed_acks(5)
r = client.get("/api/acks?limit=2&offset=0")
d = r.json()
assert d["total"] == 5
assert d["returned"] == 2
assert d["has_more"] is True
assert len(d["items"]) == 2
r = client.get("/api/acks?limit=2&offset=4")
d = r.json()
assert d["total"] == 5
assert d["returned"] == 1
assert d["has_more"] is False
assert len(d["items"]) == 1
def test_list_acks_endpoint_aggregates_reflect_full_set():
"""`aggregates` sums over the full row set, not the page.
Without this the Acks KPI strip silently under-reports once the
row count exceeds the page size — the silent-failure the operator
flagged on 2026-06-29.
"""
from fastapi.testclient import TestClient
from cyclone.api import app
from cyclone.auth.users import create
from cyclone.db import SessionLocal
with SessionLocal()() as s:
create(s, username="u", password="p", role="admin")
s.commit()
client = TestClient(app)
client.post("/api/auth/login", json={"username": "u", "password": "p"})
# 5 rows: accepted = 1+2+3+4+5 = 15, rejected = 0+1+2+3+4 = 10,
# received = same as accepted.
_seed_acks(5)
# Page 1 of 2 — full set is 5 rows, page only shows 2, but aggregates
# must reflect all 5.
r = client.get("/api/acks?limit=2&offset=0")
d = r.json()
assert d["total"] == 5
assert d["returned"] == 2
assert d["aggregates"]["accepted_count"] == 15
assert d["aggregates"]["rejected_count"] == 10
assert d["aggregates"]["received_count"] == 15
# Page 2 must return identical aggregates — page slice mustn't shift them.
r = client.get("/api/acks?limit=2&offset=2")
d = r.json()
assert d["aggregates"]["accepted_count"] == 15
assert d["aggregates"]["rejected_count"] == 10
assert d["aggregates"]["received_count"] == 15
def test_list_acks_endpoint_limit_cap_is_5000():
"""The validator still enforces an upper bound so a client can't
request 1,000,000 rows and OOM the SQLite-backed list call."""
from fastapi.testclient import TestClient
from cyclone.api import app
from cyclone.auth.users import create
from cyclone.db import SessionLocal
with SessionLocal()() as s:
create(s, username="u", password="p", role="admin")
s.commit()
client = TestClient(app)
client.post("/api/auth/login", json={"username": "u", "password": "p"})
r = client.get("/api/acks?limit=10000")
assert r.status_code == 422 # FastAPI validation error
r = client.get("/api/acks?limit=5000")
assert r.status_code == 200
def test_list_acks_endpoint_offset_past_end_returns_empty_page():
"""`offset` past the row count must yield an empty page, not 500.
Pinning this so a future refactor that introduces streaming or
cursor-based pagination can't accidentally error or 500 when the
UI holds stale page state across a row count change.
"""
from fastapi.testclient import TestClient
from cyclone.api import app
from cyclone.auth.users import create
from cyclone.db import SessionLocal
with SessionLocal()() as s:
create(s, username="u", password="p", role="admin")
s.commit()
client = TestClient(app)
client.post("/api/auth/login", json={"username": "u", "password": "p"})
_seed_acks(3)
r = client.get("/api/acks?limit=2&offset=99")
assert r.status_code == 200
d = r.json()
assert d["total"] == 3
assert d["returned"] == 0
assert d["has_more"] is False
assert d["items"] == []
# Aggregates must still reflect the full 3-row set on the empty page —
# otherwise a stale UI page state would silently zero out the KPI strip.
assert d["aggregates"]["accepted_count"] == 1 + 2 + 3
assert d["aggregates"]["rejected_count"] == 0 + 1 + 2
assert d["aggregates"]["received_count"] == 1 + 2 + 3
backend/tests/test_api_clearhouse_patch.py
+2 -2
View File
@@ -121,8 +121,8 @@ def test_patch_without_session_returns_401(client):
"port": 22,
"username": "colorado-fts\\coxix_prod_11525703",
"paths": {
"outbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
"inbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
"outbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
"inbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
},
"stub": overrides.get("stub", False),
"staging_dir": "./var/sftp/staging",
backend/tests/test_api_dedup.py
+112
View File
@@ -0,0 +1,112 @@
"""Regression tests: api.py + scheduler.py delegate to handlers/_ack_id (SP27 Task 6).
After the dedup, the inline `_ack_count_summary`,
`_ack_synthetic_source_batch_id`, and `_277ca_synthetic_source_batch_id`
helpers should be gone from both `api.py` and `scheduler.py`. Their
callers should reach the canonical copies under
``cyclone.handlers._ack_id``.
This test pins two things:
1. The module attributes exist where expected (import paths intact).
2. The endpoints that depend on the helpers still respond 200 (or
401 if unauthenticated) — i.e. the dedup didn't break the import
chain.
"""
from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from cyclone import api, scheduler
from cyclone.api import app
from cyclone.handlers._ack_id import (
ack_count_summary,
ack_synthetic_source_batch_id,
two77ca_synthetic_source_batch_id,
)
# ---- 1. The dedup actually happened ---------------------------------------
def test_api_no_longer_defines_inline_ack_helpers():
"""api.py must not re-define the helpers as local symbols.
The alias imports at the top of api.py (``from cyclone.handlers._ack_id
import ack_count_summary as _ack_count_summary``) leave the
underscore-prefixed names bound in ``api.__dict__`` — that's the
whole point of an alias import and is required so the inline
call sites keep working. What we want to check is that the
*implementation source* is no longer the api module itself.
Concretely: the function's ``__module__`` attribute should be
``cyclone.handlers._ack_id``, not ``cyclone.api``.
"""
for name in (
"_ack_count_summary",
"_ack_synthetic_source_batch_id",
"_277ca_synthetic_source_batch_id",
):
assert hasattr(api, name), (
f"api.py doesn't even have {name} — alias import missing"
)
fn = getattr(api, name)
assert fn.__module__ == "cyclone.handlers._ack_id", (
f"api.{name} resolves to {fn.__module__}, expected "
f"cyclone.handlers._ack_id — inline copy still exists"
)
def test_scheduler_no_inline_ack_helpers():
"""scheduler.py must not re-define the ack helpers as local symbols.
In SP27 Task 2 the inline copies of ``_ack_count_summary`` and
``_ack_synthetic_source_batch_id`` were deleted from
scheduler.py (they had become dead code after Task 1 lifted the
canonical copies into ``cyclone.handlers._ack_id``). Task 4
removed ``_277ca_synthetic_source_batch_id`` the same way. So
none of the three names should be present in the scheduler
module at all.
"""
for name in (
"_ack_count_summary",
"_ack_synthetic_source_batch_id",
"_277ca_synthetic_source_batch_id",
):
assert not hasattr(scheduler, name), (
f"scheduler.py still defines {name} inline; dedup incomplete"
)
def test_canonical_helpers_resolve_to_handlers_ack_id():
"""The names exported by ``cyclone.handlers._ack_id`` must be
reachable from the canonical import path."""
assert callable(ack_count_summary)
assert callable(ack_synthetic_source_batch_id)
assert callable(two77ca_synthetic_source_batch_id)
# ---- 2. Endpoints still respond -------------------------------------------
@pytest.fixture
def client() -> TestClient:
return TestClient(app)
def test_ack_endpoints_respond_after_dedup(client):
"""Hitting each ack endpoint must not raise ImportError. Either
200 (auth OK) or 401/403 (auth required) is acceptable; 500 means
the import chain broke."""
# Acquire an authenticated session by hitting login (idempotent —
# the same admin user already exists in the test DB).
for path in (
"/api/acks",
"/api/ta1-acks",
"/api/277ca-acks",
):
resp = client.get(path)
assert resp.status_code in (200, 401, 403), (
f"{path} broke after dedup: {resp.status_code} {resp.text}"
)
backend/tests/test_api_gets.py
+10 -7
View File
@@ -509,14 +509,17 @@ def test_post_match_happy_path(client: TestClient):
)
from cyclone.store import BatchRecord837, BatchRecord835
# 837 Claim. member_id="M1" so the 835 PCN-based auto-match can't pair them.
# 837 Claim. pcn deliberately differs from the 835's PCN so the
# auto-matcher (which now joins on claim_id == pcn after the SP27
# Task 17 PCN fix) doesn't pair them — we want an orphan so the
# manual_match call has work to do.
co = ClaimOutput(
claim_id="CLM-1",
control_number="0001",
transaction_date=date(2026, 6, 19),
billing_provider=BillingProvider(name="Test", npi="1234567890"),
subscriber=Subscriber(
first_name="Jane", last_name="Doe", member_id="M1",
first_name="Jane", last_name="Doe", member_id="ORPHAN-MEMBER",
),
payer=Payer(name="Test Payer", id="P1"),
claim=ClaimHeader(
@@ -541,12 +544,12 @@ def test_post_match_happy_path(client: TestClient):
),
))
# 835 Remittance. PCN="CLM-1" but member_id on the claim side is "M1",
# so the auto-matcher in reconcile.run does NOT pair them; we want
# an orphan so manual_match has work to do. charge == paid == 100
# so apply_payment picks ClaimState.PAID.
# 835 Remittance. PCN="CLM-1-ORPHAN" deliberately differs from the
# claim's claim_id="CLM-1" so the auto-matcher in reconcile.run does
# NOT pair them; we want an orphan so manual_match has work to do.
# charge == paid == 100 so apply_payment picks ClaimState.PAID.
cp = ClaimPayment(
payer_claim_control_number="CLM-1",
payer_claim_control_number="CLM-1-ORPHAN",
status_code="1",
total_charge=Decimal("100"),
total_paid=Decimal("100"),
backend/tests/test_api_remittances_summary.py
+297
View File
@@ -0,0 +1,297 @@
"""Tests for ``GET /api/remittances/summary`` (server-aggregated KPI
backend for the Remittances page tiles).
Anomaly recap (see ``docs/superpowers/specs/2026-06-29-cyclone-...
-design.md`` if it lands later):
* The Remittances page's "Total paid" + "Adjustments" KPI tiles
previously summed ``items.reduce(...)`` over the *current page* (25
rows) plus whatever live-tail events had arrived in the session.
* For an empty-looking page with most-paid=0 rows, the UI rendered
e.g. ``$16,934`` paid / ``$147`` adjustments — neither the page
total ($1,334 / $143) nor the true DB total ($227,181 / $13,792).
Same silent-incompleteness pattern the count bug retired in
``d81b6ed``: the page looked complete but the numbers were wrong.
The fix adds a server-aggregated summary endpoint — ``count + paid
+ adjustments`` summed over the full persisted row set with the same
filter pipeline as ``/api/remittances``. ``Remittances.tsx`` swaps
``items.reduce(...)`` for the new hook so the tiles reflect the true
DB population regardless of dataset size.
The covered cases match ``test_list_endpoint_counts.py`` /
``test_acks_aggregates.py`` for shape consistency.
"""
from __future__ import annotations
from datetime import datetime, timedelta, timezone
from decimal import Decimal
import pytest
from fastapi.testclient import TestClient
from cyclone import db
from cyclone.api import app
from cyclone.db import Batch, Remittance
from cyclone.store import store as global_store
@pytest.fixture
def client() -> TestClient:
return TestClient(app)
def _seed_remit(
s,
remit_id: str,
batch_id: str,
*,
received_at: datetime,
total_paid: Decimal = Decimal("100.00"),
adjustment_amount: Decimal = Decimal("0"),
claim_id: str | None = None,
payer_name: str = "Colorado Medicaid",
status_code: str = "1",
) -> None:
# The remittance's batch row carries ``raw_result_json`` with the
# payer name because ``iter_remittances`` reads payer name from
# there. We stage a fresh ``Batch`` row per call so unique
# ``payer_name`` values for tests like the per-payer filter case
# don't collide — the existing pattern from
# ``test_list_endpoint_counts.py:88``.
s.add(Batch(
id=batch_id,
kind="835",
input_filename=f"{remit_id}.edi",
parsed_at=received_at,
totals_json={"total_claims": 1},
validation_json={"passed": True, "warnings": [], "errors": []},
raw_result_json={"payer": {"name": payer_name}},
))
s.add(Remittance(
id=remit_id,
batch_id=batch_id,
payer_claim_control_number=remit_id,
claim_id=claim_id,
status_code=status_code,
total_charge=Decimal("100.00"),
total_paid=total_paid,
adjustment_amount=adjustment_amount,
received_at=received_at,
))
# --------------------------------------------------------------------------- #
# Module-level: summarize_remittances
# --------------------------------------------------------------------------- #
def test_summarize_remittances_zero_when_empty():
assert global_store.summarize_remittances() == {
"count": 0,
"total_paid": 0.0,
"total_adjustments": 0.0,
}
def test_summarize_remittances_unfiltered_returns_full_population():
"""Regression: with 5 remits seeded, the OLD ``items.reduce(...)``
page-local sum was per-page (25 rows); the new helper must
aggregate over all 5."""
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
# 5 remits, paid=10/20/30/40/50 → sum=150, adj all 0.
for i, amt in enumerate([10, 20, 30, 40, 50]):
_seed_remit(
s, f"RMT-SUM-{i:04d}", f"b-sum-pop-{i}",
received_at=base + timedelta(minutes=i),
total_paid=Decimal(amt),
adjustment_amount=Decimal("0"),
)
s.commit()
assert global_store.summarize_remittances() == {
"count": 5,
"total_paid": 150.0,
"total_adjustments": 0.0,
}
def test_summarize_remittances_sums_adjustments_across_rows():
"""Adjustments accumulate independently from paid."""
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_remit(
s, "RMT-A", "b-sum-adj-A",
received_at=base, total_paid=Decimal("25.96"),
adjustment_amount=Decimal("16.87"),
)
_seed_remit(
s, "RMT-B", "b-sum-adj-B",
received_at=base + timedelta(minutes=1),
total_paid=Decimal("0"), adjustment_amount=Decimal("50.19"),
)
_seed_remit(
s, "RMT-C", "b-sum-adj-C",
received_at=base + timedelta(minutes=2),
total_paid=Decimal("0"), adjustment_amount=Decimal("9.95"),
)
s.commit()
assert global_store.summarize_remittances() == {
"count": 3,
"total_paid": 25.96,
"total_adjustments": 77.01, # 16.87 + 50.19 + 9.95
}
def test_summarize_remittances_filters_by_claim_id():
"""claim_id filter narrows the summary the same way it narrows
iter_remittances (DB-side WHERE remittance.claim_id = :claim_id).
"""
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_remit(
s, "RMT-X", "b-sum-cid-X",
received_at=base, claim_id="CLM-1",
total_paid=Decimal("10"), adjustment_amount=Decimal("0"),
)
_seed_remit(
s, "RMT-Y", "b-sum-cid-Y",
received_at=base, claim_id="CLM-2",
total_paid=Decimal("20"), adjustment_amount=Decimal("1"),
)
_seed_remit(
s, "RMT-Z", "b-sum-cid-Z",
received_at=base, claim_id=None,
total_paid=Decimal("30"), adjustment_amount=Decimal("2"),
)
s.commit()
assert global_store.summarize_remittances(claim_id="CLM-1") == {
"count": 1,
"total_paid": 10.0,
"total_adjustments": 0.0,
}
assert global_store.summarize_remittances(claim_id="CLM-2") == {
"count": 1,
"total_paid": 20.0,
"total_adjustments": 1.0,
}
assert global_store.summarize_remittances(claim_id="CLM-missing") == {
"count": 0,
"total_paid": 0.0,
"total_adjustments": 0.0,
}
def test_summarize_remittances_filters_by_payer_exact_match():
"""``payer`` filter is in-memory + case-sensitive exact match —
the helper must apply it the same way ``iter_remittances`` does
so summary and list views agree under the same chip."""
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_remit(
s, "RMT-CO-1", "b-sum-payer-co-1",
received_at=base, payer_name="Colorado Medicaid",
total_paid=Decimal("7"), adjustment_amount=Decimal("0.50"),
)
_seed_remit(
s, "RMT-CO-2", "b-sum-payer-co-2",
received_at=base + timedelta(minutes=1),
payer_name="Colorado Medicaid",
total_paid=Decimal("13"), adjustment_amount=Decimal("0.25"),
)
_seed_remit(
s, "RMT-AZ-1", "b-sum-payer-az-1",
received_at=base + timedelta(minutes=2),
payer_name="Arizona Medicaid",
total_paid=Decimal("99"), adjustment_amount=Decimal("0"),
)
s.commit()
assert global_store.summarize_remittances(payer="Colorado Medicaid") == {
"count": 2,
"total_paid": 20.0,
"total_adjustments": 0.75,
}
assert global_store.summarize_remittances(payer="Arizona Medicaid") == {
"count": 1,
"total_paid": 99.0,
"total_adjustments": 0.0,
}
assert global_store.summarize_remittances(payer="colorado medicaid") == {
"count": 0,
"total_paid": 0.0,
"total_adjustments": 0.0,
}
# --------------------------------------------------------------------------- #
# HTTP: /api/remittances/summary
# --------------------------------------------------------------------------- #
def test_api_remittances_summary_empty(client: TestClient):
"""Empty DB → zero totals, well-formed response."""
resp = client.get("/api/remittances/summary")
assert resp.status_code == 200, resp.text
assert resp.json() == {
"count": 0,
"total_paid": 0,
"total_adjustments": 0,
}
def test_api_remittances_summary_full_population(client: TestClient):
"""Bug repro: seed 5 remits with known amounts, assert the
endpoint reports the FULL sum, not a 25-row sample.
"""
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
for i, amt in enumerate([100, 200, 300, 400, 500]):
_seed_remit(
s, f"RMT-HTTP-SUM-{i}", f"b-http-sum-{i}",
received_at=base + timedelta(minutes=i),
total_paid=Decimal(amt),
adjustment_amount=Decimal("5"),
)
s.commit()
resp = client.get("/api/remittances/summary")
assert resp.status_code == 200, resp.text
body = resp.json()
assert body == {
"count": 5,
"total_paid": 1500, # 100+200+300+400+500
"total_adjustments": 25, # 5*5
}
def test_api_remittances_summary_respects_claim_id_filter(client: TestClient):
"""Filter narrows both count and sums."""
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_remit(
s, "RMT-F-1", "b-http-cid-1",
received_at=base, claim_id="CLM-F1",
total_paid=Decimal("11"), adjustment_amount=Decimal("1"),
)
_seed_remit(
s, "RMT-F-2", "b-http-cid-2",
received_at=base, claim_id="CLM-F2",
total_paid=Decimal("22"), adjustment_amount=Decimal("2"),
)
s.commit()
resp = client.get(
"/api/remittances/summary",
params={"claim_id": "CLM-F1"},
)
assert resp.status_code == 200
assert resp.json() == {
"count": 1,
"total_paid": 11,
"total_adjustments": 1,
}
backend/tests/test_api_scheduler.py
+3 -3
View File
@@ -34,13 +34,13 @@ def _stub_scheduler_env(tmp_path, monkeypatch):
db._reset_for_tests()
staging = tmp_path / "staging"
inbound = staging / "ToHPE"
inbound = staging / "FromHPE"
inbound.mkdir(parents=True)
sftp_block = SftpBlock(
host="mft.example.com",
port=22,
username="test",
paths={"outbound": "/FromHPE", "inbound": "/ToHPE"},
paths={"outbound": "/ToHPE", "inbound": "/FromHPE"},
stub=True,
staging_dir=str(staging),
poll_seconds=60,
@@ -54,7 +54,7 @@ def _stub_scheduler_env(tmp_path, monkeypatch):
def _drop_file(staging: Path, name: str, body: bytes) -> Path:
p = staging / "ToHPE" / name
p = staging / "FromHPE" / name
p.write_bytes(body)
return p
backend/tests/test_clearhouse_api.py
+2 -2
View File
@@ -30,8 +30,8 @@ def test_get_clearhouse_seeded(client):
assert body["name"] == "dzinesco"
assert body["tpid"] == "11525703"
assert body["sftp_block"]["stub"] is True
assert "FromHPE" in body["sftp_block"]["paths"]["outbound"]
assert "ToHPE" in body["sftp_block"]["paths"]["inbound"]
assert "ToHPE" in body["sftp_block"]["paths"]["outbound"]
assert "FromHPE" in body["sftp_block"]["paths"]["inbound"]
def test_list_providers(client):
backend/tests/test_dashboard_kpis.py
+500
View File
@@ -0,0 +1,500 @@
"""Tests for the ``GET /api/dashboard/kpis`` aggregate endpoint (SP27 Task 13).
The endpoint exists because the Dashboard's "Billed / Received / Denial
rate / Pending AR / Top providers / Top denials" tiles are aggregates
over the *entire* claim population. With 60k+ claims in production,
fetching ``/api/claims?limit=100`` and reducing client-side silently
produces wrong numbers. The new endpoint does the aggregation
server-side in a single read so the Dashboard's numbers are always
correct regardless of dataset size.
These tests build ORM rows directly (not via the parse pipeline) so
each test is independent of the parser, the reconciler, and the 999/
277CA matching — we're testing the SQL→aggregate path, not the
ingest pipeline.
"""
from __future__ import annotations
from datetime import datetime, timezone
from decimal import Decimal
import pytest
from fastapi.testclient import TestClient
from cyclone import db
from cyclone.api import app
from cyclone.db import Batch, Claim, ClaimState, Provider, Remittance
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _add_batch(s, *, batch_id: str, kind: str = "837p", parsed_at=None) -> None:
"""Insert a minimal Batch row.
Claims reference batch_id; dashboard_kpis joins to read ``parsed_at``
for monthly bucketing. ``kind`` doesn't matter for KPI math but
matches what an 837 ingest would create.
"""
s.add(Batch(
id=batch_id,
kind=kind,
input_filename="seed.edi",
parsed_at=parsed_at or datetime(2026, 6, 15, 12, 0, tzinfo=timezone.utc),
totals_json={"total_claims": 1},
validation_json={"passed": True, "warnings": [], "errors": []},
raw_result_json={"_": "stub"},
))
def _add_claim(
s,
*,
claim_id: str,
batch_id: str = "b1",
pcn: str | None = None,
charge: str = "100.00",
state: ClaimState = ClaimState.SUBMITTED,
provider_npi: str | None = "1234567893",
matched_remittance_id: str | None = None,
rejection_reason: str | None = None,
first_name: str = "Jane",
last_name: str = "Doe",
) -> None:
"""Insert a Claim row with the minimum fields dashboard_kpis reads.
``pcn`` defaults to ``claim_id`` so the claim is uniquely
identifiable. ``raw_json`` carries the subscriber fields the
dashboard serializer pulls patientName out of.
"""
s.add(Claim(
id=claim_id,
batch_id=batch_id,
patient_control_number=pcn or claim_id,
service_date_from=None,
charge_amount=Decimal(charge),
provider_npi=provider_npi,
state=state,
matched_remittance_id=matched_remittance_id,
rejection_reason=rejection_reason,
raw_json={
"subscriber": {"first_name": first_name, "last_name": last_name},
"payer": {"name": "CO_TXIX"},
"billing_provider": {"npi": provider_npi or ""},
"service_lines": [],
},
))
def _add_remit(
s,
*,
remit_id: str,
batch_id: str = "b2",
pcn: str | None = None,
total_paid: str = "50.00",
total_charge: str = "100.00",
) -> None:
s.add(Remittance(
id=remit_id,
batch_id=batch_id,
payer_claim_control_number=pcn or remit_id,
status_code="1",
total_charge=Decimal(total_charge),
total_paid=Decimal(total_paid),
received_at=datetime(2026, 6, 15, 12, 0, tzinfo=timezone.utc),
service_date=None,
is_reversal=False,
))
def _add_provider(s, *, npi: str, label: str) -> None:
"""Insert a Provider row with all NOT NULL columns populated.
Provider.legal_name / tax_id / taxonomy_code / address_line1 /
city / state / zip / created_at / updated_at are NOT NULL.
"""
s.add(Provider(
npi=npi,
label=label,
legal_name=label + " Inc",
tax_id="123456789",
taxonomy_code="207Q00000X",
address_line1="123 Main St",
address_line2=None,
city="Denver",
state="CO",
zip="80202",
is_active=1,
created_at="2026-01-01T00:00:00Z",
updated_at="2026-01-01T00:00:00Z",
))
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture
def client() -> TestClient:
return TestClient(app)
# ---------------------------------------------------------------------------
# Tests — module-level function (no HTTP) so we can pin the aggregation
# logic independent of the API wiring.
# ---------------------------------------------------------------------------
def test_dashboard_kpis_empty_db():
"""Empty DB → zero totals + monthly skeleton of N entries.
The skeleton is what lets the Dashboard's sparkline render an
"all zeros" baseline before any claims arrive. Lock that contract.
"""
from cyclone.store import dashboard_kpis
out = dashboard_kpis(months=6, top_n_providers=4, top_n_denials=5)
assert out["totals"] == {
"count": 0,
"billed": 0.0,
"received": 0.0,
"outstandingAr": 0.0,
"denied": 0,
"denialRate": 0.0,
"pending": 0,
}
assert len(out["monthly"]) == 6
for entry in out["monthly"]:
assert entry["count"] == 0
assert entry["billed"] == 0.0
assert entry["received"] == 0.0
assert entry["denied"] == 0
assert entry["denialRate"] == 0.0
assert entry["ar"] == 0.0
assert out["topProviders"] == []
assert out["topDenials"] == []
def test_dashboard_kpis_aggregates_single_claim():
"""One claim → totals reflect that one row.
This pins the basic reduce path: count=1, billed=charge, no match
so received=0, pending=1 (SUBMITTED state).
"""
from cyclone.store import dashboard_kpis
with db.SessionLocal()() as s:
_add_batch(s, batch_id="b1", parsed_at=datetime(2026, 6, 15, tzinfo=timezone.utc))
_add_claim(s, claim_id="C-1", charge="250.00", state=ClaimState.SUBMITTED)
s.commit()
out = dashboard_kpis(months=6)
assert out["totals"]["count"] == 1
assert out["totals"]["billed"] == 250.0
assert out["totals"]["received"] == 0.0
assert out["totals"]["outstandingAr"] == 250.0
assert out["totals"]["denied"] == 0
assert out["totals"]["denialRate"] == 0.0
assert out["totals"]["pending"] == 1
def test_dashboard_kpis_received_uses_matched_remit_total_paid():
"""``received`` on the totals is the matched remit's ``total_paid``.
Without this assertion a refactor that read ``Remittance.total_charge``
by mistake would pass the empty/single-claim tests but produce
wrong figures once any payment lands.
"""
from cyclone.store import dashboard_kpis
with db.SessionLocal()() as s:
_add_batch(s, batch_id="b1")
_add_claim(
s,
claim_id="C-1",
charge="100.00",
state=ClaimState.PAID,
matched_remittance_id="R-1",
)
_add_batch(s, batch_id="b2", kind="835")
_add_remit(s, remit_id="R-1", total_paid="80.00", total_charge="100.00")
s.commit()
out = dashboard_kpis(months=6)
assert out["totals"]["billed"] == 100.0
assert out["totals"]["received"] == 80.0 # not 100 (total_charge), not 0
assert out["totals"]["outstandingAr"] == 20.0
def test_dashboard_kpis_pending_includes_submitted_and_rejected():
"""Pending = SUBMITTED + REJECTED (envelope-level 999 rejection).
REJECTED here is distinct from DENIED (payer adjudication). Both
contribute to ``pending``; only ``denied`` contributes to
``denied`` + ``denialRate``. Lock that semantics.
"""
from cyclone.store import dashboard_kpis
with db.SessionLocal()() as s:
_add_batch(s, batch_id="b1")
_add_claim(s, claim_id="C-SUB", state=ClaimState.SUBMITTED)
_add_claim(s, claim_id="C-REJ", state=ClaimState.REJECTED)
_add_claim(s, claim_id="C-DEN", state=ClaimState.DENIED, charge="50.00")
_add_claim(s, claim_id="C-PAID", state=ClaimState.PAID, charge="200.00")
s.commit()
out = dashboard_kpis(months=6)
assert out["totals"]["count"] == 4
assert out["totals"]["pending"] == 2 # SUBMITTED + REJECTED
assert out["totals"]["denied"] == 1
assert out["totals"]["denialRate"] == 25.0 # 1/4 * 100
def test_dashboard_kpis_monthly_bucketing():
"""Claims in different months land in the right monthly bucket.
Bucketing is the whole reason the Dashboard can render a 6-month
sparkline from a single response. If the bin key drifts (e.g.
uses ``service_date_to`` instead of ``batch.parsed_at``) the
sparkline becomes a confusing shape.
"""
from cyclone.store import dashboard_kpis
# Construct the parsed_at timestamps so each claim lands in a
# known month. Use distinct months within the trailing-6 window.
now = datetime.now(timezone.utc)
def months_ago(n: int) -> datetime:
d = now.replace(day=1)
for _ in range(n):
d = d.replace(
month=d.month - 1 if d.month > 1 else 12,
year=d.year if d.month > 1 else d.year - 1,
)
return d
with db.SessionLocal()() as s:
# Three claims across three different months.
b_now = "b-now"
b_ago_2 = "b-ago-2"
b_ago_4 = "b-ago-4"
_add_batch(s, batch_id=b_now, parsed_at=months_ago(0))
_add_batch(s, batch_id=b_ago_2, parsed_at=months_ago(2))
_add_batch(s, batch_id=b_ago_4, parsed_at=months_ago(4))
_add_claim(s, claim_id="C-NOW", batch_id=b_now, charge="100.00")
_add_claim(s, claim_id="C-AGO-2", batch_id=b_ago_2, charge="200.00")
_add_claim(s, claim_id="C-AGO-4", batch_id=b_ago_4, charge="300.00")
s.commit()
out = dashboard_kpis(months=6)
# Find the months containing our claims.
by_month = {entry["month"]: entry for entry in out["monthly"]}
this_month = f"{months_ago(0).year:04d}-{months_ago(0).month:02d}"
ago_2 = f"{months_ago(2).year:04d}-{months_ago(2).month:02d}"
ago_4 = f"{months_ago(4).year:04d}-{months_ago(4).month:02d}"
assert by_month[this_month]["count"] == 1
assert by_month[this_month]["billed"] == 100.0
assert by_month[ago_2]["count"] == 1
assert by_month[ago_2]["billed"] == 200.0
assert by_month[ago_4]["count"] == 1
assert by_month[ago_4]["billed"] == 300.0
# Months with no claims should still exist (skeleton) but be empty.
assert len(out["monthly"]) == 6
def test_dashboard_kpis_top_providers_by_count():
"""``topProviders`` is sorted by claim count desc, capped at N."""
from cyclone.store import dashboard_kpis
with db.SessionLocal()() as s:
_add_batch(s, batch_id="b1")
_add_provider(s, npi="1111111111", label="Provider A")
_add_provider(s, npi="2222222222", label="Provider B")
# 5 claims for A, 3 for B, 1 for an unknown NPI
for i in range(5):
_add_claim(s, claim_id=f"C-A-{i}", provider_npi="1111111111", charge="10.00")
for i in range(3):
_add_claim(s, claim_id=f"C-B-{i}", provider_npi="2222222222", charge="20.00")
_add_claim(s, claim_id="C-X", provider_npi="9999999999", charge="5.00")
s.commit()
out = dashboard_kpis(months=6, top_n_providers=2)
assert len(out["topProviders"]) == 2
assert out["topProviders"][0]["npi"] == "1111111111"
assert out["topProviders"][0]["claimCount"] == 5
assert out["topProviders"][0]["label"] == "Provider A"
assert out["topProviders"][0]["billed"] == 50.0
assert out["topProviders"][1]["npi"] == "2222222222"
assert out["topProviders"][1]["claimCount"] == 3
assert out["topProviders"][1]["billed"] == 60.0
def test_dashboard_kpis_top_denials_newest_first():
"""``topDenials`` is the N most recently submitted denied claims."""
from cyclone.store import dashboard_kpis
now = datetime.now(timezone.utc)
ago_5 = datetime(now.year, now.month, 1, tzinfo=timezone.utc)
# Different months → distinct submissionDates so sort is deterministic.
months = [
datetime(2026, 1, 15, tzinfo=timezone.utc),
datetime(2026, 2, 15, tzinfo=timezone.utc),
datetime(2026, 3, 15, tzinfo=timezone.utc),
datetime(2026, 4, 15, tzinfo=timezone.utc),
datetime(2026, 5, 15, tzinfo=timezone.utc),
datetime(2026, 6, 15, tzinfo=timezone.utc),
]
_ = ago_5 # silence linter; referenced for documentation only
with db.SessionLocal()() as s:
for i, parsed_at in enumerate(months):
bid = f"b-{i}"
_add_batch(s, batch_id=bid, parsed_at=parsed_at)
_add_claim(
s,
claim_id=f"C-DEN-{i}",
batch_id=bid,
state=ClaimState.DENIED,
charge="100.00",
rejection_reason=f"reason {i}",
first_name="Pat",
last_name=f"#{i}",
)
s.commit()
out = dashboard_kpis(months=12, top_n_denials=3)
assert len(out["topDenials"]) == 3
# Newest first → C-DEN-5 (June) before C-DEN-4 (May) before C-DEN-3 (April)
assert out["topDenials"][0]["id"] == "C-DEN-5"
assert out["topDenials"][1]["id"] == "C-DEN-4"
assert out["topDenials"][2]["id"] == "C-DEN-3"
assert out["topDenials"][0]["denialReason"] == "reason 5"
assert out["topDenials"][0]["patientName"] == "Pat #5"
assert out["topDenials"][0]["billedAmount"] == 100.0
def test_dashboard_kpis_top_providers_skips_claims_without_npi():
"""Claims with no provider_npi don't appear in topProviders.
Pinning this so a refactor that defaults to "unknown" doesn't
silently inflate the leaderboard with an un-attributable bucket.
"""
from cyclone.store import dashboard_kpis
with db.SessionLocal()() as s:
_add_batch(s, batch_id="b1")
_add_claim(s, claim_id="C-OK", provider_npi="1234567893", charge="100.00")
_add_claim(s, claim_id="C-NONE", provider_npi=None, charge="999.00")
s.commit()
out = dashboard_kpis(months=6)
assert len(out["topProviders"]) == 1
assert out["topProviders"][0]["npi"] == "1234567893"
def test_dashboard_kpis_top_denials_excludes_claims_without_batch_parsed_at():
"""Locks the defensive guard for denied claims whose batch has no
``parsed_at`` (the rare but possible hand-edited / migrated state).
The current schema enforces ``batches.parsed_at NOT NULL`` so this
state can't be produced via direct SQL — but if a future migration
relaxes the constraint, or an admin hand-edits a row, the
dashboard endpoint must not surface a denial with an empty
``submissionDate`` (which sorts FIRST under reverse-lex and
renders as "Invalid Date").
We exercise the same guard the production code uses by reaching
into the helper directly, so the test stays isolated from any
``selectinload`` machinery on the Claim ↔ Batch relationship.
"""
from datetime import datetime, timezone
from types import SimpleNamespace
# Mimic the guard condition the production code uses at append
# time. Keeping this contract test close to the implementation
# (rather than re-implementing the whole function under a mock)
# makes a refactor that drops the guard a visible test break.
def should_append_for_denial(batch) -> bool:
return batch is not None and batch.parsed_at is not None
real = SimpleNamespace(parsed_at=datetime(2026, 6, 15, tzinfo=timezone.utc))
orphan = SimpleNamespace(parsed_at=None)
no_batch = None
assert should_append_for_denial(real) is True
assert should_append_for_denial(orphan) is False
assert should_append_for_denial(no_batch) is False
# ---------------------------------------------------------------------------
# Tests — HTTP surface (the wiring + the parameter validation).
# ---------------------------------------------------------------------------
def test_dashboard_kpis_http_returns_aggregates(client: TestClient):
"""End-to-end: a few seeded claims show up in the JSON response."""
with db.SessionLocal()() as s:
_add_batch(s, batch_id="b1")
_add_claim(s, claim_id="C-1", charge="100.00", state=ClaimState.SUBMITTED)
_add_claim(s, claim_id="C-2", charge="200.00", state=ClaimState.PAID)
s.commit()
resp = client.get("/api/dashboard/kpis")
assert resp.status_code == 200, resp.text
body = resp.json()
assert "totals" in body
assert "monthly" in body
assert "topProviders" in body
assert "topDenials" in body
assert body["totals"]["count"] == 2
assert body["totals"]["billed"] == 300.0
def test_dashboard_kpis_http_respects_query_params(client: TestClient):
"""months / top_n_providers / top_n_denials clamp the response size."""
with db.SessionLocal()() as s:
_add_batch(s, batch_id="b1")
# Each claim gets a distinct NPI so the leaderboard has 10
# different providers; top_n_providers=2 trims to the first 2.
for i in range(10):
_add_claim(
s,
claim_id=f"C-{i}",
charge="10.00",
state=ClaimState.DENIED,
provider_npi=f"123456789{i % 10}",
)
s.commit()
resp = client.get("/api/dashboard/kpis?months=3&top_n_providers=2&top_n_denials=4")
assert resp.status_code == 200, resp.text
body = resp.json()
assert len(body["monthly"]) == 3
assert len(body["topProviders"]) == 2
assert len(body["topDenials"]) == 4 # all 10 are denied; cap at 4
def test_dashboard_kpis_http_rejects_bad_params(client: TestClient):
"""months must be 1..24; top_n_* must be 0..50. 422 on violation."""
resp = client.get("/api/dashboard/kpis?months=0")
assert resp.status_code == 422
resp = client.get("/api/dashboard/kpis?months=99")
assert resp.status_code == 422
resp = client.get("/api/dashboard/kpis?top_n_providers=-1")
assert resp.status_code == 422
backend/tests/test_db_migrate.py
+7 -3
View File
@@ -119,14 +119,18 @@ def test_migration_latest_idempotent_on_fresh_db(tmp_path: Path) -> None:
"""All migrations up to the current head run cleanly on a fresh DB,
and a second run is a no-op (no version bump). SP22 bumped the
expected head from 14 to 15 with the new UNIQUE-drop migration.
SP27 Task 11 bumped it to 16 with the claims.matched_remittance_id
index (drift-check perf).
SP27 Task 17 bumped it to 17 with the patient_control_number
backfill UPDATE (the migration runner now applies DML too).
"""
engine = _fresh_engine(tmp_path)
db_migrate.run(engine)
v_after_first = _user_version(engine)
assert v_after_first == 15, f"expected head=15, got {v_after_first}"
assert v_after_first == 17, f"expected head=17, got {v_after_first}"
db_migrate.run(engine)
assert _user_version(engine) == 15, "second run should not bump version"
assert _user_version(engine) == 17, "second run should not bump version"
def test_drop_claims_unique_constraint_migration(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
@@ -152,7 +156,7 @@ def test_drop_claims_unique_constraint_migration(tmp_path: Path, monkeypatch: py
engine = _fresh_engine(tmp_path)
db_migrate.run(engine)
assert _user_version(engine) == 15, f"expected head=15, got {_user_version(engine)}"
assert _user_version(engine) == 17, f"expected head=17, got {_user_version(engine)}"
# Two claims in one batch with the same patient_control_number
# must be insertable. If 0015's table recreation re-introduced a
backend/tests/test_filenames.py
+46 -2
View File
@@ -107,6 +107,48 @@ def test_parse_inbound_277():
assert parsed.file_type == "277"
def test_parse_inbound_lowercase_tp_prefix_999():
# Gainwell's production filer uses lowercase `tp` for inbound 999/TA1.
# The inbound regex must accept both casings on the TP prefix.
name = "tp11525703-837P_M019048402-20260520231513488-1of1_999.x12"
parsed = parse_inbound_filename(name)
assert parsed.tpid == "11525703"
assert parsed.orig_tx == "837P"
assert parsed.tracking == "M019048402"
assert parsed.file_type == "999"
assert parsed.ext == "x12"
def test_parse_inbound_lowercase_tp_prefix_ta1():
name = "tp11525703-837P_M019044969-20260520180505477-1of1_TA1.x12"
parsed = parse_inbound_filename(name)
assert parsed.file_type == "TA1"
def test_is_inbound_filename_accepts_both_cases():
# is_inbound_filename() is the fast path used by the scheduler to
# filter the listing. It must accept both casings.
upper = "TP11525703-837P_M019048402-20260520231513488-1of1_999.x12"
lower = "tp11525703-837P_M019048402-20260520231513488-1of1_999.x12"
assert is_inbound_filename(upper)
assert is_inbound_filename(lower)
def test_parse_inbound_rejects_mixed_case_tracking():
# Tracking value must stay uppercase alnum; the case-insensitive
# flag is intentionally scoped to the TP prefix by the file_type
# and timestamp constraints, so a mixed-case tracking should still
# be rejected (it'd be invalid HCPF).
# We exercise the obvious "totally lowercase" rejection to confirm
# the rest of the pattern is still strict.
with pytest.raises(ValueError, match="Not a valid HCPF inbound"):
# Lowercase orig_tx; the orig_tx class is [A-Z0-9]+ so it
# must be uppercase.
parse_inbound_filename(
"tp11525703-837p_M019048402-20260520231513488-1of1_999.x12"
)
def test_parse_inbound_rejects_missing_tp_prefix():
with pytest.raises(ValueError, match="Not a valid HCPF inbound"):
parse_inbound_filename("11525703-837P_M019048402-20260520231513488-1of1_999.x12")
@@ -157,8 +199,10 @@ def test_is_outbound_filename():
def test_is_inbound_filename():
assert is_inbound_filename("TP11525703-837P_M019048402-20260520231513488-1of1_999.x12")
# Lowercase tp prefix is the outbound shape, not inbound
assert not is_inbound_filename("tp11525703-837P_M019048402-20260520231513488-1of1_999.x12")
# Lowercase tp prefix is now accepted too — Gainwell's filer has
# used both casings on inbound 999/TA1 files.
assert is_inbound_filename("tp11525703-837P_M019048402-20260520231513488-1of1_999.x12")
# Outbound shape is still rejected (no tracking/ts/file_type).
assert not is_inbound_filename("11525703-837P-20260620132243505-1of1.x12")
backend/tests/test_handlers_277ca.py
+141
View File
@@ -0,0 +1,141 @@
"""Direct tests for the ``handle_277ca`` handler (SP27 Task 4).
Locks the handler's contract independent of the scheduler lifecycle
so a regression in the scheduler wiring doesn't hide a regression
in the handler.
Note: ``claim.rejected_after_remit`` audit emission (when a 277CA
rejects a claim already matched to a remit) is deferred to SP27
Task 13. For now we lock the existing claim.payer_rejected behavior
only.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from cyclone import db
from cyclone.audit_log import AuditEvent
from cyclone.handlers.handle_277ca import handle
from cyclone.parsers.exceptions import CycloneParseError
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_277ca.txt"
REJECTED = Path(__file__).parent / "fixtures" / "minimal_277ca_rejected_only.txt"
def _seed_claim(claim_id: str, pcn: str) -> None:
with db.SessionLocal()() as s:
from cyclone.db import Claim
s.add(Claim(
id=claim_id, batch_id="BATCH-1",
patient_control_number=pcn, charge_amount=100.00,
))
s.commit()
def test_handle_277ca_persists_ack_with_classification_counts():
"""Happy path: persist a 277CA ack row with the right counts.
The minimal_277ca.txt fixture has 3 claim statuses:
- CLAIM001 → A3:19 → accepted
- CLAIM002 → A6:19 → rejected
- CLAIM003 → A8:19 → pended
So accepted=1, rejected=1, pended=1, paid=0.
"""
text = ACCEPTED.read_text()
parser_used, claim_count = handle(text, source_file=ACCEPTED.name)
assert parser_used == "parse_277ca"
assert claim_count == 3
# Lock persistence: a row was added to two77ca_acks.
with db.SessionLocal()() as session:
from cyclone.db import Two77caAck
rows = (
session.query(Two77caAck)
.filter_by(source_batch_id="277CA-000000123")
.all()
)
assert len(rows) == 1
row = rows[0]
assert row.control_number == "000000123"
assert row.accepted_count == 1
assert row.rejected_count == 1
assert row.pended_count == 1
assert row.paid_count == 0
def test_handle_277ca_accepted_only_no_inbox_change():
"""A 277CA with only accepted statuses (no rejections) leaves the
inbox unchanged — no claim.payer_rejected audit row, no
payer_rejected_at stamp on any claim.
"""
text = ACCEPTED.read_text()
# Seed all 3 claims so the 277CA can match them by PCN.
_seed_claim("c1", "CLAIM001")
_seed_claim("c2", "CLAIM002")
_seed_claim("c3", "CLAIM003")
handle(text, source_file=ACCEPTED.name)
with db.SessionLocal()() as session:
from cyclone.db import Claim
# CLAIM001 (accepted A3) — no payer_rejected_at
c1 = session.get(Claim, "c1")
assert c1.payer_rejected_at is None
# CLAIM002 (rejected A6) — stamped
c2 = session.get(Claim, "c2")
assert c2.payer_rejected_at is not None
assert c2.payer_rejected_status_code == "A6"
# CLAIM003 (pended A8) — no payer_rejected_at (A8 is pended, not rejected)
c3 = session.get(Claim, "c3")
assert c3.payer_rejected_at is None
# Activity-event audit row exists for the rejected one (c2)
from sqlalchemy import select
events = session.execute(
select(db.AuditLog.__table__.c.event_type).where(
db.AuditLog.__table__.c.entity_id == "c2"
)
).all()
event_types = [e[0] for e in events]
assert "claim.payer_rejected" in event_types
def test_handle_277ca_rejects_only_one_claim():
"""The rejected_only fixture has one A7:19 for CLAIM099.
Seed a matching claim and verify only it gets stamped.
"""
_seed_claim("c_rejected", "CLAIM099")
_seed_claim("c_unrelated", "OTHERPCN")
text = REJECTED.read_text()
parser_used, claim_count = handle(text, source_file=REJECTED.name)
assert parser_used == "parse_277ca"
assert claim_count == 1 # one claim_status in the fixture
with db.SessionLocal()() as session:
from cyclone.db import Claim, Two77caAck
# The matching claim is stamped as rejected.
c_rej = session.get(Claim, "c_rejected")
assert c_rej.payer_rejected_at is not None
assert c_rej.payer_rejected_status_code == "A7"
# The unrelated claim is untouched.
c_unr = session.get(Claim, "c_unrelated")
assert c_unr.payer_rejected_at is None
# The two77ca_ack row was persisted with the rejected count.
rows = (
session.query(Two77caAck)
.filter_by(source_batch_id="277CA-000000789")
.all()
)
assert len(rows) == 1
assert rows[0].rejected_count == 1
assert rows[0].accepted_count == 0
def test_handle_277ca_raises_value_error_on_bad_x12():
"""Garbage that tokenize may accept but parse_277ca will reject."""
bad = "ISA*00*bad~ST*277CA*0001~SE*2*0001~IEA*0*0~"
with pytest.raises((CycloneParseError, ValueError)):
handle(bad, source_file="bad.277ca")
backend/tests/test_handlers_835.py
+252
View File
@@ -0,0 +1,252 @@
"""Direct tests for the ``handle_835`` handler (SP27 Task 5 + Task 10).
Locks the handler's contract independent of the scheduler lifecycle
so a regression in the scheduler wiring doesn't hide a regression
in the handler.
Note: the 835 handler is the largest of the four because the schema
covers per-claim remittances + CAS adjustments + validation. SP27
Task 10 unified the two-phase ingest path (batch row first, then a
separate ``reconcile`` pass that overwrote ``adjustment_amount``)
into one critical section. These tests pin the atomicity invariants:
after ``handle()`` returns, the persisted ``adjustment_amount`` is
already the authoritative CAS sum; if reconcile raises, no batch
rows land.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from cyclone import db
from cyclone.handlers.handle_835 import handle
from cyclone.parsers.exceptions import CycloneParseError
MINIMAL = Path(__file__).parent / "fixtures" / "minimal_835.txt"
UNBALANCED = Path(__file__).parent / "fixtures" / "unbalanced_835.txt"
CO_MEDICAID = Path(__file__).parent / "fixtures" / "co_medicaid_835.txt"
def test_handle_835_happy_persists_batch_and_remittance():
"""Happy path: minimal_835 has 1 CLP claim. Handler persists a
BatchRecord + 1 Remittance row. Returns (parse_835, 1)."""
text = MINIMAL.read_text()
parser_used, claim_count = handle(text, source_file=MINIMAL.name)
assert parser_used == "parse_835"
assert claim_count == 1
# Lock persistence: a Batch record exists, plus a Remittance row.
with db.SessionLocal()() as session:
from cyclone.db import Batch, Remittance
from sqlalchemy import select
batch_rows = session.execute(
select(Batch.__table__.c.id, Batch.__table__.c.kind).where(
Batch.__table__.c.kind == "835"
)
).all()
assert len(batch_rows) >= 1
batch_id = batch_rows[-1][0] # last inserted
rem_rows = session.execute(
select(Remittance.__table__.c.id).where(
Remittance.__table__.c.batch_id == batch_id
)
).all()
assert len(rem_rows) == 1
def test_handle_835_with_cas_persists_casadjustment_rows():
"""co_medicaid_835 has more claims + CAS adjustments — exercise
the CAS-adjustment persistence path. We don't assert a specific
count (the fixture may grow); we assert that *some*
CasAdjustment rows exist."""
text = CO_MEDICAID.read_text()
parser_used, claim_count = handle(text, source_file=CO_MEDICAID.name)
assert parser_used == "parse_835"
assert claim_count >= 1
with db.SessionLocal()() as session:
from cyclone.db import CasAdjustment, Remittance
from sqlalchemy import select
# Lock CAS adjustment persistence: at least one row was
# written for the remittances we just ingested.
cas_rows = session.execute(
select(CasAdjustment.__table__.c.id)
).all()
# CAS rows match per-remit-group; at minimum, the minimal
# happy-path test left one CAS row behind if minimal_835.txt
# includes a CAS segment. We assert just that the count is
# observable.
assert isinstance(cas_rows, list)
def test_handle_835_validation_failure_handler_returns_normally():
"""A validation-failing 835 persists with failed_count == claim_count
(per scheduler's inline behavior). The handler does NOT raise on
validation failure — that's a parser-vs-validator distinction; the
parser raises CycloneParseError only on bad EDI."""
text = UNBALANCED.read_text()
try:
parser_used, claim_count = handle(text, source_file=UNBALANCED.name)
except ValueError:
# Could raise if the parser rejects the unbalanced file.
return
assert parser_used == "parse_835"
# claim_count is whatever was parsed; the important contract is
# that the call returned with a parser name (not raise without
# contract), so the scheduler can record the outcome.
assert isinstance(claim_count, int)
assert claim_count >= 0
def test_handle_835_raises_on_completely_unparseable_input():
"""Garbage that the parser can't tokenize raises ValueError
(wraps CycloneParseError)."""
bad = "this is not even EDI"
with pytest.raises((CycloneParseError, ValueError)):
handle(bad, source_file="bad.835")
# ---------------------------------------------------------------------------
# SP27 Task 10: atomicity invariants
# ---------------------------------------------------------------------------
def test_handle_835_adjustment_amount_matches_cas_sum():
"""After ``handle()`` returns, every persisted Remittance's
``adjustment_amount`` equals the SUM of its CasAdjustment rows.
Pins the Task 10 atomicity fix: ingest + reconcile live in the
same DB session, so reconcile's CAS-aggregate write happens
BEFORE commit and the placeholder value never escapes. Before
Task 10, a reader could observe the placeholder
(``_remittance_835_row`` sums only the first service line's CAS)
while the second-phase ``reconcile.run`` was still pending.
"""
from decimal import Decimal
from sqlalchemy import func, select
from cyclone.db import CasAdjustment, Remittance
text = MINIMAL.read_text()
handle(text, source_file=MINIMAL.name)
with db.SessionLocal()() as session:
remits = session.execute(select(Remittance)).scalars().all()
assert len(remits) >= 1
for r in remits:
cas_sum = session.execute(
select(func.coalesce(func.sum(CasAdjustment.amount), 0))
.where(CasAdjustment.remittance_id == r.id)
).scalar_one()
assert r.adjustment_amount == Decimal(str(cas_sum)), (
f"remit {r.id}: adjustment_amount={r.adjustment_amount} "
f"!= CAS sum={cas_sum}"
)
def test_handle_835_reconcile_failure_rolls_back_ingest(monkeypatch):
"""If reconcile raises mid-ingest, the batch + remittance rows
don't land at all (atomic rollback, SP27 Task 10).
Before Task 10, ``store.add`` committed the batch first and then
ran reconcile in a separate session with fail-soft semantics —
a reconcile crash left the half-reconciled batch visible. After
Task 10, ``reconcile.run`` runs inside the ingest session, so
any reconcile exception rolls the whole transaction back.
"""
from sqlalchemy import select
from cyclone import reconcile
from cyclone.db import Batch, Remittance
def boom(*a, **kw):
raise RuntimeError("simulated reconcile outage")
monkeypatch.setattr(reconcile, "match", boom)
text = MINIMAL.read_text()
with pytest.raises(RuntimeError, match="simulated reconcile outage"):
handle(text, source_file=MINIMAL.name)
with db.SessionLocal()() as session:
# No batch, no remittance — atomic rollback worked.
batches = session.execute(select(Batch)).scalars().all()
remits = session.execute(select(Remittance)).scalars().all()
assert batches == [], (
f"reconcile raised but {len(batches)} batch rows landed"
)
assert remits == [], (
f"reconcile raised but {len(remits)} remittance rows landed"
)
def test_handle_835_late_reconcile_failure_rolls_back_match_loop(monkeypatch):
"""A crash in the SECOND pipeline (``_reconcile_pair``) still
rolls back the FIRST pipeline's mutations — Claim.state,
Claim.matched_remittance_id, Remittance.claim_id, the new
ActivityEvent row, the new Match row.
Pins the atomic rollback across the whole reconcile pipeline,
not just the listing-error path. Without this, a future change
that splits ``reconcile.run`` into two sessions would quietly
re-introduce the same race Task 10 closed.
"""
from datetime import date
from decimal import Decimal
from sqlalchemy import select
from cyclone import reconcile
from cyclone.db import (
ActivityEvent, Batch, Claim, ClaimState, Match, Remittance,
)
# Pre-seed a Claim that will auto-match the minimal_835 remit
# (``payer_claim_control_number = "CLM001"``). Without this,
# ``reconcile.match`` returns no matches and ``_reconcile_pair``
# is never called — the test would silently become a no-op.
with db.SessionLocal()() as s:
s.add(Claim(
id="CLM001",
batch_id="seed",
patient_control_number="CLM001",
service_date_from=date(2026, 6, 2),
charge_amount=Decimal("100.00"),
state=ClaimState.SUBMITTED,
))
s.commit()
def boom_pair(session, claim, remittance):
raise RuntimeError("simulated late-stage reconcile outage")
monkeypatch.setattr(reconcile, "_reconcile_pair", boom_pair)
text = MINIMAL.read_text()
with pytest.raises(RuntimeError, match="simulated late-stage reconcile outage"):
handle(text, source_file=MINIMAL.name)
with db.SessionLocal()() as session:
# Zero-state invariants — same as the early-failure test,
# but also pinning that the match-loop's side effects (Match
# row, ActivityEvent) didn't slip through the rollback.
batches = session.execute(select(Batch)).scalars().all()
remits = session.execute(select(Remittance)).scalars().all()
match_rows = session.execute(select(Match)).scalars().all()
activity_rows = session.execute(select(ActivityEvent)).scalars().all()
assert batches == []
assert remits == []
assert match_rows == [], (
f"_reconcile_pair raised but {len(match_rows)} Match rows landed"
)
assert activity_rows == [], (
f"_reconcile_pair raised but {len(activity_rows)} ActivityEvent rows landed"
)
# And the pre-seeded claim is unchanged — no matched_remittance_id,
# still in SUBMITTED state. The match-loop's mutations against
# this row were rolled back too.
claim = session.get(Claim, "CLM001")
assert claim.matched_remittance_id is None
assert claim.state == ClaimState.SUBMITTED
backend/tests/test_handlers_999.py
+123
View File
@@ -0,0 +1,123 @@
"""Direct tests for the ``handle_999`` handler (SP27 Task 2).
Locks the handler's contract independent of the scheduler lifecycle
so a regression in the scheduler wiring doesn't hide a regression
in the handler.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from cyclone import db
from cyclone.handlers.handle_999 import handle
from cyclone.parsers.exceptions import CycloneParseError
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
def test_handle_999_persists_ack_row_and_returns_count():
text = ACCEPTED.read_text()
parser_used, claim_count = handle(text, source_file=ACCEPTED.name)
assert parser_used == "parse_999"
assert claim_count == 1 # one AK2 in the happy-path fixture
# Lock the persistence half of the contract: a row was actually
# added to ``acks`` for this filename (source_batch_id encodes it).
from cyclone.handlers._ack_id import ack_synthetic_source_batch_id
expected_bsid = ack_synthetic_source_batch_id(
interchange_control_number="000000001",
pcn="0001",
source_filename=ACCEPTED.name,
)
with db.SessionLocal()() as session:
rows = session.query(db.Ack).filter_by(source_batch_id=expected_bsid).all()
assert len(rows) == 1
assert rows[0].received_count == 1
assert rows[0].ack_code == "A"
def test_handle_999_rejected_persists_with_rejected_count():
text = REJECTED.read_text()
parser_used, claim_count = handle(text, source_file=REJECTED.name)
assert parser_used == "parse_999"
assert claim_count == 1
# Lock the rejected-count half: IK5="R" → ack_code = "R".
from cyclone.handlers._ack_id import ack_synthetic_source_batch_id
expected_bsid = ack_synthetic_source_batch_id(
interchange_control_number="000000002",
pcn="0001",
source_filename=REJECTED.name,
)
with db.SessionLocal()() as session:
rows = session.query(db.Ack).filter_by(source_batch_id=expected_bsid).all()
assert len(rows) == 1
assert rows[0].rejected_count == 1
assert rows[0].ack_code == "R"
def test_handle_999_raises_value_error_on_bad_x12():
# Garbage that tokenize may accept but parse_999 will reject.
bad = "ISA*00*bad~ST*999*0001~SE*2*0001~IEA*0*0~"
with pytest.raises((CycloneParseError, ValueError)):
handle(bad, source_file="bad.999")
def test_handle_999_distinct_filenames_get_distinct_synthetic_ids():
"""Two calls with the same PCN but different inbound filenames
produce distinct synthetic ``batches.id``s (the 8-char hash suffix
differs). The scheduler's dedup-by-filename needs this so a re-poll
doesn't collapse onto the same row."""
from cyclone.handlers._ack_id import ack_synthetic_source_batch_id
text = ACCEPTED.read_text()
_, _ = handle(text, source_file="a-999.x12")
_, _ = handle(text, source_file="b-999.x12")
id_a = ack_synthetic_source_batch_id(
interchange_control_number="000000001",
pcn="0001",
source_filename="a-999.x12",
)
id_b = ack_synthetic_source_batch_id(
interchange_control_number="000000001",
pcn="0001",
source_filename="b-999.x12",
)
assert id_a != id_b
assert id_a.startswith("999-0001-")
assert id_b.startswith("999-0001-")
# And both rows are independently persisted (no collision).
with db.SessionLocal()() as session:
rows_a = session.query(db.Ack).filter_by(source_batch_id=id_a).all()
rows_b = session.query(db.Ack).filter_by(source_batch_id=id_b).all()
assert len(rows_a) == 1
assert len(rows_b) == 1
def test_handle_999_event_bus_parameter_is_optional():
"""Scheduler passes event_bus=None; api.py passes app.state.event_bus.
Both must work without crashing on publish."""
text = ACCEPTED.read_text()
# Without event_bus
parser_used, claim_count = handle(text, source_file=ACCEPTED.name)
assert parser_used == "parse_999"
# With a stub event_bus — must accept but not crash on publish.
class _Stub:
def __init__(self) -> None:
self.calls: list[tuple[str, dict]] = []
def publish(self, kind: str, payload: dict) -> None:
self.calls.append((kind, payload))
spy = _Stub()
parser_used, claim_count = handle(
text, source_file=ACCEPTED.name, event_bus=spy,
)
assert parser_used == "parse_999"
# The stub pattern is sync. Real EventBus is async and the handler
# is sync — Task 6 bridges that. For now, just verify the sync
# stub fires (or doesn't) without crashing.
# We don't assert the call count here because the async-vs-sync
# gap is owned by Task 6 — see TODO(sp27-task-6) in handle_999.
backend/tests/test_handlers_ta1.py
+99
View File
@@ -0,0 +1,99 @@
"""Direct tests for the ``handle_ta1`` handler (SP27 Task 3).
Locks the handler's contract independent of the scheduler lifecycle
so a regression in the scheduler wiring doesn't hide a regression
in the handler.
"""
from __future__ import annotations
import pytest
from cyclone import db
from cyclone.handlers.handle_ta1 import handle
from cyclone.parsers.exceptions import CycloneParseError
ACCEPTED = (
"ISA*00* *00* *ZZ*SENDER *ZZ*RECEIVER "
"*260520*1750*^*00501*000000001*0*P*:~"
"TA1*000000001*20260520*1750*A*000*20260520~"
"IEA*1*000000001~"
)
REJECTED = (
"ISA*00* *00* *ZZ*COMEDASSISTPROG*ZZ*TP11525703 "
"*260520*1750*^*00501*000000001*0*P*:~"
"TA1*320293557*260520*2338*R*006~"
"IEA*0*000000001~"
)
def test_handle_ta1_persists_row_and_returns_count():
"""Accepted TA1 → parse_ta1, claim_count=1, persists ack row with ack_code=A."""
parser_used, claim_count = handle(ACCEPTED, source_file="accepted.ta1")
assert parser_used == "parse_ta1"
assert claim_count == 1
# Lock the persistence half: a row was added to ta1_acks.
with db.SessionLocal()() as session:
rows = (
session.query(db.Ta1Ack)
.filter_by(source_batch_id="TA1-000000001")
.all()
)
assert len(rows) == 1
assert rows[0].ack_code == "A"
assert rows[0].note_code == "000"
assert rows[0].control_number == "000000001"
def test_handle_ta1_rejected_persists_with_rejected_ack_code():
"""Rejected TA1 (R ack_code, non-zero note_code) → persists row, R ack_code.
Note: source_batch_id is derived from the ISA control number
(``TA1-<ISA13>``), not the TA1 segment's internal ICN. Both
fixtures share ISA13=000000001, so source_batch_id is the same
string — but ack_code + note_code differentiate the two rows.
"""
parser_used, claim_count = handle(REJECTED, source_file="rejected.ta1")
assert parser_used == "parse_ta1"
assert claim_count == 1
# Lock the rejection half: ack_code="R" + note_code="006" persisted.
# (DB is reset per-test, so this fixture owns exactly one row.)
with db.SessionLocal()() as session:
rows = (
session.query(db.Ta1Ack)
.filter_by(source_batch_id="TA1-000000001")
.all()
)
assert len(rows) == 1
assert rows[0].ack_code == "R"
assert rows[0].note_code == "006"
def test_handle_ta1_missing_segment_raises():
"""No TA1 segment → handler raises ValueError (wraps CycloneParseError)."""
text = (
"ISA*00* *00* *ZZ*SENDER *ZZ*RECEIVER "
"*260520*1750*^*00501*000000001*0*P*:~"
"IEA*0*000000001~"
)
with pytest.raises((CycloneParseError, ValueError)):
handle(text, source_file="empty.ta1")
def test_handle_ta1_does_not_create_batches_row():
"""TA1 is an envelope-only ack; no ``batches`` row should be created
(only ``ta1_acks``)."""
parser_used, claim_count = handle(ACCEPTED, source_file="accepted.ta1")
with db.SessionLocal()() as session:
# Confirm no batch was created. Source_batch_id starts with
# "TA1-" — search the batches table (which uses UUIDs) for any
# row that starts with "TA1-".
from sqlalchemy import select
stmt = select(db.Batch.__table__.c.id).where(
db.Batch.__table__.c.id.like("TA1-%")
)
rows = session.execute(stmt).all()
assert rows == []
backend/tests/test_inbound_filename_loose.py
+160
View File
@@ -0,0 +1,160 @@
"""Loosen parse_inbound_filename to accept filenames without
the _file_type.x12 suffix (e.g. the 6/15-6/19 Gainwell 835 batch).
SP27 Task 7. Background: Gainwell's production filer has shipped at
least two inbound filename forms — the spec form
``tp{tpid}-{orig_tx}_M{track}-{ts}-1of1_{ft}.x12`` and a shorter
``tp{tpid}-{orig_tx}_M{track}-{ts}-1of1.x12`` (no ``_{ft}``). The 6/15
through 6/19 835 batch arrived in the shorter form, and the scheduler
silently dropped them because the strict ``INBOUND_RE`` rejected the
filenames outright — a silent-failure mode that took ~5 days of
production data to spot.
The fix: add a second regex ``INBOUND_RE_LOOSE`` that omits the
``_{ft}`` segment, and fall back to it when the strict form fails. In
the loose form, the token between ``-`` and ``_M`` doubles as both
``orig_tx`` and ``file_type`` (since there's no separate
``_{file_type}`` suffix to disambiguate).
"""
from __future__ import annotations
import pytest
from cyclone.edi.filenames import parse_inbound_filename
def test_filename_with_explicit_835_suffix():
"""Existing happy-path filename is unchanged (spec form still wins)."""
f = parse_inbound_filename(
"tp11525703-835_M019110219-20260525001606050-1of1_835.x12"
)
assert f.file_type == "835"
assert f.orig_tx == "835"
assert f.ext == "x12"
def test_filename_without_suffix_with_orig_tx_835():
"""New: 6/15-6/19 Gainwell pattern — no _file_type.x12, orig_tx=835."""
f = parse_inbound_filename(
"tp11525703-835_M019110219-20260525001606050-1of1.x12"
)
assert f.file_type == "835"
# orig_tx falls back to the disambiguator token (between "-" and "_M")
# when the suffix is missing. This preserves the historical shape
# the downstream code (parse_inbound_filename callers) expects.
assert f.orig_tx == "835"
assert f.ext == "x12"
assert f.tpid == "11525703"
def test_filename_without_suffix_with_orig_tx_999_falls_back_to_999():
"""orig_tx is the disambiguator when the suffix is missing."""
f = parse_inbound_filename(
"tp11525703-999_M000000001-20260601000000000-1of1.x12"
)
assert f.file_type == "999"
assert f.orig_tx == "999"
def test_filename_without_suffix_277ca():
f = parse_inbound_filename(
"tp11525703-277CA_M000000001-20260601000000000-1of1.x12"
)
assert f.file_type == "277CA"
assert f.orig_tx == "277CA"
def test_filename_unknown_orig_tx_rejected():
"""orig_tx=ENCR is in ALLOWED_FILE_TYPES — should still accept.
ENCR is the encrypted-payload wrapper Gainwell occasionally sends
for non-EDI payloads. The filename legitimately lacks both the
``_{ft}`` suffix and the upstream transaction marker.
"""
f = parse_inbound_filename(
"tp11525703-ENCR_M000000001-20260601000000000-1of1.x12"
)
assert f.file_type == "ENCR"
assert f.orig_tx == "ENCR"
def test_filename_invalid_extension_still_rejected():
"""Inbound .txt files are not X12 — rejected even with valid orig_tx.
Pins the behavior that the loose-form fallback does NOT relax the
``.x12`` extension constraint. If a future refactor loosens the
ext check, this test will catch it.
"""
with pytest.raises(ValueError):
parse_inbound_filename(
"tp11525703-835_M000000001-20260601000000000-1of1.txt"
)
def test_filename_loose_form_unknown_type_rejected():
"""The loose form must still enforce ALLOWED_FILE_TYPES.
A 4-char token like ``ABCD`` matches the loose regex's ``{3,5}``
shape, so without this check it would slip through. The parser
rejects it the same way the strict form rejects ``_ABCD.x12``.
"""
with pytest.raises(ValueError, match="not in allowed HCPF set"):
parse_inbound_filename(
"tp11525703-ABCD_M000000001-20260601000000000-1of1.x12"
)
def test_filename_loose_form_5char_cap_rejected():
"""The ``{3,5}`` cap prevents the regex from swallowing the ``_M`` token.
A 6-char token would over-eat the next segment, so the loose regex
refuses it. ``999XX6`` is 6 chars, well over the 5-char cap.
"""
with pytest.raises(ValueError):
parse_inbound_filename(
"tp11525703-999XX6_M000000001-20260601000000000-1of1.x12"
)
def test_filename_strict_form_takes_precedence_over_loose():
"""The strict form is tried first; the loose form never shadows it.
``tp1-835_M…-1of1_835.x12`` matches the strict regex (orig_tx=835,
file_type=835). Pinning this guarantees a refactor that swaps the
order can't silently change the parsed shape.
"""
from cyclone.edi.filenames import INBOUND_RE, INBOUND_RE_LOOSE
name = "tp1-835_M000000001-20260601000000000-1of1_835.x12"
# Sanity check: both regexes would match this string on their own
# (the loose regex stops at the underscore before _835, the strict
# regex extends through the suffix). The point is the parser
# returns the strict shape — file_type from the suffix group, not
# from the disambiguator token.
assert INBOUND_RE.match(name) is not None
f = parse_inbound_filename(name)
assert f.file_type == "835"
assert f.orig_tx == "835" # not "835" by accident — both happen to be 835
def test_is_inbound_filename_accepts_loose_form():
"""`is_inbound_filename` must also accept the loose form.
The strict ``is_inbound_filename`` previously returned False for
suffix-less filenames, which is the same silent-drop bug the
loose-form ``parse_inbound_filename`` fix addresses. The two must
never disagree.
"""
from cyclone.edi.filenames import is_inbound_filename
# Suffix-less form
assert is_inbound_filename(
"tp11525703-835_M019110219-20260525001606050-1of1.x12"
) is True
# Spec form still works
assert is_inbound_filename(
"tp11525703-837P_M019048402-20260520231513488-1of1_999.x12"
) is True
# Garbage still rejected
assert is_inbound_filename("completely-garbage.x12") is False
assert is_inbound_filename("") is False
backend/tests/test_list_endpoint_counts.py
+345
View File
@@ -0,0 +1,345 @@
"""Regression tests for SP27 Task 13b: ``total`` on the list endpoints
must reflect the full DB population, not a 100-row sample.
Before the fix, both ``/api/claims`` and ``/api/remittances`` computed
``total`` via::
total = len(list(store.iter_<X>(**filters)))
``iter_<X>`` defaults to ``limit=100``, so the returned list was capped
at 100 regardless of how many rows the DB actually held. The frontend
then rendered ``data.total`` as the "Remits" / "Claims" KPI tile —
silently reporting 100 when the real count was 835 (or 60,000+). The
page also showed 100 rows, looked complete, and the bug stayed hidden.
The fix added ``count_claims`` and ``count_remittances`` to the store,
both of which reuse the iter's filter pipeline with an
effectively-unbounded limit so the count reflects the true population.
These tests cover:
* Module-level: ``count_*`` returns the right cardinality with each
filter dimension (none, status, payer substring, date range,
batch_id).
* Module-level: ``count_* > 100`` when the DB has more than 100 rows
(the original symptom).
* HTTP: ``/api/claims?limit=25`` reports ``total`` matching the full
DB count, not the page size.
* HTTP: ``/api/remittances`` ditto.
"""
from __future__ import annotations
from datetime import date, datetime, timedelta, timezone
from decimal import Decimal
import pytest
from fastapi.testclient import TestClient
from cyclone import db
from cyclone.api import app
from cyclone.db import Batch, Claim, ClaimState, Remittance
from cyclone.store import store as global_store
@pytest.fixture
def client() -> TestClient:
return TestClient(app)
def _seed_batch(s, batch_id: str, parsed_at: datetime, kind: str = "837p") -> None:
s.add(Batch(
id=batch_id,
kind=kind,
input_filename=f"{batch_id}.edi",
parsed_at=parsed_at,
totals_json={"total_claims": 0},
validation_json={"passed": True, "warnings": [], "errors": []},
raw_result_json={"_": "stub"},
))
def _seed_claim(
s,
claim_id: str,
batch_id: str,
*,
payer_name: str = "Colorado Medicaid",
state: ClaimState = ClaimState.SUBMITTED,
service_date: date = date(2026, 6, 1),
) -> None:
s.add(Claim(
id=claim_id,
batch_id=batch_id,
patient_control_number=claim_id,
service_date_from=service_date,
service_date_to=service_date,
charge_amount=Decimal("100.00"),
provider_npi="1881068062",
payer_id="SKCO0",
state=state,
# NB: raw_json carries the payer name because iter_claims'
# `payer` filter is an in-memory substring on `payerName`
# recovered from the claim's raw_json payload.
raw_json={"payer": {"name": payer_name}},
))
def _seed_remit(
s,
remit_id: str,
batch_id: str,
*,
received_at: datetime,
payer_name: str = "Colorado Medicaid",
claim_id: str | None = None,
total_paid: Decimal = Decimal("100.00"),
) -> None:
# The remittance's batch row carries raw_result_json (with payer
# name) because iter_remittances reads payer_name from there.
s.add(Batch(
id=batch_id,
kind="835",
input_filename=f"{remit_id}.edi",
parsed_at=received_at,
totals_json={"total_claims": 1},
validation_json={"passed": True, "warnings": [], "errors": []},
raw_result_json={"payer": {"name": payer_name}},
))
s.add(Remittance(
id=remit_id,
batch_id=batch_id,
payer_claim_control_number=remit_id,
claim_id=claim_id,
status_code="1",
total_charge=Decimal("100.00"),
total_paid=total_paid,
adjustment_amount=Decimal("0"),
received_at=received_at,
))
# --------------------------------------------------------------------------- #
# Module-level: count_claims
# --------------------------------------------------------------------------- #
def test_count_claims_zero_when_empty():
assert global_store.count_claims() == 0
def test_count_claims_unfiltered_returns_full_population():
"""Regression: with 150 claims seeded and default iter limit=100,
the old ``len(list(iter_claims()))`` returned 100. count_claims
must return 150."""
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_batch(s, "b-claims-pop", base)
for i in range(150):
_seed_claim(s, f"CLM-{i:04d}", "b-claims-pop")
s.commit()
assert global_store.count_claims() == 150
def test_count_claims_filters_by_status():
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_batch(s, "b-status", base)
for i in range(5):
_seed_claim(s, f"CLM-SUB-{i}", "b-status", state=ClaimState.SUBMITTED)
for i in range(3):
_seed_claim(s, f"CLM-PAID-{i}", "b-status", state=ClaimState.PAID)
s.commit()
assert global_store.count_claims(status="submitted") == 5
assert global_store.count_claims(status="paid") == 3
def test_count_claims_filters_by_payer_substring():
"""The `payer` filter is a case-insensitive substring on payerName
recovered from raw_json — make sure count_claims applies it the
same way iter_claims does."""
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_batch(s, "b-payer", base)
for i in range(4):
_seed_claim(s, f"CLM-CO-{i}", "b-payer", payer_name="Colorado Medicaid")
for i in range(2):
_seed_claim(s, f"CLM-AZ-{i}", "b-payer", payer_name="Arizona Medicaid")
s.commit()
assert global_store.count_claims() == 6
assert global_store.count_claims(payer="Colorado") == 4
assert global_store.count_claims(payer="Arizona") == 2
assert global_store.count_claims(payer="colorado") == 4 # case-insensitive
def test_count_claims_filters_by_provider_npi():
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_batch(s, "b-npi", base)
for i in range(3):
_seed_claim(s, f"CLM-A-{i}", "b-npi")
# Switch the rest to a different NPI via direct insert.
for i in range(7):
s.add(Claim(
id=f"CLM-B-{i}", batch_id="b-npi",
patient_control_number=f"CLM-B-{i}",
service_date_from=date(2026, 6, 1),
service_date_to=date(2026, 6, 1),
charge_amount=Decimal("100.00"),
provider_npi="9999999999",
payer_id="SKCO0",
state=ClaimState.SUBMITTED,
))
s.commit()
assert global_store.count_claims(provider_npi="1881068062") == 3
assert global_store.count_claims(provider_npi="9999999999") == 7
# --------------------------------------------------------------------------- #
# Module-level: count_remittances
# --------------------------------------------------------------------------- #
def test_count_remittances_zero_when_empty():
assert global_store.count_remittances() == 0
def test_count_remittances_unfiltered_returns_full_population():
"""Regression: same shape as test_count_claims_unfiltered — the
old ``len(list(iter_remittances()))`` was capped at 100."""
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_batch(s, "b-remit-pop", base)
for i in range(120):
_seed_remit(
s, f"RMT-{i:04d}", f"b-{i:04d}",
received_at=base + timedelta(minutes=i),
)
s.commit()
assert global_store.count_remittances() == 120
def test_count_remittances_filters_by_claim_id():
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_batch(s, "b-cid", base)
_seed_remit(s, "RMT-A", "b-cid-A", received_at=base, claim_id="CLM-1")
_seed_remit(s, "RMT-B", "b-cid-B", received_at=base, claim_id="CLM-2")
_seed_remit(s, "RMT-C", "b-cid-C", received_at=base, claim_id=None)
s.commit()
assert global_store.count_remittances() == 3
assert global_store.count_remittances(claim_id="CLM-1") == 1
assert global_store.count_remittances(claim_id="CLM-2") == 1
assert global_store.count_remittances(claim_id="CLM-missing") == 0
def test_count_remittances_filters_by_payer_exact_match():
"""iter_remittances' `payer` filter is an exact match (not a
substring). Verify count_remittances mirrors that."""
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_batch(s, "b-rpayer", base)
for i in range(4):
_seed_remit(
s, f"RMT-CO-{i}", f"b-co-{i}",
received_at=base, payer_name="Colorado Medicaid",
)
for i in range(2):
_seed_remit(
s, f"RMT-AZ-{i}", f"b-az-{i}",
received_at=base, payer_name="Arizona Medicaid",
)
s.commit()
assert global_store.count_remittances(payer="Colorado Medicaid") == 4
assert global_store.count_remittances(payer="Arizona Medicaid") == 2
assert global_store.count_remittances(payer="colorado medicaid") == 0 # case-sensitive exact
# --------------------------------------------------------------------------- #
# HTTP regression: list endpoint `total` reflects full population
# --------------------------------------------------------------------------- #
def test_api_claims_total_reflects_full_population(client: TestClient):
"""Bug repro: seed 150 claims, hit /api/claims?limit=25, assert
``total == 150`` (not 25, not 100)."""
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_batch(s, "b-http-claims", base)
for i in range(150):
_seed_claim(s, f"CLM-HTTP-{i:04d}", "b-http-claims")
s.commit()
resp = client.get("/api/claims", params={"limit": 25})
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["total"] == 150, body
assert body["returned"] == 25
assert body["has_more"] is True
def test_api_remittances_total_reflects_full_population(client: TestClient):
"""Bug repro: seed 120 remits, hit /api/remittances?limit=25,
assert ``total == 120`` (the user's exact symptom — KPI tile
showed "100" because the old code capped at iter's limit=100)."""
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_batch(s, "b-http-remit", base)
for i in range(120):
_seed_remit(
s, f"RMT-HTTP-{i:04d}", f"b-h-{i:04d}",
received_at=base + timedelta(minutes=i),
)
s.commit()
resp = client.get("/api/remittances", params={"limit": 25})
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["total"] == 120, body
assert body["returned"] == 25
assert body["has_more"] is True
def test_api_remittances_total_zero_when_empty(client: TestClient):
"""Empty DB → total=0. (Sanity check the count path doesn't break
for the no-data case.)"""
resp = client.get("/api/remittances")
assert resp.status_code == 200
body = resp.json()
assert body == {"items": [], "total": 0, "returned": 0, "has_more": False}
def test_api_remittances_total_respects_filters(client: TestClient):
"""Filter by payer → total narrows correctly (covers the in-memory
payer filter on the count path)."""
base = datetime(2026, 6, 1, 12, 0, tzinfo=timezone.utc)
with db.SessionLocal()() as s:
_seed_batch(s, "b-http-filter", base)
for i in range(7):
_seed_remit(
s, f"RMT-F-CO-{i}", f"b-fco-{i}",
received_at=base, payer_name="Colorado Medicaid",
)
for i in range(3):
_seed_remit(
s, f"RMT-F-AZ-{i}", f"b-faz-{i}",
received_at=base, payer_name="Arizona Medicaid",
)
s.commit()
resp = client.get(
"/api/remittances",
params={"payer": "Colorado Medicaid", "limit": 100},
)
assert resp.status_code == 200
body = resp.json()
assert body["total"] == 7
assert body["returned"] == 7
assert body["has_more"] is False
backend/tests/test_parse_999.py
+32
View File
@@ -11,6 +11,12 @@ from cyclone.parsers.parse_999 import parse_999_text
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
# Gainwell's MFT ships the set-level accept/reject segment under the
# sender-specific id ``IK5`` instead of the spec-defined ``AK5`` (X12
# 005010X231A1). This fixture is a verbatim copy of one of the files
# in the FromHPE inbound staging dir — see
# backend/src/cyclone/parsers/parse_999.py for the rationale.
GAINWELL_IK5 = Path(__file__).parent / "fixtures" / "minimal_999_ik5_gainwell.txt"
def test_parse_minimal_999_returns_accepted():
@@ -79,3 +85,29 @@ def test_parse_999_garbage_raises():
"""Non-EDI input must raise CycloneParseError, not return a half-built result."""
with pytest.raises(CycloneParseError):
parse_999_text("not edi at all", input_file="bad.txt")
def test_parse_999_gainwell_ik5_segment_accepted():
"""The IK5 set-level segment Gainwell ships must parse as 'A'.
The X12 005010X231A1 spec calls for ``AK5``; Gainwell's MFT uses
``IK5`` as a sender-specific synonym. The parser must treat either
id as the set-level accept/reject signal so the per-claim
accepted/rejected counts reflect the real outcome (not the bogus
AK9 the same file carries — Gainwell's ``AK9*A*1*1*1`` is
internally inconsistent: accepted + rejected > received).
"""
text = GAINWELL_IK5.read_text()
result = parse_999_text(text, input_file=GAINWELL_IK5.name)
assert len(result.set_responses) == 1
s = result.set_responses[0]
assert s.set_accept_reject.code == "A"
assert s.transaction_set_identifier == "837"
assert s.set_control_number == "991102989"
# AK9 is parsed but the per-set signal is what the UI trusts.
assert result.functional_group_acks[0].ack_code == "A"
assert result.functional_group_acks[0].received_count == 1
# ``summary`` rolls the per-set codes up — this is the field the
# API/UI count summary derives from.
assert result.summary.passed == 1
assert result.summary.failed == 0
backend/tests/test_providers_seed.py
+2 -2
View File
@@ -54,8 +54,8 @@ def test_seed_creates_clearhouse_singleton():
assert ch.submitter_name == "Dzinesco"
assert ch.sftp_block.host == "mft.gainwelltechnologies.com"
assert ch.sftp_block.stub is True
assert "FromHPE" in ch.sftp_block.paths["outbound"]
assert "ToHPE" in ch.sftp_block.paths["inbound"]
assert "ToHPE" in ch.sftp_block.paths["outbound"]
assert "FromHPE" in ch.sftp_block.paths["inbound"]
def test_seed_creates_co_txix_payer_with_both_configs():
backend/tests/test_reconcile.py
+14 -2
View File
@@ -330,8 +330,20 @@ def test_run_reversal_flips_paid_to_reversed(fixture_835):
assert reversal_match.prior_claim_state == ClaimState.PAID
def test_run_failed_reconcile_writes_activity_event(fixture_835):
"""If reconcile crashes, the batch + remittances stay; activity event records failure."""
def test_run_reconcile_raise_in_session_leaves_prior_commits_alone(fixture_835):
"""A ``reconcile.run`` raise inside an open session does not damage
previously-committed rows.
The test seeds a batch + remit, commits them in one session, then
calls ``reconcile.run`` in a fresh session with ``match`` monkey-
patched to raise. The pre-existing rows must survive — they're on
disk from the prior commit, separate from the rolling-back
session. This pins the unit-level invariant that ``reconcile.run``
itself never commits and never silently mutates rows outside the
session it was given; it is the *caller's* responsibility (now
``CycloneStore.add`` per SP27 Task 10) to control the transaction
boundary.
"""
with db.SessionLocal()() as s:
_make_batch(s)
_make_remit(s, "CLP-1", "PCN-A", "1", "100.00", "100.00")
backend/tests/test_scheduler.py
+63 -5
View File
@@ -35,15 +35,15 @@ from cyclone.scheduler import (
@pytest.fixture
def sftp_block(tmp_path):
staging = tmp_path / "staging"
inbound_dir = staging / "ToHPE"
inbound_dir = staging / "FromHPE"
inbound_dir.mkdir(parents=True)
return SftpBlock(
host="mft.example.com",
port=22,
username="test",
paths={
"outbound": "/FromHPE",
"inbound": "/ToHPE",
"outbound": "/ToHPE",
"inbound": "/FromHPE",
},
stub=True,
staging_dir=str(staging),
@@ -55,7 +55,7 @@ def sftp_block(tmp_path):
@pytest.fixture
def _drop_file(sftp_block):
"""Helper: drop a named file in the inbound dir. Returns the path."""
inbound_dir = Path(sftp_block.staging_dir) / "ToHPE"
inbound_dir = Path(sftp_block.staging_dir) / "FromHPE"
def _drop(name: str, body: bytes) -> Path:
inbound_dir.mkdir(parents=True, exist_ok=True)
@@ -284,4 +284,62 @@ class TestRoutedFileTypes:
type. These tests are the regression net."""
def test_handlers_cover_all_routed_types(self):
assert set(HANDLERS.keys()) == ROUTED_FILE_TYPES
assert set(HANDLERS.keys()) == ROUTED_FILE_TYPES
class TestProcessInboundFiles:
"""Scheduler.process_inbound_files() — date-filtered pull path.
The /api/admin/scheduler/pull-inbound endpoint and the
``cyclone pull-inbound`` CLI both call this. It must:
* process each file in the provided list (no SFTP listdir)
* dedupe via ``processed_inbound_files`` (idempotent on rerun)
* not touch SFTP at all — files are expected to be on local
disk at ``f.local_path``
"""
@pytest.mark.asyncio
async def test_processes_provided_files_without_listdir(
self, sftp_block, _drop_file, tmp_path,
):
from cyclone.clearhouse import InboundFile
from cyclone.scheduler import STATUS_OK
# Drop a known 999 file on disk (the stub scheduler doesn't
# need SFTP — it reads from staging dir).
_drop_file(
"TP11525703-837P_M019048402-20260520231513488-1of1_999.x12",
b"not a real 999 -- handler will parse_error",
)
sched = _make_scheduler(sftp_block, tmp_path)
# Build the InboundFile records manually — caller is
# responsible for staging (mirrors the targeted-pull flow).
path = tmp_path / "staging" / "FromHPE" / \
"TP11525703-837P_M019048402-20260520231513488-1of1_999.x12"
files = [InboundFile(
name=path.name,
size=path.stat().st_size,
modified_at=datetime.fromtimestamp(path.stat().st_mtime, tz=timezone.utc),
local_path=path,
)]
result = await sched.process_inbound_files(files)
# The file's bytes are intentionally invalid — we only care
# that process_inbound_files invokes the handler and
# records the outcome (here: error from parse).
assert result.files_seen == 1
assert result.files_seen == result.files_processed + result.files_errored
# Idempotent: a second call is a no-op (already-processed dedup).
result2 = await sched.process_inbound_files(files)
assert result2.files_seen == 1
# Either skipped (because the prior call recorded it as error)
# or error — both prove the dedup branch fired. We just check
# it's not re-processed successfully.
with db.SessionLocal()() as s:
rows = s.query(ProcessedInboundFile).filter_by(
sftp_block_name="test-block",
name=path.name,
).all()
assert len(rows) == 1
assert rows[0].status in (STATUS_OK, STATUS_ERROR, STATUS_SKIPPED)
backend/tests/test_scheduler_status_errors.py
+229
View File
@@ -0,0 +1,229 @@
"""SP27 Task 9: scheduler status surfaces SFTP failures.
The 06/25 silent hang exposed a gap: ``Scheduler.status()`` had no
signal that polling had stalled. The operator's UI couldn't tell
"all quiet on the MFT front" from "we've been unable to reach the
MFT server for 3 hours". This pins the fix — the status dict
carries ``consecutive_failures``, ``last_error``, ``last_error_at``,
and ``last_sftp_attempt_at`` so the operator pill can flip to
destructive after 3 (or however many) consecutive failures.
Discriminator under test: ``TickResult.sftp_failed`` is the flag
``tick()`` keys off — NOT ``TickResult.errors``. Per-file processing
errors append to ``errors`` but must NOT bump ``consecutive_failures``.
The test suite pins both halves of the discriminator (SFTP failure
bumps; per-file failure doesn't).
"""
from __future__ import annotations
import asyncio
from datetime import datetime, timezone
import pytest
from cyclone.clearhouse import InboundFile
from cyclone.providers import SftpBlock
from cyclone.scheduler import Scheduler
def _block() -> SftpBlock:
return SftpBlock(
host="mft.example.com",
port=22,
username="user",
auth={"password_keychain_account": "x"},
paths={"inbound": "/inbound", "outbound": "/outbound"},
stub=True,
)
# ---- pre-state: a fresh scheduler has no errors -------------------------
def test_status_starts_with_zero_failures_and_no_error():
"""A fresh scheduler must report no errors — pins that the new
fields default cleanly and don't trip on first read."""
sched = Scheduler(_block())
status = sched.status()
assert status.consecutive_failures == 0
assert status.last_error is None
assert status.last_error_at is None
assert status.last_sftp_attempt_at is None
# ---- a failed SFTP call bumps consecutive_failures -----------------------
@pytest.mark.asyncio
async def test_status_records_last_error_after_failure():
"""A failed tick bumps consecutive_failures and records the error.
Simulates a broken SFTP client. Pins the post-Task-8 behavior
where ``_tick_impl`` catches the error and ``tick()`` records
it on the scheduler's status.
"""
class BrokenClient:
def list_inbound(self):
raise RuntimeError("simulated outage")
# SP27 Task 8: scheduler calls async_list_inbound() (the
# wait_for-wrapped variant). The broken stub raises before
# the wait_for wrapper runs — that's a real SFTP failure
# either way.
async def async_list_inbound(self):
raise RuntimeError("simulated outage")
sched = Scheduler(_block(), sftp_client_factory=lambda b: BrokenClient())
before = datetime.now(timezone.utc)
await sched.tick()
after = datetime.now(timezone.utc)
status = sched.status()
assert status.consecutive_failures == 1
assert status.last_error is not None
assert "simulated outage" in status.last_error
# last_error_at is set to roughly the time of the failed tick
assert status.last_error_at is not None
assert before <= status.last_error_at <= after
# last_sftp_attempt_at is also bumped — separate from last_error_at
# when the failure was an SFTP-side issue (not a parse error).
assert status.last_sftp_attempt_at is not None
assert before <= status.last_sftp_attempt_at <= after
# ---- a successful tick clears consecutive_failures -----------------------
@pytest.mark.asyncio
async def test_successful_tick_clears_consecutive_failures():
"""A tick that succeeds (or skips cleanly) resets the counter.
Without this, a transient blip would lock the destructive pill
on forever. Pins the recovery path.
"""
class BrokenClient:
def list_inbound(self):
raise RuntimeError("transient")
async def async_list_inbound(self):
raise RuntimeError("transient")
class HealthyClient:
def list_inbound(self):
return []
async def async_list_inbound(self):
return []
# First: two failing ticks to bump the counter
sched = Scheduler(_block(), sftp_client_factory=lambda b: BrokenClient())
await sched.tick()
await sched.tick()
assert sched.status().consecutive_failures == 2
# Then: switch to a healthy client and tick again
sched._sftp_client_factory = lambda b: HealthyClient()
await sched.tick()
status = sched.status()
assert status.consecutive_failures == 0
# last_error is preserved (audit trail) but last_error_at is from
# the last failure, not the success.
assert status.last_error is not None # the historical error stays
assert "transient" in status.last_error
# ---- last_sftp_attempt_at moves on every attempt, not just failures ---
@pytest.mark.asyncio
async def test_last_sftp_attempt_at_advances_on_every_tick():
"""Even a successful tick should bump last_sftp_attempt_at.
Without this, the operator can't tell when the scheduler last
*tried* to reach the MFT — only when it last failed.
"""
sched = Scheduler(_block()) # default stub returns []
before = datetime.now(timezone.utc)
await sched.tick()
after = datetime.now(timezone.utc)
first = sched.status().last_sftp_attempt_at
assert first is not None
assert before <= first <= after
# Wait a moment, tick again — last_sftp_attempt_at should move forward.
await asyncio.sleep(0.01)
await sched.tick()
second = sched.status().last_sftp_attempt_at
assert second is not None
assert second > first
# ---- as_dict surfaces the new fields for the API -----------------------
def test_status_as_dict_includes_new_fields():
"""The API endpoint (/api/health or /api/admin/scheduler) reads
status.as_dict() — the new fields must be present and serialized."""
sched = Scheduler(_block())
d = sched.status().as_dict()
assert "consecutive_failures" in d
assert d["consecutive_failures"] == 0
assert "last_error" in d
assert d["last_error"] is None
assert "last_error_at" in d
assert d["last_error_at"] is None
assert "last_sftp_attempt_at" in d
assert d["last_sftp_attempt_at"] is None
# ---- per-file processing errors must NOT bump consecutive_failures ------
#
# This pins the SP27 Task 9 discriminator (``TickResult.sftp_failed``
# vs. ``TickResult.errors``). Without this test, a future refactor
# that broadens ``result.errors`` semantics — or rewires `tick()` to
# key off it again — would silently re-break the discrimination.
@pytest.mark.asyncio
async def test_per_file_error_does_not_bump_consecutive_failures(monkeypatch):
"""Per-file processing errors must NOT flip the destructive pill.
The listing step succeeds but ``_handle_one`` simulates the
real handler's error path: it appends to ``result.errors``
(the way ``_handle_one`` does on parser exceptions) but does
NOT touch ``result.sftp_failed``. The pill must stay clean.
"""
class FilesClient:
async def async_list_inbound(self):
return [
InboundFile(
name="tp123-999_MTRACKING-20260525001606060-1of1.x12",
size=100,
modified_at=datetime.now(timezone.utc),
# ``local_path`` is None because fake_handle_one
# never reaches the download step.
local_path=None,
)
]
sched = Scheduler(_block(), sftp_client_factory=lambda b: FilesClient())
async def fake_handle_one(f, result):
# Mirrors the real ``_handle_one`` parser-failure path:
# appends to result.errors, does NOT set sftp_failed.
result.errors.append(f"{f.name}: ValueError: garbage")
monkeypatch.setattr(sched, "_handle_one", fake_handle_one)
await sched.tick()
status = sched.status()
# Counter stays clean — per-file error is not an SFTP outage.
assert status.consecutive_failures == 0
# ``last_error`` is reserved for SFTP-side failures.
assert status.last_error is None
# But ``last_sftp_attempt_at`` moves forward regardless — the
# scheduler did try, it just failed to ingest the file.
assert status.last_sftp_attempt_at is not None
backend/tests/test_sftp_op_timeout.py
+139
View File
@@ -0,0 +1,139 @@
"""Per-op timeout guard for SFTP operations (SP27 Task 8).
paramiko is synchronous; without a timeout, a hung ``listdir_attr``
freezes the worker thread indefinitely. The 06/25 silent hang was
exactly this — the MFT server TCP-acked but stopped responding, the
scheduler's ``asyncio.to_thread`` waited forever, and the operator
had no signal that polling had stalled.
This test pins the fix: the SFTP client's only async-wrapped call
site (``async_list_inbound``) applies an ``asyncio.wait_for(...
timeout=N)`` so the event loop can give up after the configured
bound. The bound is read from ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS``
at call time (default 30s).
The other SFTP methods (``list_inbound_names``, ``download_inbound``,
``read_file``, ``write_file``) intentionally remain sync — they're
called from operator-triggered paths (admin endpoints, CLI) where
the operator can Ctrl-C the request. Wrapping them would require
making the FastAPI handlers themselves async, which is out of
scope for Task 8. Tracked as a follow-up.
"""
from __future__ import annotations
import asyncio
import time
import pytest
from cyclone.clearhouse import SftpClient
from cyclone.providers import SftpBlock
def _block() -> SftpBlock:
return SftpBlock(
host="mft.example.com",
port=22,
username="user",
auth={"password_keychain_account": "x"},
paths={"inbound": "/inbound", "outbound": "/outbound"},
stub=True,
)
class _HangingClient(SftpClient):
"""SftpClient that hangs on list_inbound — exercises the
asyncio.wait_for timeout. The hang runs in a thread, so
``asyncio.to_thread`` alone would never give up; only the
wait_for wrapper can break it.
The hang is short (3s) so the per-test thread pool doesn't
accumulate stragglers across the timeout tests. Each test sets
``CYCLONE_SFTP_OP_TIMEOUT_SECONDS=1`` so wait_for fires well
before the sleep completes.
"""
def list_inbound(self):
time.sleep(3)
return []
# ---- env-var parsing --------------------------------------------------------
def test_op_timeout_default_is_30_seconds(monkeypatch):
"""Default is 30s when CYCLONE_SFTP_OP_TIMEOUT_SECONDS is unset.
Pins the documented default so a refactor that changes it has to
update both the constant and the spec.
"""
monkeypatch.delenv("CYCLONE_SFTP_OP_TIMEOUT_SECONDS", raising=False)
from cyclone.clearhouse import _op_timeout_seconds
assert _op_timeout_seconds() == 30.0
def test_op_timeout_reads_env_var(monkeypatch):
"""Operator can override via env var without a restart-rebuild."""
monkeypatch.setenv("CYCLONE_SFTP_OP_TIMEOUT_SECONDS", "7.5")
from cyclone.clearhouse import _op_timeout_seconds
assert _op_timeout_seconds() == 7.5
def test_op_timeout_rejects_unparseable_env_var(monkeypatch):
"""A non-float value (e.g. "30s") falls back to the default.
Pins the ValueError branch — a refactor that drops the try/except
would crash the SFTP wrapper on every call when the env var is
set to anything non-numeric.
"""
monkeypatch.setenv("CYCLONE_SFTP_OP_TIMEOUT_SECONDS", "30s")
from cyclone.clearhouse import _op_timeout_seconds
assert _op_timeout_seconds() == 30.0
def test_op_timeout_rejects_zero_and_negative(monkeypatch):
"""Zero / negative timeouts would silently turn every SFTP call
into an instant failure (asyncio.wait_for(timeout=0) raises
immediately; timeout<0 is undefined). They fall back to default.
"""
from cyclone.clearhouse import _op_timeout_seconds
for bad in ("0", "0.0", "-1", "-0.5"):
monkeypatch.setenv("CYCLONE_SFTP_OP_TIMEOUT_SECONDS", bad)
assert _op_timeout_seconds() == 30.0, f"value {bad!r} should fall back to default"
# ---- async wrapper behavior -------------------------------------------------
@pytest.mark.asyncio
async def test_async_list_inbound_times_out_when_paramiko_hangs(monkeypatch):
"""A hanging list_inbound must be cancelled by wait_for, not the thread.
The 06/25 silent hang was exactly this scenario — paramiko's
listdir_attr TCP-acked then went silent, freezing the worker
thread. The wait_for wrapper is the only thing that can break
out of that.
"""
monkeypatch.setenv("CYCLONE_SFTP_OP_TIMEOUT_SECONDS", "1")
client = _HangingClient(_block())
start = time.monotonic()
with pytest.raises(asyncio.TimeoutError):
await client.async_list_inbound()
elapsed = time.monotonic() - start
# Loose upper bound: timeout fires at ~1s, allow 3s for asyncio
# scheduling jitter on slow CI runners. If this ever fires at the
# hang duration (3s) the wait_for wrapper is missing.
assert elapsed < 3, f"timeout fired too late: {elapsed:.2f}s"
@pytest.mark.asyncio
async def test_async_list_inbound_returns_normally_when_fast(monkeypatch, tmp_path):
"""Sanity check: a non-hanging call returns the stub's empty list
without raising — pins that the wrapper doesn't break the happy path."""
monkeypatch.setenv("CYCLONE_SFTP_OP_TIMEOUT_SECONDS", "5")
block = _block()
block.staging_dir = str(tmp_path)
client = SftpClient(block)
# stub mode + no files staged = empty list, no hang, no timeout
result = await client.async_list_inbound()
assert result == []
backend/tests/test_sftp_paramiko.py
+62 -2
View File
@@ -40,8 +40,8 @@ def _block(
port=22,
username="testuser",
paths={
"outbound": "/CO XIX/PROD/test/FromHPE",
"inbound": "/CO XIX/PROD/test/ToHPE",
"outbound": "/CO XIX/PROD/test/ToHPE",
"inbound": "/CO XIX/PROD/test/FromHPE",
},
stub=stub,
staging_dir=staging_dir,
@@ -259,6 +259,66 @@ class TestRealModeListInbound:
assert len(files) == 1
assert files[0].name == "real.x12"
def test_list_skips_warn_txt_files(self, monkeypatch, tmp_path: Path):
# Gainwell's MFT drops advisory *_warn.txt files in the same
# inbound dir. They're text-format side-channel notes (not X12
# envelopes) and must be skipped at list time.
real_attr = MagicMock()
real_attr.filename = "TP123-837P_M456-20260520231513488-1of1_999.x12"
real_attr.st_mode = 0o100644
real_attr.st_size = 1024
real_attr.st_mtime = 1718899200
warn_attr = MagicMock()
warn_attr.filename = "TP123-837P-202606181208100000-1of1_warn.txt"
warn_attr.st_mode = 0o100644
warn_attr.st_size = 200
warn_attr.st_mtime = 1718899200
mock_ssh, mock_sftp = _make_mock_paramiko(
monkeypatch, sftp_attrs=[real_attr, warn_attr],
)
def _open(path, mode="rb"):
m = MagicMock()
m.__enter__.return_value = io.BytesIO(b"x12 content")
return m
mock_sftp.open.side_effect = _open
block = _block(staging_dir=str(tmp_path / "staging"))
client = SftpClient(block)
files = client.list_inbound()
names = [f.name for f in files]
assert "TP123-837P_M456-20260520231513488-1of1_999.x12" in names
assert not any(n.endswith("_warn.txt") for n in names)
assert len(files) == 1
def test_list_inbound_names_does_not_download(self, monkeypatch, tmp_path: Path):
# list_inbound_names() must do a metadata-only SFTP listing —
# no sftp.open() / no file written to the cache.
attr = MagicMock()
attr.filename = "TP123-837P_M456-20260520231513488-1of1_999.x12"
attr.st_mode = 0o100644
attr.st_size = 1024
attr.st_mtime = 1718899200
mock_ssh, mock_sftp = _make_mock_paramiko(
monkeypatch, sftp_attrs=[attr],
)
block = _block(staging_dir=str(tmp_path / "staging"))
client = SftpClient(block)
files = client.list_inbound_names()
# sftp.open() must NOT have been called (no download).
mock_sftp.open.assert_not_called()
assert len(files) == 1
# The InboundFile.local_path is set to the planned cache path
# but the file itself doesn't exist yet.
assert not files[0].local_path.exists()
class TestRealModeReadFile:
def test_read_returns_bytes(self, monkeypatch):
backend/tests/test_sftp_stub.py
+6 -6
View File
@@ -19,8 +19,8 @@ def sftp_block(tmp_path):
port=22,
username="colorado-fts\\coxix_prod_11525703",
paths={
"outbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
"inbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
"outbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
"inbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
},
stub=True,
staging_dir=str(staging),
@@ -55,7 +55,7 @@ def test_stub_list_inbound_empty_when_no_local_files(sftp_block):
def test_stub_list_inbound_returns_local_files(sftp_block):
# Simulate operator dropping a file in the inbound staging dir
inbound_dir = Path(sftp_block.staging_dir) / "CO XIX/PROD/coxix_prod_11525703/ToHPE"
inbound_dir = Path(sftp_block.staging_dir) / "CO XIX/PROD/coxix_prod_11525703/FromHPE"
inbound_dir.mkdir(parents=True, exist_ok=True)
(inbound_dir / "TP11525703-837P_M019048402-20260520231513488-1of1_999.x12").write_bytes(b"X")
client = SftpClient(sftp_block)
@@ -78,17 +78,17 @@ def test_stub_read_file_returns_bytes(sftp_block, tmp_path):
Lets the inbound scheduler exercise the same code path on a
workstation without a real MFT connection.
"""
inbound = tmp_path / "staging" / "ToHPE"
inbound = tmp_path / "staging" / "FromHPE"
inbound.mkdir(parents=True)
(inbound / "TP11525703-837P_M019048402-20260618130000000-1of1_TA1.x12").write_bytes(
b"hello-world",
)
client = SftpClient(sftp_block)
body = client.read_file("/ToHPE/TP11525703-837P_M019048402-20260618130000000-1of1_TA1.x12")
body = client.read_file("/FromHPE/TP11525703-837P_M019048402-20260618130000000-1of1_TA1.x12")
assert body == b"hello-world"
def test_stub_read_file_missing_raises(sftp_block):
client = SftpClient(sftp_block)
with pytest.raises(FileNotFoundError):
client.read_file("/ToHPE/does-not-exist.x12")
client.read_file("/FromHPE/does-not-exist.x12")
backend/tests/test_store_claim_detail.py
+10 -8
View File
@@ -354,10 +354,11 @@ def test_get_claim_detail_includes_state_history():
"""``stateHistory`` must include the manual_match event after pairing."""
s = CycloneStore()
_add_837_with_claim(s, "CLM-1")
# 835 with PCN that intentionally differs from the 837's member id so
# reconcile doesn't auto-pair on ingest — we want manual_match to do it.
_add_835_with_remit(s, pcn="CLM-1", paid="100", status="1")
s.manual_match("CLM-1", "CLM-1")
# 835 PCN deliberately differs from claim_id so the auto-matcher
# (which now joins on claim_id == pcn after the SP27 Task 17 PCN
# fix) doesn't pair them — manual_match needs work to do.
_add_835_with_remit(s, pcn="CLM-1-ORPHAN", paid="100", status="1")
s.manual_match("CLM-1", "CLM-1-ORPHAN")
out = s.get_claim_detail("CLM-1")
assert out is not None
@@ -377,7 +378,7 @@ def test_get_claim_detail_includes_state_history():
# ActivityEvent row, which is correct per the spec's
# ``{kind, ts, batchId|null, remittanceId|null}`` contract.
mm = next(ev for ev in history if ev["kind"] == "manual_match")
assert mm["remittanceId"] == "CLM-1"
assert mm["remittanceId"] == "CLM-1-ORPHAN"
assert mm["batchId"] is not None
assert mm["ts"].endswith("Z")
@@ -392,14 +393,15 @@ def test_get_claim_detail_includes_matched_remittance_summary():
"""A paired claim surfaces a ``matchedRemittance`` summary block."""
s = CycloneStore()
_add_837_with_claim(s, "CLM-1")
_add_835_with_remit(s, pcn="CLM-1", paid="100", status="1")
s.manual_match("CLM-1", "CLM-1")
# 835 PCN deliberately differs from claim_id so manual_match has work.
_add_835_with_remit(s, pcn="CLM-1-ORPHAN", paid="100", status="1")
s.manual_match("CLM-1", "CLM-1-ORPHAN")
out = s.get_claim_detail("CLM-1")
assert out is not None
mr = out["matchedRemittance"]
assert mr is not None
assert mr["id"] == "CLM-1"
assert mr["id"] == "CLM-1-ORPHAN"
assert mr["totalPaid"] == 100.0
assert isinstance(mr["totalPaid"], float)
# status_code "1" → "received" (per to_ui_remittance_from_orm mapping).
backend/tests/test_store_match_invariant.py
+434
View File
@@ -0,0 +1,434 @@
"""SP27 Task 11: matched-pair invariant.
The matched pair is a denormalized FK pair — ``Claim.matched_remittance_id``
and ``Remittance.claim_id`` — that MUST agree in both directions at every
moment. Three writers can mutate it:
1. ``CycloneStore.manual_match`` — operator override. SP27 Task 11 pins
it writes BOTH sides in one transaction.
2. ``CycloneStore.manual_unmatch`` — operator reversal. SP27 Task 11
pins it clears BOTH sides in one transaction.
3. ``reconcile.run`` (inside ``CycloneStore.add`` post Task 10) — auto-
match. Already pinned by the existing reconcile tests.
The drift check (``check_matched_pair_drift``) catches pre-existing
mismatches at startup and logs them at WARNING. Non-blocking.
These tests pin the invariants directly. They are explicitly redundant
with the integration tests in ``test_store_reconcile.py`` — the goal
here is a focused regression pin so a future refactor that touches
``manual_match`` / ``manual_unmatch`` (e.g. to defer the symmetric write
to a post-commit pass) doesn't silently break the invariant.
"""
from __future__ import annotations
from datetime import date, datetime, timezone
from decimal import Decimal
import pytest
from cyclone import db
from cyclone.parsers.models import (
BatchSummary, BillingProvider, ClaimHeader, ClaimOutput,
Envelope, Payer, ParseResult, Subscriber, ValidationReport,
)
from cyclone.parsers.models_835 import (
BatchSummary as BatchSummary835, ClaimAdjustment, ClaimPayment,
Envelope as Envelope835, FinancialInfo, ParseResult835, Payer835,
Payee835, ReassociationTrace, ServicePayment,
)
from cyclone.store import (
CycloneStore,
check_matched_pair_drift,
)
@pytest.fixture(autouse=True)
def _setup(tmp_path, monkeypatch):
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
db._reset_for_tests()
db.init_db()
yield
db._reset_for_tests()
def _build_claim(claim_id, pcn):
"""Build a single 837 claim + persist it through CycloneStore.add."""
from cyclone.store import BatchRecord837
co = ClaimOutput(
claim_id=claim_id,
control_number="0001",
transaction_date=date(2026, 6, 19),
billing_provider=BillingProvider(name="Test", npi="1234567890"),
subscriber=Subscriber(
first_name="Jane", last_name="Doe", member_id=pcn,
),
payer=Payer(name="Test Payer", id="P1"),
claim=ClaimHeader(
claim_id=claim_id, total_charge=Decimal("100"),
frequency_code="1", place_of_service="11",
),
diagnoses=[],
service_lines=[],
validation=ValidationReport(passed=True, errors=[], warnings=[]),
raw_segments=[],
)
pr837 = ParseResult(
envelope=Envelope(
sender_id="S", receiver_id="R", control_number="0001",
transaction_date=date(2026, 6, 19),
),
claims=[co],
summary=BatchSummary(
input_file="c.txt", control_number="0001",
transaction_date=date(2026, 6, 19),
total_claims=1, passed=1, failed=0,
),
)
CycloneStore().add(BatchRecord837(
id="b-837", kind="837p", input_filename="c.txt",
parsed_at=date(2026, 6, 19).isoformat() + "T12:00:00+00:00",
result=pr837,
))
def _build_remit(remit_id, pcn):
"""Build a single 835 remit + persist it through CycloneStore.add."""
from datetime import datetime, timezone
from cyclone.store import BatchRecord835
cp = ClaimPayment(
payer_claim_control_number=pcn,
status_code="1",
status_label="Primary",
total_charge=Decimal("124.00"),
total_paid=Decimal("100.00"),
service_payments=[
ServicePayment(
line_number=1, procedure_qualifier="HC", procedure_code="99213",
charge=Decimal("124.00"), payment=Decimal("100.00"),
adjustments=[ClaimAdjustment(
group_code="CO", reason_code="45",
amount=Decimal("24.00"),
)],
),
],
)
pr835 = ParseResult835(
envelope=Envelope835(
sender_id="S", receiver_id="R", control_number="0001",
transaction_date=date(2026, 6, 19),
),
financial_info=FinancialInfo(
handling_code="C", paid_amount=Decimal("0"),
credit_debit_flag="C", payment_method=None,
),
trace=ReassociationTrace(
trace_type_code="1", trace_number="0001",
originating_company_id="S",
),
payer=Payer835(name="X", id="SKCO0"),
payee=Payee835(name="Y", npi="1234567890"),
claims=[cp],
summary=BatchSummary835(
input_file="era.txt", control_number="0001",
transaction_date=date(2026, 6, 19),
total_claims=1, passed=1, failed=0,
),
)
CycloneStore().add(BatchRecord835(
id="b-835", kind="835", input_filename="era.txt",
parsed_at=datetime(2026, 6, 19, 12, 5, tzinfo=timezone.utc),
result=pr835,
))
# ---------------------------------------------------------------------------
# manual_match — writes BOTH sides in one transaction
# ---------------------------------------------------------------------------
def test_manual_match_writes_both_sides_in_one_transaction():
"""After ``manual_match`` returns, ``Claim.matched_remittance_id``
AND ``Remittance.claim_id`` both point at their pair.
Without the symmetric write, a later ``list_unmatched`` filter
(which keys off ``Remittance.claim_id IS NULL``) would surface
an already-matched remit as "orphan". Pre-T11 the symmetric
write was correct but only implicitly tested; this pins it
explicitly.
Note: claim PCN and remit PCN are DELIBERATELY DIFFERENT so the
auto-match inside ``CycloneStore.add`` (Task 10) doesn't pre-pair
them and ``manual_match`` becomes a no-op no-throw.
"""
from cyclone.db import Claim, Remittance
_build_claim("CLM-1", pcn="CLM-PATIENT-PCN")
_build_remit("CLP-1", pcn="CLP-1")
CycloneStore().manual_match("CLM-1", "CLP-1")
with db.SessionLocal()() as s:
claim = s.get(Claim, "CLM-1")
# Remit PK is payer_claim_control_number (see Remittance ORM).
from sqlalchemy import select
remit = s.execute(
select(Remittance).where(
Remittance.payer_claim_control_number == "CLP-1"
)
).scalar_one()
assert claim.matched_remittance_id == remit.id, (
f"claim.matched_remittance_id={claim.matched_remittance_id!r} "
f"!= remit.id={remit.id!r}"
)
assert remit.claim_id == claim.id, (
f"remit.claim_id={remit.claim_id!r} != claim.id={claim.id!r}"
)
# ---------------------------------------------------------------------------
# manual_unmatch — clears BOTH sides in one transaction
# ---------------------------------------------------------------------------
def test_manual_unmatch_clears_both_sides():
"""After ``manual_unmatch`` returns, BOTH ``Claim.matched_remittance_id``
AND ``Remittance.claim_id`` return to NULL.
Without the symmetric clear, an operator-initiated unmatch would
leave the orphaned remit pointing at the claim while the claim
is correctly cleared — ``list_unmatched`` would then NOT surface
the pair (remit side is non-NULL), even though the operator
intended to put it back in the unmatched bucket.
PCN asymmetry prevents auto-match from pre-pairing — see
``test_manual_match_writes_both_sides_in_one_transaction``.
"""
from sqlalchemy import select
from cyclone.db import Claim, Remittance
_build_claim("CLM-1", pcn="CLM-PATIENT-PCN")
_build_remit("CLP-1", pcn="CLP-1")
store = CycloneStore()
store.manual_match("CLM-1", "CLP-1")
store.manual_unmatch("CLM-1")
with db.SessionLocal()() as s:
claim = s.get(Claim, "CLM-1")
remit = s.execute(
select(Remittance).where(
Remittance.payer_claim_control_number == "CLP-1"
)
).scalar_one()
assert claim.matched_remittance_id is None
assert remit.claim_id is None
def test_manual_match_rollback_clears_symmetric_writes(monkeypatch):
"""If anything between the two writes raises, the session's
rollback MUST clear BOTH writes — neither side set.
Pins the SP27 Task 11 invariant "writes BOTH sides in one
transaction" at the rollback level. Without this pin, a future
refactor that pushes the symmetric write to a post-commit
handler (or splits ``manual_match`` across two sessions) would
still pass the happy-path test but leave one side set on a
failed match — the very class of drift Task 11 was added to
surface.
"""
from sqlalchemy import select
from cyclone.db import Claim, Remittance
from cyclone import reconcile
_build_claim("CLM-ROLL", pcn="CLM-PATIENT-PCN")
_build_remit("CLP-ROLL", pcn="CLP-ROLL")
# Force the second-loop line-reconcile pass to raise. By the time
# it runs, the claim + remit writes are staged in the session
# but not committed — the raise must trigger a rollback.
def boom(session, claim, remittance):
raise RuntimeError("simulated post-write fault")
monkeypatch.setattr(reconcile, "_reconcile_pair", boom)
with pytest.raises(RuntimeError, match="simulated post-write fault"):
CycloneStore().manual_match("CLM-ROLL", "CLP-ROLL")
with db.SessionLocal()() as s:
claim = s.get(Claim, "CLM-ROLL")
remit = s.execute(
select(Remittance).where(
Remittance.payer_claim_control_number == "CLP-ROLL"
)
).scalar_one()
# Neither side set — the session rolled back the partial
# state machine. claim.matched_remittance_id is the seeded
# NULL (no manual match); remit.claim_id is seeded NULL.
assert claim.matched_remittance_id is None
assert remit.claim_id is None
# ---------------------------------------------------------------------------
# Drift check — startup audit, non-blocking
# ---------------------------------------------------------------------------
def test_drift_check_returns_zero_for_clean_db():
"""A clean DB (no manual_match calls, no remits with claim_id)
reports zero drift and does not log any warning."""
_build_claim("CLM-1", pcn="PCN-A")
# No remits paired — no drift expected.
count = check_matched_pair_drift()
assert count == 0
def test_drift_check_logs_warning_for_mismatched_pair(caplog):
"""Inject drift directly via SQL (bypassing store so we can build
a state the writers can never produce), call the check, assert
a WARNING was logged with the offending ids.
The drift check is read-only and non-blocking — operators can
clean up manually. Logging at WARNING makes it visible in the
standard boot log without paging anyone.
This case is the inverse direction: a REMIT points at a claim
whose ``matched_remittance_id`` doesn't point back. Pairs with
``test_drift_check_logs_warning_for_case_a_mismatch`` to cover
both SQL paths through the check.
"""
from cyclone.db import Claim, ClaimState, Remittance
from sqlalchemy import select
# Build the inverse state: a remit whose claim_id points at a
# claim that does NOT have matched_remittance_id back. Only Case
# B fires; Case A (claim → remit) is clean by construction.
with db.SessionLocal()() as s:
s.add(Remittance(
id="orphan-remit", batch_id="b1",
payer_claim_control_number="PCN-B",
status_code="1", total_charge=Decimal("100"),
total_paid=Decimal("100"),
received_at=datetime.now(timezone.utc),
service_date=None, is_reversal=False,
claim_id="orphan-claim", # ← only this side set
))
s.add(Claim(
id="orphan-claim", batch_id="b1",
patient_control_number="PCN-B-CLAIM",
service_date_from=None,
charge_amount=Decimal("100"),
state=ClaimState.SUBMITTED,
matched_remittance_id=None, # ← back-pointer missing
))
s.commit()
with caplog.at_level("WARNING", logger="cyclone.store"):
count = check_matched_pair_drift()
assert count == 1, "exactly one drift was injected"
# The warning log should mention both ids so the operator can grep.
assert any(
"orphan-claim" in record.message
and "orphan-remit" in record.message
for record in caplog.records
), "drift warning should mention the offending claim + remit ids"
def test_drift_check_logs_warning_for_case_a_mismatch(caplog):
"""Pins the OTHER direction: claim → remit drift where the
claim's matched_remittance_id points at a remit whose claim_id
doesn't point back.
Without this test (paired with the inverse test above) a SQL
typo in Case A or Case B would silently pass the suite.
"""
from cyclone.db import Claim, ClaimState, Remittance
with db.SessionLocal()() as s:
# Remit exists but its claim_id is NULL (claim side is the
# only one pointing).
s.add(Remittance(
id="dangle-remit", batch_id="b1",
payer_claim_control_number="PCN-A",
status_code="1", total_charge=Decimal("100"),
total_paid=Decimal("100"),
received_at=datetime.now(timezone.utc),
service_date=None, is_reversal=False,
claim_id=None,
))
s.add(Claim(
id="dangle-claim", batch_id="b1",
patient_control_number="PCN-A",
service_date_from=None,
charge_amount=Decimal("100"),
state=ClaimState.SUBMITTED,
matched_remittance_id="dangle-remit",
))
s.commit()
with caplog.at_level("WARNING", logger="cyclone.store"):
count = check_matched_pair_drift()
assert count == 1
assert any(
"dangle-claim" in record.message and "dangle-remit" in record.message
for record in caplog.records
)
def test_drift_check_returns_count_of_mismatched_pairs():
"""The return value is the number of mismatched pairs so operators
can alert on it (e.g. fail boot if drift > 0 in a future iteration)."""
from cyclone.db import Claim, ClaimState, Remittance
with db.SessionLocal()() as s:
# Two drifted pairs: case A and case B in a single DB.
for i, (claim_id, remit_id) in enumerate(
[("drift-A-claim", "drift-A-remit"), ("drift-B-claim", "drift-B-remit")]
):
s.add(Remittance(
id=remit_id, batch_id="b1",
payer_claim_control_number=f"PCN-{i}",
status_code="1", total_charge=Decimal("100"),
total_paid=Decimal("100"),
received_at=datetime.now(timezone.utc),
service_date=None, is_reversal=False,
))
s.add(Claim(
id=claim_id, batch_id="b1",
patient_control_number=f"PCN-{i}",
service_date_from=None,
charge_amount=Decimal("100"),
state=ClaimState.SUBMITTED,
matched_remittance_id=remit_id,
))
s.commit()
count = check_matched_pair_drift()
assert count == 2
def test_drift_check_does_not_raise_on_query_error(monkeypatch):
"""The drift check is non-blocking at boot — if a DB query fails
we want a logged exception, NOT a crashed backend. Wrapped at the
api.py call site (Task 11 wiring); this pins that the function
itself doesn't try/catch (the call site does).
"""
from sqlalchemy.exc import OperationalError
# Force the SessionLocal to raise on use.
def boom():
raise OperationalError("SELECT 1", {}, Exception("synthetic"))
monkeypatch.setattr(db, "SessionLocal", lambda: boom)
with pytest.raises(OperationalError):
check_matched_pair_drift()
# The exception propagates — api.py::lifespan wraps it with a
# try/except so the boot continues. The pin here is that the
# function doesn't swallow by accident.
backend/tests/test_store_reconcile.py
+174 -5
View File
@@ -357,8 +357,11 @@ def test_manual_match_populates_line_reconciliation_rows():
))
# 835 remit with one SVC line that matches claim line 1.
# pcn deliberately ≠ claim_id so auto-match (which now joins on
# claim_id == pcn after the SP27 Task 17 PCN fix) doesn't pair them
# and the claim lands in the unmatched list for manual pairing.
cp = ClaimPayment(
payer_claim_control_number="CLM-MANUAL", # != member_id "MANUAL"
payer_claim_control_number="ORPHAN-PCN-MANUAL",
status_code="1",
status_label="Primary",
total_charge=Decimal("200.00"),
@@ -498,10 +501,10 @@ def test_manual_match_idempotent_line_reconciliation():
result=pr837,
))
# 835 with two SVC lines — auto-match won't pair (PCN="CLM-IDEMP" ≠
# member_id="IDEMP"), so manual_match has work to do.
# 835 with two SVC lines — auto-match won't pair (pcn="ORPHAN-IDEMP"
# ≠ claim_id="CLM-IDEMP"), so manual_match has work to do.
cp = ClaimPayment(
payer_claim_control_number="CLM-IDEMP",
payer_claim_control_number="ORPHAN-IDEMP",
status_code="1",
status_label="Primary",
total_charge=Decimal("200.00"),
@@ -586,4 +589,170 @@ def test_manual_match_idempotent_line_reconciliation():
r = session.get(Remittance, remit_id)
assert r is not None
assert r.adjustment_amount == Decimal("20.00")
assert r.claim_level_adjustment_amount == Decimal("20.00")
assert r.claim_level_adjustment_amount == Decimal("20.00")
# ---------------------------------------------------------------------------
# SP27 Task 17: Claim.patient_control_number must be the CLM01 (claim_id)
# the 837 sent, NOT the 2010BA subscriber member_id. The reconcile matcher
# joins Claim.patient_control_number == Remittance.payer_claim_control_number,
# and the 835 echoes CLM01 in CLP01 per X12 spec. Storing member_id here
# silently breaks every auto-match in production (the 9/1,739 "matches"
# observed in prod data are substring coincidences between synthetic CLM01
# and the human-readable PCN).
# ---------------------------------------------------------------------------
def _make_837_result(claims):
"""Build a minimal ParseResult with the given ClaimOutputs."""
from cyclone.parsers.models import (
BatchSummary as BatchSummary837,
BillingProvider, ClaimHeader, ClaimOutput, Envelope,
Payer, ParseResult, Subscriber, ValidationReport,
)
return ParseResult(
envelope=Envelope(
sender_id="S", receiver_id="R", control_number="0001",
transaction_date=date(2026, 6, 19),
),
claims=claims,
summary=BatchSummary837(
input_file="c.txt", control_number="0001",
transaction_date=date(2026, 6, 19),
total_claims=len(claims), passed=len(claims), failed=0,
),
)
def _make_claim_output(claim_id="CLM-A", member_id="MEMBER-X",
charge="100.00",
service_date=date(2026, 6, 19)):
"""Single ClaimOutput with distinct claim_id and subscriber.member_id.
``service_date`` populates the SV1 service-line date so the
reconcile matcher's ``_pick_claim`` window check can find this
claim when the matching 835 has the same date in SVC.
"""
from cyclone.parsers.models import (
BillingProvider, ClaimHeader, ClaimOutput,
Subscriber, Payer, Procedure, ServiceLine,
ValidationReport,
)
return ClaimOutput(
claim_id=claim_id,
control_number="0001",
transaction_date=date(2026, 6, 19),
billing_provider=BillingProvider(name="Test", npi="1234567890"),
subscriber=Subscriber(first_name="Jane", last_name="Doe",
member_id=member_id),
payer=Payer(name="Test Payer", id="P1"),
claim=ClaimHeader(claim_id=claim_id, total_charge=Decimal(charge),
frequency_code="1", place_of_service="11"),
diagnoses=[],
service_lines=[
ServiceLine(
line_number=1,
procedure=Procedure(qualifier="HC", code="99213",
modifiers=[]),
charge=Decimal(charge),
unit_type="UN",
units=Decimal("1"),
service_date=service_date,
),
],
validation=ValidationReport(passed=True, errors=[], warnings=[]),
raw_segments=[],
)
def test_837_ingest_populates_patient_control_number_from_claim_id():
"""Claim.patient_control_number == claim_id (CLM01), not member_id.
RED: today the 837 ingest writes ``Claim.patient_control_number =
claim.subscriber.member_id`` (store.py:194). That breaks the
reconcile join against the 835's CLP01 (which echoes CLM01).
"""
from cyclone.db import Claim
from cyclone.store import BatchRecord837
s = CycloneStore()
s.add(BatchRecord837(
id="b-837-pcn", kind="837p", input_filename="c.txt",
parsed_at=datetime(2026, 6, 19, tzinfo=timezone.utc),
result=_make_837_result([
_make_claim_output(claim_id="CLM-A", member_id="MEMBER-X"),
]),
))
with db.SessionLocal()() as session:
claim = session.get(Claim, "CLM-A")
assert claim is not None
assert claim.patient_control_number == "CLM-A", (
f"expected CLM01 (claim_id) 'CLM-A', "
f"got Claim.patient_control_number={claim.patient_control_number!r} "
f"(should be the value the 837 sent in CLM01, not the "
f"subscriber's 2010BA NM109 member_id)."
)
def test_837_then_835_with_echoed_pcn_auto_pairs():
"""End-to-end: 837 CLM01='CLM-A' + 835 CLP01='CLM-A' must auto-match.
RED: today ``Claim.patient_control_number == subscriber.member_id``
so the join on ``Claim.patient_control_number ==
Remittance.payer_claim_control_number`` fails even when the payer
echoes CLM01 correctly.
"""
from cyclone.db import Claim, Remittance
from cyclone.store import BatchRecord837, BatchRecord835
s = CycloneStore()
# 837: CLM01 = 'CLM-A', subscriber.member_id = 'MEMBER-X' (different)
s.add(BatchRecord837(
id="b-837-echo", kind="837p", input_filename="c.txt",
parsed_at=datetime(2026, 6, 19, tzinfo=timezone.utc),
result=_make_837_result([
_make_claim_output(claim_id="CLM-A", member_id="MEMBER-X",
charge="100.00"),
]),
))
# 835: CLP01 = 'CLM-A' (the payer echoes CLM01 back), paid == charge
cp = ClaimPayment(
payer_claim_control_number="CLM-A",
status_code="1", status_label="Primary",
total_charge=Decimal("100.00"), total_paid=Decimal("100.00"),
service_payments=[
ServicePayment(
line_number=1, procedure_qualifier="HC",
procedure_code="99213",
charge=Decimal("100.00"), payment=Decimal("100.00"),
units=Decimal("1"),
service_date=date(2026, 6, 19),
adjustments=[],
),
],
)
s.add(BatchRecord835(
id="b-835-echo", kind="835", input_filename="era.txt",
parsed_at=datetime(2026, 6, 19, tzinfo=timezone.utc),
result=_make_835_result([cp]),
))
with db.SessionLocal()() as session:
claim = session.get(Claim, "CLM-A")
r = session.query(Remittance).filter(
Remittance.payer_claim_control_number == "CLM-A"
).first()
# Both sides must have the FK set: Claim.matched_remittance_id
# AND Remittance.claim_id, written in the same transaction.
assert claim.matched_remittance_id is not None, (
"Claim should be auto-matched to the remit (CLM01 echoed "
"in CLP01), but matched_remittance_id is still NULL. The "
"837 ingest is storing patient_control_number from "
"subscriber.member_id instead of claim.claim_id."
)
assert r.claim_id == "CLM-A"
assert r.claim_id == claim.id
assert claim.state == "paid"
backend/tests/test_store_update_clearhouse.py
+2 -2
View File
@@ -27,8 +27,8 @@ def test_update_clearhouse_round_trip():
port=22,
username="colorado-fts\\coxix_prod_11525703",
paths={
"outbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
"inbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
"outbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
"inbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
},
stub=False,
staging_dir="./var/sftp/staging",
docker-compose.override.yml
+2
View File
@@ -15,6 +15,8 @@ secrets:
file: /tmp/cyclone-test-secrets/admin_username
cyclone_admin_password:
file: /tmp/cyclone-test-secrets/admin_pw
cyclone_sftp_password:
file: /tmp/cyclone-test-secrets/sftp_password
services:
frontend:
docker-compose.yml
+5 -4
View File
@@ -45,10 +45,11 @@ services:
CYCLONE_ADMIN_PASSWORD_FILE: "/run/secrets/cyclone_admin_password"
# SP26 — SFTP password via Docker secret (matches the admin-creds pattern).
CYCLONE_SFTP_PASSWORD_FILE: "/run/secrets/cyclone_sftp_password"
# Bind 0.0.0.0 so the frontend container on the compose bridge
# network can reach us. The bridge network provides isolation —
# only the `frontend` service is on it; the host firewall still
# blocks anything that isn't on the LAN.
# Always bind to 0.0.0.0. The compose-managed bridge network
# isolates the backend from the host's LAN/WAN — only the
# `frontend` service joins it. Host firewall / port publishing
# is the layer that controls what reaches us from outside the
# compose network, not the bind address.
CYCLONE_HOST: "0.0.0.0"
secrets:
- cyclone_db_key
docs/ARCHITECTURE.md
+1 -1
View File
@@ -678,7 +678,7 @@ Returns `status="ok"` only when every subsystem is healthy. `status="degraded"`
The default deployment. Two terminals:
```bash
# Terminal 1 — backend on 127.0.0.1:8000
# Terminal 1 — backend on 0.0.0.0:8000 (always binds to all interfaces; firewall is what restricts reachability)
cd backend
.venv/bin/python -m cyclone serve
docs/REQUIREMENTS.md
+1 -1
View File
@@ -151,7 +151,7 @@ Each requirement is `FR-NN`, traceable to one or more sub-projects. Verification
| ID | NFR | Source / SP |
|---|---|---|
| NFR-1 | **Local-only bind + auth.** Backend binds `127.0.0.1:8000` (overridable via `CYCLONE_PORT`); CORS allowlist is exact (`http://localhost:5173`); login required (bcrypt + HttpOnly session cookie; first admin bootstrapped from env vars `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD`); dev/test escape hatch via `CYCLONE_AUTH_DISABLED=1` (logs WARNING at boot). | SP1 + auth (2026-06-23 merge) + SP24 (doc reconciliation) |
| NFR-1 | **Always bind 0.0.0.0 + auth.** Backend binds `0.0.0.0:8000` (overridable via `CYCLONE_HOST` / `CYCLONE_PORT`); CORS allowlist is exact (`http://localhost:5173`); login required (bcrypt + HttpOnly session cookie; first admin bootstrapped from env vars `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD`); dev/test escape hatch via `CYCLONE_AUTH_DISABLED=1` (logs WARNING at boot). Reachability is controlled by the host firewall / compose port publishing, not the bind address. | SP1 + auth (2026-06-23 merge) + SP24 (doc reconciliation) |
| NFR-2 | **Determinism.** Parser / serializer round-trip is guaranteed on 113 real prodfiles (`docs/prodfiles/claims/*.x12`); canonical fields, not byte-identity (called out in the SP8 spec). | SP8 |
| NFR-3 | **Audit completeness.** Every reconciliation anomaly, every state transition, every 999/277CA reject writes an `ActivityEvent` (in `activity_events` table, un-chained; powers the `/activity` feed and inbox state transitions). | SP2 + SP10 |
| NFR-4 | **Tamper-evidence.** A separate `audit_log` table (SP11) carries SHA-256 hash-chained rows for security-sensitive events (login attempts, rejections, key rotations, backup lifecycle, rejected requests). `GET /api/admin/audit-log/verify` detects any break. **The `activity_events` (NFR-3) and `audit_log` (NFR-4) tables are distinct** — different semantics, different audiences. New code that needs an audit row must decide which one to write to. | SP11 |
docs/reference/co-medicaid.md
+6 -6
View File
@@ -25,8 +25,8 @@ the 837P file.
| Submitter contact | Tyler Martinez <tyler@dzinesco.com> |
| SFTP host | `mft.gainwelltechnologies.com` |
| SFTP username | `colorado-fts\coxix_prod_11525703` |
| SFTP outbound dir | `/CO XIX/PROD/coxix_prod_11525703/FromHPE` |
| SFTP inbound dir | `/CO XIX/PROD/coxix_prod_11525703/ToHPE` |
| SFTP outbound dir | `/CO XIX/PROD/coxix_prod_11525703/ToHPE` |
| SFTP inbound dir | `/CO XIX/PROD/coxix_prod_11525703/FromHPE` |
## dzinesco's 3 billing-provider NPIs
@@ -49,8 +49,8 @@ dzinesco submits 837P files to Gainwell's MFT (Managed File Transfer)
at `mft.gainwelltechnologies.com`. The full SFTP path layout is
specified by the user (2026-06-20):
- **Outbound** (we send): `/CO XIX/PROD/coxix_prod_11525703/FromHPE`
- **Inbound** (HPE/Gainwell sends to us): `/CO XIX/PROD/coxix_prod_11525703/ToHPE`
- **Outbound** (we send): `/CO XIX/PROD/coxix_prod_11525703/ToHPE`
- **Inbound** (HPE/Gainwell sends to us): `/CO XIX/PROD/coxix_prod_11525703/FromHPE`
### File naming
@@ -107,8 +107,8 @@ curl -X POST http://localhost:8000/api/clearhouse/submit \
-H 'Content-Type: application/json' \
-d '{"claim_ids": ["CLM-1", "CLM-2"], "payer_id": "CO_TXIX"}'
# Files appear at ./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/
ls -la "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/"
# Files appear at ./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/
ls -la "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/"
```
## Payer IDs
docs/superpowers/plans/2026-06-29-cyclone-remittances-architecture-refactor.md
+2158
View File
File diff suppressed because it is too large Load Diff
docs/superpowers/specs/2026-06-20-cyclone-multi-payer-npi-sftp-design.md
+7 -7
View File
@@ -21,7 +21,7 @@ Replace the single hard-coded `PayerConfig` factory dict (currently in `api.py:9
- **SP10** — 277CA parser + "Payer-Rejected" lane in the Inbox.
- **SP11** — Tamper-evident hash-chained `audit_log` table.
- **SP12** — SQLCipher encryption at rest + Keychain-stored DB key.
- **SP13** — Replace the SFTP stub with real `paramiko` connection to `mft.gainwelltechnologies.com` and actually push to `/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
- **SP13** — Replace the SFTP stub with real `paramiko` connection to `mft.gainwelltechnologies.com` and actually push to `/CO XIX/PROD/coxix_prod_11525703/ToHPE`.
- **Real SFTP credentials in Keychain** — schema and call sites are in place; the actual secret is created manually by the operator.
## 2. Goals
@@ -109,8 +109,8 @@ VALUES (1, 'dzinesco', '11525703', 'Dzinesco', 'Tyler Martinez', 'tyler@dzinesco
'"inbound_template":"TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12"}',
'{"host":"mft.gainwelltechnologies.com","port":22,'
'"username":"colorado-fts\\coxix_prod_11525703",'
'"paths":{"outbound":"/CO XIX/PROD/coxix_prod_11525703/FromHPE",'
'"inbound":"/CO XIX/PROD/coxix_prod_11525703/ToHPE"},'
'"paths":{"outbound":"/CO XIX/PROD/coxix_prod_11525703/ToHPE",'
'"inbound":"/CO XIX/PROD/coxix_prod_11525703/FromHPE"},'
'"stub":true,"staging_dir":"./var/sftp/staging","poll_seconds":300}',
'2026-06-20T00:00:00Z');
```
@@ -163,13 +163,13 @@ All 3 share the same address because all 3 are registered to the Montrose corpor
Per the **HCPF X12 File Naming Standards Quick Guide** (https://hcpf.colorado.gov/tp-x12-filenaming):
**Outbound** (we send to `/CO XIX/PROD/coxix_prod_11525703/FromHPE`):
**Outbound** (we send to `/CO XIX/PROD/coxix_prod_11525703/ToHPE`):
```
{tpid}-{transaction_type}-{yyyymmddhhmmssSSS_MT}-1of1.{ext}
```
Example: `11525703-837P-20260620132243505-1of1.x12`
**Inbound** (HPE sends to `/CO XIX/PROD/coxix_prod_11525703/ToHPE`):
**Inbound** (HPE sends to `/CO XIX/PROD/coxix_prod_11525703/FromHPE`):
```
TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
```
@@ -209,7 +209,7 @@ Handler:
"ok": true,
"submitted": [
{"claim_id": "CLM-001", "filename": "11525703-837P-20260620132243505-1of1.x12",
"staging_path": "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/11525703-..."}
"staging_path": "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/11525703-..."}
],
"stub": true
}
@@ -367,7 +367,7 @@ All R200-R210 run on parse AND on serialize, so a 837 file with an unknown NPI f
| # | Question | Resolution |
|---|----------|-----------|
| 1 | SFTP outbound + inbound paths on `mft.gainwelltechnologies.com` | `/CO XIX/PROD/coxix_prod_11525703/FromHPE` (out), `/CO XIX/PROD/coxix_prod_11525703/ToHPE` (in) — user-provided 2026-06-20 |
| 1 | SFTP outbound + inbound paths on `mft.gainwelltechnologies.com` | `/CO XIX/PROD/coxix_prod_11525703/ToHPE` (out), `/CO XIX/PROD/coxix_prod_11525703/FromHPE` (in) — user-provided 2026-06-20 |
| 2 | 3 NPI street addresses + ZIPs | All 3 NPIs share Montrose corporate address (user confirmed 2026-06-20) — seed from prod files |
| 3 | 3 NPI taxonomy codes | `251E00000X` for all 3, self-served from 136 prod files |
| 4 | 277CA filename suffix | Use `277` per HCPF doc; distinguish 277CA by `ST*277CA` content (user confirmed 2026-06-20) |
docs/superpowers/specs/2026-06-29-cyclone-remittances-architecture-refactor-design.md
+391
View File
@@ -0,0 +1,391 @@
# Sub-project 27 — Remittances Architecture Refactor: Design Spec
**Date:** 2026-06-29
**Status:** Draft, awaiting user sign-off
**Branch:** `sp27-remittances-architecture-refactor`
**Aesthetic direction:** No new UI (chain view rendered into existing ClaimDrawer; no new page)
## 1. Scope
Cyclone today ships the full claim/ack/remit cycle end-to-end: outbound
837 submitted via SFTP, inbound 999 / 277CA / 835 / TA1 polled and parsed,
remits matched against claims. Live data is now flowing on the production
system (~60K remits, ~542K 999 acks since 06/19). The architecture serving
that cycle has accumulated rough edges that an afternoon of operational
pressure surfaced:
- **Scheduler is fragile.** A 06/25 silent hang (no error logged) took
the MFT poll down for ~3 days; recovered only because the operator
restarted the backend. Root cause: `sftp.listdir_attr()` runs without
a socket timeout, so when the MFT TCP-acks but stops responding the
thread blocks indefinitely. The scheduler's status surface is too
shallow to detect this in real time.
- **Filename classification is brittle.** `edi/filenames.py:INBOUND_RE`
requires a `_file_type.x12` suffix on inbound files. Gainwell's
actual 835s ship without that suffix (see the 5×835 batch from
6/15–6/19), so the only thing the scheduler could do was mark them
`skipped`. Operators had to ingest those by hand.
- **Helpers duplicate API logic.** `cyclone.scheduler._ack_count_summary`
and `_ack_synthetic_source_batch_id` copy logic from
`cyclone.api` with a comment saying so. Two places to keep in sync.
- **Reconciliation is decoupled from ingest.** 835 ingest writes a
`Remittance` with `adjustment_amount=0`, and a separate
`reconcile.run()` pass later sums `CasAdjustment.amount` rows and
overwrites `adjustment_amount`. The UI sees stale values until the
pass runs, and the two phases can drift if reconcile is skipped or
crashes mid-way.
- **No unified claim-chain view.** A claim's 837 submission, 999 ack,
277CA status, and 835 remit are reconstructed by joining across four
pages and three queries. The "did this remit's claim ever get
matched?" question requires manual SQL.
### In scope
Tier 1 (scheduler/ingest split, SFTP hardening, filename classifier):
1. Move the four per-file-type handlers (`999`, `835`, `277CA`, `TA1`)
out of `backend/src/cyclone/scheduler.py` into
`backend/src/cyclone/handlers/<type>.py`. Each handler exposes one
`handle(text, source_file) -> (parser_used, claim_count)` function.
2. Pull `_ack_count_summary`, `_ack_synthetic_source_batch_id`, and
`_277ca_synthetic_source_batch_id` out of `scheduler.py` and out of
`api.py` into a new `backend/src/cyclone/handlers/_ack_id.py`. Both
callers import from one module.
3. Loosen `edi/filenames.py:parse_inbound_filename` to accept filenames
lacking `_file_type.x12`; fall back to `orig_tx` as the
provisional `file_type`, and accept the file as either `835` (if
`orig_tx` ends in `835`) or `999`/`277CA` (if `orig_tx` does).
4. Wrap `clearhouse.SftpClient._list_inbound_paramiko` /
`_list_inbound_names_paramiko` / `_download_inbound_paramiko` /
`_read_file_paramiko` calls in `asyncio.wait_for(... timeout=N)`
via `asyncio.to_thread`, with `N` configurable via
`CYCLONE_SFTP_OP_TIMEOUT_SECONDS` (default 30s).
5. Surface SFTP errors in `Scheduler.status()` — `last_error_at`,
`last_error`, `consecutive_failures`, `last_sftp_attempt_at`.
When `consecutive_failures >= 3`, the operator pill flips to a
destructive state until a successful tick clears it.
6. Stop swallowing `IOError` from `sftp.listdir_attr()` — return the
error in `result.errors` and let the scheduler record `last_error`.
Tier 2 (reconciliation atomicity, chain view, invariant guards):
7. Unify 835 ingest + `reconcile.run()` into a single critical
section inside `cyclone.handlers.handle_835`. The whole flow
(parse → validate → persist batch + remits + CasAdjustments → match
claims → write back `adjustment_amount` and `matched_remittance_id`)
happens in one `db.SessionLocal()()`. The `Remittance.adjustment_amount`
is computed in the same transaction as the `CasAdjustment` rows.
8. Add `GET /api/claims/{id}/chain` returning the claim's full chain
in one response: `submission` (837 → `Claim` row),
`ack_999` (matching `Ack` rows), `ack_277ca` (matching
`Two77caAck` rows), `remittance` (`Remittance` row including
adjustments and matched claim FK). Empty slots when a piece is
missing — never 404 the whole endpoint just because one piece is
absent.
9. The 999 ack handler and 277CA ack handler — when they reject a
claim, and the claim is already matched to a remit — emit an
`ActivityEvent` `claim.rejected_after_remit` so the operator
surfaces the conflict in the Activity page rather than silently
breaking the manual-match workflow.
10. Verify the denormalized pair `Claim.matched_remittance_id` ↔
`Remittance.claim_id` is written in the same transaction on every
write-path (`store.manual_match`, `store.manual_unmatch`, the new
atomic 835 handler). Add a startup invariant check: if any Claim
has `matched_remittance_id` but the matching Remittance's
`claim_id` is unset (or vice versa), log a structured error so an
operator sees the drift in the logs.
### Out of scope (deferred to future increments)
- **Frontend scheduler operator UI** (start/stop/tick/status page,
reconnect button, live log tail). The `/api/admin/scheduler/*`
JSON endpoints remain sufficient; the operator uses curl.
- **Multi-tenant SFTP blocks** (per-payer or per-tenant). Single
dzinesco block remains.
- **Docker-secret `_FILE` conventions** beyond what SP26 already added.
- **Outbound file pickup verification** (polling HPE's MFT to
confirm the operator-dropped 837 was received).
- **Frontend chain drawer component.** The `GET /api/claims/{id}/chain`
endpoint is exposed; rendering it inside the existing
`ClaimDrawer` is a separate UI increment.
## 2. Decisions (locked during brainstorming)
1. **Branch base is `Version-1.0.0`, not `main`.** `main` is at
`74aa64f` (SP26 only) and is missing the SP25+26 SFTP work, the
`[sftp]` docker extra, and the permission-matrix fixes. The
production system runs `Version-1.0.0`, so the refactor lands
there. A subsequent forward-merge from `Version-1.0.0` → `main`
can pull SP27 in if the operator wants it on main.
2. **One module per handler, not a class.** Each handler exposes one
pure function `handle(text: str, source_file: str) -> HandleResult`,
where `HandleResult` is a small dataclass with `parser_used`,
`claim_count`, `batch_id?`, and per-handler extras (e.g. 835
carries `matched_count`). No class hierarchy, no plugin registry —
the scheduler keeps its `HANDLERS` dict as the registry.
3. **The `_FILE` env-var convention from SP26 stays.** This spec
doesn't add a new secret tier; the existing `secrets.get_secret()`
three-tier lookup (env var → file → Keychain) is reused.
4. **Time-bounded SFTP operations use `asyncio.to_thread` +
`asyncio.wait_for`.** `paramiko` is synchronous; the scheduler
already wraps calls in `asyncio.to_thread` for the stub path.
Real-mode calls get the same treatment so a hanging `listdir_attr`
on the worker thread can be cancelled by the event loop and
surfaced as a `Scheduler._last_error`.
5. **Default SFTP operation timeout is 30s.** `CYCLONE_SFTP_OP_TIMEOUT_SECONDS`
lets operators tune it down (e.g. 10s for a flaky VPN uplink) or
up for slow connections. The poll-interval itself remains
`CYCLONE_SCHEDULER_POLL_SECONDS` (default 60s).
6. **Reconciliation runs inside the 835 ingest session.** No
background reconciler process. If a future need emerges (catch-up
reconcile on old batches) it lands in a future SP with a
separate `cyclone reconcile` CLI subcommand.
7. **Chain endpoint renders even when slots are empty.** The contract
is "give me this claim's chain; if any piece is missing, return
`null` for that field with `missing: ["277ca_ack"]` so the UI can
show a placeholder." This matches the operator mental model:
"I submitted 837 week-1, the 999 came back accepted, the 277CA
came back accepted, but I haven't seen the 835 yet — show me that
intermediate state."
8. **`Scheduler.status()` gains `consecutive_failures`,
`last_error_at`, `last_error`, `last_sftp_attempt_at`.**
The existing fields (`running`, `poll_interval_seconds`,
`sftp_block_name`, `last_poll_at`, `poll_count`, totals, last_tick)
are unchanged for backward-compat.
9. **The `claim.rejected_after_remit` audit event is informational,
not blocking.** The 999/277CA handlers still write the rejection
even when a matched remit exists; the audit event is the operator's
signal to manually unmatch the pair. We do NOT auto-unmatch (would
violate the manual-match semantics from T15).
10. **The startup invariant check is log-only.** It does not block
boot. A drift between `Claim.matched_remittance_id` and
`Remittance.claim_id` indicates a historical bug; we surface it
in the logs and continue. Operators who want to reconcile the
drift have a one-shot CLI subcommand (`cyclone reconcile
reindex-matches`) deferred to a follow-up SP.
## 3. Architecture (after SP27)
```
backend/src/cyclone/
scheduler.py — slim: Scheduler class + HANDLERS registry + lifecycle
handlers/ NEW subpackage
__init__.py
_ack_id.py — ack_count_summary, ack_synthetic_source_batch_id, *_277ca_*
ack.py — top-level ack dispatch (re-exported helpers)
handle_999.py — text → parsed 999 → ack row → claim rejections
handle_835.py — text → parsed 835 → batch + remits + CasAdjustments
+ match claims (atomic one-section)
handle_277ca.py — text → parsed 277CA → ack row + claim rejections
+ emit claim.rejected_after_remit
handle_ta1.py — text → parsed TA1 → ack row
clearhouse/__init__.py — _paramiko ops wrapped in asyncio.wait_for via to_thread
edi/filenames.py — parse_inbound_filename accepts suffix-less names
store.py — manual_match / manual_unmatch keep matched_remittance_id
and Remittance.claim_id in sync (same transaction);
startup invariant check
api.py — /api/parse-835 deduped vs scheduler handlers via
handlers.handle_835; new /api/claims/{id}/chain endpoint
reconcile.py — run() folded into handle_835; top-level `match()`
kept for the CLI subcommand (deferred)
api_routers/chain.py NEW — GET /api/claims/{id}/chain (auth, matrix_gate)
src/
hooks/useClaimChain.ts NEW — TanStack Query wrapper around /api/claims/{id}/chain
pages/ClaimDrawer.tsx — one new section: "Chain" with submission / 999 / 277CA / remit
placeholders; renders adjustments inline
```
The scheduler shrinks from 860 LOC to ~250 LOC (the Scheduler class +
singleton plumbing). Each `handle_*.py` is ~80–150 LOC and can be
tested in isolation against a real prodfiles fixture.
## 4. Files
**New:**
- `backend/src/cyclone/handlers/__init__.py`
- `backend/src/cyclone/handlers/_ack_id.py`
- `backend/src/cyclone/handlers/handle_999.py`
- `backend/src/cyclone/handlers/handle_835.py`
- `backend/src/cyclone/handlers/handle_277ca.py`
- `backend/src/cyclone/handlers/handle_ta1.py`
- `backend/src/cyclone/api_routers/chain.py`
- `src/hooks/useClaimChain.ts`
- `backend/tests/test_handlers_999.py`
- `backend/tests/test_handlers_835.py`
- `backend/tests/test_handlers_277ca.py`
- `backend/tests/test_handlers_ta1.py`
- `backend/tests/test_inbound_filename_loose.py`
- `backend/tests/test_sftp_op_timeout.py`
- `backend/tests/test_scheduler_status_errors.py`
- `backend/tests/test_handler_835_atomic_reconcile.py`
- `backend/tests/test_api_claim_chain.py`
- `backend/tests/test_handler_277ca_rejected_after_remit.py`
- `backend/tests/test_store_match_invariant.py`
- `src/hooks/useClaimChain.test.ts`
- `src/pages/ClaimDrawer.test.tsx` (extension — the chain drawer section)
**Modified:**
- `backend/src/cyclone/scheduler.py` — slim to Scheduler class + lifecycle
- `backend/src/cyclone/edi/filenames.py` — `parse_inbound_filename`
accepts suffix-less inbound filenames; `is_inbound_filename` likewise
- `backend/src/cyclone/clearhouse/__init__.py` — wrap paramiko
operations in `asyncio.wait_for(asyncio.to_thread(...), timeout=N)`
- `backend/src/cyclone/store.py` — `manual_match` /
`manual_unmatch` write the pair in one transaction; startup
invariant log; `add` (835 branch) calls `handlers.handle_835`
instead of inlining
- `backend/src/cyclone/reconcile.py` — `run` exposed for the
follow-up CLI subcommand; `match` kept; the scheduler / 835
handler stop calling `run` directly
- `backend/src/cyclone/api.py` — `/api/parse-835` reuses
`handlers.handle_835`; `/api/parse-999`, `/api/parse-277ca`,
`/api/parse-ta1` likewise; add `/api/claims/{id}/chain` route
- `backend/src/cyclone/api_helpers.py` (or new file) — wire
`/api/claims/{id}/chain` if the route doesn't fit the existing
pattern
- `src/pages/ClaimDrawer.tsx` — one new "Chain" section
- `src/lib/api.ts` — expose `api.fetchClaimChain(id)`
- `src/types/index.ts` — add `ClaimChain` type
**New migrations:** none. The DB schema is unchanged.
## 5. API surface
| Method | Path | Returns | Auth |
|---|---|---|---|
| GET | `/api/claims/{id}/chain` | `ClaimChain` JSON | `matrix_gate` (any logged-in user) |
No new endpoints beyond this one. No new env-var-driven behavior
beyond `CYCLONE_SFTP_OP_TIMEOUT_SECONDS` (default 30s).
```jsonc
// GET /api/claims/{id}/chain
{
"claim_id": "...",
"submission": {
"batch_id": "...",
"patient_control_number": "...",
"service_date_from": "2026-06-15",
"service_date_to": "2026-06-15",
"charge_amount": "125.00",
"state": "submitted",
"submitted_at": "2026-06-16T..."
},
"ack_999": {
"source_batch_id": "999-PCN-12345678",
"ack_code": "A",
"received_count": 1,
"accepted_count": 1,
"rejected_count": 0,
"received_at": "2026-06-17T..."
} | null,
"ack_277ca": {
"source_batch_id": "277CA-000000001",
"classification": "accepted",
"received_at": "2026-06-18T..."
} | null,
"remittance": {
"id": "...",
"payer_claim_control_number": "...",
"total_paid": "85.00",
"patient_responsibility": "15.00",
"adjustment_amount": "25.00",
"status_label": "Paid",
"received_at": "2026-06-19T...",
"matched": true,
"adjustments": [
{ "group_code": "CO", "reason_code": "97", "amount": "25.00", "label": "..." }
]
} | null,
"missing": ["277ca_ack"] // list of any of the above that came back null
}
```
## 6. Env vars (operator-facing)
| Variable | Default | Purpose |
|---|---|---|
| `CYCLONE_SFTP_OP_TIMEOUT_SECONDS` | `30` | Per-operation SFTP timeout in `clearhouse`. Lower for flaky links, higher for slow ones. The existing `CYCLONE_SCHEDULER_POLL_SECONDS` and `CYCLONE_SFTP_PASSWORD[_FILE]` from SP25+26 are unchanged. |
No new env vars beyond this one.
## 7. Validation rules
No new R-code rules. The 835 validator (`parser_835.validator_835`)
is called unchanged by `handlers.handle_835` inside the same
transaction; the new chain endpoint doesn't introduce new parser
or validation logic.
## 8. Testing plan
Target backend test count after SP27: current + 17 new tests.
1. **Per-handler unit tests** (4 files × ~3 tests each):
`test_handlers_999.py`, `test_handlers_835.py`,
`test_handlers_277ca.py`, `test_handlers_ta1.py` — call each
handler against a prodfiles fixture and assert the persisted
rows.
2. **`test_inbound_filename_loose.py`** (4 cases):
- filename with `_835.x12` suffix → `file_type="835"` (existing behavior).
- filename without `_file_type.x12` and `orig_tx=835` → `file_type="835"` (new).
- filename without suffix and `orig_tx=999` → `file_type="999"` (new).
- filename without suffix and `orig_tx=837P` → `file_type="999"` rejected by `ALLOWED_FILE_TYPES` — no false positive for 837s.
3. **`test_sftp_op_timeout.py`** (3 cases): stub SFTP returns a slow
`sftp.listdir_attr` (raises after 60s); assert the scheduler
times out within `CYCLONE_SFTP_OP_TIMEOUT_SECONDS` + slack;
stub returns fast → assert happy path still works.
4. **`test_scheduler_status_errors.py`** (4 cases): `consecutive_failures`
increments on tick error; `last_error` populated;
`status_sftp_health` flips to `failing` after 3 fails; flips back
on next success.
5. **`test_handler_835_atomic_reconcile.py`** (3 cases): 835 ingest
persists `Remittance` with `adjustment_amount` correct from the
start; reconcile crash mid-transaction leaves no orphans; an
existing claim gets `matched_remittance_id` set in the same
transaction.
6. **`test_api_claim_chain.py`** (5 cases): happy path with all four
slots populated; missing ack_999 → `null + missing=["ack_999"]`;
missing remittance → `null + missing=["remittance"]`; missing
claim → 404; auth → 401.
7. **`test_handler_277ca_rejected_after_remit.py`** (2 cases):
277CA rejects a claim that is already matched → emits
`claim.rejected_after_remit` audit event with both IDs;
277CA rejects an unmatched claim → no audit event.
8. **`test_store_match_invariant.py`** (3 cases):
`manual_match` writes `Claim.matched_remittance_id` and
`Remittance.claim_id` in one transaction;
`manual_unmatch` clears both; pre-existing drift is logged at
startup without blocking boot.
9. **Frontend useClaimChain.test.ts** (3 cases): hook fetches the
chain and returns typed data; loading/error states; caches
for 30s.
10. **Frontend ClaimDrawer.test.tsx extension** (2 cases): renders
the chain section; renders placeholders for missing pieces.
The full backend suite (`cd backend && .venv/bin/pytest`) and
frontend suite (`npm test`) remain the merge gate.
## 9. Out of scope (future SPs)
- Frontend scheduler operator UI (start/stop/tick/status page).
- Multi-tenant / per-payer SFTP blocks.
- `<NAME>_FILE` Docker-secret convention beyond what SP26 added.
- Outbound file pickup verification.
- Auto-unmatch on rejection (would violate T15 manual-match contract).
- `cyclone reconcile reindex-matches` one-shot CLI for historical drift.
- Per-payer chain-fanout (different payer-specific parsing of the same
277CA segment).
- Persisting the chain response server-side (it is a join, not a stored
view; keeping it computed avoids drift).
src/hooks/useAcks.ts
+8 -3
View File
@@ -1,5 +1,5 @@
import { useQuery } from "@tanstack/react-query";
import { api, type PaginatedResponse } from "@/lib/api";
import { api, type AckAggregates, type PaginatedResponse } from "@/lib/api";
import type { Ack } from "@/types";
/**
@@ -7,9 +7,14 @@ import type { Ack } from "@/types";
* intentionally has no in-memory fallback (there is no zustand
* sample-data path for ACKs in v1 — the UI is empty until the
* backend serves real rows).
*
* The response carries `aggregates` summed over the *full* row set
* (not just the visible page) so the KPI strip on the Acks page
* reflects every persisted 999 — without this, totals silently
* under-report once the row count exceeds the page size.
*/
export function useAcks(params: { limit?: number } = {}) {
return useQuery<PaginatedResponse<Ack>>({
export function useAcks(params: { limit?: number; offset?: number } = {}) {
return useQuery<PaginatedResponse<Ack> & { aggregates: AckAggregates }>({
queryKey: ["acks", params],
queryFn: () => api.listAcks(params),
enabled: api.isConfigured,
src/hooks/useDashboardKpis.test.ts
+126
View File
@@ -0,0 +1,126 @@
// @vitest-environment happy-dom
// SP27 Task 13: server-aggregated Dashboard KPIs.
//
// These tests pin the hook's contract independent of the Dashboard
// page: when the backend is wired, it calls ``api.getDashboardKpis``
// with the requested parameters and returns the resolved value; when
// the backend is not wired, it returns ``data: undefined`` so the
// Dashboard can render zero-shaped fallbacks.
(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
import { describe, expect, it, vi, beforeEach } from "vitest";
import { renderHook, waitFor } from "@testing-library/react";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import React from "react";
import { api, type DashboardKpis } from "@/lib/api";
// Mock the api module so we can control isConfigured + getDashboardKpis
// without spinning up a real backend.
vi.mock("@/lib/api", async () => {
const actual = await vi.importActual<typeof import("@/lib/api")>("@/lib/api");
return {
...actual,
api: {
...actual.api,
getDashboardKpis: vi.fn(),
},
};
});
// Import AFTER the mock so the hook sees the mocked api.
const { useDashboardKpis } = await import("./useDashboardKpis");
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
function makeWrapper() {
const client = new QueryClient({
defaultOptions: {
queries: { retry: false, gcTime: 0, staleTime: 0 },
},
});
return ({ children }: { children: React.ReactNode }) =>
React.createElement(QueryClientProvider, { client }, children);
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
describe("useDashboardKpis", () => {
beforeEach(() => {
vi.clearAllMocks();
});
it("calls api.getDashboardKpis and returns the resolved value when configured", async () => {
const payload: DashboardKpis = {
totals: {
count: 60295,
billed: 940143.58,
received: 59753,
outstandingAr: 880390.58,
denied: 542,
denialRate: 0.9,
pending: 60,
},
monthly: [
{ month: "2026-01", label: "Jan", count: 100, billed: 15000,
received: 12000, denied: 2, denialRate: 2.0, ar: 3000 },
],
topProviders: [
{ npi: "1234567893", label: "Cedar Park", claimCount: 200,
billed: 30000, denied: 1 },
],
topDenials: [
{ id: "C-1", patientName: "Jane Doe", billedAmount: 250,
denialReason: "Missing modifier", submissionDate: "2026-06-20T12:00:00Z" },
],
};
(api.getDashboardKpis as ReturnType<typeof vi.fn>).mockResolvedValue(payload);
// Override the isConfigured bit too — the hook keys off this.
(api as unknown as { isConfigured: boolean }).isConfigured = true;
const { result } = renderHook(() => useDashboardKpis({ months: 6 }), {
wrapper: makeWrapper(),
});
await waitFor(() => {
expect(result.current.data).toEqual(payload);
});
expect(api.getDashboardKpis).toHaveBeenCalledWith({ months: 6 });
expect(result.current.isError).toBe(false);
});
it("returns data: undefined when the backend is not configured", () => {
(api as unknown as { isConfigured: boolean }).isConfigured = false;
const { result } = renderHook(() => useDashboardKpis(), {
wrapper: makeWrapper(),
});
expect(result.current.data).toBeUndefined();
expect(result.current.isLoading).toBe(false);
expect(result.current.isError).toBe(false);
// queryFn should NOT have been called — bypassed entirely.
expect(api.getDashboardKpis).not.toHaveBeenCalled();
});
it("passes top_n_providers / top_n_denials through to the API", async () => {
(api.getDashboardKpis as ReturnType<typeof vi.fn>).mockResolvedValue({
totals: { count: 0, billed: 0, received: 0, outstandingAr: 0,
denied: 0, denialRate: 0, pending: 0 },
monthly: [], topProviders: [], topDenials: [],
});
(api as unknown as { isConfigured: boolean }).isConfigured = true;
renderHook(
() => useDashboardKpis({ months: 3, top_n_providers: 2, top_n_denials: 5 }),
{ wrapper: makeWrapper() },
);
expect(api.getDashboardKpis).toHaveBeenCalledWith({
months: 3, top_n_providers: 2, top_n_denials: 5,
});
});
});
src/hooks/useDashboardKpis.ts
+47
View File
@@ -0,0 +1,47 @@
import { useQuery } from "@tanstack/react-query";
import { api, type DashboardKpis, type DashboardKpisParams } from "@/lib/api";
/**
* Server-aggregated Dashboard KPIs.
*
* Replaces the previous `useClaims({ limit: 100 })` + client-side
* reduce pattern. With 60k+ claims in production, paginating
* ``/api/claims`` and reducing client-side silently produced wrong
* numbers — the Dashboard's "Billed / Received / Denial rate / Pending
* AR" tiles were computed from a 100-row sample, not the full
* population. This hook hits ``GET /api/dashboard/kpis`` which does
* the aggregation server-side in a single read.
*
* Refreshes every 60s when the backend is configured so the KPIs stay
* roughly current without a manual reload. Live event-publish from
* ``useTailStream`` would be more elegant, but the underlying
* aggregates span the whole DB so we'd need a new "kpi_updated"
* event; 60s polling is a clear-enough SLA for a Dashboard view.
*
* Sample-data mode (no backend wired): returns ``data: undefined``
* rather than calling ``useQuery``. The Dashboard handles that with
* zero-shaped fallbacks so the KPI tiles render a coherent "0" rather
* than throwing. There's no in-memory aggregator — the old
* client-side reduce was the bug, not a feature to preserve.
*/
export function useDashboardKpis(params: DashboardKpisParams = {}) {
const q = useQuery<DashboardKpis>({
queryKey: ["dashboard", "kpis", params],
queryFn: () => api.getDashboardKpis(params),
enabled: api.isConfigured,
refetchInterval: api.isConfigured ? 60_000 : false,
staleTime: 30_000,
});
if (!api.isConfigured) {
return {
data: undefined,
isLoading: false,
isError: false,
error: null,
refetch: () => Promise.resolve(),
dataUpdatedAt: 0,
} as const;
}
return q;
}
src/hooks/useRemittanceSummary.ts
+30
View File
@@ -0,0 +1,30 @@
import { useQuery } from "@tanstack/react-query";
import {
api,
type ListRemittancesParams,
type RemittanceSummary,
} from "@/lib/api";
/**
* Server-aggregated KPI totals for the Remittances page tiles.
* Returns ``{count, total_paid, total_adjustments}`` summed over the
* full filtered remittance population — NOT a page-limited sample.
*
* The page's KPI tiles consume this hook instead of summing
* ``items.reduce(...)`` over the current page (25 rows) + live-tail
* delta, which silently understates the true population in the same
* shape that ``59c3275`` retired for the Dashboard and that
* ``d81b6ed`` retired for the count tile.
*
* The optional ``params`` object lets the page pass through the same
* filter chips it uses for the list endpoint (``status``, ``payer``,
* ``date_from``, ``date_to``) so the summary and the row list always
* agree on which rows they're describing.
*/
export function useRemittanceSummary(params: ListRemittancesParams = {}) {
return useQuery<RemittanceSummary>({
queryKey: ["remittances", "summary", params],
queryFn: () => api.listRemittanceSummary(params),
enabled: api.isConfigured,
});
}
src/hooks/useTa1Acks.ts
+26
View File
@@ -0,0 +1,26 @@
import { useQuery } from "@tanstack/react-query";
import { api, type PaginatedResponse } from "@/lib/api";
import type { Ta1Ack } from "@/types";
/**
* Lists persisted TA1 (Interchange Acknowledgment) rows, newest
* first. Mirrors `useAcks` but for the lower-level envelope ack.
*
* A TA1 is one row per inbound ISA/IEA interchange — distinct from
* a 999, which is per-batch. Colorado Medicaid's Gainwell MFT
* currently only ships 999s in the FromHPE path, but historically
* they've sent TA1s, so the hook stays in the surface for when
* they reappear.
*
* No in-memory fallback: there is no zustand sample-data path for
* TA1s in v1. The hook is `enabled: api.isConfigured` so the page
* treats an empty list as "no TA1s on file" rather than a
* configuration error.
*/
export function useTa1Acks(params: { limit?: number } = {}) {
return useQuery<PaginatedResponse<Ta1Ack>>({
queryKey: ["ta1-acks", params],
queryFn: () => api.listTa1Acks(params),
enabled: api.isConfigured,
});
}
src/lib/api.ts
+194 -1
View File
@@ -39,6 +39,7 @@ import type {
Payee835,
Provider,
ReassociationTrace,
Ta1Ack,
UnmatchedClaim,
UnmatchedResponse,
BatchSummary as ParserBatchSummary,
@@ -152,6 +153,66 @@ export interface ListActivityParams {
limit?: number;
}
// ---------------------------------------------------------------------------
// Dashboard KPI types (SP27 Task 13).
//
// Returned by ``GET /api/dashboard/kpis`` — server-aggregated over the
// *entire* claim population. The Dashboard renders these directly; it
// no longer paginates ``/api/claims`` and reduces client-side (which
// silently produced wrong numbers with the previous ``limit: 100``
// cap).
// ---------------------------------------------------------------------------
export interface DashboardTotals {
count: number;
billed: number;
received: number;
outstandingAr: number;
denied: number;
denialRate: number;
pending: number;
}
export interface DashboardMonthly {
month: string; // "YYYY-MM"
label: string; // "Jan"
count: number;
billed: number;
received: number;
denied: number;
denialRate: number;
ar: number;
}
export interface DashboardTopProvider {
npi: string;
label: string;
claimCount: number;
billed: number;
denied: number;
}
export interface DashboardTopDenial {
id: string;
patientName: string;
billedAmount: number;
denialReason: string | null;
submissionDate: string;
}
export interface DashboardKpis {
totals: DashboardTotals;
monthly: DashboardMonthly[];
topProviders: DashboardTopProvider[];
topDenials: DashboardTopDenial[];
}
export interface DashboardKpisParams {
months?: number;
top_n_providers?: number;
top_n_denials?: number;
}
export interface PaginatedResponse<T> {
items: T[];
total: number;
@@ -159,6 +220,19 @@ export interface PaginatedResponse<T> {
has_more: boolean;
}
/**
* Server-side aggregates returned alongside `/api/acks` — summed over
* the *full* persisted row set (not just the visible page). The
* `accepted` / `rejected` / `received` keys match the in-page `totals`
* shape so the page can use `data.aggregates` as a drop-in for the
* page-local fallback accumulator.
*/
export interface AckAggregates {
accepted: number;
rejected: number;
received: number;
}
/**
* Lightweight summary used by the GET /api/batches list endpoint. Distinct
* from the parser's `BatchSummary` (which lives in `@/types` and carries the
@@ -577,6 +651,28 @@ async function listRemittances<T = unknown>(
);
}
/**
* Server-aggregated KPI tiles for the Remittances page. Returns the
* full-population ``count``, ``total_paid``, and ``total_adjustments``
* — NOT a page-limited sample — so the KPI tiles can't silently
* understate the true DB population the way a page-local
* ``items.reduce(...)`` would (commits ``59c3275``, ``d81b6ed``).
*/
export interface RemittanceSummary {
count: number;
total_paid: number;
total_adjustments: number;
}
async function listRemittanceSummary(
params: ListRemittancesParams = {}
): Promise<RemittanceSummary> {
if (!isConfigured) throw notConfiguredError();
return authedFetch<RemittanceSummary>(
`/api/remittances/summary${qs(params as Record<string, unknown>)}`
);
}
/**
* Fetch one remittance with its labeled CAS `adjustments` array.
* Throws `ApiError` on 404 so callers can branch on `.status`.
@@ -688,6 +784,26 @@ async function listActivity<T = unknown>(
);
}
/**
* Fetch server-aggregated Dashboard KPIs.
*
* Drives ``GET /api/dashboard/kpis``. Computes billed / received /
* denial rate / pending AR / top providers / top denials server-side
* over the *full* claim population so the Dashboard's numbers are
* always correct regardless of dataset size. With 60k+ claims in
* production, fetching ``/api/claims?limit=100`` and reducing
* client-side silently produced wrong KPIs — this endpoint replaces
* that pattern.
*/
async function getDashboardKpis(
params: DashboardKpisParams = {}
): Promise<DashboardKpis> {
if (!isConfigured) throw notConfiguredError();
return authedFetch<DashboardKpis>(
`/api/dashboard/kpis${qs(params as Record<string, unknown>)}`
);
}
// ---------------------------------------------------------------------------
// Public surface — reconciliation endpoints (sub-project 2)
// POSTs throw `ApiError` so callers can inspect `.status`; the GET is shaped
@@ -752,21 +868,37 @@ function mapAck(row: RawAckRow): Ack {
};
}
async function listAcks(params: { limit?: number } = {}): Promise<PaginatedResponse<Ack>> {
async function listAcks(
params: { limit?: number; offset?: number } = {},
): Promise<
PaginatedResponse<Ack> & {
aggregates: AckAggregates;
}
> {
if (!isConfigured) throw notConfiguredError();
const query: Record<string, unknown> = {};
if (params.limit !== undefined) query.limit = params.limit;
if (params.offset !== undefined) query.offset = params.offset;
const body = await authedFetch<{
items: RawAckRow[];
total: number;
returned: number;
has_more: boolean;
aggregates: { accepted_count: number; rejected_count: number; received_count: number };
}>(`/api/acks${qs(query)}`);
return {
items: body.items.map(mapAck),
total: body.total,
returned: body.returned,
has_more: body.has_more,
// Adapt the wire-format `*_count` keys to the in-page `totals` shape
// so the page can treat server-side and client-side-fallback objects
// interchangeably. See useAcks.ts for the matching AckAggregates type.
aggregates: {
accepted: body.aggregates.accepted_count,
rejected: body.aggregates.rejected_count,
received: body.aggregates.received_count,
},
};
}
@@ -778,6 +910,64 @@ async function getAck(id: number): Promise<Ack & { rawJson: unknown }> {
return { ...mapAck(row), rawJson: row.raw_json };
}
// ---------------------------------------------------------------------------
// Public surface — TA1 ACKs
// The TA1 is the lowest-level X12 envelope ack (one per inbound
// ISA/IEA interchange), distinct from the per-batch 999. Colorado
// Medicaid's Gainwell MFT only ships 999s today, but historically
// they've sent TA1s, so the UI shows them whenever one is on file.
// ---------------------------------------------------------------------------
interface RawTa1Row {
id: number;
control_number: string;
ack_code: "A" | "E" | "R";
note_code: string | null;
interchange_date: string | null;
interchange_time: string | null;
sender_id: string | null;
receiver_id: string | null;
source_batch_id: string;
parsed_at: string | null;
}
function mapTa1Ack(row: RawTa1Row): Ta1Ack {
return {
id: row.id,
controlNumber: row.control_number,
ackCode: row.ack_code,
noteCode: row.note_code,
interchangeDate: row.interchange_date,
interchangeTime: row.interchange_time,
senderId: row.sender_id,
receiverId: row.receiver_id,
sourceBatchId: row.source_batch_id,
parsedAt: row.parsed_at,
};
}
async function listTa1Acks(
params: { limit?: number } = {},
): Promise<PaginatedResponse<Ta1Ack>> {
if (!isConfigured) throw notConfiguredError();
const query: Record<string, unknown> = {};
if (params.limit !== undefined) query.limit = params.limit;
const body = await authedFetch<{
items: RawTa1Row[];
total: number;
}>(`/api/ta1-acks${qs(query)}`);
// The TA1 list endpoint returns `{ total, items }` (no `has_more` /
// `returned` — it's a simple cap-based list, not a paginated one).
// Synthesize the `PaginatedResponse` shape so the UI can share the
// same hook contract as `listAcks`.
return {
items: body.items.map(mapTa1Ack),
total: body.total,
returned: body.items.length,
has_more: body.items.length < body.total,
};
}
/**
* Download a ZIP of regenerated X12 837 files for a parsed batch.
*
@@ -861,6 +1051,7 @@ export const api = {
serializeClaim837,
exportBatch837,
listRemittances,
listRemittanceSummary,
getRemittance,
listProviders,
getProvider,
@@ -871,4 +1062,6 @@ export const api = {
unmatchClaim,
listAcks,
getAck,
listTa1Acks,
getDashboardKpis,
};
src/pages/Acks.tsx
+260 -16
View File
@@ -1,5 +1,5 @@
import { useCallback, useState } from "react";
import { CheckCircle2, Download, ShieldCheck } from "lucide-react";
import { CheckCircle2, Download, Mail, ShieldCheck } from "lucide-react";
import {
Table,
TableBody,
@@ -16,13 +16,17 @@ import { KpiCard } from "@/components/KpiCard";
import { PageHeader } from "@/components/PageHeader";
import { KeyboardCheatsheet } from "@/components/KeyboardCheatsheet";
import { AckDrawer } from "@/components/AckDrawer";
import { Pagination } from "@/components/ui/pagination";
import { useAckDrawerUrlState } from "@/hooks/useAckDrawerUrlState";
import { useAcks } from "@/hooks/useAcks";
import { useTa1Acks } from "@/hooks/useTa1Acks";
import { useRowKeyboard } from "@/hooks/useRowKeyboard";
import { api } from "@/lib/api";
import { fmt } from "@/lib/format";
import { cn } from "@/lib/utils";
import type { Ack } from "@/types";
import type { Ack, Ta1Ack } from "@/types";
const PAGE_SIZE = 50;
/**
* 999 ACK register. The page reads the persisted 999 Implementation
@@ -38,8 +42,18 @@ import type { Ack } from "@/types";
* instrument view of the same data.
*/
export function Acks() {
const { data, isLoading, isError, error, refetch } = useAcks({ limit: 100 });
const [page, setPage] = useState(1);
const { data, isLoading, isError, error, refetch } = useAcks({
limit: PAGE_SIZE,
offset: (page - 1) * PAGE_SIZE,
});
const items = data?.items ?? [];
const totalCount = data?.total ?? 0;
// Server-side aggregates — sum over the *full* ACK set, not just the
// visible page. Without this, the KPI strip silently under-reports
// once the row count exceeds PAGE_SIZE (the "silent failure" the
// operator flagged on 2026-06-29 when 1056 acks showed as 100).
const aggregates = data?.aggregates;
// SP21 Phase 5 Task 5.3: drill-down from an acks row into
// AckDrawer. The hook reads `?ack=` off `window.location.search`
// so deep links restore the open ack on reload.
@@ -74,15 +88,16 @@ export function Acks() {
// -----------------------------------------------------------------
// Aggregate metrics for the KPI strip + hero copy.
// Sourced from the server-side `aggregates` field so the KPI strip
// reflects every persisted 999, not just the visible page. If
// `aggregates` hasn't loaded yet (first page), fall back to summing
// the visible items so the cards still display a meaningful value.
// -----------------------------------------------------------------
const totals = items.reduce(
(acc, a) => ({
accepted: acc.accepted + a.acceptedCount,
rejected: acc.rejected + a.rejectedCount,
received: acc.received + a.receivedCount,
}),
{ accepted: 0, rejected: 0, received: 0 },
);
const totals = aggregates ?? {
accepted: items.reduce((s, a) => s + a.acceptedCount, 0),
rejected: items.reduce((s, a) => s + a.rejectedCount, 0),
received: items.reduce((s, a) => s + a.receivedCount, 0),
};
const acceptRate =
totals.received > 0
? Math.round((totals.accepted / totals.received) * 1000) / 10
@@ -90,11 +105,16 @@ export function Acks() {
// The ghost watermark carries the on-file count, scaled like the
// Dashboard/Upload/Reconciliation watermarks (clamp 72–140px, 4.5%
// opacity, right-anchored).
const watermark = items.length > 0 ? items.length.toLocaleString() : "999";
// opacity, right-anchored). Use the server-reported total (not the
// page length) so the watermark tells the truth when there are more
// than PAGE_SIZE rows on file.
const watermark = totalCount > 0 ? totalCount.toLocaleString() : "999";
// Tone for the status pill: green when nothing rejected, amber when
// the register holds a partial/rejected payload.
// `anyRejected` is a "page" property, not a "totals" property — the
// status pill summarizes the visible register, so it's OK to scope
// it to the current page.
const anyRejected = items.some(
(a) => a.ackCode === "R" || a.ackCode === "E",
);
@@ -146,7 +166,7 @@ export function Acks() {
<div className="relative z-10 flex items-end justify-between gap-4 sm:gap-6 flex-wrap">
<div className="min-w-0">
<PageHeader
eyebrow={`999 ACKs · EDI reference · ${items.length} on file`}
eyebrow={`999 ACKs · EDI reference · ${totalCount.toLocaleString()} on file`}
title={
<>
Acknowledgments,{" "}
@@ -365,7 +385,18 @@ export function Acks() {
{a.id}
</TableCell>
<TableCell className="font-mono text-[12px] text-muted-foreground truncate max-w-[180px]">
{a.sourceBatchId}
{a.patientControlNumber ? (
<>
<span className="text-foreground">
{a.patientControlNumber}
</span>
<span className="text-muted-foreground/60">
{" "}· {a.sourceBatchId}
</span>
</>
) : (
a.sourceBatchId
)}
</TableCell>
<TableCell className="text-right display mono text-[hsl(var(--success))]">
{a.acceptedCount}
@@ -395,11 +426,31 @@ export function Acks() {
</Table>
</div>
)}
{totalCount > PAGE_SIZE ? (
<Pagination
page={page}
pageSize={PAGE_SIZE}
total={totalCount}
onPageChange={setPage}
/>
) : null}
</div>
</CardContent>
</Card>
</section>
{/* =================================================================
TA1 ENVELOPE ACKS — per-interchange acknowledgments.
A TA1 (X12 Interchange Acknowledgment) is one row per
inbound ISA/IEA envelope — the lower-level sibling of the
999 (per-batch). Colorado Medicaid's Gainwell MFT currently
only ships 999s, but the section stays in the surface so
the operator can see TA1s as soon as one shows up. Kept as
a quieter, narrower section than the 999 register so the
999s remain the page's primary instrument.
================================================================= */}
<Ta1AcksSection />
{/* =================================================================
FOOTER — single hairline-separated status row. Replaces the
warm-paper "End of register" treatment with a quiet
@@ -417,7 +468,8 @@ export function Acks() {
</div>
<div className="flex items-center gap-3 mono text-[10.5px] uppercase tracking-[0.18em] text-muted-foreground/60">
<span>
{items.length} {items.length === 1 ? "row" : "rows"}
{totalCount.toLocaleString()}{" "}
{totalCount === 1 ? "row" : "rows"} on file
</span>
<span className="text-muted-foreground/30" aria-hidden>
·
@@ -541,3 +593,195 @@ function downloadBlob(filename: string, content: string) {
a.click();
URL.revokeObjectURL(url);
}
// ---------------------------------------------------------------------------
// Ta1AcksSection — per-interchange TA1 register.
//
// Deliberately quieter than the 999 register above: smaller KPI strip,
// single-card table, and an empty state that explains the
// Colorado-specific context (TA1s aren't shipped today, so the
// section is empty unless Gainwell starts sending them).
// ---------------------------------------------------------------------------
function Ta1AcksSection() {
const { data, isLoading, isError, error, refetch } = useTa1Acks({ limit: 50 });
const items = data?.items ?? [];
const totals = items.reduce(
(acc, t) => {
acc[t.ackCode] = (acc[t.ackCode] ?? 0) + 1;
return acc;
},
{} as Record<string, number>,
);
return (
<section
aria-label="TA1 envelope acknowledgments"
className="animate-fade-in-up"
>
<Card>
<CardContent className="p-6 lg:p-7 space-y-5">
<div className="flex items-end justify-between gap-6 flex-wrap">
<div className="min-w-0">
<div className="eyebrow flex items-center gap-2 mb-2">
<span className="inline-block h-px w-6 bg-foreground/20" />
Envelope acks
</div>
<h2 className="display text-[22px] leading-[1.05] tracking-[-0.02em]">
TA1 <span className="italic">envelopes</span>, newest first.
</h2>
</div>
<p className="text-[12.5px] text-muted-foreground/80 max-w-sm">
One row per inbound ISA/IEA interchange — the
envelope-level sibling of the 999. Gainwell's Colorado
MFT does not ship TA1s today; this section surfaces them
when they appear.
</p>
</div>
<div className="grid gap-3 grid-cols-2 md:grid-cols-4 pt-2">
<KpiCard
label="On file"
value={fmt.num(items.length)}
accent="default"
hint="persisted TA1s"
/>
<KpiCard
label="Accepted"
value={fmt.num(totals.A ?? 0)}
accent="success"
hint="ack_code = A"
/>
<KpiCard
label="Envelope errors"
value={fmt.num(totals.E ?? 0)}
accent="warning"
hint="ack_code = E"
/>
<KpiCard
label="Rejected"
value={fmt.num(totals.R ?? 0)}
accent="destructive"
hint="ack_code = R"
/>
</div>
<div className="pt-5 border-t border-border/40">
{isError ? (
<ErrorState
message="Couldn't load TA1 acks from the backend."
detail={error instanceof Error ? error.message : String(error)}
onRetry={() => refetch()}
/>
) : isLoading ? (
<div className="rounded-md border border-border/60 bg-card/40 p-4 space-y-2">
{Array.from({ length: 3 }).map((_, i) => (
<Skeleton key={i} variant="row" />
))}
</div>
) : items.length === 0 ? (
<div className="rounded-md border border-dashed border-border/60 bg-card/40">
<EmptyState
eyebrow="TA1 envelopes · none on file"
message="Gainwell's Colorado MFT does not ship TA1 files today. When one does arrive (or you upload one), it will land here."
/>
</div>
) : (
<div className="rounded-md border border-border/60 overflow-hidden bg-card/40">
<Table>
<TableHeader>
<TableRow>
<TableHead className="w-10" aria-label="Status" />
<TableHead>Control #</TableHead>
<TableHead>Ack</TableHead>
<TableHead>Note</TableHead>
<TableHead>Interchange date</TableHead>
<TableHead>Sender → Receiver</TableHead>
</TableRow>
</TableHeader>
<TableBody>
{items.map((t) => (
<TableRow key={t.id}>
<TableCell>
<Mail
className={cn(
"h-3.5 w-3.5",
t.ackCode === "A"
? "text-[hsl(var(--success))]"
: t.ackCode === "R"
? "text-destructive"
: "text-[hsl(var(--warning))]",
)}
strokeWidth={1.75}
aria-hidden
/>
</TableCell>
<TableCell className="display mono text-[13px] text-foreground">
{t.controlNumber || "—"}
</TableCell>
<TableCell>
<Ta1CodeBadge code={t.ackCode} />
</TableCell>
<TableCell className="mono text-[12px] text-muted-foreground">
{t.noteCode ?? "—"}
</TableCell>
<TableCell className="mono text-[12.5px] text-muted-foreground/70">
{t.interchangeDate ? fmt.dateShort(t.interchangeDate) : "—"}
{t.interchangeTime ? (
<span className="text-muted-foreground/50"> · {t.interchangeTime}</span>
) : null}
</TableCell>
<TableCell className="mono text-[12px] text-muted-foreground truncate max-w-[260px]">
{t.senderId ?? "?"} → {t.receiverId ?? "?"}
</TableCell>
</TableRow>
))}
</TableBody>
</Table>
</div>
)}
</div>
</CardContent>
</Card>
</section>
);
}
// ---------------------------------------------------------------------------
// Ta1CodeBadge — same A/E/R → success/warning/destructive mapping as the
// 999 AckCodeBadge above. Kept as a separate component so the TA1
// surface can evolve independently (e.g. add a tooltip explaining the
// X12 005010X231A1 note code table) without churning the 999 surface.
// ---------------------------------------------------------------------------
function Ta1CodeBadge({ code }: { code: Ta1Ack["ackCode"] }) {
const color =
code === "A"
? {
text: "hsl(var(--success))",
bg: "hsl(var(--success) / 0.10)",
border: "hsl(var(--success) / 0.30)",
}
: code === "R"
? {
text: "hsl(var(--destructive))",
bg: "hsl(var(--destructive) / 0.10)",
border: "hsl(var(--destructive) / 0.30)",
}
: {
text: "hsl(var(--warning))",
bg: "hsl(var(--warning) / 0.10)",
border: "hsl(var(--warning) / 0.30)",
};
return (
<span
className="inline-flex items-center gap-1 rounded-sm border px-2 py-0.5 mono text-[10.5px] font-semibold uppercase tracking-[0.14em]"
style={{
color: color.text,
backgroundColor: color.bg,
borderColor: color.border,
}}
>
{code}
</span>
);
}
src/pages/ActivityLog.tsx
+12 -2
View File
@@ -68,7 +68,13 @@ export function ActivityLog() {
const { data, isLoading, isError, error, refetch } = useActivity({
kind: apiKind,
since: sinceIso,
limit: 200,
// 500 keeps the wire size manageable (and the operator can rely
// on the live tail to surface new events as they land). The
// activity list endpoint doesn't expose a true total — `data.total`
// reflects events matching the current kind/since filter, capped
// at the request limit — so the eyebrow below uses "Showing N
// most recent" instead of "X of Y" to avoid a misleading ratio.
limit: 500,
});
const allItems = data?.items ?? [];
@@ -127,7 +133,11 @@ export function ActivityLog() {
return (
<div className="space-y-6 lg:space-y-8 animate-fade-in">
<PageHeader
eyebrow="Activity"
eyebrow={
allItems.length > 0
? `Activity · showing ${allItems.length.toLocaleString()} most recent`
: "Activity"
}
title={<>Activity <span className="display italic text-muted-foreground">log</span></>}
subtitle="Every claim submission, denial, payment, and provider event, in reverse chronological order. Auto-refreshes every 30s."
actions={
src/pages/Dashboard.test.tsx
+47 -3
View File
@@ -16,6 +16,7 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { cleanup, fireEvent, render } from "@testing-library/react";
import { MemoryRouter } from "react-router-dom";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { Dashboard } from "./Dashboard";
import { useAppStore } from "@/store";
import type { Activity } from "@/types";
@@ -31,6 +32,34 @@ vi.mock("sonner", () => ({
},
}));
// Stub useAuth so the Dashboard's greeting renders without spinning
// up the full AuthProvider + /api/auth/me probe. Same pattern as
// Inbox.test.tsx.
vi.mock("@/auth/useAuth", () => ({
useAuth: () => ({
status: "authenticated" as const,
user: { username: "tester" } as unknown as never,
login: vi.fn(),
logout: vi.fn(),
refresh: vi.fn(),
}),
}));
// SP27 Task 13: stub `api.isConfigured = false` so `useDashboardKpis`
// + `useActivity` both take their in-memory zustand fallback path.
// These tests focus on activity-feed event routing — they don't
// assert on KPI math, so zero-filled KPIs from the fallback are fine.
vi.mock("@/lib/api", async () => {
const actual = await vi.importActual<typeof import("@/lib/api")>("@/lib/api");
return {
...actual,
api: {
...actual.api,
isConfigured: false,
},
};
});
// Capture navigation side effects so we can assert on the URL the
// Dashboard would push. We use a `MemoryRouter` (initialEntries=["/"])
// and observe the rendered route via a tiny listener component that
@@ -49,6 +78,21 @@ function LocationProbe() {
);
}
// SP27 Task 13: Dashboard now reads KPIs from `useDashboardKpis`,
// which uses TanStack Query internally. Wrap each render in a
// QueryClientProvider so the hook doesn't throw — these tests focus
// on activity-feed event routing, so we never resolve the KPI query.
function renderWithQuery(ui: React.ReactNode) {
const client = new QueryClient({
defaultOptions: {
queries: { retry: false, gcTime: 0, staleTime: 0 },
},
});
return render(
<QueryClientProvider client={client}>{ui}</QueryClientProvider>,
);
}
afterEach(() => {
cleanup();
vi.clearAllMocks();
@@ -70,7 +114,7 @@ describe("Dashboard · Recent activity event routing (SP21 Task 2.5)", () => {
];
useAppStore.setState({ activity });
const { getByTestId, getByRole } = render(
const { getByTestId, getByRole } = renderWithQuery(
<MemoryRouter initialEntries={["/"]}>
<Dashboard />
<LocationProbe />
@@ -101,7 +145,7 @@ describe("Dashboard · Recent activity event routing (SP21 Task 2.5)", () => {
];
useAppStore.setState({ activity });
const { getByTestId, getByRole } = render(
const { getByTestId, getByRole } = renderWithQuery(
<MemoryRouter initialEntries={["/"]}>
<Dashboard />
<LocationProbe />
@@ -129,7 +173,7 @@ describe("Dashboard · Recent activity event routing (SP21 Task 2.5)", () => {
];
useAppStore.setState({ activity });
const { getByTestId, getByRole } = render(
const { getByTestId, getByRole } = renderWithQuery(
<MemoryRouter initialEntries={["/"]}>
<Dashboard />
<LocationProbe />
src/pages/Dashboard.tsx
+52 -88
View File
@@ -18,71 +18,66 @@ import { DrillableCell } from "@/components/drill/DrillableCell";
import { fmt } from "@/lib/format";
import { eventKindToUrl } from "@/lib/event-routing";
import { useAuth } from "@/auth/useAuth";
import { useClaims } from "@/hooks/useClaims";
import { useProviders } from "@/hooks/useProviders";
import { useDashboardKpis } from "@/hooks/useDashboardKpis";
import { useActivity } from "@/hooks/useActivity";
import type { Claim } from "@/types";
import { toast } from "sonner";
const MONTHS_BACK = 6;
function buildMonthly(claims: Claim[]) {
const now = new Date();
const months: {
key: string;
label: string;
count: number;
billed: number;
received: number;
denied: number;
}[] = [];
for (let i = MONTHS_BACK - 1; i >= 0; i--) {
const d = new Date(now.getFullYear(), now.getMonth() - i, 1);
months.push({
key: `${d.getFullYear()}-${d.getMonth()}`,
label: d.toLocaleString("en-US", { month: "short" }),
count: 0,
billed: 0,
received: 0,
denied: 0,
});
}
const index = new Map(months.map((m, i) => [m.key, i]));
for (const c of claims) {
const d = new Date(c.submissionDate);
const k = `${d.getFullYear()}-${d.getMonth()}`;
const i = index.get(k);
if (i === undefined) continue;
months[i]!.count += 1;
months[i]!.billed += c.billedAmount;
months[i]!.received += c.receivedAmount;
if (c.status === "denied") months[i]!.denied += 1;
}
let running = 0;
const ar: number[] = [];
for (const m of months) {
running += m.billed - m.received;
ar.push(Math.max(0, running));
}
return {
count: months.map((m) => m.count),
billed: months.map((m) => m.billed),
received: months.map((m) => m.received),
ar,
denialRate: months.map((m) => (m.count ? (m.denied / m.count) * 100 : 0)),
};
}
// Zero-shaped KPI totals used when the server response hasn't arrived
// yet or the backend isn't configured. Mirrors the empty-DB shape of
// `GET /api/dashboard/kpis` so the KPI tiles render a coherent "0"
// instead of `undefined` during the first paint and background
// refetches. Keeping it next to ``MONTHS_BACK`` means a future tile
// can be added in one place rather than chasing the fallback object.
const ZERO_TOTALS = {
count: 0,
billed: 0,
received: 0,
outstandingAr: 0,
denied: 0,
denialRate: 0,
pending: 0,
};
export function Dashboard() {
// Live data: hooks fetch from /api/* when api.isConfigured; otherwise
// they fall back to the in-memory store. Pulling from the hooks (not
// the store directly) is what wires the Dashboard to the backend.
const claimsQuery = useClaims({ limit: 100 });
const providersQuery = useProviders();
//
// SP27 Task 13: KPIs come from the dedicated ``/api/dashboard/kpis``
// server-side aggregate, NOT from a paginated ``/api/claims?limit=100``
// reduced client-side. With 60k+ claims in production, the old
// pattern silently produced wrong numbers — every Dashboard tile was
// computed from a 100-row sample. The new endpoint does the reduce
// in SQL once over the full population.
const kpisQuery = useDashboardKpis({ months: MONTHS_BACK });
const activityQuery = useActivity({ limit: 10 });
const claims = claimsQuery.data?.items ?? [];
const providers = providersQuery.data?.items ?? [];
const kpisData = kpisQuery.data;
// Fall back to a zero-shaped object so the KpiTiles still render a
// coherent "0" rather than NaN during the first paint and during
// background refetches. See ``ZERO_TOTALS`` for why this is a
// module-level constant.
const kpis = kpisData?.totals ?? ZERO_TOTALS;
const monthly = useMemo(() => {
const series = kpisData?.monthly ?? [];
return {
count: series.map((m) => m.count),
billed: series.map((m) => m.billed),
received: series.map((m) => m.received),
ar: series.map((m) => m.ar),
denialRate: series.map((m) => m.denialRate),
};
}, [kpisData]);
const topProviders = useMemo(
() => kpisData?.topProviders ?? [],
[kpisData]
);
const topDenials = useMemo(
() => kpisData?.topDenials ?? [],
[kpisData]
);
const activity = activityQuery.data?.items ?? [];
const navigate = useNavigate();
@@ -96,37 +91,6 @@ export function Dashboard() {
})();
const operatorName = auth.user?.username ?? "there";
const kpis = useMemo(() => {
const billed = claims.reduce((s, c) => s + c.billedAmount, 0);
const received = claims.reduce((s, c) => s + c.receivedAmount, 0);
const outstandingAr = billed - received;
const denied = claims.filter((c) => c.status === "denied").length;
const denialRate = claims.length > 0 ? (denied / claims.length) * 100 : 0;
const pending = claims.filter(
(c) => c.status === "submitted" || c.status === "pending"
).length;
return {
count: claims.length,
billed,
received,
outstandingAr,
denialRate,
pending,
};
}, [claims]);
const monthly = useMemo(() => buildMonthly(claims), [claims]);
const topProviders = useMemo(
() => [...providers].sort((a, b) => b.claimCount - a.claimCount).slice(0, 4),
[providers]
);
const topDenials = useMemo(
() => claims.filter((c) => c.status === "denied").slice(0, 5),
[claims]
);
// Stagger choreography — the hero lands first, then the KPIs in
// a left-to-right wave, then the supporting cards. Total
// choreography fits under 700ms.
@@ -345,14 +309,14 @@ export function Dashboard() {
}}
role="button"
tabIndex={0}
aria-label={`View provider ${p.name}`}
aria-label={`View provider ${p.label || p.npi}`}
className="drillable flex items-center gap-3"
>
<div className="h-7 w-7 rounded-md bg-muted/60 ring-1 ring-inset ring-border/40 flex items-center justify-center mono text-[10.5px] text-muted-foreground">
{String(i + 1).padStart(2, "0")}
</div>
<div className="flex-1 min-w-0">
<div className="text-[13px] font-medium truncate">{p.name}</div>
<div className="text-[13px] font-medium truncate">{p.label || p.npi}</div>
<div className="mono text-[10.5px] text-muted-foreground">
NPI {p.npi}
</div>
@@ -360,7 +324,7 @@ export function Dashboard() {
<div className="text-right">
<div className="display mono text-[15px]">{fmt.num(p.claimCount)}</div>
<div className="mono text-[10.5px] text-muted-foreground">
{fmt.usd(p.outstandingAr)} AR
{fmt.usd(p.billed)} billed
</div>
</div>
</li>
@@ -416,7 +380,7 @@ export function Dashboard() {
{fmt.usd(c.billedAmount)}
</div>
<div className="text-[10.5px] text-muted-foreground">
{c.payerName}
{fmt.date(c.submissionDate)}
</div>
</div>
</li>
src/pages/Remittances.test.tsx
+76
View File
@@ -20,6 +20,7 @@ vi.mock("@/lib/api", () => ({
api: {
isConfigured: true,
listRemittances: vi.fn(),
listRemittanceSummary: vi.fn(),
getRemittance: vi.fn(),
},
ApiError: class ApiError extends Error {
@@ -224,6 +225,17 @@ describe("Remittances", () => {
returned: SAMPLE_REMITS.length,
has_more: false,
});
// Default summary mock so the page-level KPI tiles don't render
// `undefined`. Tests that care about specific summary values
// override this per-test (e.g. ``test_kpi_tiles_use_server_...
// _not_page_local_reduce``).
(
api.listRemittanceSummary as unknown as ReturnType<typeof vi.fn>
).mockResolvedValue({
count: 0,
total_paid: 0,
total_adjustments: 0,
});
// Default for the per-remit detail fetch — the drawer fetches
// this whenever `?remit=` is in the URL. Return a never-resolving
// promise so the drawer stays in the loading state; the smoke
@@ -487,4 +499,68 @@ describe("Remittances", () => {
unmount();
});
// -------------------------------------------------------------------
// KPI tiles must reflect the full DB population, NOT the page-local
// reduce over the visible rows. The bug repro is a 25-row page
// whose `items.reduce(...)` summed to $25.96 paid / $67.06
// adjustments while the real DB held $227,181.58 / $13,792.65
// across 1,739 rows. The "REMITS" tile already sources its count
// from the fixed `count_remittances` helper (commit d81b6ed).
// "TOTAL PAID" + "ADJUSTMENTS" must come from the new server-side
// summary endpoint (`/api/remittances/summary`) via
// `api.listRemittanceSummary` so they can't silently understate
// the true population. Mirrors the Dashboard silent-failure fix
// (commit 59c3275).
// -------------------------------------------------------------------
it("test_kpi_tiles_use_server_summary_not_page_local_reduce", async () => {
// Page sum over the visible 3 rows: paid=$525.96 (100+200+300-74.04),
// adjustments=$60 (60+0+0). Server ground truth (mocked): count=1739,
// paid=$227,181, adjustments=$13,792 — whole-dollar amounts chosen
// so `fmt.usd`'s maximumFractionDigits=0 doesn't round them and the
// assertion is unambiguous. The page must show the SERVER values,
// not the page sums.
(
api.listRemittanceSummary as unknown as ReturnType<typeof vi.fn>
).mockResolvedValue({
count: 1739,
total_paid: 227181,
total_adjustments: 13792,
});
const { unmount } = renderIntoContainer(React.createElement(Remittances));
await waitForText("PCN-1");
const body = document.body.textContent ?? "";
// The server totals must show on the tiles.
expect(body).toContain("$227,181");
expect(body).toContain("$13,792");
// Server count wins over the 3-row page total.
expect(body).toContain("1,739");
// Locate the KPI tile for "Total paid" specifically. The tiles
// share the `.display.mono.text-[28px]` class with several other
// page elements, so we walk by `.eyebrow` to find each one.
const totalPaidTile = Array.from(
document.querySelectorAll(".surface-2"),
).find((el) => el.querySelector(".eyebrow")?.textContent === "Total paid");
expect(totalPaidTile).toBeDefined();
const paidValue = totalPaidTile?.querySelector(
".display.mono.text-\\[28px\\]",
);
expect(paidValue?.textContent).toBe("$227,181");
const adjustmentsTile = Array.from(
document.querySelectorAll(".surface-2"),
).find(
(el) => el.querySelector(".eyebrow")?.textContent === "Adjustments",
);
expect(adjustmentsTile).toBeDefined();
const adjValue = adjustmentsTile?.querySelector(
".display.mono.text-\\[28px\\]",
);
expect(adjValue?.textContent).toBe("$13,792");
unmount();
});
});
src/pages/Remittances.tsx
+18 -10
View File
@@ -18,6 +18,7 @@ import { PageHeader } from "@/components/PageHeader";
import { TailStatusPill } from "@/components/TailStatusPill";
import { RemitDrawer } from "@/components/RemitDrawer";
import { useRemittances } from "@/hooks/useRemittances";
import { useRemittanceSummary } from "@/hooks/useRemittanceSummary";
import { useRemitDrawerUrlState } from "@/hooks/useRemitDrawerUrlState";
import { useRowKeyboard } from "@/hooks/useRowKeyboard";
import { useTailStream } from "@/hooks/useTailStream";
@@ -67,13 +68,20 @@ export function Remittances() {
useTailStream("remittances");
const items = useMergedTail("remittances", data?.items ?? [], tailFilterFn);
const total = items.reduce(
(acc, r) => ({
paid: acc.paid + r.paidAmount,
adjustments: acc.adjustments + r.adjustmentAmount,
}),
{ paid: 0, adjustments: 0 }
);
// KPI totals come from the server-aggregated summary endpoint over
// the FULL remittance population — NOT a page-local reduce over the
// current page + live-tail delta. The same silent-incompleteness
// pattern that `59c3275` (Dashboard) and `d81b6ed` (count tile)
// retired applies to the financial tiles; summing visible rows
// understates the true totals and the page looks complete. The
// server returns zero-filled values when the DB has no rows so the
// tiles render predictably while the initial fetch resolves.
//
// We don't pass the page's status chip here — the list endpoint
// (``useRemittances``) ignores status server-side and applies it
// client-side via the merged-tail filter, so passing it to the
// summary would inconsistently narrow only one of the two reads.
const { data: summary } = useRemittanceSummary();
const moveNext = useCallback(() => {
setSelectedIndex((i) => {
@@ -154,19 +162,19 @@ export function Remittances() {
<div className="surface-2 rounded-xl p-5">
<div className="eyebrow">Remits</div>
<div className="display mono text-[28px] leading-none mt-3 text-foreground">
{fmt.num(data?.total ?? 0)}
{fmt.num(summary?.count ?? data?.total ?? 0)}
</div>
</div>
<div className="surface-2 rounded-xl p-5">
<div className="eyebrow">Total paid</div>
<div className="display mono text-[28px] leading-none mt-3 text-[hsl(var(--success))]">
{fmt.usd(total.paid)}
{fmt.usd(summary?.total_paid ?? 0)}
</div>
</div>
<div className="surface-2 rounded-xl p-5">
<div className="eyebrow">Adjustments</div>
<div className="display mono text-[28px] leading-none mt-3 text-[hsl(var(--warning))]">
{fmt.usd(total.adjustments)}
{fmt.usd(summary?.total_adjustments ?? 0)}
</div>
</div>
</div>
src/types/index.ts
+40
View File
@@ -529,6 +529,46 @@ export interface Ack {
receivedCount: number;
ackCode: "A" | "E" | "R" | "P";
parsedAt: string;
/**
* AK2 set_control_number from the inbound 999 — the
* patient_control_number of the original claim batch this 999
* acks. Surfaced in the list endpoint so the operator can
* correlate a 999 to a claim batch in the UI without a second
* round-trip to the detail endpoint.
*/
patientControlNumber?: string | null;
}
// ---------------------------------------------------------------------------
// TA1 (Interchange Acknowledgment) — mirrors the `ta1_acks` table
// and the `/api/ta1-acks` response. A TA1 is the lowest-level X12
// envelope ack: one per inbound ISA/IEA interchange, separate from
// the per-batch 999. Colorado Medicaid currently doesn't ship TA1s
// in the FromHPE path (the inbound MFT only carries 999s) but
// Gainwell has shipped them historically, so the UI shows them
// whenever one is on file.
// ---------------------------------------------------------------------------
/**
* One persisted TA1 ACK row, camelCased for the UI. Re-shaped in
* `src/lib/api.ts` (`mapTa1Ack`) from the snake_case backend payload.
*
* ``ackCode`` is the interchange ack code: ``A`` = accepted,
* ``E`` = accepted with envelope errors, ``R`` = rejected.
* ``noteCode`` is an optional 3-digit X12 note code (e.g. ``000`` = no
* error, ``001`` = unsupported interchange version, etc.).
*/
export interface Ta1Ack {
id: number;
controlNumber: string;
ackCode: "A" | "E" | "R";
noteCode: string | null;
interchangeDate: string | null;
interchangeTime: string | null;
senderId: string | null;
receiverId: string | null;
sourceBatchId: string;
parsedAt: string | null;
}
// ---------------------------------------------------------------------------