Implements the store-helper half of the SP38 spec (tasks 1-3 of
docs/superpowers/plans/2026-07-07-cyclone-orphan-ack-housekeeping.md).
Two new CycloneStore methods + one extraction helper in claim_acks.py:
- find_ack_orphan_st02_summary() -> list[dict]
Per-ST02 breakdown of orphan 999 acks. Returns one row per
distinct (st02, ack_count, has_batch, batch_id). Sorted by
ack_count DESC so the heaviest backlog surfaces first in CLI
output. 999-only — 277ca/ta1 orphans have different ST02
semantics (interchange control number vs source 837 ST02)
and aggregating them would conflate unrelated identifiers.
- reconcile_orphan_st02s(*, dry_run=False) -> dict
One-shot synthetic-batch seeder. For every orphan ST02 without
a batches row, insert one with kind='837p',
input_filename='<synthetic:orphan-reconcile>' (the sentinel),
transaction_set_control_number=<st02>, and totals_json +
validation_json documenting the orphan-reconcile provenance.
Idempotent: re-running after a successful pass is a no-op.
dry_run=True returns the plan shape without writing.
- _iter_orphan_999_st02s() -> Iterator[(st02, count)]
Extracted walk helper. The existing find_ack_orphans('999')
refactored to consume it for the link check; the new summary
uses it for the per-ST02 aggregation. Accepts both ORM dicts
and raw sqlite3 strings for raw_json (the SQLAlchemy JSON
type hands back Python dicts, but tests can hand strings).
11 new tests (test_ack_orphan_summary.py) cover:
- per-ST02 counts with duplicates
- has_batch=True when a batches row covers the ST02
- sort order (heaviest first)
- exclusion of 999s with a claim_acks link
- skip of malformed/empty raw_json
- empty-DB case
- reconcile creates synthetic rows for missing ST02s only
- reconcile skips ST02s with existing batches
- idempotency
- --dry-run flag
- ack_count preserved in totals_json
Live-verified against ~/.local/share/cyclone/cyclone.db: 4 distinct
orphan ST02s, 804 total orphan rows. After reconcile, all 4 marked
has_batch=yes.
Audit on 2026-07-07 found three production bugs and six stale matrix
entries:
Three registered routes were DENIED in production (fail-closed):
- DELETE /api/acks/{kind}/{ack_id}/match-claim/{claim_id}
(Inbox unlink-a-wrong-match action; wired from the frontend)
- GET /api/dashboard/kpis (dashboard summary cards)
- GET /api/inbox/ack-orphans (Inbox 'Ack orphans' lane; already
wired into src/hooks/useAckOrphans.ts + AckOrphansLane.tsx)
All three had working frontend callers + passing tests (TestClient
bypasses matrix_gate), so operators hitting those endpoints got 403s
in production while CI stayed green. Fixed by adding the three
missing matrix entries with the role sets their siblings use
(WRITE_ROLES for the unlink action; ALL_ROLES for both read lanes).
Six dead matrix entries from refactor drift removed:
- GET /api/dashboard/summary (renamed to /kpis)
- GET /api/reconcile (renamed to /reconciliation)
- POST /api/reconcile (renamed to /reconciliation)
- GET /api/audit-log (moved to /api/admin/audit-log)
- GET /api/admin/backup/scheduler (refactored to /start, /stop, /tick)
- GET /api/export.csv (renamed to /api/inbox/export.csv)
Plus a one-line doc fix at api_routers/claim_acks.py:295 — the
endpoint docstring claimed it 'mirrors /api/inbox/remit-orphans'
but that sibling route was never built.
Guard test: tests/test_routes_have_auth_coverage.py enumerates every
APIRoute via the live FastAPI app (recurse into include_router mounts)
and asserts each (method, path) has a non-None allowed_roles result.
Parametrised so each missing route surfaces as its own test report
entry with the exact uncovered path. 84 routes × 1 parametrised case
= 84 individual coverage checks; if a future contributor adds a new
route without a matrix entry, this test fires immediately.
The SftpClient wrapper previously exposed write_file, list_inbound,
list_inbound_names, download_inbound, and read_file — but NO stat().
The canonical cyclone.submission.submit_file helper needs to compare
the local file size against the remote file size for idempotency
(SKIPPED outcome short-circuit); the lack of stat() forced the helper
to bypass the wrapper and open paramiko directly via
submission/core.py:_default_sftp_factory.
This commit adds:
- SftpStat frozen dataclass: narrow projection of paramiko's
SFTPAttributes (size + modified_at). Callers don't need paramiko.
- SftpClient.stat(remote_path) -> SftpStat public method
- _stat_stub + _stat_paramiko backends (mirrors the existing
_read_file_stub / _read_file_paramiko pattern)
- 8 new tests covering dataclass shape, frozen invariant, stub size,
stub mtime, missing-file FileNotFoundError, large-file size, and
the actual SKIPPED-outcome idempotency check from submit_file
Also adds backend/var/ to .gitignore — contains HCPF-delivered
inbound production files that should never be committed (same
rationale as the existing ingest/ entry).
Followup #4 from the SP37 final-state tracker. The previous
BACKFILL_SQL constant in test_migration_0020.py was a hand-copied
duplicate of the migration's UPDATE statement. A future contributor
could edit one without the other and the test would silently
replay a different SQL than production — defeating the regression.
Fix: tests now load the migration file at test time and extract
its UPDATE via the same splitter db_migrate.run() uses (strip
'--' comments, split on ';'). The test can never disagree with
what production runs.
Changes:
* test_migration_0020.py:
- Remove the hand-copied BACKFILL_SQL constant
- Add _migration_0020_path(), _extract_update_statements(),
and _load_migration_0020_backfill_sql() helpers
- Replace 3 BACKFILL_SQL references with helper calls
* test_migration_0020_no_drift.py (new, 3 tests):
- test_migration_0020_backfill_sql_uses_migration_file
(asserts the extracted SQL targets the right column + path)
- test_migration_0020_backfill_sql_is_non_empty_single_statement
- test_migration_0020_has_exactly_one_update (guardrail against
future contributors adding a second UPDATE — the extraction
fails loudly so the test author can decide which is the
backfill)
Tests: 43/43 pass in 1.46s (full SP37 followup chain).
Imports across test files match the existing pattern (test_store.py
imports from test_store_reconcile.py).
Followup #3 from the SP37 final-state tracker. The router's
per-file try/except previously coerced every unexpected exception
to SubmitOutcome.SFTP_FAILED — misleading because a RuntimeError
from cycl_store.add has nothing to do with SFTP.
Changes:
* submission/result.py — add UNEXPECTED_ERROR = 'unexpected_error'
enum value. Docstring distinguishes it from the typed SFTP_FAILED.
* api_routers/submission.py — exception handler now returns
UNEXPECTED_ERROR (was SFTP_FAILED).
* test_api_submit_batch.py::test_submit_batch_unexpected_exception_recorded_in_results
— updated assertion (was sftp_failed, now unexpected_error).
* test_unexpected_error_outcome.py (new, 4 tests):
- enum value exists
- distinct from typed failure values
- API surfaces 'unexpected_error' for uncaught exceptions
- typed SFTP_FAILED path still surfaces 'sftp_failed' (regression lock)
Tests: 40/40 pass in 1.41s (full SP37 chain + new file).
Live curl: POST /api/submit-batch in stub mode → HTTP 409 (unchanged
behavior, no regression on the config-level guards).
Followup #2 from the SP37 final-state tracker. Replaces the
subprocess.run-based CLI help test with click.testing.CliRunner
(matches the pattern already used in test_cli.py).
Speedup: CliRunner doesn't spawn a subprocess, so the test runs in
~0.5s vs ~0.2s+ for subprocess (and the subprocess version was
susceptible to fork overhead, environment leakage between tests,
and PATH/CWD surprises on different hosts).
Same assertions:
* result.exit_code == 0 (matches subprocess.returncode == 0)
* '--ingest-dir' in result.output (matches stdout check)
Test passes in isolation and as part of the full SP37 chain
(36 tests in 1.29s — same speedup as the single test).
Followup #1 from the SP37 final-state tracker. The pre-existing
('POST', '/api/resubmit'): WRITE_ROLES entry only matched the exact
path /api/resubmit (no children — the prefix-match logic requires
the trailing slash for sub-paths). No route is registered at that
path; the actual resubmit lives at /api/inbox/rejected/resubmit
(which has its own entry).
Removing the dead entry is fail-closed: any future request to
/api/resubmit (or its descendants) returns None from allowed_roles,
which means deny. Without this cleanup, the matrix silently granted
WRITE_ROLES to a non-existent route.
Tests (test_auth_permissions_matrix.py, 5 cases):
* Exact path /api/resubmit not in PERMISSIONS
* allowed_roles('POST', '/api/resubmit') is None
* /api/resubmit/, /api/resubmit/anything all deny
Pin the invariant so a future contributor can't re-add a dead prefix.
TDD: tests written first, watched fail (5 fail with the dead entry),
removed the line, tests pass (5 pass). Full SP37 chain (36 tests)
passes with the change.
Thin HTTP wrapper around `cyclone.submission.submit_file` — the
HTTP twin of the `cyclone submit-batch` CLI (SP37 Task 5). Same
walker pattern, same `._*` AppleDouble skip, same `limit`
semantics, same per-file outcomes; both surfaces now agree on DB +
SFTP state byte-for-byte.
Body: `{ingest_dir, validate?, actor?, limit?}`. Response:
`{submitted, skipped, failed, results:[{file, outcome, batch_id,
error}]}`. Status codes:
- 200 on completed runs (per-file failures live in the body)
- 401 unauthenticated (matrix_gate)
- 404 no clearhouse seeded
- 409 clearhouse SFTP block in stub mode
- 422 ingest_dir missing on disk or Pydantic body error
Does NOT inject `sftp_client_factory` — submit_file uses its
default paramiko factory so SKIPPED is reachable in production.
Tests monkey-patch submit_file itself.
Permissions entry: `POST /api/submit-batch` → WRITE_ROLES (admin +
user), matching `POST /api/resubmit` posture.
Tests (11): happy path, AppleDouble skip, limit, empty ingest_dir
422, mixed-outcome 200, unexpected-exception-as-failed,
Pydantic-missing-422, on-disk-missing-422, no-clearhouse-404,
stub-mode-409, auth-gate-401.
Walks batch-*-claims/*.x12 under --ingest-dir, calls submit_file per
file, prints submitted/skipped/failed counts. Exits 0 even on
per-file failures (details in stdout); exits 2 on config-level
errors (no clearhouse, stub mode, missing dir).
This is the canonical outbound path; resubmit-rejected-claims
remains for one-off cases where no DB row is wanted.
SftpClient wrapper lacks a stat() method, so the previous default
factory produced a client whose stat() raised AttributeError — making
the SKIPPED outcome unreachable in production and silently
misclassifying all uploads as SFTP_FAILED. The helper now opens a
paramiko SFTP session directly (same pattern as
resubmit-rejected-claims in cli.py:620-650), so stat() and write_file()
both work end-to-end.
Also: tighten typing (SftpBlock instead of Any), add PAYER_MISMATCH
test, drop duplicate _db fixture, rename misleading test, add class
docstrings to SubmitResult/SubmitOutcome.
Owns the parse → DB write → SFTP upload sequence in one place. CLI
and HTTP thin-call it. DB-first invariant: if add_record raises, no
SFTP call is made. Idempotency: add_record dedupes via s.get(Claim,
claim_id); SFTP layer dedupes via stat().st_size match.
- test_acks.py:test_migration_latest_idempotent_on_fresh_db was still
asserting user_version == 19; migration 0020 (Task 1) bumped it to 20.
Without this fix, the full pytest suite fails the moment SP37 ships.
- claim_acks.py module docstring + store/__init__.py facade docstring
still advertised the index as single-key {envelope.control_number:
batch.id}. SP37 Task 3 made it dual-key (ISA13 + ST02); the docs now
match the implementation.
Each 837p batch row contributes up to two entries to the join-key
index (envelope.control_number + transaction_set_control_number). One
idx.get(set_control_number) call resolves either, so 999 acks whose
AK201 echoes the source 837's ST02 now hit Pass 1 where they
previously fell through to Pass 2 (and to the orphan log).
Backward compat preserved: rows with transaction_set_control_number
NULL (pre-SP37 batches) still resolve by ISA13.
The parser now captures the source 837's ST02 (transaction set control
number) on the Envelope model as 'transaction_set_control_number'.
add_record reads it off the envelope and stores it on the Batch row,
mirroring how 'envelope.control_number' (ISA13) was already handled.
Unlocks the SP37 join-key update so 999 set_control_number (AK201)
values resolve back to the right batch via Pass 1.
Migration 0020 adds the column additively (nullable, no default) and
backfills from raw_result_json.envelope.transaction_set_control_number
for any existing batch rows that already carry it. Required for the
SP37 join-key update so 999 acks can resolve by ST02 (the source 837's
transaction set control number) instead of just ISA13.
Moves the entire /api/admin/* namespace (audit-log ×2, db/rotate-key ×1,
backup ×10, scheduler ×6, reload-config ×1) out of api.py and into the
existing api_routers/admin.py, which already owned /api/admin/validate-provider.
- api.py: 4294 → 3547 LOC (Δ -747). Removed the orphaned # --- separator
left behind by the cut, the orphaned 'return {ok,loaded,errors}'
tail of reload-config, and the now-unused cyclone.clearhouse.InboundFile
top-level import.
- api_routers/admin.py: 60 → 851 LOC. Added the imports the moved
routes need (json, logging, threading, time, AuditEvent, verify_chain,
InboundFile) and stripped the redundant top-level imports of
symbols that the route bodies reach via the inline _db_crypto /
_secrets / _backup_svc_mod / _backup_sched_mod / _scheduler_mod
/ _audit aliases (those aliases stay — tests rely on being able to
monkeypatch cyclone.api_routers.admin._X.method). Hoisted the inline
'import time' from inside pull_inbound to module scope. Dropped
the redundant 'import threading as _threading' alias — top-level
'import threading' already covers it.
- tests/test_api_rotate_key.py: 4 monkeypatch targets migrated from
cyclone.api._db_crypto / cyclone.api._secrets to
cyclone.api_routers.admin._db_crypto / cyclone.api_routers.admin._secrets
to match the new home of the rotate-key endpoint. Lock acquire/release
also migrated.
The non-admin /api/config/* and /api/payers/{id}/summary routes that
bracketed the moved admin blocks stay in api.py — they're extracted
in Tasks 5 & 6.
Behaviour-preserving: pytest = 20 failed / 1253 passed / 10 skipped
= baseline match. Admin-only subset (audit-log + backup + scheduler +
rotate-key + reload-config) = 34 passed.
The 999, 277CA, and TA1 parsers already enforce envelope correctness at
the parser level (parse_999.py line 290 raises 'No AK9 segment found';
parse_277ca.py line 298 raises 'Expected ST*277 or ST*277CA'; parse_ta1.py
line 111 raises 'Expected TA1, got <other>'). These tests lock the HTTP
surface contract: a wrong-kind file POSTed to those endpoints must come
back as 400, never as 200 or 500.
Tests added:
- test_api_999.py: rejects_837_input, rejects_835_input
- test_api_277ca.py: rejects_835_input, rejects_837_input
- test_api_ta1.py: rejects_837_input, rejects_835_input
If a future PR relaxes any of those parser-level guards, the
corresponding regression lock fires immediately.
Mirror of the parse-837 SP35 guards. Same defense-in-depth shape:
tokenize first, reject anything whose ST01 doesn't start with '835'
(400 'Mismatched file kind'), and after parse refuse to persist a
batch with zero CLP segments (400 'No claims parsed').
Reuses the _transaction_set_id_from_segments helper added by the
parse-837 commit.
New tests in tests/test_api_835.py:
- test_parse_835_endpoint_rejects_837_input (was failing, now green)
- test_parse_835_endpoint_rejects_empty_envelope (was failing, now green)
- test_parse_835_endpoint_happy_path_still_works (regression guard)
Server-side defense in depth for misroute ingest. Before SP35, posting
an 835 file (or any other X12 with a parseable ISA envelope) to
/api/parse-837 silently produced a BatchRecord with claims=[] and a
bogus row on the History tab. The 837 parser only required an ISA
envelope; it didn't check the ST transaction-set id.
Two new guards run before persistence:
1. Envelope check: tokenize first, read ST01, reject anything that
doesn't start with '837'. 400 with error='Mismatched file kind',
expected='837p', detected_st=<actual>. Catches an 835/999/270/etc
routed to the wrong endpoint.
2. Empty-claims check: even with the right envelope, if the parser
produces zero CLM segments, return 400 'No claims parsed' and do
NOT persist.
New tests in tests/test_api.py:
- test_parse_837_endpoint_rejects_835_input (was failing, now green)
- test_parse_837_endpoint_rejects_empty_envelope (was failing, now green)
- test_parse_837_endpoint_happy_path_still_works (regression guard)
Helper _transaction_set_id_from_segments reused by the 835 mirror.
- Adds `cyclone resubmit-rejected-claims` to push corrected single-claim
837 files to the Gainwell ToHPE SFTP dir. Idempotent (stat-then-skip by
byte size). One persistent paramiko session per batch with
reconnect-every=50 to dodge MOVEit's silent per-session file cap
(~200 puts/session, no exception).
- Validates each file via `parse_837` before upload and rejects any
whose payer_id is not `CO_TXIX` (catches a bad byte-fix early).
- Refreshes test fixtures (`minimal_837p.txt`, `co_medicaid_837p.txt`,
`co_medicaid_837p_with_renderer.txt`) and the corresponding test
assertions (`test_payer.py`, `test_payer_summary.py`,
`test_co_medicaid_fixture.py`, `test_parse_837.py`) from the old
`SKCO0`/`COHCPF` payer IDs to `CO_TXIX`, matching
PayerConfig.co_medicaid() and the HCPF 837P Companion Guide.
- Adds `ingest/` to .gitignore — local scratch / production-data
staging only.
The 999 handler's rejection pass was looking up claims by patient
control number, but Gainwell rejects at the SET level (ST envelope)
when the whole batch fails the NM109=CO_TXIX rule. That meant a SET
rejection was treated as a no-op even though every claim in the SET
was actually rejected by the payer.
Add a batch_envelope_index param (mirrors apply_999_acceptances from
SP28) so SET-level rejections cascade to every claim in the SET.
Falls back to the legacy PCN lookup when the index has no entry.
Also tightens test_payer.py: PayerConfig.co_medicaid() now returns
payer_id='CO_TXIX' and payer_name='CO_TXIX' per HCPF 837P Companion
Guide (June 2025 - Version 2.5).
- backend: GET /api/batches now returns acceptedCount/rejectedCount/
pendingCount/billedTotal/topRejectionReason/hasProblem per item
(one GROUP BY query + one ordered scan, no N+1)
- backend: 2 tests pin the 837p full-bucket case + the 835 zero case
- frontend: BatchSummary extended with 6 optional fields
(backwards compat preserved)
- frontend: new RecentBatchesWidget renders one row per batch with
status icon + billed total + accepted count + top rejection reason
- frontend: 837p rows show $ total + 'N/M accepted'; 835 rows show
payment count (Remittance has no batch-level total_charge)
- frontend: full-width row between KPI tiles and Activity on the
Dashboard; click navigates to /batches?batch=ID
- frontend: 3 component tests cover empty/clean/problem/835 branches
- backend: _ack_summary_for_claims helper attaches claim_acks payload
to inbox rejected-lane rows (3 tests)
- frontend: InboxRow renders newest 3 AK2 chips + '+N more' overflow
under the rejection reason, with '999 not linked' marker when an
ack couldn't be linked to a claim
- frontend: per-row Resubmit button downloads a single corrected 837
via api.serializeClaim837 (no bulk modal, no zip — one click, one
.x12 file)
- frontend: 2 new tests for the chip rendering and the per-row
download flow
The SP28 helpers (batch_envelope_index + lookup_claims_for_ack_set_response
+ two _batch_lookup closures in handle_ta1 + api.py) all filter
Batch.kind == "837", but prod Batch rows are persisted with
kind="837p" (lowercase p — see api.py:443 / store/records.py:58).
Result: the D10 batch-envelope index was empty on prod and zero 999 /
277CA / TA1 acks auto-linked to their claims via Pass 1 (ST02 via
batch.envelope.control_number). Pass 2 (PCN) was the only path firing,
which on this codebase matches 0 acks (Gainwell's 999 echoes the 837's
ST02, not its CLM01 — see spec §D10).
Fix: 4 production sites swapped to Batch.kind == "837p". 3 test
files updated to use the prod value (test_apply_claim_ack_links.py +
test_api_claim_acks.py + test_e2e_999_to_claim_drawer.py). All 25 SP28
tests + 20 handler tests pass. Expected prod effect: claim
t991102989o1c120d's 143 incoming 999 AK2 acks now auto-link via the
batch with envelope.control_number=991102989 (Pass 1), plus 727/1,398
total acks across all batches.
Phase 5 of SP28 (Ack↔Claim Auto-Link). Adds the public HTTP
surface for the claim_acks join table and wires it into the
existing claims + acks list endpoints.
New router (registered with matrix_gate auth, any-logged-in-user):
* GET /api/claims/{id}/acks — per-claim links
* GET /api/claims/{id}/acks/stream — NDJSON live-tail
* GET /api/acks/{kind}/{id}/claims — per-ack links
* GET /api/acks/{kind}/{id}/claims/stream — NDJSON live-tail
* POST /api/acks/{kind}/{id}/match-claim — manual link (D5)
* DELETE /api/acks/{kind}/{id}/match-claim/{claim_id} — unlink
* GET /api/inbox/ack-orphans — orphan reconciliation
Manual match is any-logged-in-user (D5/D9), idempotent on the
(claim_id, ack_kind, ack_id) dedup key, rejects with 409 when the
claim is in a terminal state (REVERSED), and publishes
claim_ack_written / claim_ack_dropped on the bus so live-tail
subscribers refresh.
Existing list endpoints extended:
* /api/acks — each item gains linked_claim_ids
* /api/ta1-acks — each item gains linked_claim_ids
* /api/277ca-acks — each item gains linked_claim_ids
* /api/claims/{id} — body gains ack_links (compact form)
All three list extensions use a single batched SELECT against
claim_acks to avoid N+1.
Tests:
* tests/test_api_claim_acks.py — 8 tests covering all 7
endpoints + the spec §6 named tests for the extended
surfaces (claim_detail_includes_ack_links,
acks_list_includes_linked_claim_ids,
claim_acks_stream_emits_claim_ack_written). The stream test
uses the established direct-endpoint-invocation pattern
from test_api_stream_live.py so it gets true byte-streaming
+ clean async cancellation.
Phase 4 of SP28 (Ack↔Claim Auto-Link). Refactors the per-AK2
helpers in cyclone.claim_acks to return ClaimAckLinkRow dataclasses
instead of mutating the session directly — the orchestrator (the
999 / 277CA / TA1 handlers + the matching parse-* API endpoints)
now persists each row via cycl_store.add_claim_ack so the
publish-from-store contract owns the live-tail event emission.
* handle_999 / handle_277ca / handle_ta1 — build batch envelope
index outside the work session (SQLite + concurrent sessions
causes 'database is locked'), call apply_X to get
ClaimAckLinkRow dataclasses, snapshot the rows before committing
the work session, then call cycl_store.add_claim_ack per row
in fresh sessions.
* /api/parse-999 / /api/parse-277ca / /api/parse-ta1 — mirror the
handler chain with event_bus passed through so live-tail
subscribers on the claim and ack sides see the new rows the
moment they land. Adds a 'claim_ack_links_count' field to each
ack response (spec §4).
* lookup_claims_for_ack_set_response — now accepts either a
callable OR a plain dict as batch_envelope_index (the store
returns a dict; tests pass callables).
* test_apply_claim_ack_links.py — 15 tests updated to assert on
the dataclass shape and exercise the full helper→add_claim_ack
cycle (so idempotency is verified at the store layer).
* test_e2e_999_to_claim_drawer.py — 2 new tests covering the
D10 two-pass join end-to-end via FastAPI TestClient (Pass 1
via ST02 + Pass 2 via PCN fallback).
The persistence layer for the SP28 join table lands in
cyclone/store/claim_acks.py:
- add_claim_ack — insert + publish 'claim_ack_written' on the bus
(mirrors the publish-from-store pattern used by the ACKs paths)
- list_acks_for_claim / list_claims_for_ack — read helpers for the
two list endpoints
- find_ack_orphans — Inbox ack-orphans lane source: every ack row
whose ack_id has no ClaimAck row tied to it
- remove_claim_ack — unlink + publish 'claim_ack_dropped'
- batch_envelope_index — D10 in-memory map of
{envelope.control_number: batch.id}, called once per ingest (cost
is ~16 lookups today; trivially cheap)
Plus the to_ui_claim_ack serializer in store/ui.py mirroring
to_ui_ack shape — single source of truth for the wire format so the
live-tail event payload matches the list endpoint byte-for-byte.
Includes a single SELECT against claims for the claim_state field
(TA1 batch-level rows return 'n/a').
The CycloneStore facade in store/__init__.py re-exports all six
methods (add_claim_ack, list_acks_for_claim, list_claims_for_ack,
find_ack_orphans, remove_claim_ack, batch_envelope_index) plus
to_ui_claim_ack from the ui module.
Migration-version assertions in test_acks.py and test_db_migrate.py
bumped 17 → 18 to match the new migration head.
Steps 3.1/3.2/3.3/3.4 of the SP28 implementation plan.
The auto-linker closes the operator gap where every inbound 999 /
277CA / TA1 ack was persisted but never linked back to the claim it
acknowledges. Five pure helpers land in cyclone/claim_acks.py:
- lookup_claims_for_ack_set_response — D10 two-pass join. Primary is
Batch.envelope.control_number (== source 837 ST02 for Gainwell
batches); fallback is Claim.patient_control_number. Pass 1 wins,
the two paths cannot both fire.
- apply_999_acceptances — walks parsed_999.set_responses, emits one
ClaimAck per AK2 per matched claim (one-ack-to-many supported).
Both accepted AND rejected AK2s link; per-AK2 granularity.
- apply_277ca_acks — same shape for parsed_277ca.claim_statuses.
STC category code carried on the link row's
set_accept_reject_code so the UI can render the lane inline
without re-parsing raw_json.
- apply_ta1_envelope_link — envelope-level link. The link row has
claim_id NULL + batch_id populated (the spec's batch-level TA1
trace). Sender/receiver matching is delegated to a closure the
caller supplies.
- link_manual — manually link an ack to a claim. Used by the new
/api/acks/{kind}/{id}/match-claim endpoint. Idempotent.
All five helpers are pure (callers own the session); idempotent via
the partial unique index ux_claim_acks_dedup (helpers pre-check to
avoid IntegrityError log noise on re-ingest); flush-only (callers
commit).
14 of 15 named tests pass (the facade surface check belongs in
Phase 3 once store/claim_acks.py lands).
Steps 2.1/2.2/2.3/2.4/2.5 of the SP28 implementation plan.
Adds two streaming endpoints that match the live-tail wire format
established by /api/claims/stream, /api/remittances/stream, and
/api/activity/stream:
* /api/acks/stream — subscribes to ack_received
* /api/ta1-acks/stream — subscribes to ta1_ack_received
Both yield a snapshot of existing rows (newest first, capped by the
`limit` query param), then a `snapshot_end` line, then forward
live events from the bus. They are registered before the
/{ack_id} path-param endpoints so the literal `stream` segment
isn't matched as an id.
Tests use the same direct-coroutine pattern as test_api_stream_live.py
because httpx.ASGITransport buffers the response body and never
delivers a disconnect message — iterating body_iterator directly
with body_iterator.aclose() simulates a client disconnect.
The Remittances page's three KPI tiles — REMITS / TOTAL PAID /
ADJUSTMENTS — were computed page-locally via items.reduce(...) over
the merged tail of the current page + live delta. With a 100-row
default limit, a 1,739-row population showed count=100, paid=$16,934,
adjustments=$147 — silently understating reality because the page
hadn't loaded the remaining rows yet.
This change mirrors the silent-incompleteness fix that
/api/dashboard/kpis (commit 59c3275) and /api/remittances (commit
d81b6ed) made for their tiles:
* CycloneStore.summarize_remittances() iterates the full filtered
remittance population (no limit) and returns
{count, total_paid, total_adjustments}. Mirrors iter_remittances
with limit=_ITER_UNBOUNDED.
* GET /api/remittances/summary — server endpoint with the same
filter parameters as /api/remittances. Registered BEFORE the
/api/remittances/stream handler so FastAPI doesn't treat
'summary' as a stream sub-path.
* api.listRemittanceSummary + useRemittanceSummary hook.
* Remittances.tsx swaps off items.reduce, consumes the server
summary. Tiles render the server totals so the values reflect
the entire DB population, not the page-local sample.
Verified live: /api/remittances/summary returns
{count: 1739, total_paid: 227181.58, total_adjustments: 13792.65},
which matches DB ground truth exactly.
Tests: 8 new backend tests in test_api_remittances_summary.py; 1 new
frontend test in Remittances.test.tsx (kpi_tiles_use_server_summary_not
_page_local_reduce) plus the page-level test for the zero-valued mock
default.
The 837 ingest path (_claim_837_row in store.py:194) populated
Claim.patient_control_number from claim.subscriber.member_id — the
subscriber's 2010BA NM109 Medicaid ID — instead of from
claim.claim_id (CLM01, the claim submitter's identifier the 837
actually sent).
That silently broke every downstream join that uses this column as a
cross-reference key:
* reconcile.match() (reconcile.py:74) — joins
Claim.patient_control_number against
Remittance.payer_claim_control_number (which is parsed from CLP01,
the 835's echo of CLM01 per X12 spec). Member_id never matches
CLP01, so auto-match always fails.
* apply_999_rejections — same lookup, same broken key.
* apply_277ca_rejections — same lookup, same broken key.
* scoring.score_pair — same broken key.
Live DB probe (1,739 remits, 337 claims):
* Claim.id == Remit.payer_claim_control_number → 0 matches
* Claim.patient_control_number == Remit.payer_claim_control_number
→ 9 matches (substring coincidences; synthetic PCN strings share
alphanumeric characters with the human-readable member_ids)
* Claim.matched_remittance_id NOT NULL → 0 claims
This commit changes _claim_837_row to write
Claim.patient_control_number = claim.claim_id (= CLM01). The
reconcile matcher's existing join now hits the row the 835 echoes
back. Companion migration 0017 backfills the 337 pre-fix rows so
they're on equal footing with new ingests (UPDATE claims SET
patient_control_number = id WHERE patient_control_number IS DISTINCT
FROM id — idempotent).
Tests:
* 2 new RED→GREEN tests in test_store_reconcile.py:
- test_837_ingest_populates_patient_control_number_from_claim_id
pins the field semantics directly
- test_837_then_835_with_echoed_pcn_auto_pairs proves the full
end-to-end auto-match now fires
* 3 existing manual_match tests that relied on the bug (had setup
using member_id != claim_id to deliberately prevent auto-match)
updated to use distinct PCNs explicitly so the tests still
exercise the manual-match path with a real orphan pair.
* 3 store_claim_detail / api_gets tests adjusted for the same
reason.
Verified: 1,176 / 1,177 backend tests pass; the one failure is the
pre-existing flake in test_provider_extended_response.py noted before
this work started.
Caveat for the existing dev DB: the 1,739 remits already in the DB
were ingested from 835 fixtures whose CLP01 is a different synthetic
identifier than the 837's CLM01 (the test fixtures never echoed
CLM01 — a fixture-data limitation, not a code bug). After this fix,
new 837+835 ingest pairs whose payer echoes CLM01 in CLP01 will
auto-match as expected. The pre-existing 1,739 remits will continue
to land in the unmatched bucket; that can only be fixed by
regenerating the test fixtures (out of scope for this SP).
Before this fix, the list endpoints computed `total` via
`len(list(store.iter_*(**common)))`. Both `iter_claims` and
`iter_remittances` default to `limit=100`, so the reported total
silently capped at 100 even when the DB held 60k claims or 835
remits. The frontend rendered a `data.total` of 100 in the KPI
tile and a 100-row table, the page looked complete, and the bug
stayed hidden — exactly the Dashboard silent-failure pattern that
59c3275 (server-aggregated KPIs) was meant to retire.
The Remittances page symptom reported on Jun 29 ('REMITS 100',
empty CLAIM column on 100 rows) was this bug. The empty CLAIM is
a separate matching concern (those remits have `claim_id = NULL`
in the DB); the count fix addresses the population-size lie.
Fix:
* `CycloneStore.count_claims` + `count_remittances` reuse the
iter's filter pipeline (DB filters + in-memory payer/date
filters) with an effectively-unbounded limit so the count
reflects the true DB population.
* `list_claims` + `list_remittances` call the new count
helpers instead of slicing `iter_*` twice.
* 13 new tests in `test_list_endpoint_counts.py` cover the
empty/filter/per-dimension/cardinality cases plus the HTTP
regression (seed 150/120, assert total matches).
Bonus fix:
* `conftest._reset_rate_limit_buckets` walks the middleware
stack and clears `RateLimitMiddleware._buckets` between
tests. Without this, the full suite tripped the 300 req/60s
rate limiter mid-run and 9 later tests (4 of mine, 5
pre-existing in test_inbox_endpoints / test_payer_summary)
got 429s. All 1167 tests now pass.
Follow-ups noted but out of scope:
* `/api/admin/audit-log`, `/api/admin/backup/list`,
`/api/admin/scheduler/processed-files` use the same
`len(rows)`-after-`.limit(limit)` pattern. Admin-only,
no `has_more` consumption, so impact is bounded.
* `count_*` could short-circuit to `func.count()` to skip
the UI-dict build, but iter already calls `q.all()` so
the win is modest.
The Dashboard was hardcoded to useClaims({ limit: 100 }) and reduce
KPIs client-side. With 60k+ claims in production, every tile
(Billed $940K, Received $59K, Denial rate, Pending AR, monthly
sparkline, top providers, recent denials) was computed from a
0.16% sample of the dataset — silently wrong numbers on the
operator's primary view.
Fix: add GET /api/dashboard/kpis that aggregates server-side in
one read over the entire claim population. The new useDashboardKpis
hook consumes it and polls every 60s. Dashboard.tsx drops
useClaims({limit:100}) + useProviders() + the client-side
buildMonthly reduce.
Backend:
- store.py: dashboard_kpis() — one Claim query (selectinload on
batch to avoid N+1) + one bulk Remittance lookup, Python reduce
over the full population. Zero-filled response for empty DB.
- api.py: GET /api/dashboard/kpis behind matrix_gate, query-param
clamps (1..24 months, 0..50 top_n_*).
Frontend:
- api.ts: DashboardKpis types + getDashboardKpis() wrapper.
- useDashboardKpis.ts: TanStack Query hook, 60s refetchInterval,
bypass to data:undefined when not configured.
- Dashboard.tsx: switched to useDashboardKpis, extracted
ZERO_TOTALS constant, dropped the buildMonthly helper.
Tests:
- backend/tests/test_dashboard_kpis.py: 12 tests covering empty DB,
matched-remit math, pending-state semantics, monthly binning,
top-providers/top-denials sort + cap, orphan-claim defensive
guard, HTTP wiring + param validation.
- src/hooks/useDashboardKpis.test.ts: 3 tests for the hook
contract (configured path, unconfigured fallback, param
passthrough).
- src/pages/Dashboard.test.tsx: wrapped renders in
QueryClientProvider + stubbed useAuth + isConfigured=false. This
fixes 3 pre-existing Dashboard test failures (the page never had
a QueryClient set up because useClaims/useProviders were the
first useQuery hooks in the page).
Reviewer fixes (same commit):
1. topDenials sort placed empty submissionDate claims first under
reverse-lex. Drop them at append time.
2. r.batch lazy-load → N+1 on 60k rows. selectinload(Claim.batch).
3. pending_states rebuilt per call as a mutable set — moved to
module-level _DASHBOARD_PENDING_STATES frozenset.
4. Module-level ProviderORM import inconsistent with the
"local import inside the function" pattern — moved inline.
Add check_matched_pair_drift() at module level in store.py — read-only
audit of the Claim.matched_remittance_id ↔ Remittance.claim_id FK pair.
Logs WARNING on drift with up to 5 examples of each of two cases
(claim-side and remit-side), returns the count of drifted rows. Wired
into api.py::lifespan right after db.init_db() and ensure_clearhouse_seeded(),
wrapped in try/except so a query failure logs but doesn't crash boot.
The symmetric-write + symmetric-clear invariants on
manual_match / manual_unmatch were already correct; this commit pins
them with focused regression tests in test_store_match_invariant.py
(8 tests covering both directions, both happy-path, rollback pin, and
drift-check unit behavior). PCN asymmetry in the fixture prevents the
auto-match inside CycloneStore.add (Task 10) from pre-pairing, so
manual_match is the only writer.
Migration 0016 adds the missing ix_claims_matched_remittance_id index —
the drift check scans WHERE matched_remittance_id IS NOT NULL on every
boot and would become a full-table scan past ~10k claims. Symmetric
with ix_remittances_claim_id (added in 0007). Migration tests bumped
from head=15 to head=16.
Nora
Cyclone SP37
sp37 bot