Followup #1 from the SP37 final-state tracker. The pre-existing
('POST', '/api/resubmit'): WRITE_ROLES entry only matched the exact
path /api/resubmit (no children — the prefix-match logic requires
the trailing slash for sub-paths). No route is registered at that
path; the actual resubmit lives at /api/inbox/rejected/resubmit
(which has its own entry).
Removing the dead entry is fail-closed: any future request to
/api/resubmit (or its descendants) returns None from allowed_roles,
which means deny. Without this cleanup, the matrix silently granted
WRITE_ROLES to a non-existent route.
Tests (test_auth_permissions_matrix.py, 5 cases):
* Exact path /api/resubmit not in PERMISSIONS
* allowed_roles('POST', '/api/resubmit') is None
* /api/resubmit/, /api/resubmit/anything all deny
Pin the invariant so a future contributor can't re-add a dead prefix.
TDD: tests written first, watched fail (5 fail with the dead entry),
removed the line, tests pass (5 pass). Full SP37 chain (36 tests)
passes with the change.
Thin HTTP wrapper around `cyclone.submission.submit_file` — the
HTTP twin of the `cyclone submit-batch` CLI (SP37 Task 5). Same
walker pattern, same `._*` AppleDouble skip, same `limit`
semantics, same per-file outcomes; both surfaces now agree on DB +
SFTP state byte-for-byte.
Body: `{ingest_dir, validate?, actor?, limit?}`. Response:
`{submitted, skipped, failed, results:[{file, outcome, batch_id,
error}]}`. Status codes:
- 200 on completed runs (per-file failures live in the body)
- 401 unauthenticated (matrix_gate)
- 404 no clearhouse seeded
- 409 clearhouse SFTP block in stub mode
- 422 ingest_dir missing on disk or Pydantic body error
Does NOT inject `sftp_client_factory` — submit_file uses its
default paramiko factory so SKIPPED is reachable in production.
Tests monkey-patch submit_file itself.
Permissions entry: `POST /api/submit-batch` → WRITE_ROLES (admin +
user), matching `POST /api/resubmit` posture.
Tests (11): happy path, AppleDouble skip, limit, empty ingest_dir
422, mixed-outcome 200, unexpected-exception-as-failed,
Pydantic-missing-422, on-disk-missing-422, no-clearhouse-404,
stub-mode-409, auth-gate-401.
Walks batch-*-claims/*.x12 under --ingest-dir, calls submit_file per
file, prints submitted/skipped/failed counts. Exits 0 even on
per-file failures (details in stdout); exits 2 on config-level
errors (no clearhouse, stub mode, missing dir).
This is the canonical outbound path; resubmit-rejected-claims
remains for one-off cases where no DB row is wanted.
SftpClient wrapper lacks a stat() method, so the previous default
factory produced a client whose stat() raised AttributeError — making
the SKIPPED outcome unreachable in production and silently
misclassifying all uploads as SFTP_FAILED. The helper now opens a
paramiko SFTP session directly (same pattern as
resubmit-rejected-claims in cli.py:620-650), so stat() and write_file()
both work end-to-end.
Also: tighten typing (SftpBlock instead of Any), add PAYER_MISMATCH
test, drop duplicate _db fixture, rename misleading test, add class
docstrings to SubmitResult/SubmitOutcome.
Owns the parse → DB write → SFTP upload sequence in one place. CLI
and HTTP thin-call it. DB-first invariant: if add_record raises, no
SFTP call is made. Idempotency: add_record dedupes via s.get(Claim,
claim_id); SFTP layer dedupes via stat().st_size match.
- test_acks.py:test_migration_latest_idempotent_on_fresh_db was still
asserting user_version == 19; migration 0020 (Task 1) bumped it to 20.
Without this fix, the full pytest suite fails the moment SP37 ships.
- claim_acks.py module docstring + store/__init__.py facade docstring
still advertised the index as single-key {envelope.control_number:
batch.id}. SP37 Task 3 made it dual-key (ISA13 + ST02); the docs now
match the implementation.
Each 837p batch row contributes up to two entries to the join-key
index (envelope.control_number + transaction_set_control_number). One
idx.get(set_control_number) call resolves either, so 999 acks whose
AK201 echoes the source 837's ST02 now hit Pass 1 where they
previously fell through to Pass 2 (and to the orphan log).
Backward compat preserved: rows with transaction_set_control_number
NULL (pre-SP37 batches) still resolve by ISA13.
The parser now captures the source 837's ST02 (transaction set control
number) on the Envelope model as 'transaction_set_control_number'.
add_record reads it off the envelope and stores it on the Batch row,
mirroring how 'envelope.control_number' (ISA13) was already handled.
Unlocks the SP37 join-key update so 999 set_control_number (AK201)
values resolve back to the right batch via Pass 1.
Migration 0020 adds the column additively (nullable, no default) and
backfills from raw_result_json.envelope.transaction_set_control_number
for any existing batch rows that already carry it. Required for the
SP37 join-key update so 999 acks can resolve by ST02 (the source 837's
transaction set control number) instead of just ISA13.
Moves the entire /api/admin/* namespace (audit-log ×2, db/rotate-key ×1,
backup ×10, scheduler ×6, reload-config ×1) out of api.py and into the
existing api_routers/admin.py, which already owned /api/admin/validate-provider.
- api.py: 4294 → 3547 LOC (Δ -747). Removed the orphaned # --- separator
left behind by the cut, the orphaned 'return {ok,loaded,errors}'
tail of reload-config, and the now-unused cyclone.clearhouse.InboundFile
top-level import.
- api_routers/admin.py: 60 → 851 LOC. Added the imports the moved
routes need (json, logging, threading, time, AuditEvent, verify_chain,
InboundFile) and stripped the redundant top-level imports of
symbols that the route bodies reach via the inline _db_crypto /
_secrets / _backup_svc_mod / _backup_sched_mod / _scheduler_mod
/ _audit aliases (those aliases stay — tests rely on being able to
monkeypatch cyclone.api_routers.admin._X.method). Hoisted the inline
'import time' from inside pull_inbound to module scope. Dropped
the redundant 'import threading as _threading' alias — top-level
'import threading' already covers it.
- tests/test_api_rotate_key.py: 4 monkeypatch targets migrated from
cyclone.api._db_crypto / cyclone.api._secrets to
cyclone.api_routers.admin._db_crypto / cyclone.api_routers.admin._secrets
to match the new home of the rotate-key endpoint. Lock acquire/release
also migrated.
The non-admin /api/config/* and /api/payers/{id}/summary routes that
bracketed the moved admin blocks stay in api.py — they're extracted
in Tasks 5 & 6.
Behaviour-preserving: pytest = 20 failed / 1253 passed / 10 skipped
= baseline match. Admin-only subset (audit-log + backup + scheduler +
rotate-key + reload-config) = 34 passed.
The 999, 277CA, and TA1 parsers already enforce envelope correctness at
the parser level (parse_999.py line 290 raises 'No AK9 segment found';
parse_277ca.py line 298 raises 'Expected ST*277 or ST*277CA'; parse_ta1.py
line 111 raises 'Expected TA1, got <other>'). These tests lock the HTTP
surface contract: a wrong-kind file POSTed to those endpoints must come
back as 400, never as 200 or 500.
Tests added:
- test_api_999.py: rejects_837_input, rejects_835_input
- test_api_277ca.py: rejects_835_input, rejects_837_input
- test_api_ta1.py: rejects_837_input, rejects_835_input
If a future PR relaxes any of those parser-level guards, the
corresponding regression lock fires immediately.
Mirror of the parse-837 SP35 guards. Same defense-in-depth shape:
tokenize first, reject anything whose ST01 doesn't start with '835'
(400 'Mismatched file kind'), and after parse refuse to persist a
batch with zero CLP segments (400 'No claims parsed').
Reuses the _transaction_set_id_from_segments helper added by the
parse-837 commit.
New tests in tests/test_api_835.py:
- test_parse_835_endpoint_rejects_837_input (was failing, now green)
- test_parse_835_endpoint_rejects_empty_envelope (was failing, now green)
- test_parse_835_endpoint_happy_path_still_works (regression guard)
Server-side defense in depth for misroute ingest. Before SP35, posting
an 835 file (or any other X12 with a parseable ISA envelope) to
/api/parse-837 silently produced a BatchRecord with claims=[] and a
bogus row on the History tab. The 837 parser only required an ISA
envelope; it didn't check the ST transaction-set id.
Two new guards run before persistence:
1. Envelope check: tokenize first, read ST01, reject anything that
doesn't start with '837'. 400 with error='Mismatched file kind',
expected='837p', detected_st=<actual>. Catches an 835/999/270/etc
routed to the wrong endpoint.
2. Empty-claims check: even with the right envelope, if the parser
produces zero CLM segments, return 400 'No claims parsed' and do
NOT persist.
New tests in tests/test_api.py:
- test_parse_837_endpoint_rejects_835_input (was failing, now green)
- test_parse_837_endpoint_rejects_empty_envelope (was failing, now green)
- test_parse_837_endpoint_happy_path_still_works (regression guard)
Helper _transaction_set_id_from_segments reused by the 835 mirror.
- Adds `cyclone resubmit-rejected-claims` to push corrected single-claim
837 files to the Gainwell ToHPE SFTP dir. Idempotent (stat-then-skip by
byte size). One persistent paramiko session per batch with
reconnect-every=50 to dodge MOVEit's silent per-session file cap
(~200 puts/session, no exception).
- Validates each file via `parse_837` before upload and rejects any
whose payer_id is not `CO_TXIX` (catches a bad byte-fix early).
- Refreshes test fixtures (`minimal_837p.txt`, `co_medicaid_837p.txt`,
`co_medicaid_837p_with_renderer.txt`) and the corresponding test
assertions (`test_payer.py`, `test_payer_summary.py`,
`test_co_medicaid_fixture.py`, `test_parse_837.py`) from the old
`SKCO0`/`COHCPF` payer IDs to `CO_TXIX`, matching
PayerConfig.co_medicaid() and the HCPF 837P Companion Guide.
- Adds `ingest/` to .gitignore — local scratch / production-data
staging only.
The 999 handler's rejection pass was looking up claims by patient
control number, but Gainwell rejects at the SET level (ST envelope)
when the whole batch fails the NM109=CO_TXIX rule. That meant a SET
rejection was treated as a no-op even though every claim in the SET
was actually rejected by the payer.
Add a batch_envelope_index param (mirrors apply_999_acceptances from
SP28) so SET-level rejections cascade to every claim in the SET.
Falls back to the legacy PCN lookup when the index has no entry.
Also tightens test_payer.py: PayerConfig.co_medicaid() now returns
payer_id='CO_TXIX' and payer_name='CO_TXIX' per HCPF 837P Companion
Guide (June 2025 - Version 2.5).
- backend: GET /api/batches now returns acceptedCount/rejectedCount/
pendingCount/billedTotal/topRejectionReason/hasProblem per item
(one GROUP BY query + one ordered scan, no N+1)
- backend: 2 tests pin the 837p full-bucket case + the 835 zero case
- frontend: BatchSummary extended with 6 optional fields
(backwards compat preserved)
- frontend: new RecentBatchesWidget renders one row per batch with
status icon + billed total + accepted count + top rejection reason
- frontend: 837p rows show $ total + 'N/M accepted'; 835 rows show
payment count (Remittance has no batch-level total_charge)
- frontend: full-width row between KPI tiles and Activity on the
Dashboard; click navigates to /batches?batch=ID
- frontend: 3 component tests cover empty/clean/problem/835 branches
- backend: _ack_summary_for_claims helper attaches claim_acks payload
to inbox rejected-lane rows (3 tests)
- frontend: InboxRow renders newest 3 AK2 chips + '+N more' overflow
under the rejection reason, with '999 not linked' marker when an
ack couldn't be linked to a claim
- frontend: per-row Resubmit button downloads a single corrected 837
via api.serializeClaim837 (no bulk modal, no zip — one click, one
.x12 file)
- frontend: 2 new tests for the chip rendering and the per-row
download flow
The SP28 helpers (batch_envelope_index + lookup_claims_for_ack_set_response
+ two _batch_lookup closures in handle_ta1 + api.py) all filter
Batch.kind == "837", but prod Batch rows are persisted with
kind="837p" (lowercase p — see api.py:443 / store/records.py:58).
Result: the D10 batch-envelope index was empty on prod and zero 999 /
277CA / TA1 acks auto-linked to their claims via Pass 1 (ST02 via
batch.envelope.control_number). Pass 2 (PCN) was the only path firing,
which on this codebase matches 0 acks (Gainwell's 999 echoes the 837's
ST02, not its CLM01 — see spec §D10).
Fix: 4 production sites swapped to Batch.kind == "837p". 3 test
files updated to use the prod value (test_apply_claim_ack_links.py +
test_api_claim_acks.py + test_e2e_999_to_claim_drawer.py). All 25 SP28
tests + 20 handler tests pass. Expected prod effect: claim
t991102989o1c120d's 143 incoming 999 AK2 acks now auto-link via the
batch with envelope.control_number=991102989 (Pass 1), plus 727/1,398
total acks across all batches.
Phase 5 of SP28 (Ack↔Claim Auto-Link). Adds the public HTTP
surface for the claim_acks join table and wires it into the
existing claims + acks list endpoints.
New router (registered with matrix_gate auth, any-logged-in-user):
* GET /api/claims/{id}/acks — per-claim links
* GET /api/claims/{id}/acks/stream — NDJSON live-tail
* GET /api/acks/{kind}/{id}/claims — per-ack links
* GET /api/acks/{kind}/{id}/claims/stream — NDJSON live-tail
* POST /api/acks/{kind}/{id}/match-claim — manual link (D5)
* DELETE /api/acks/{kind}/{id}/match-claim/{claim_id} — unlink
* GET /api/inbox/ack-orphans — orphan reconciliation
Manual match is any-logged-in-user (D5/D9), idempotent on the
(claim_id, ack_kind, ack_id) dedup key, rejects with 409 when the
claim is in a terminal state (REVERSED), and publishes
claim_ack_written / claim_ack_dropped on the bus so live-tail
subscribers refresh.
Existing list endpoints extended:
* /api/acks — each item gains linked_claim_ids
* /api/ta1-acks — each item gains linked_claim_ids
* /api/277ca-acks — each item gains linked_claim_ids
* /api/claims/{id} — body gains ack_links (compact form)
All three list extensions use a single batched SELECT against
claim_acks to avoid N+1.
Tests:
* tests/test_api_claim_acks.py — 8 tests covering all 7
endpoints + the spec §6 named tests for the extended
surfaces (claim_detail_includes_ack_links,
acks_list_includes_linked_claim_ids,
claim_acks_stream_emits_claim_ack_written). The stream test
uses the established direct-endpoint-invocation pattern
from test_api_stream_live.py so it gets true byte-streaming
+ clean async cancellation.
Phase 4 of SP28 (Ack↔Claim Auto-Link). Refactors the per-AK2
helpers in cyclone.claim_acks to return ClaimAckLinkRow dataclasses
instead of mutating the session directly — the orchestrator (the
999 / 277CA / TA1 handlers + the matching parse-* API endpoints)
now persists each row via cycl_store.add_claim_ack so the
publish-from-store contract owns the live-tail event emission.
* handle_999 / handle_277ca / handle_ta1 — build batch envelope
index outside the work session (SQLite + concurrent sessions
causes 'database is locked'), call apply_X to get
ClaimAckLinkRow dataclasses, snapshot the rows before committing
the work session, then call cycl_store.add_claim_ack per row
in fresh sessions.
* /api/parse-999 / /api/parse-277ca / /api/parse-ta1 — mirror the
handler chain with event_bus passed through so live-tail
subscribers on the claim and ack sides see the new rows the
moment they land. Adds a 'claim_ack_links_count' field to each
ack response (spec §4).
* lookup_claims_for_ack_set_response — now accepts either a
callable OR a plain dict as batch_envelope_index (the store
returns a dict; tests pass callables).
* test_apply_claim_ack_links.py — 15 tests updated to assert on
the dataclass shape and exercise the full helper→add_claim_ack
cycle (so idempotency is verified at the store layer).
* test_e2e_999_to_claim_drawer.py — 2 new tests covering the
D10 two-pass join end-to-end via FastAPI TestClient (Pass 1
via ST02 + Pass 2 via PCN fallback).
The persistence layer for the SP28 join table lands in
cyclone/store/claim_acks.py:
- add_claim_ack — insert + publish 'claim_ack_written' on the bus
(mirrors the publish-from-store pattern used by the ACKs paths)
- list_acks_for_claim / list_claims_for_ack — read helpers for the
two list endpoints
- find_ack_orphans — Inbox ack-orphans lane source: every ack row
whose ack_id has no ClaimAck row tied to it
- remove_claim_ack — unlink + publish 'claim_ack_dropped'
- batch_envelope_index — D10 in-memory map of
{envelope.control_number: batch.id}, called once per ingest (cost
is ~16 lookups today; trivially cheap)
Plus the to_ui_claim_ack serializer in store/ui.py mirroring
to_ui_ack shape — single source of truth for the wire format so the
live-tail event payload matches the list endpoint byte-for-byte.
Includes a single SELECT against claims for the claim_state field
(TA1 batch-level rows return 'n/a').
The CycloneStore facade in store/__init__.py re-exports all six
methods (add_claim_ack, list_acks_for_claim, list_claims_for_ack,
find_ack_orphans, remove_claim_ack, batch_envelope_index) plus
to_ui_claim_ack from the ui module.
Migration-version assertions in test_acks.py and test_db_migrate.py
bumped 17 → 18 to match the new migration head.
Steps 3.1/3.2/3.3/3.4 of the SP28 implementation plan.
The auto-linker closes the operator gap where every inbound 999 /
277CA / TA1 ack was persisted but never linked back to the claim it
acknowledges. Five pure helpers land in cyclone/claim_acks.py:
- lookup_claims_for_ack_set_response — D10 two-pass join. Primary is
Batch.envelope.control_number (== source 837 ST02 for Gainwell
batches); fallback is Claim.patient_control_number. Pass 1 wins,
the two paths cannot both fire.
- apply_999_acceptances — walks parsed_999.set_responses, emits one
ClaimAck per AK2 per matched claim (one-ack-to-many supported).
Both accepted AND rejected AK2s link; per-AK2 granularity.
- apply_277ca_acks — same shape for parsed_277ca.claim_statuses.
STC category code carried on the link row's
set_accept_reject_code so the UI can render the lane inline
without re-parsing raw_json.
- apply_ta1_envelope_link — envelope-level link. The link row has
claim_id NULL + batch_id populated (the spec's batch-level TA1
trace). Sender/receiver matching is delegated to a closure the
caller supplies.
- link_manual — manually link an ack to a claim. Used by the new
/api/acks/{kind}/{id}/match-claim endpoint. Idempotent.
All five helpers are pure (callers own the session); idempotent via
the partial unique index ux_claim_acks_dedup (helpers pre-check to
avoid IntegrityError log noise on re-ingest); flush-only (callers
commit).
14 of 15 named tests pass (the facade surface check belongs in
Phase 3 once store/claim_acks.py lands).
Steps 2.1/2.2/2.3/2.4/2.5 of the SP28 implementation plan.
Adds two streaming endpoints that match the live-tail wire format
established by /api/claims/stream, /api/remittances/stream, and
/api/activity/stream:
* /api/acks/stream — subscribes to ack_received
* /api/ta1-acks/stream — subscribes to ta1_ack_received
Both yield a snapshot of existing rows (newest first, capped by the
`limit` query param), then a `snapshot_end` line, then forward
live events from the bus. They are registered before the
/{ack_id} path-param endpoints so the literal `stream` segment
isn't matched as an id.
Tests use the same direct-coroutine pattern as test_api_stream_live.py
because httpx.ASGITransport buffers the response body and never
delivers a disconnect message — iterating body_iterator directly
with body_iterator.aclose() simulates a client disconnect.
The Remittances page's three KPI tiles — REMITS / TOTAL PAID /
ADJUSTMENTS — were computed page-locally via items.reduce(...) over
the merged tail of the current page + live delta. With a 100-row
default limit, a 1,739-row population showed count=100, paid=$16,934,
adjustments=$147 — silently understating reality because the page
hadn't loaded the remaining rows yet.
This change mirrors the silent-incompleteness fix that
/api/dashboard/kpis (commit 59c3275) and /api/remittances (commit
d81b6ed) made for their tiles:
* CycloneStore.summarize_remittances() iterates the full filtered
remittance population (no limit) and returns
{count, total_paid, total_adjustments}. Mirrors iter_remittances
with limit=_ITER_UNBOUNDED.
* GET /api/remittances/summary — server endpoint with the same
filter parameters as /api/remittances. Registered BEFORE the
/api/remittances/stream handler so FastAPI doesn't treat
'summary' as a stream sub-path.
* api.listRemittanceSummary + useRemittanceSummary hook.
* Remittances.tsx swaps off items.reduce, consumes the server
summary. Tiles render the server totals so the values reflect
the entire DB population, not the page-local sample.
Verified live: /api/remittances/summary returns
{count: 1739, total_paid: 227181.58, total_adjustments: 13792.65},
which matches DB ground truth exactly.
Tests: 8 new backend tests in test_api_remittances_summary.py; 1 new
frontend test in Remittances.test.tsx (kpi_tiles_use_server_summary_not
_page_local_reduce) plus the page-level test for the zero-valued mock
default.
The 837 ingest path (_claim_837_row in store.py:194) populated
Claim.patient_control_number from claim.subscriber.member_id — the
subscriber's 2010BA NM109 Medicaid ID — instead of from
claim.claim_id (CLM01, the claim submitter's identifier the 837
actually sent).
That silently broke every downstream join that uses this column as a
cross-reference key:
* reconcile.match() (reconcile.py:74) — joins
Claim.patient_control_number against
Remittance.payer_claim_control_number (which is parsed from CLP01,
the 835's echo of CLM01 per X12 spec). Member_id never matches
CLP01, so auto-match always fails.
* apply_999_rejections — same lookup, same broken key.
* apply_277ca_rejections — same lookup, same broken key.
* scoring.score_pair — same broken key.
Live DB probe (1,739 remits, 337 claims):
* Claim.id == Remit.payer_claim_control_number → 0 matches
* Claim.patient_control_number == Remit.payer_claim_control_number
→ 9 matches (substring coincidences; synthetic PCN strings share
alphanumeric characters with the human-readable member_ids)
* Claim.matched_remittance_id NOT NULL → 0 claims
This commit changes _claim_837_row to write
Claim.patient_control_number = claim.claim_id (= CLM01). The
reconcile matcher's existing join now hits the row the 835 echoes
back. Companion migration 0017 backfills the 337 pre-fix rows so
they're on equal footing with new ingests (UPDATE claims SET
patient_control_number = id WHERE patient_control_number IS DISTINCT
FROM id — idempotent).
Tests:
* 2 new RED→GREEN tests in test_store_reconcile.py:
- test_837_ingest_populates_patient_control_number_from_claim_id
pins the field semantics directly
- test_837_then_835_with_echoed_pcn_auto_pairs proves the full
end-to-end auto-match now fires
* 3 existing manual_match tests that relied on the bug (had setup
using member_id != claim_id to deliberately prevent auto-match)
updated to use distinct PCNs explicitly so the tests still
exercise the manual-match path with a real orphan pair.
* 3 store_claim_detail / api_gets tests adjusted for the same
reason.
Verified: 1,176 / 1,177 backend tests pass; the one failure is the
pre-existing flake in test_provider_extended_response.py noted before
this work started.
Caveat for the existing dev DB: the 1,739 remits already in the DB
were ingested from 835 fixtures whose CLP01 is a different synthetic
identifier than the 837's CLM01 (the test fixtures never echoed
CLM01 — a fixture-data limitation, not a code bug). After this fix,
new 837+835 ingest pairs whose payer echoes CLM01 in CLP01 will
auto-match as expected. The pre-existing 1,739 remits will continue
to land in the unmatched bucket; that can only be fixed by
regenerating the test fixtures (out of scope for this SP).
Before this fix, the list endpoints computed `total` via
`len(list(store.iter_*(**common)))`. Both `iter_claims` and
`iter_remittances` default to `limit=100`, so the reported total
silently capped at 100 even when the DB held 60k claims or 835
remits. The frontend rendered a `data.total` of 100 in the KPI
tile and a 100-row table, the page looked complete, and the bug
stayed hidden — exactly the Dashboard silent-failure pattern that
59c3275 (server-aggregated KPIs) was meant to retire.
The Remittances page symptom reported on Jun 29 ('REMITS 100',
empty CLAIM column on 100 rows) was this bug. The empty CLAIM is
a separate matching concern (those remits have `claim_id = NULL`
in the DB); the count fix addresses the population-size lie.
Fix:
* `CycloneStore.count_claims` + `count_remittances` reuse the
iter's filter pipeline (DB filters + in-memory payer/date
filters) with an effectively-unbounded limit so the count
reflects the true DB population.
* `list_claims` + `list_remittances` call the new count
helpers instead of slicing `iter_*` twice.
* 13 new tests in `test_list_endpoint_counts.py` cover the
empty/filter/per-dimension/cardinality cases plus the HTTP
regression (seed 150/120, assert total matches).
Bonus fix:
* `conftest._reset_rate_limit_buckets` walks the middleware
stack and clears `RateLimitMiddleware._buckets` between
tests. Without this, the full suite tripped the 300 req/60s
rate limiter mid-run and 9 later tests (4 of mine, 5
pre-existing in test_inbox_endpoints / test_payer_summary)
got 429s. All 1167 tests now pass.
Follow-ups noted but out of scope:
* `/api/admin/audit-log`, `/api/admin/backup/list`,
`/api/admin/scheduler/processed-files` use the same
`len(rows)`-after-`.limit(limit)` pattern. Admin-only,
no `has_more` consumption, so impact is bounded.
* `count_*` could short-circuit to `func.count()` to skip
the UI-dict build, but iter already calls `q.all()` so
the win is modest.
The Dashboard was hardcoded to useClaims({ limit: 100 }) and reduce
KPIs client-side. With 60k+ claims in production, every tile
(Billed $940K, Received $59K, Denial rate, Pending AR, monthly
sparkline, top providers, recent denials) was computed from a
0.16% sample of the dataset — silently wrong numbers on the
operator's primary view.
Fix: add GET /api/dashboard/kpis that aggregates server-side in
one read over the entire claim population. The new useDashboardKpis
hook consumes it and polls every 60s. Dashboard.tsx drops
useClaims({limit:100}) + useProviders() + the client-side
buildMonthly reduce.
Backend:
- store.py: dashboard_kpis() — one Claim query (selectinload on
batch to avoid N+1) + one bulk Remittance lookup, Python reduce
over the full population. Zero-filled response for empty DB.
- api.py: GET /api/dashboard/kpis behind matrix_gate, query-param
clamps (1..24 months, 0..50 top_n_*).
Frontend:
- api.ts: DashboardKpis types + getDashboardKpis() wrapper.
- useDashboardKpis.ts: TanStack Query hook, 60s refetchInterval,
bypass to data:undefined when not configured.
- Dashboard.tsx: switched to useDashboardKpis, extracted
ZERO_TOTALS constant, dropped the buildMonthly helper.
Tests:
- backend/tests/test_dashboard_kpis.py: 12 tests covering empty DB,
matched-remit math, pending-state semantics, monthly binning,
top-providers/top-denials sort + cap, orphan-claim defensive
guard, HTTP wiring + param validation.
- src/hooks/useDashboardKpis.test.ts: 3 tests for the hook
contract (configured path, unconfigured fallback, param
passthrough).
- src/pages/Dashboard.test.tsx: wrapped renders in
QueryClientProvider + stubbed useAuth + isConfigured=false. This
fixes 3 pre-existing Dashboard test failures (the page never had
a QueryClient set up because useClaims/useProviders were the
first useQuery hooks in the page).
Reviewer fixes (same commit):
1. topDenials sort placed empty submissionDate claims first under
reverse-lex. Drop them at append time.
2. r.batch lazy-load → N+1 on 60k rows. selectinload(Claim.batch).
3. pending_states rebuilt per call as a mutable set — moved to
module-level _DASHBOARD_PENDING_STATES frozenset.
4. Module-level ProviderORM import inconsistent with the
"local import inside the function" pattern — moved inline.
Add check_matched_pair_drift() at module level in store.py — read-only
audit of the Claim.matched_remittance_id ↔ Remittance.claim_id FK pair.
Logs WARNING on drift with up to 5 examples of each of two cases
(claim-side and remit-side), returns the count of drifted rows. Wired
into api.py::lifespan right after db.init_db() and ensure_clearhouse_seeded(),
wrapped in try/except so a query failure logs but doesn't crash boot.
The symmetric-write + symmetric-clear invariants on
manual_match / manual_unmatch were already correct; this commit pins
them with focused regression tests in test_store_match_invariant.py
(8 tests covering both directions, both happy-path, rollback pin, and
drift-check unit behavior). PCN asymmetry in the fixture prevents the
auto-match inside CycloneStore.add (Task 10) from pre-pairing, so
manual_match is the only writer.
Migration 0016 adds the missing ix_claims_matched_remittance_id index —
the drift check scans WHERE matched_remittance_id IS NOT NULL on every
boot and would become a full-table scan past ~10k claims. Symmetric
with ix_remittances_claim_id (added in 0007). Migration tests bumped
from head=15 to head=16.
Move 'reconcile.run(s, record.id)' inside CycloneStore.add()'s ingest
session, before s.commit(). The placeholder adjustment_amount set by
_remittance_835_row is overwritten by reconcile's CAS-aggregate pass
in the same transaction — readers never see a half-reconciled
Remittance row. If reconcile raises, the entire 835 ingest rolls back
via the session's __exit__.
The previous flow committed batch + remittance rows in session-1,
then opened session-2 to run reconcile.fail-soft. Two visible
problems closed: a race window where readers fetched the placeholder
adjustment_amount, and a half-reconciled state left visible if
reconcile crashed.
Deviation from the plan (N4 in autoreview): the reconcile call lives
in cycl_store.add() rather than handle_835 calling a new
reconcile.run_now(batch_id) helper after add(). Same end state, one
fewer module surface, handler stays a thin wrapper over the store.
The 06/25 silent hang was a hung ``sftp.listdir_attr()`` on the
worker thread — paramiko TCP-acked then went silent, the scheduler's
``asyncio.to_thread`` waited forever, and the operator had no
signal that polling had stalled. Every poll cycle since was
suspected of the same failure mode.
``backend/src/cyclone/clearhouse/__init__.``
- New ``_op_timeout_seconds()`` helper reads
``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` (default 30s). Rejects
unparseable, zero, and negative values — ``asyncio.wait_for(timeout=0)``
raises immediately and ``timeout<0`` is undefined per the asyncio
docs, so a typo would silently turn every SFTP call into an
instant failure. Logs a WARNING and falls back to the default.
- New ``async_list_inbound()`` async wrapper applies
``asyncio.wait_for(asyncio.to_thread(self.list_inbound),
timeout=N)``. ``wait_for`` cancels the awaiter after N
seconds but the worker thread keeps running until paramiko returns
on its own (paramiko is not asyncio-aware, can't be cancelled
cleanly). ``asyncio.TimeoutError`` propagates so the scheduler
can surface it as a transient SFTP error.
``backend/src/cyclone/scheduler.py``
- ``_tick_impl`` now calls ``client.async_list_inbound()``
instead of ``asyncio.to_thread(self._list_inbound)``. Catches
``asyncio.TimeoutError`` explicitly with a clear error message
("list_inbound: timeout") — separate from the generic
``Exception`` catch-all so the operator's tick-result error
reads as the actual cause.
- Deleted the now-dead ``Scheduler._list_inbound()`` shim.
``backend/tests/test_sftp_op_timeout.py`` (new, 6 tests):
1. Default timeout is 30s when the env var is unset.
2. Operator can override via env var without restart-rebuild.
3. Unparseable value (e.g. "30s") falls back to default.
4. Zero and negative values fall back to default.
5. A hanging list_inbound raises ``asyncio.TimeoutError`` within
the configured bound (the 06/25 hang pin).
6. A non-hanging call returns the stub's empty list normally
(sanity check that the wrapper doesn't break the happy path).
Scope note: only ``async_list_inbound`` is wired up. The other
SFTP methods (``list_inbound_names``, ``download_inbound``,
``read_file``, ``write_file``) are called from
operator-triggered paths (admin endpoints, CLI, claim submission)
where the operator can Ctrl-C the request, so a hang is at least
visible. Wrapping them would require making the FastAPI handlers
async, which is out of scope for Task 8. Tracked as a follow-up —
worth folding into the same shape when those endpoints are next
touched.
Gainwell's production filer has shipped at least two inbound filename
shapes — the spec form with a trailing `_{file_type}.x12` suffix and
a shorter suffix-less form where the disambiguator token (between
`-` and `_M`) doubles as both orig_tx and file_type. The 6/15–6/19
835 batch arrived in the suffix-less form, and the strict
`INBOUND_RE` rejected them outright — the scheduler silently
dropped 5 days of production data without logging an error.
`backend/src/cyclone/edi/filenames.py`
- New `INBOUND_RE_LOOSE` regex: same prefix/tpid/tracking/ts/seq
rules as `INBOUND_RE`, but the disambiguator token is required
to be 3–5 uppercase alnum (covers 999, TA1, 835, 277CA, ENCR; the
5-char cap stops the engine from over-eating the next `_M` token
in a degenerate input).
- `parse_inbound_filename` now tries `INBOUND_RE` first
(preserves historical behavior for every existing caller) and
falls back to `INBOUND_RE_LOOSE`. In the loose form, orig_tx is
set to the disambiguator token so the parsed shape matches what
the strict form produces when orig_tx == file_type. The
`ALLOWED_FILE_TYPES` check is enforced in both branches.
- `is_inbound_filename` is loosened in the same shape so the two
never disagree — a refactor that pre-filters a directory listing
with `is_inbound_filename` then re-parses with
`parse_inbound_filename` would otherwise see the suffix-less
files rejected twice.
`backend/tests/test_inbound_filename_loose.py` (new, 10 tests):
1. Spec form with explicit `_835.x12` suffix still wins (strict
path unchanged).
2-5. Suffix-less 835 / 999 / 277CA / ENCR all parse correctly.
6. Suffix-less `.txt` is still rejected (the `.x12` ext check
applies to both forms).
7. Suffix-less unknown type (4-char `ABCD`) is rejected by
ALLOWED_FILE_TYPES — the loose regex's `{3,5}` shape would
otherwise let it through.
8. Suffix-less 6-char token (`999XX6`) is rejected by the
5-char cap, preventing the engine from swallowing the next
`_M` token.
9. Strict form takes precedence over the loose form when both
match — pinning the parser's branch order so a refactor can't
silently change the parsed shape.
10. `is_inbound_filename` accepts the loose form, the spec form,
and rejects garbage.
Live smoke: 5/5 suffix-less Gainwell patterns now parse correctly
(were ValueError before); spec form unchanged; 4 invalid forms
still rejected. Full backend suite: 1085/1121 — the 36 pre-existing
failures (test_serialize_837, test_api_stream_live,
test_inbox_endpoints) are unrelated and confirmed pre-existing by
running them on stashed pre-change code.
The Acks page silently capped at 100 of 1056 rows in the operator's
DB: the eyebrow read `${items.length} on file` (not the server's
`total`), the KPI strip summed from the 100 visible items, and the
endpoint accepted no `offset` — so the user had no signal that 956
more acks existed. Same class of bug on the Activity page (cap=200,
no 'X of Y' hint).
Backend
- /api/acks (acks.py): add `offset`, bump `le=1000`→`le=5000`,
slice `rows[offset:offset+limit]`, return server-side
`aggregates` (accepted/rejected/received summed over the full
row set, not the page) so the KPI strip reflects every persisted
999 instead of just the visible 50.
- /api/activity list + stream (api.py): bump `le=500`→`le=5000` so
the page can ask for a denser snapshot.
- /api/277ca-acks (api.py): bump `le=1000`→`le=5000` for
consistency.
- /api/ta1-acks: left at `le=1000` — TA1s aren't shipped today and
the structural fix (offset + aggregates) wasn't applied, so a
larger cap would just make the same latent silent-failure easier
to hit. (TODO: fold in the same shape when Gainwell starts
shipping TA1s.)
Frontend
- listAcks (api.ts): accept `offset`, surface `aggregates`,
adapt wire `*_count` keys to the in-page
`accepted`/`rejected`/`received` shape so the page can use
`data.aggregates` as a drop-in for the page-local fallback
accumulator.
- useAcks (hooks): pass `offset` through; return type carries
`aggregates`.
- Acks.tsx: add `page` state (PAGE_SIZE=50), use `data.total`
for eyebrow + watermark (not `items.length`), use
`data.aggregates` for the KPI strip (with in-page fallback
accumulator on first paint), render `<Pagination>` when
`totalCount > PAGE_SIZE`. Footer row reads "N rows on file"
instead of "N rows".
- ActivityLog.tsx: bump `limit: 200`→`limit: 500`, eyebrow reads
"Activity · showing N most recent" to make the bounded-window
semantics honest (the endpoint doesn't expose a true total — it
reports events matching the current kind/since filter, capped at
the request limit).
Tests
- test_acks.py: 4 new tests pin the fix:
1. `offset` walks the full set; `has_more` flips at the
boundary.
2. `aggregates` reflects the full row set, not the page (the
silent-failure pin) — and stays stable across page slices.
3. `limit` cap of 5000 is enforced (422 above it).
4. `offset` past the end returns an empty page with stable
aggregates (a stale UI page state across a row count change
must not 500 or zero the KPIs).
Live smoke-verified: /api/acks?limit=2&offset=0 vs ?offset=2 return
the expected row slices, aggregates stable at 15/10/15 for 5 seeded
rows, /api/acks?limit=10000 rejected with 422.
Triage note: the TA1 section (`Ta1AcksSection`, lines 609-616 of
Acks.tsx) has the same latent silent-failure pattern (page-sums
KPIs, no offset on /api/ta1-acks). Left untouched because the
empty-state copy says Gainwell doesn't ship TA1s today and the
larger structural fix belongs in a follow-up.
Nora
Cyclone SP37
sp37 bot