Before this fix, the list endpoints computed `total` via
`len(list(store.iter_*(**common)))`. Both `iter_claims` and
`iter_remittances` default to `limit=100`, so the reported total
silently capped at 100 even when the DB held 60k claims or 835
remits. The frontend rendered a `data.total` of 100 in the KPI
tile and a 100-row table, the page looked complete, and the bug
stayed hidden — exactly the Dashboard silent-failure pattern that
59c3275 (server-aggregated KPIs) was meant to retire.
The Remittances page symptom reported on Jun 29 ('REMITS 100',
empty CLAIM column on 100 rows) was this bug. The empty CLAIM is
a separate matching concern (those remits have `claim_id = NULL`
in the DB); the count fix addresses the population-size lie.
Fix:
* `CycloneStore.count_claims` + `count_remittances` reuse the
iter's filter pipeline (DB filters + in-memory payer/date
filters) with an effectively-unbounded limit so the count
reflects the true DB population.
* `list_claims` + `list_remittances` call the new count
helpers instead of slicing `iter_*` twice.
* 13 new tests in `test_list_endpoint_counts.py` cover the
empty/filter/per-dimension/cardinality cases plus the HTTP
regression (seed 150/120, assert total matches).
Bonus fix:
* `conftest._reset_rate_limit_buckets` walks the middleware
stack and clears `RateLimitMiddleware._buckets` between
tests. Without this, the full suite tripped the 300 req/60s
rate limiter mid-run and 9 later tests (4 of mine, 5
pre-existing in test_inbox_endpoints / test_payer_summary)
got 429s. All 1167 tests now pass.
Follow-ups noted but out of scope:
* `/api/admin/audit-log`, `/api/admin/backup/list`,
`/api/admin/scheduler/processed-files` use the same
`len(rows)`-after-`.limit(limit)` pattern. Admin-only,
no `has_more` consumption, so impact is bounded.
* `count_*` could short-circuit to `func.count()` to skip
the UI-dict build, but iter already calls `q.all()` so
the win is modest.
The Dashboard was hardcoded to useClaims({ limit: 100 }) and reduce
KPIs client-side. With 60k+ claims in production, every tile
(Billed $940K, Received $59K, Denial rate, Pending AR, monthly
sparkline, top providers, recent denials) was computed from a
0.16% sample of the dataset — silently wrong numbers on the
operator's primary view.
Fix: add GET /api/dashboard/kpis that aggregates server-side in
one read over the entire claim population. The new useDashboardKpis
hook consumes it and polls every 60s. Dashboard.tsx drops
useClaims({limit:100}) + useProviders() + the client-side
buildMonthly reduce.
Backend:
- store.py: dashboard_kpis() — one Claim query (selectinload on
batch to avoid N+1) + one bulk Remittance lookup, Python reduce
over the full population. Zero-filled response for empty DB.
- api.py: GET /api/dashboard/kpis behind matrix_gate, query-param
clamps (1..24 months, 0..50 top_n_*).
Frontend:
- api.ts: DashboardKpis types + getDashboardKpis() wrapper.
- useDashboardKpis.ts: TanStack Query hook, 60s refetchInterval,
bypass to data:undefined when not configured.
- Dashboard.tsx: switched to useDashboardKpis, extracted
ZERO_TOTALS constant, dropped the buildMonthly helper.
Tests:
- backend/tests/test_dashboard_kpis.py: 12 tests covering empty DB,
matched-remit math, pending-state semantics, monthly binning,
top-providers/top-denials sort + cap, orphan-claim defensive
guard, HTTP wiring + param validation.
- src/hooks/useDashboardKpis.test.ts: 3 tests for the hook
contract (configured path, unconfigured fallback, param
passthrough).
- src/pages/Dashboard.test.tsx: wrapped renders in
QueryClientProvider + stubbed useAuth + isConfigured=false. This
fixes 3 pre-existing Dashboard test failures (the page never had
a QueryClient set up because useClaims/useProviders were the
first useQuery hooks in the page).
Reviewer fixes (same commit):
1. topDenials sort placed empty submissionDate claims first under
reverse-lex. Drop them at append time.
2. r.batch lazy-load → N+1 on 60k rows. selectinload(Claim.batch).
3. pending_states rebuilt per call as a mutable set — moved to
module-level _DASHBOARD_PENDING_STATES frozenset.
4. Module-level ProviderORM import inconsistent with the
"local import inside the function" pattern — moved inline.
Add check_matched_pair_drift() at module level in store.py — read-only
audit of the Claim.matched_remittance_id ↔ Remittance.claim_id FK pair.
Logs WARNING on drift with up to 5 examples of each of two cases
(claim-side and remit-side), returns the count of drifted rows. Wired
into api.py::lifespan right after db.init_db() and ensure_clearhouse_seeded(),
wrapped in try/except so a query failure logs but doesn't crash boot.
The symmetric-write + symmetric-clear invariants on
manual_match / manual_unmatch were already correct; this commit pins
them with focused regression tests in test_store_match_invariant.py
(8 tests covering both directions, both happy-path, rollback pin, and
drift-check unit behavior). PCN asymmetry in the fixture prevents the
auto-match inside CycloneStore.add (Task 10) from pre-pairing, so
manual_match is the only writer.
Migration 0016 adds the missing ix_claims_matched_remittance_id index —
the drift check scans WHERE matched_remittance_id IS NOT NULL on every
boot and would become a full-table scan past ~10k claims. Symmetric
with ix_remittances_claim_id (added in 0007). Migration tests bumped
from head=15 to head=16.
Move 'reconcile.run(s, record.id)' inside CycloneStore.add()'s ingest
session, before s.commit(). The placeholder adjustment_amount set by
_remittance_835_row is overwritten by reconcile's CAS-aggregate pass
in the same transaction — readers never see a half-reconciled
Remittance row. If reconcile raises, the entire 835 ingest rolls back
via the session's __exit__.
The previous flow committed batch + remittance rows in session-1,
then opened session-2 to run reconcile.fail-soft. Two visible
problems closed: a race window where readers fetched the placeholder
adjustment_amount, and a half-reconciled state left visible if
reconcile crashed.
Deviation from the plan (N4 in autoreview): the reconcile call lives
in cycl_store.add() rather than handle_835 calling a new
reconcile.run_now(batch_id) helper after add(). Same end state, one
fewer module surface, handler stays a thin wrapper over the store.
The 06/25 silent hang was a hung ``sftp.listdir_attr()`` on the
worker thread — paramiko TCP-acked then went silent, the scheduler's
``asyncio.to_thread`` waited forever, and the operator had no
signal that polling had stalled. Every poll cycle since was
suspected of the same failure mode.
``backend/src/cyclone/clearhouse/__init__.``
- New ``_op_timeout_seconds()`` helper reads
``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` (default 30s). Rejects
unparseable, zero, and negative values — ``asyncio.wait_for(timeout=0)``
raises immediately and ``timeout<0`` is undefined per the asyncio
docs, so a typo would silently turn every SFTP call into an
instant failure. Logs a WARNING and falls back to the default.
- New ``async_list_inbound()`` async wrapper applies
``asyncio.wait_for(asyncio.to_thread(self.list_inbound),
timeout=N)``. ``wait_for`` cancels the awaiter after N
seconds but the worker thread keeps running until paramiko returns
on its own (paramiko is not asyncio-aware, can't be cancelled
cleanly). ``asyncio.TimeoutError`` propagates so the scheduler
can surface it as a transient SFTP error.
``backend/src/cyclone/scheduler.py``
- ``_tick_impl`` now calls ``client.async_list_inbound()``
instead of ``asyncio.to_thread(self._list_inbound)``. Catches
``asyncio.TimeoutError`` explicitly with a clear error message
("list_inbound: timeout") — separate from the generic
``Exception`` catch-all so the operator's tick-result error
reads as the actual cause.
- Deleted the now-dead ``Scheduler._list_inbound()`` shim.
``backend/tests/test_sftp_op_timeout.py`` (new, 6 tests):
1. Default timeout is 30s when the env var is unset.
2. Operator can override via env var without restart-rebuild.
3. Unparseable value (e.g. "30s") falls back to default.
4. Zero and negative values fall back to default.
5. A hanging list_inbound raises ``asyncio.TimeoutError`` within
the configured bound (the 06/25 hang pin).
6. A non-hanging call returns the stub's empty list normally
(sanity check that the wrapper doesn't break the happy path).
Scope note: only ``async_list_inbound`` is wired up. The other
SFTP methods (``list_inbound_names``, ``download_inbound``,
``read_file``, ``write_file``) are called from
operator-triggered paths (admin endpoints, CLI, claim submission)
where the operator can Ctrl-C the request, so a hang is at least
visible. Wrapping them would require making the FastAPI handlers
async, which is out of scope for Task 8. Tracked as a follow-up —
worth folding into the same shape when those endpoints are next
touched.
Gainwell's production filer has shipped at least two inbound filename
shapes — the spec form with a trailing `_{file_type}.x12` suffix and
a shorter suffix-less form where the disambiguator token (between
`-` and `_M`) doubles as both orig_tx and file_type. The 6/15–6/19
835 batch arrived in the suffix-less form, and the strict
`INBOUND_RE` rejected them outright — the scheduler silently
dropped 5 days of production data without logging an error.
`backend/src/cyclone/edi/filenames.py`
- New `INBOUND_RE_LOOSE` regex: same prefix/tpid/tracking/ts/seq
rules as `INBOUND_RE`, but the disambiguator token is required
to be 3–5 uppercase alnum (covers 999, TA1, 835, 277CA, ENCR; the
5-char cap stops the engine from over-eating the next `_M` token
in a degenerate input).
- `parse_inbound_filename` now tries `INBOUND_RE` first
(preserves historical behavior for every existing caller) and
falls back to `INBOUND_RE_LOOSE`. In the loose form, orig_tx is
set to the disambiguator token so the parsed shape matches what
the strict form produces when orig_tx == file_type. The
`ALLOWED_FILE_TYPES` check is enforced in both branches.
- `is_inbound_filename` is loosened in the same shape so the two
never disagree — a refactor that pre-filters a directory listing
with `is_inbound_filename` then re-parses with
`parse_inbound_filename` would otherwise see the suffix-less
files rejected twice.
`backend/tests/test_inbound_filename_loose.py` (new, 10 tests):
1. Spec form with explicit `_835.x12` suffix still wins (strict
path unchanged).
2-5. Suffix-less 835 / 999 / 277CA / ENCR all parse correctly.
6. Suffix-less `.txt` is still rejected (the `.x12` ext check
applies to both forms).
7. Suffix-less unknown type (4-char `ABCD`) is rejected by
ALLOWED_FILE_TYPES — the loose regex's `{3,5}` shape would
otherwise let it through.
8. Suffix-less 6-char token (`999XX6`) is rejected by the
5-char cap, preventing the engine from swallowing the next
`_M` token.
9. Strict form takes precedence over the loose form when both
match — pinning the parser's branch order so a refactor can't
silently change the parsed shape.
10. `is_inbound_filename` accepts the loose form, the spec form,
and rejects garbage.
Live smoke: 5/5 suffix-less Gainwell patterns now parse correctly
(were ValueError before); spec form unchanged; 4 invalid forms
still rejected. Full backend suite: 1085/1121 — the 36 pre-existing
failures (test_serialize_837, test_api_stream_live,
test_inbox_endpoints) are unrelated and confirmed pre-existing by
running them on stashed pre-change code.
The Acks page silently capped at 100 of 1056 rows in the operator's
DB: the eyebrow read `${items.length} on file` (not the server's
`total`), the KPI strip summed from the 100 visible items, and the
endpoint accepted no `offset` — so the user had no signal that 956
more acks existed. Same class of bug on the Activity page (cap=200,
no 'X of Y' hint).
Backend
- /api/acks (acks.py): add `offset`, bump `le=1000`→`le=5000`,
slice `rows[offset:offset+limit]`, return server-side
`aggregates` (accepted/rejected/received summed over the full
row set, not the page) so the KPI strip reflects every persisted
999 instead of just the visible 50.
- /api/activity list + stream (api.py): bump `le=500`→`le=5000` so
the page can ask for a denser snapshot.
- /api/277ca-acks (api.py): bump `le=1000`→`le=5000` for
consistency.
- /api/ta1-acks: left at `le=1000` — TA1s aren't shipped today and
the structural fix (offset + aggregates) wasn't applied, so a
larger cap would just make the same latent silent-failure easier
to hit. (TODO: fold in the same shape when Gainwell starts
shipping TA1s.)
Frontend
- listAcks (api.ts): accept `offset`, surface `aggregates`,
adapt wire `*_count` keys to the in-page
`accepted`/`rejected`/`received` shape so the page can use
`data.aggregates` as a drop-in for the page-local fallback
accumulator.
- useAcks (hooks): pass `offset` through; return type carries
`aggregates`.
- Acks.tsx: add `page` state (PAGE_SIZE=50), use `data.total`
for eyebrow + watermark (not `items.length`), use
`data.aggregates` for the KPI strip (with in-page fallback
accumulator on first paint), render `<Pagination>` when
`totalCount > PAGE_SIZE`. Footer row reads "N rows on file"
instead of "N rows".
- ActivityLog.tsx: bump `limit: 200`→`limit: 500`, eyebrow reads
"Activity · showing N most recent" to make the bounded-window
semantics honest (the endpoint doesn't expose a true total — it
reports events matching the current kind/since filter, capped at
the request limit).
Tests
- test_acks.py: 4 new tests pin the fix:
1. `offset` walks the full set; `has_more` flips at the
boundary.
2. `aggregates` reflects the full row set, not the page (the
silent-failure pin) — and stays stable across page slices.
3. `limit` cap of 5000 is enforced (422 above it).
4. `offset` past the end returns an empty page with stable
aggregates (a stale UI page state across a row count change
must not 500 or zero the KPIs).
Live smoke-verified: /api/acks?limit=2&offset=0 vs ?offset=2 return
the expected row slices, aggregates stable at 15/10/15 for 5 seeded
rows, /api/acks?limit=10000 rejected with 422.
Triage note: the TA1 section (`Ta1AcksSection`, lines 609-616 of
Acks.tsx) has the same latent silent-failure pattern (page-sums
KPIs, no offset on /api/ta1-acks). Left untouched because the
empty-state copy says Gainwell doesn't ship TA1s today and the
larger structural fix belongs in a follow-up.
CycloneStore split precedent (SP21) — lift three helpers from
scheduler.py + api.py into one module. Both callers will switch
to this in Task 6. The new package also defines HandleResult
and the HANDLERS registry; handle_999 / handle_ta1 / handle_277ca
/ handle_835 fill in over Tasks 2-5.
The registry is lazy + best-effort: register_handlers() catches
broad Exception so a partial-modification SyntaxError in one
in-flight handler module can't break scheduler or API import.
11 tests added; existing suite unchanged (1 failed + 2 errors
pre-existing in test_provider_extended_response.py are not
introduced by this commit).
Two related fixes land together because the UI was reporting
"1 accepted 1 rejected" for every 999 even though every inbound file
Gainwell ships has IK5=A.
1. Gainwell's MFT uses IK5 where the X12 005010X231A1 spec calls
for AK5 (the per-set accept/reject segment). The parser only
recognized AK5, so set_responses[0].set_accept_reject.code
defaulted to 'R' and the count summary showed all rejections.
_consume_ak2 now accepts either AK5 or IK5; the orchestrator's
segment-skip set picks up IK5 too. A new fixture
(minimal_999_ik5_gainwell.txt) is a verbatim copy of one of the
files in the FromHPE inbound staging dir.
2. _ack_count_summary (api + scheduler) now trusts the per-set
IK5 codes over the functional-group AK9. Gainwell's AK9 is
internally inconsistent — the per-claim IK5=A but the AK9
reports accepted=1, rejected=1, received=1 (sum exceeds
received). Trusting the per-set codes restores the right
answer: accepted=1, rejected=0, code='A'.
3. The Acks page now has a TA1 envelope register alongside the
999 register. TA1s are the lower-level sibling of the 999
(one row per inbound ISA/IEA). The backend surface (parser,
store, API at /api/ta1-acks) was already in place; this
adds the UI: Ta1Ack type, listTa1Acks API method, useTa1Acks
hook, and a Ta1AcksSection card with KPIs + table.
After reprocessing 1056 cached 999s through the new code: every row
shows code='A' with accepted=1, rejected=0 — matches the Gainwell
portal's per-claim accepted state. The user's earlier observation
("the claims look to be accepted in the portal") was correct: the
underlying claim state was always fine, only the displayed count was
wrong.
- backend/src/cyclone/parsers/parse_999.py | 21 ++-
- backend/src/cyclone/api.py | 11 +-
- backend/src/cyclone/scheduler.py | 13 +-
- backend/tests/test_parse_999.py | 32 ++++
- backend/tests/fixtures/minimal_999_ik5_gainwell.txt
- src/types/index.ts | 32 ++++
- src/lib/api.ts | 62 +++++-
- src/hooks/useTa1Acks.ts | 26 +++ (new)
- src/pages/Acks.tsx | 209 +++++++++++++++++++-
Three changes that unblock the daily inbound pull from Gainwell's
FromHPE MFT path:
1. INBOUND_RE now accepts both 'TP' and 'tp' prefixes via inline
(?i:TP) scoping — case-folding the whole pattern would let
lowercase '837p' / tracking IDs through, which is invalid HCPF.
The rest of the pattern is still case-sensitive.
2. _list_inbound_paramiko skips *_warn.txt entries. Gainwell's MFT
drops ~583 advisory text-format notes in the same inbound dir;
they come first alphabetically and were padding every poll with
~80 min of pointless downloads.
3. New SftpClient.list_inbound_names() + download_inbound() pair
gives a metadata-only listing and on-demand fetch. The
scheduler's existing full-listing path still works (now without
the warn padding); the new path is what the new
/api/admin/scheduler/pull-inbound endpoint and the
'cyclone pull-inbound' CLI use to fast-target a date range
without paying the cost of a full ~6000-file download.
Scheduler.process_inbound_files() runs the same per-file pipeline
as a regular tick on the pre-fetched list, so dedup via
processed_inbound_files still applies.
Tests added in test_filenames.py (lowercase + mixed-case cases),
test_sftp_paramiko.py (warn skip + no-download listing), and
test_scheduler.py (process_inbound_files idempotency).
With this, the daily 385-file pull for 20260624 completes in
seconds via 'docker exec cyclone-backend-1 python -m cyclone
pull-inbound --date 20260624 --block dzinesco' (or the equivalent
POST to /api/admin/scheduler/pull-inbound?date=20260624).
The dzinesco SP9 seed had `paths.inbound` and `paths.outbound` mapped to
the wrong Gainwell MFT directories:
- paths.outbound was FromHPE/ (HPE sends files FROM here TO us — inbound)
- paths.inbound was ToHPE/ (we send files TO here — outbound)
So `/api/clearhouse/submit` was writing 837P claims to FromHPE (where
HPE puts acks/835s) and the SP16 scheduler was polling ToHPE (where our
claims go). 999 / TA1 / 835 files in the real FromHPE inbox were
unreachable.
Semantics per operator (2026-06-24):
- FromHPE = HPE/Gainwell → us = 999, TA1, 835 (inbound)
- ToHPE = us → HPE/Gainwell = 837P claims (outbound)
**Runtime code (the actual fix):**
- backend/src/cyclone/store.py — SP9 seed paths flipped
- backend/src/cyclone/edi/filenames.py — docstring corrected
**Test fixtures + assertions (would otherwise fail on the new seed):**
- backend/tests/test_clearhouse_api.py
- backend/tests/test_providers_seed.py
- backend/tests/test_sftp_stub.py — incl. inbound dir paths
- backend/tests/test_sftp_paramiko.py
- backend/tests/test_store_update_clearhouse.py
- backend/tests/test_api_clearhouse_patch.py
- backend/tests/test_scheduler.py
- backend/tests/test_api_scheduler.py
**Docs (text-only — keeps the codebase self-consistent):**
- README.md
- docs/reference/co-medicaid.md
- docs/superpowers/specs/2026-06-20-cyclone-multi-payer-npi-sftp-design.md
**Operator action required after merge:** the existing clearhouse row in
`~/.local/share/cyclone/cyclone.db` was seeded with the old wrong
paths. Easiest recovery:
1. `rm ~/.local/share/cyclone/cyclone.db` and let `ensure_clearhouse_seeded` re-run on next boot, OR
2. PATCH /api/clearhouse with the new `paths` block (the SP25
reconfigure hook picks it up live, including by the running scheduler).
**Verification:**
- 82 tests in the affected files pass (test_clearhouse_api, test_providers_seed,
test_sftp_stub, test_sftp_paramiko, test_store_update_clearhouse,
test_api_clearhouse_patch, test_scheduler, test_api_scheduler, test_filenames).
- Full backend suite: 1029 pass + 36 pre-existing order-dependent flakes
unrelated to this change (verified by running the same tests in isolation).
Bring the SP25 SFTP Polling Enablement (PATCH /api/clearhouse,
scheduler hot-reload, env-var-first secret lookup) and SP26
SFTP Password File Companion (Docker secret _FILE tier) onto
the v1.0.0 release branch so the production stack can actually
flip the Gainwell MFT poll from stub → real.
Conflict resolved in docker-compose.yml: kept both the SP26
CYCLONE_SFTP_PASSWORD_FILE env var and the new 'Always bind
to 0.0.0.0' comment.
The PERMISSIONS matrix only listed the POST /api/acks entry (parse-999
ingest). The GET list + detail endpoints for 999 / TA1 / 277CA acks
were missing, so the matrix_gate (default-deny) returned 403 to every
authenticated role, breaking the 999 ACKs / TA1 / 277CA inbox pages.
Add the three GET entries as ALL_ROLES — they're read-only metadata
surfaces that every authenticated operator needs to see. Add a
regression test that logs in as admin via the public route and
exercises the matrix; the existing test_existing_endpoints_require_auth
suite only checks the unauthenticated 401 path, which is why this slipped
through.
Fixes the live-verification 'Couldn't load ACKs from the backend /
forbidden' error reported against the UI.
Add a second tab to the Upload page that surfaces the persisted batch
archive and lets the user re-download any 837P batch as a ZIP without
re-parsing the original file.
- Backend: /api/batches now carries per-row claimIds (837P only).
835 batches return an empty list, which the UI uses as the signal
to hide the Re-export button on those rows. Avoids an extra
round-trip to /api/batches/{id} per row.
- Frontend: BatchSummary.claimIds added to the list-endpoint type.
- Upload page: page body wrapped in Tabs.Root with a History trigger
that mirrors ?tab= in the URL for deep-link round-trip. The
History tab renders UploadHistory → HistoryTable → HistoryRow with
a one-click Re-export ZIP button per 837P row. The button calls
POST /api/batches/{id}/export-837 with the row's claim ids and
downloads the ZIP via downloadBlob. Falls back to the in-memory
parsedBatches store when the backend returns no rows so the tab
stays useful in sample-data mode.
- Backend tests: claimIds present on 837P rows, empty on 835 rows.
- Frontend tests: 13 tests covering tab switching, URL deep-link,
loading/error/empty states, the 837P-vs-835 button visibility
split, the Re-export happy path, and the failure toast.
Two test-only fixes after running test_compose_up_brings_up_healthy_stack
on a non-CI host for the first time:
1. The test only used -f docker-compose.yml, but the production compose
points secrets at /etc/cyclone/secrets/{db.key,admin_username,
admin_pw} — which requires sudo to create. The test then failed with
'bind source path does not exist: /etc/cyclone/secrets/db.key' even
though the stack itself was correctly configured.
Fix: if docker-compose.override.yml exists at the repo root, the
test uses `-f compose.yml -f override.yml` so secrets come from
/tmp/cyclone-test-secrets/. Production CI skips the override.
2. The stack publishes host port 8080. If another local service (e.g.
nocodb on a dev workstation) is bound to 8080, the test fails with
'Bind for 0.0.0.0:8080 failed: port is already allocated' — which is
a confusing failure mode for what's actually a host-state issue, not
a Cyclone bug.
Fix: probe 127.0.0.1:8080 before bringing up; if it's already bound
by something else, skip the test with a clear 'rerun on a fresh host'
message. CI workers don't have this conflict.
Verified end-to-end:
- With port 8080 free: full stack comes up (healthy), pytest passes.
- With port 8080 bound: pytest skips cleanly with the message above.
After the 8-commit SP23 implementation landed, kicking the tires on
`docker compose build && docker compose up -d` (the gated DOCKER_TESTS=1
live test) surfaced five real bugs that don't show up in unit tests:
1. Backend wheel was built from the stub `__init__.py`, not the real
source. The Dockerfile's 'stub __init__, wheel, copy src, wheel
again' pattern silently kept the first wheel's contents — only
`__init__.py` got re-stubbed. The installed package had an empty
`__init__.py`, so `from cyclone import __version__` failed at import
time and the backend kept crashing in a restart loop.
Fix: single `COPY src/` + single `pip wheel`. Comment explains
why the stub trick is gone for good.
2. Backend binds to 127.0.0.1 (intentional — local-only by design,
see CLAUDE.md). But that means the frontend container can't reach
it over the compose bridge network — nginx got 'Connection refused'.
Fix: `CYCLONE_HOST` env var, defaults to 127.0.0.1 (preserves local
posture for non-Docker runs), set to 0.0.0.0 by the docker-compose
backend service. Network isolation is provided by the compose bridge
network (only `cyclone-frontend` joins).
3. Healthcheck probed `/api/healthz` (404 — the route is `/api/health`).
Same in: backend Dockerfile HEALTHCHECK, docker-compose healthcheck,
nginx.conf doesn't have one (frontend proxies through), RUNBOOK.md,
scripts/post-deploy.sh, scripts/smoke.sh.
Fix: `/api/healthz` → `/api/health` everywhere SP23 owns.
4. The auth matrix in `cyclone.auth.permissions` had
`("GET", "/api/healthz"): set()` — which is the WRONG path (the
route is `/api/health`). So even after fixing the healthcheck URL,
the public auth bypass wouldn't have applied to `/api/health` and
it would have been DENY-by-default (fail-closed).
Fix: matrix entry updated to `/api/health`.
5. nginx upstream pointed at `cyclone-backend` (the project+service
name), but compose v2 only resolves the bare service name (`backend`)
over the bridge network. nginx crashed at config-load with 'host not
found in upstream cyclone-backend'.
Fix: `cyclone-backend:8000` → `backend:8000` in nginx.conf + spec
+ plan.
6. Frontend HEALTHCHECK used `http://localhost:8080/`. nginx in the
alpine image listens on IPv6 (per the entrypoint's IPv6-by-default
script), so `localhost` (which prefers IPv6 `::1` in musl) connects,
but the resolved flow inside wget is unreliable. `127.0.0.1` works.
Fix: HEALTHCHECK uses `http://127.0.0.1:8080/`.
Also moves `frontend/Dockerfile` → `Dockerfile.frontend` and
`frontend/nginx.conf` → `nginx.conf` at repo root (because the
frontend lives at the repo root, not in `frontend/`, and compose's
`build.context: .` needs them at the same root as compose.yml).
The frontend's pre-existing `.dockerignore` was empty/unused, so it's
dropped — the root `.dockerignore` covers it.
Adds `docker-compose.override.yml` for local bring-up testing on a
host without sudo. Production uses `/etc/cyclone/secrets/` directly.
Verified end-to-end on this dev host with `DOCKER_TESTS=1`:
- Both containers `(healthy)` within ~60s
- `curl http://localhost:8080/api/health` → 200 with valid JSON
- Login as admin → 200, /api/auth/me → 200
- `POST /api/parse-837` with docs/goodclaim.x12 → 200, batch created
- Full backend test suite: 1014 passed, 9 skipped (prodfiles gitignored)
Eight tests: compose file exists, docker compose config validates, services declare required volumes/restart/healthcheck/ports/depends_on, secrets and volumes tables include the names the operator expects, backend wires CYCLONE_BACKUP_AUTOSTART=1, both Dockerfiles pass . The live bring-up test (compose up + wait for healthy) is gated on DOCKER_TESTS=1 so it skips on bare CI. PyYAML was already a hard dep.
Add _read_secret() helper that prefers *_FILE env vars (the standard Docker-secret pattern) over bare env vars. Strips trailing whitespace from file contents so printf/echo newlines don't break bcrypt verify. Existing CYCLONE_ADMIN_USERNAME + CYCLONE_ADMIN_PASSWORD still work for non-Docker deployments. Add tests/test_auth_bootstrap_file.py covering: file-path read, file-overrides-bare, bare-fallback, whitespace-stripping, missing-file raises.
- test_migration_latest_idempotent_on_fresh_db now uses the pytest
tmp_path fixture instead of a literal /tmp path that did not exist,
so sqlite can create the test DB.
- test_drop_claims_unique_constraint_migration now inserts the NOT
NULL batches.kind / input_filename / parsed_at columns when seeding
the parent row for the two same-PCN claims.
Adds migration 0015_drop_claims_unique_constraint.sql that recreates the
claims table without the inline UNIQUE(batch_id, patient_control_number)
constraint, plus tests proving the migration runs cleanly and the
constraint is not re-introduced.
Before this change, every claim — matched or not — came back from
`/api/claims` with `receivedAmount: 0.0`. `to_ui_claim_from_orm`
hardcoded the value, so the Dashboard's 'Received' KPI and the
claim detail drawer's paid amount always read $0 regardless of
whether the claim had been paired with a paying remittance.
The fix threads the real value through the read path:
- `to_ui_claim_from_orm` accepts a new `received_total: float`
kwarg (default 0.0 for callers that don't have a remittance in
scope).
- `iter_claims` bulk-loads `Remittance.total_paid` for every
matched claim id in the result set (single SQL roundtrip via
`IN` filter — no N+1) and stamps the sum onto each claim dict.
- `manual_match` passes `float(remit.total_paid)` from the remit
it's already holding.
- `manual_unmatch` reads `paired_remit.total_paid` before clearing
the FK so the response still reports what was paid pre-unpair.
- `add` (write path) and `list_unmatched` (filters matched-remit
is NULL) pass `0.0` explicitly for readability.
The dev seed CLI now also generates matched Remittance rows for
every PAID/PARTIAL claim — `matched_remittance_id` is set on the
claim, and the remittance's `total_paid` is derived from the
billed amount using the same ratios the frontend's old sample
fixtures used (60–100% for PAID, 20–50% for PARTIAL). That gives
the Dashboard a non-zero 'Received' KPI on a fresh dev DB.
New tests:
- tests/test_iter_claims_received.py — 4 cases covering matched,
unmatched, orphan FK, and bulk-load paths. Catches regressions
if anyone re-introduces the hardcoded 0.0 or breaks the bulk
query.
- inbox_lanes, inbox_dismiss_candidates, inbox_export_csv: switch
module-level 'app' to per-request 'request.app' so per-endpoint
state stays consistent with the test's TestClient target across
importlib.reload() (test_api.py::test_cors_extra_origins_via_env
reloads the api module to mutate CORS allow-lists; pre-reload
endpoints then mutate the wrong app instance).
- _http_exc_handler: when HTTPException.detail is a dict, wrap under
'detail' so the standard envelope stays stable and callers can
branch on body['detail']['error'].
- conftest._auto_init_db: re-resolve cyclone.api.app each fixture
invocation (instead of caching at module load) so the reload pattern
doesn't leave event_bus set on a stale app.
- test_acks.test_migration_latest_idempotent_on_fresh_db: bump
user_version assertion from 12 to 14 after auth migration renumber
(0013 users+sessions, 0014 audit_log.user_id).
Also installs sqlcipher3 + paramiko into the backend venv so the
capability tests can run; both modules were optional deps that
test_db_crypto and test_sftp_paramiko assume are present.
Backend test results: 1008 passed, 9 skipped (gitignored prodfile
fixtures), 0 failed.
- New test_existing_endpoints_require_auth.py: spot-check that existing
/api/* endpoints now require auth (gated via Depends(matrix_gate)) when
AUTH_DISABLED is False. Health remains public.
- conftest.py: flip AUTH_DISABLED=True for the suite so the legacy
pre-auth tests keep passing without login. Auth tests flip it back
off via their own autouse fixture (now patched to use monkeypatch
for cleanup).
Verified: 53 auth tests pass; 222 pre-existing non-auth failures are
unchanged.
Backend:
- New POST /api/batches/{id}/export-837: regenerate X12 837 files
for a list of claim_ids into a ZIP using HCPF file naming standards,
with a unique interchange/group control number per export. Wire
the clearhouse Loop 1000A (NM1*41 + PER) and per-payer receiver
(NM1*40) blocks so the serializer no longer falls back to
CYCLONE / RECEIVER placeholders.
- /api/parse-837 and /api/parse-835 now surface the server-side
batch_id in both JSON and NDJSON response shapes so the frontend
can hit batch-scoped endpoints without an extra listBatches
round-trip.
- Filename helpers and the 837 serializer updated to match the new
HCPF envelope; tests cover batch export, parse batch_id, and the
serializer's control-number uniqueness guarantee.
Frontend:
- New shared components: ClaimCard, ClaimCard837, DominantKpiCard,
EditorialNote, ExportBar, TickerTape, and a charts/ set
(BarChart, HBarChart, SegmentedBar, AgingBars).
- New useBatchExport hook driving ExportBar's download flow against
the new endpoint.
- ClaimDrawer, Lane, and Layout migrated from raw CSS-variable
colors to Tailwind theme tokens (bg-card, text-foreground,
border/60, etc.) for consistency with the rest of the instrument
chrome; the active tab indicator gains a subtle accent glow.
- Upload, Inbox, Batches, BatchDiff, Reconciliation, and Acks pages
reworked to compose the new shared components and consume the new
batch-scoped API surface (notably ExportBar wired into Batches).
Tooling / Docs:
- Add audit-uiux.mjs and a docs/goodclaim.x12 sample fixture.
- Update ClaimDrawer testids and add coverage for the new
components and the useBatchExport hook.
Rolls up into the v0.2.0 release tag.
Three independent improvements that fix real browser-facing bugs:
1. CORS: allow 127.0.0.1:5173 in addition to localhost:5173. Both
resolve to the same Vite dev server but CORS treats them as distinct
origins, so tabs opened via the IP form silently break.
2. CORS: support CYCLONE_ALLOWED_ORIGINS env var (comma-separated) for
LAN / staging hosts. The middleware reads it at module import.
3. Catch-all exception handler: returns JSON 500 with CORS headers
instead of a bare Uvicorn text/plain response. Without this, any
unhandled exception is misreported by browsers as a CORS error
because the body can't be read without the allow-origin header.
4. IntegrityError → 409: when (batch_id, patient_control_number) is
UNIQUE-constrained and a duplicate collides, return 409 with the
batch id instead of letting the exception 500. Same problem as (3)
for the most common ingest failure mode.
Tests added:
- test_cors_headers_present_for_loopback_ip
- test_cors_extra_origins_via_env (uses importlib.reload because the
allow-list is built at module import)
Adds pure local validators for the 10-digit NPI Luhn checksum (CMS-
published algorithm with the '80840' NPPES prefix) and 9-digit EIN
format (rejects reserved prefixes 00/07/80-89). No NPPES round-trip,
no IRS e-file lookup — catches the 99% typo case at parse time.
Surface:
- cyclone.npi.is_valid_npi / is_valid_tax_id / normalize_tax_id
- CLI: 'cyclone validate-npi <npi>' and 'cyclone validate-tax-id <ein>'
- API: GET /api/admin/validate-provider?npi=&tax_id=
- Parser validator: new R021_npi_checksum rule (warning, not error,
to keep test fixtures with placeholder NPIs ingestible)
- minimal_837p.txt fixture NPI updated from '1234567890' to the
Luhn-valid '1993999998' so strict-mode CLI parses still pass
Tests:
- test_npi.py — 27 cases (Luhn math, valid/invalid NPIs, EIN cases,
normalize_tax_id edge cases)
- test_api_validate_provider.py — 4 cases (both valid, both invalid,
omitted NPI, omitted tax_id)
- test_cli_validate.py — 8 cases (valid/invalid for both subcommands,
exit codes, malformed inputs)
- test_validator.py — 4 new R021 cases (valid Luhn silent, bad Luhn
warning, skipped when format bad, skipped when NPI missing)
Total: 923 tests pass.
Three pure-ASGI middlewares close completeness-review gaps §3.1.4
(no body/rate limits) and §3.1.25 (no security headers):
- BodySizeLimitMiddleware — rejects oversized uploads (50 MB
default, CYCLONE_MAX_BODY_BYTES override). 413 on over-cap
Content-Length and on chunked reads that cross the cap.
- RateLimitMiddleware — sliding-window per-IP limiter (300/min
default, CYCLONE_RATE_LIMIT_PER_MIN override). 429 over the
window. /api/health is exempt.
- SecurityHeadersMiddleware — stamps X-Content-Type-Options,
X-Frame-Options, Referrer-Policy, Permissions-Policy, and a
strict Content-Security-Policy on every response.
Every 413/429 also writes a tamper-evident api.request_rejected
event into the SP11 audit chain so an operator can correlate
rejections with the SP18 JSON logs.
GET /api/health is rewritten to return a subsystem snapshot:
DB connectivity (SELECT 1), MFT scheduler state, backup scheduler
state, live pubsub subscriber counts, last batch id + timestamp.
Returns status='degraded' if any subsystem is unhappy; per-subsystem
errors surfaced in the respective dict.
Cyclone.pubsub.EventBus.stats() — new method for live subscriber
counts.
13 new tests (test_security.py) + 1 updated (test_api.py health
endpoint). All 883 backend tests pass.
Cyclone previously emitted stdlib default-formatted log lines that
operators couldn't parse with anything beyond grep. SP18 replaces
that with:
- JsonFormatter: newline-delimited JSON, ISO-8601 ms timestamps,
structured "extra" dict, exception tracebacks serialized as a
single string.
- CycloneDevFormatter: tabular format for tail -f in dev.
- PiiScrubber: logging.Filter that redacts NPIs, SSNs, DOBs,
patient names — both inline in the message ("npi 1881068062")
and via PHI-keyed extras ({"dob": "1980-04-12"}).
- setup_logging(): idempotent entry point used by the API lifespan
and CLI main; respects CYCLONE_LOG_LEVEL/FILE/JSON/NO_PII_SCRUB.
- CLI --log-format=json|dev + --log-file=… flags.
- Migrate 9 highest-value log sites in scheduler/backup_scheduler/
backup_service to use extra={...} (input_filename, claims, parser,
backup_id, db_fingerprint, etc.).
34 new tests (test_logging_formatter + test_logging_scrubber +
test_logging_setup). All 867 backend tests pass.
Nora
tyler
tyler
Cyclone
Tyler