Behaviour-preserving structural split of the 2,995-LOC store.py into
a 14-module subpackage with a thin facade. CycloneStore class keeps
its full method surface as 1-line delegations to module functions.
14 modules:
exceptions, records, orm_builders, ui, write,
batches, claim_detail, kpis, acks,
backups, inbox, providers
(+ __init__.py facade)
Facade re-exports 12 public symbols (CycloneStore, store, BatchRecord*,
BatchKind, AlreadyMatchedError/NotMatchedError/InvalidStateError, utcnow,
dashboard_kpis, check_matched_pair_drift) + 3 private helpers
(_claim_status_from_validation, _persist_835_remit, _remittance_835_row)
to preserve the 4 test files that import them.
Zero public API changes, zero test changes, zero importer changes.
CycloneStore._lock and CycloneStore._batches.clear() remain intact for
the 7 test files still using the cleanup idiom.
Verified: 1,176 / 1 (pre-existing isolation flake) / 10 tests pass —
identical to baseline.
The Remittances page's three KPI tiles — REMITS / TOTAL PAID /
ADJUSTMENTS — were computed page-locally via items.reduce(...) over
the merged tail of the current page + live delta. With a 100-row
default limit, a 1,739-row population showed count=100, paid=$16,934,
adjustments=$147 — silently understating reality because the page
hadn't loaded the remaining rows yet.
This change mirrors the silent-incompleteness fix that
/api/dashboard/kpis (commit 59c3275) and /api/remittances (commit
d81b6ed) made for their tiles:
* CycloneStore.summarize_remittances() iterates the full filtered
remittance population (no limit) and returns
{count, total_paid, total_adjustments}. Mirrors iter_remittances
with limit=_ITER_UNBOUNDED.
* GET /api/remittances/summary — server endpoint with the same
filter parameters as /api/remittances. Registered BEFORE the
/api/remittances/stream handler so FastAPI doesn't treat
'summary' as a stream sub-path.
* api.listRemittanceSummary + useRemittanceSummary hook.
* Remittances.tsx swaps off items.reduce, consumes the server
summary. Tiles render the server totals so the values reflect
the entire DB population, not the page-local sample.
Verified live: /api/remittances/summary returns
{count: 1739, total_paid: 227181.58, total_adjustments: 13792.65},
which matches DB ground truth exactly.
Tests: 8 new backend tests in test_api_remittances_summary.py; 1 new
frontend test in Remittances.test.tsx (kpi_tiles_use_server_summary_not
_page_local_reduce) plus the page-level test for the zero-valued mock
default.
The 837 ingest path (_claim_837_row in store.py:194) populated
Claim.patient_control_number from claim.subscriber.member_id — the
subscriber's 2010BA NM109 Medicaid ID — instead of from
claim.claim_id (CLM01, the claim submitter's identifier the 837
actually sent).
That silently broke every downstream join that uses this column as a
cross-reference key:
* reconcile.match() (reconcile.py:74) — joins
Claim.patient_control_number against
Remittance.payer_claim_control_number (which is parsed from CLP01,
the 835's echo of CLM01 per X12 spec). Member_id never matches
CLP01, so auto-match always fails.
* apply_999_rejections — same lookup, same broken key.
* apply_277ca_rejections — same lookup, same broken key.
* scoring.score_pair — same broken key.
Live DB probe (1,739 remits, 337 claims):
* Claim.id == Remit.payer_claim_control_number → 0 matches
* Claim.patient_control_number == Remit.payer_claim_control_number
→ 9 matches (substring coincidences; synthetic PCN strings share
alphanumeric characters with the human-readable member_ids)
* Claim.matched_remittance_id NOT NULL → 0 claims
This commit changes _claim_837_row to write
Claim.patient_control_number = claim.claim_id (= CLM01). The
reconcile matcher's existing join now hits the row the 835 echoes
back. Companion migration 0017 backfills the 337 pre-fix rows so
they're on equal footing with new ingests (UPDATE claims SET
patient_control_number = id WHERE patient_control_number IS DISTINCT
FROM id — idempotent).
Tests:
* 2 new RED→GREEN tests in test_store_reconcile.py:
- test_837_ingest_populates_patient_control_number_from_claim_id
pins the field semantics directly
- test_837_then_835_with_echoed_pcn_auto_pairs proves the full
end-to-end auto-match now fires
* 3 existing manual_match tests that relied on the bug (had setup
using member_id != claim_id to deliberately prevent auto-match)
updated to use distinct PCNs explicitly so the tests still
exercise the manual-match path with a real orphan pair.
* 3 store_claim_detail / api_gets tests adjusted for the same
reason.
Verified: 1,176 / 1,177 backend tests pass; the one failure is the
pre-existing flake in test_provider_extended_response.py noted before
this work started.
Caveat for the existing dev DB: the 1,739 remits already in the DB
were ingested from 835 fixtures whose CLP01 is a different synthetic
identifier than the 837's CLM01 (the test fixtures never echoed
CLM01 — a fixture-data limitation, not a code bug). After this fix,
new 837+835 ingest pairs whose payer echoes CLM01 in CLP01 will
auto-match as expected. The pre-existing 1,739 remits will continue
to land in the unmatched bucket; that can only be fixed by
regenerating the test fixtures (out of scope for this SP).
Before this fix, the list endpoints computed `total` via
`len(list(store.iter_*(**common)))`. Both `iter_claims` and
`iter_remittances` default to `limit=100`, so the reported total
silently capped at 100 even when the DB held 60k claims or 835
remits. The frontend rendered a `data.total` of 100 in the KPI
tile and a 100-row table, the page looked complete, and the bug
stayed hidden — exactly the Dashboard silent-failure pattern that
59c3275 (server-aggregated KPIs) was meant to retire.
The Remittances page symptom reported on Jun 29 ('REMITS 100',
empty CLAIM column on 100 rows) was this bug. The empty CLAIM is
a separate matching concern (those remits have `claim_id = NULL`
in the DB); the count fix addresses the population-size lie.
Fix:
* `CycloneStore.count_claims` + `count_remittances` reuse the
iter's filter pipeline (DB filters + in-memory payer/date
filters) with an effectively-unbounded limit so the count
reflects the true DB population.
* `list_claims` + `list_remittances` call the new count
helpers instead of slicing `iter_*` twice.
* 13 new tests in `test_list_endpoint_counts.py` cover the
empty/filter/per-dimension/cardinality cases plus the HTTP
regression (seed 150/120, assert total matches).
Bonus fix:
* `conftest._reset_rate_limit_buckets` walks the middleware
stack and clears `RateLimitMiddleware._buckets` between
tests. Without this, the full suite tripped the 300 req/60s
rate limiter mid-run and 9 later tests (4 of mine, 5
pre-existing in test_inbox_endpoints / test_payer_summary)
got 429s. All 1167 tests now pass.
Follow-ups noted but out of scope:
* `/api/admin/audit-log`, `/api/admin/backup/list`,
`/api/admin/scheduler/processed-files` use the same
`len(rows)`-after-`.limit(limit)` pattern. Admin-only,
no `has_more` consumption, so impact is bounded.
* `count_*` could short-circuit to `func.count()` to skip
the UI-dict build, but iter already calls `q.all()` so
the win is modest.
The Dashboard was hardcoded to useClaims({ limit: 100 }) and reduce
KPIs client-side. With 60k+ claims in production, every tile
(Billed $940K, Received $59K, Denial rate, Pending AR, monthly
sparkline, top providers, recent denials) was computed from a
0.16% sample of the dataset — silently wrong numbers on the
operator's primary view.
Fix: add GET /api/dashboard/kpis that aggregates server-side in
one read over the entire claim population. The new useDashboardKpis
hook consumes it and polls every 60s. Dashboard.tsx drops
useClaims({limit:100}) + useProviders() + the client-side
buildMonthly reduce.
Backend:
- store.py: dashboard_kpis() — one Claim query (selectinload on
batch to avoid N+1) + one bulk Remittance lookup, Python reduce
over the full population. Zero-filled response for empty DB.
- api.py: GET /api/dashboard/kpis behind matrix_gate, query-param
clamps (1..24 months, 0..50 top_n_*).
Frontend:
- api.ts: DashboardKpis types + getDashboardKpis() wrapper.
- useDashboardKpis.ts: TanStack Query hook, 60s refetchInterval,
bypass to data:undefined when not configured.
- Dashboard.tsx: switched to useDashboardKpis, extracted
ZERO_TOTALS constant, dropped the buildMonthly helper.
Tests:
- backend/tests/test_dashboard_kpis.py: 12 tests covering empty DB,
matched-remit math, pending-state semantics, monthly binning,
top-providers/top-denials sort + cap, orphan-claim defensive
guard, HTTP wiring + param validation.
- src/hooks/useDashboardKpis.test.ts: 3 tests for the hook
contract (configured path, unconfigured fallback, param
passthrough).
- src/pages/Dashboard.test.tsx: wrapped renders in
QueryClientProvider + stubbed useAuth + isConfigured=false. This
fixes 3 pre-existing Dashboard test failures (the page never had
a QueryClient set up because useClaims/useProviders were the
first useQuery hooks in the page).
Reviewer fixes (same commit):
1. topDenials sort placed empty submissionDate claims first under
reverse-lex. Drop them at append time.
2. r.batch lazy-load → N+1 on 60k rows. selectinload(Claim.batch).
3. pending_states rebuilt per call as a mutable set — moved to
module-level _DASHBOARD_PENDING_STATES frozenset.
4. Module-level ProviderORM import inconsistent with the
"local import inside the function" pattern — moved inline.
Add check_matched_pair_drift() at module level in store.py — read-only
audit of the Claim.matched_remittance_id ↔ Remittance.claim_id FK pair.
Logs WARNING on drift with up to 5 examples of each of two cases
(claim-side and remit-side), returns the count of drifted rows. Wired
into api.py::lifespan right after db.init_db() and ensure_clearhouse_seeded(),
wrapped in try/except so a query failure logs but doesn't crash boot.
The symmetric-write + symmetric-clear invariants on
manual_match / manual_unmatch were already correct; this commit pins
them with focused regression tests in test_store_match_invariant.py
(8 tests covering both directions, both happy-path, rollback pin, and
drift-check unit behavior). PCN asymmetry in the fixture prevents the
auto-match inside CycloneStore.add (Task 10) from pre-pairing, so
manual_match is the only writer.
Migration 0016 adds the missing ix_claims_matched_remittance_id index —
the drift check scans WHERE matched_remittance_id IS NOT NULL on every
boot and would become a full-table scan past ~10k claims. Symmetric
with ix_remittances_claim_id (added in 0007). Migration tests bumped
from head=15 to head=16.
Move 'reconcile.run(s, record.id)' inside CycloneStore.add()'s ingest
session, before s.commit(). The placeholder adjustment_amount set by
_remittance_835_row is overwritten by reconcile's CAS-aggregate pass
in the same transaction — readers never see a half-reconciled
Remittance row. If reconcile raises, the entire 835 ingest rolls back
via the session's __exit__.
The previous flow committed batch + remittance rows in session-1,
then opened session-2 to run reconcile.fail-soft. Two visible
problems closed: a race window where readers fetched the placeholder
adjustment_amount, and a half-reconciled state left visible if
reconcile crashed.
Deviation from the plan (N4 in autoreview): the reconcile call lives
in cycl_store.add() rather than handle_835 calling a new
reconcile.run_now(batch_id) helper after add(). Same end state, one
fewer module surface, handler stays a thin wrapper over the store.
The 06/25 silent hang was a hung ``sftp.listdir_attr()`` on the
worker thread — paramiko TCP-acked then went silent, the scheduler's
``asyncio.to_thread`` waited forever, and the operator had no
signal that polling had stalled. Every poll cycle since was
suspected of the same failure mode.
``backend/src/cyclone/clearhouse/__init__.``
- New ``_op_timeout_seconds()`` helper reads
``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` (default 30s). Rejects
unparseable, zero, and negative values — ``asyncio.wait_for(timeout=0)``
raises immediately and ``timeout<0`` is undefined per the asyncio
docs, so a typo would silently turn every SFTP call into an
instant failure. Logs a WARNING and falls back to the default.
- New ``async_list_inbound()`` async wrapper applies
``asyncio.wait_for(asyncio.to_thread(self.list_inbound),
timeout=N)``. ``wait_for`` cancels the awaiter after N
seconds but the worker thread keeps running until paramiko returns
on its own (paramiko is not asyncio-aware, can't be cancelled
cleanly). ``asyncio.TimeoutError`` propagates so the scheduler
can surface it as a transient SFTP error.
``backend/src/cyclone/scheduler.py``
- ``_tick_impl`` now calls ``client.async_list_inbound()``
instead of ``asyncio.to_thread(self._list_inbound)``. Catches
``asyncio.TimeoutError`` explicitly with a clear error message
("list_inbound: timeout") — separate from the generic
``Exception`` catch-all so the operator's tick-result error
reads as the actual cause.
- Deleted the now-dead ``Scheduler._list_inbound()`` shim.
``backend/tests/test_sftp_op_timeout.py`` (new, 6 tests):
1. Default timeout is 30s when the env var is unset.
2. Operator can override via env var without restart-rebuild.
3. Unparseable value (e.g. "30s") falls back to default.
4. Zero and negative values fall back to default.
5. A hanging list_inbound raises ``asyncio.TimeoutError`` within
the configured bound (the 06/25 hang pin).
6. A non-hanging call returns the stub's empty list normally
(sanity check that the wrapper doesn't break the happy path).
Scope note: only ``async_list_inbound`` is wired up. The other
SFTP methods (``list_inbound_names``, ``download_inbound``,
``read_file``, ``write_file``) are called from
operator-triggered paths (admin endpoints, CLI, claim submission)
where the operator can Ctrl-C the request, so a hang is at least
visible. Wrapping them would require making the FastAPI handlers
async, which is out of scope for Task 8. Tracked as a follow-up —
worth folding into the same shape when those endpoints are next
touched.
Gainwell's production filer has shipped at least two inbound filename
shapes — the spec form with a trailing `_{file_type}.x12` suffix and
a shorter suffix-less form where the disambiguator token (between
`-` and `_M`) doubles as both orig_tx and file_type. The 6/15–6/19
835 batch arrived in the suffix-less form, and the strict
`INBOUND_RE` rejected them outright — the scheduler silently
dropped 5 days of production data without logging an error.
`backend/src/cyclone/edi/filenames.py`
- New `INBOUND_RE_LOOSE` regex: same prefix/tpid/tracking/ts/seq
rules as `INBOUND_RE`, but the disambiguator token is required
to be 3–5 uppercase alnum (covers 999, TA1, 835, 277CA, ENCR; the
5-char cap stops the engine from over-eating the next `_M` token
in a degenerate input).
- `parse_inbound_filename` now tries `INBOUND_RE` first
(preserves historical behavior for every existing caller) and
falls back to `INBOUND_RE_LOOSE`. In the loose form, orig_tx is
set to the disambiguator token so the parsed shape matches what
the strict form produces when orig_tx == file_type. The
`ALLOWED_FILE_TYPES` check is enforced in both branches.
- `is_inbound_filename` is loosened in the same shape so the two
never disagree — a refactor that pre-filters a directory listing
with `is_inbound_filename` then re-parses with
`parse_inbound_filename` would otherwise see the suffix-less
files rejected twice.
`backend/tests/test_inbound_filename_loose.py` (new, 10 tests):
1. Spec form with explicit `_835.x12` suffix still wins (strict
path unchanged).
2-5. Suffix-less 835 / 999 / 277CA / ENCR all parse correctly.
6. Suffix-less `.txt` is still rejected (the `.x12` ext check
applies to both forms).
7. Suffix-less unknown type (4-char `ABCD`) is rejected by
ALLOWED_FILE_TYPES — the loose regex's `{3,5}` shape would
otherwise let it through.
8. Suffix-less 6-char token (`999XX6`) is rejected by the
5-char cap, preventing the engine from swallowing the next
`_M` token.
9. Strict form takes precedence over the loose form when both
match — pinning the parser's branch order so a refactor can't
silently change the parsed shape.
10. `is_inbound_filename` accepts the loose form, the spec form,
and rejects garbage.
Live smoke: 5/5 suffix-less Gainwell patterns now parse correctly
(were ValueError before); spec form unchanged; 4 invalid forms
still rejected. Full backend suite: 1085/1121 — the 36 pre-existing
failures (test_serialize_837, test_api_stream_live,
test_inbox_endpoints) are unrelated and confirmed pre-existing by
running them on stashed pre-change code.
The Acks page silently capped at 100 of 1056 rows in the operator's
DB: the eyebrow read `${items.length} on file` (not the server's
`total`), the KPI strip summed from the 100 visible items, and the
endpoint accepted no `offset` — so the user had no signal that 956
more acks existed. Same class of bug on the Activity page (cap=200,
no 'X of Y' hint).
Backend
- /api/acks (acks.py): add `offset`, bump `le=1000`→`le=5000`,
slice `rows[offset:offset+limit]`, return server-side
`aggregates` (accepted/rejected/received summed over the full
row set, not the page) so the KPI strip reflects every persisted
999 instead of just the visible 50.
- /api/activity list + stream (api.py): bump `le=500`→`le=5000` so
the page can ask for a denser snapshot.
- /api/277ca-acks (api.py): bump `le=1000`→`le=5000` for
consistency.
- /api/ta1-acks: left at `le=1000` — TA1s aren't shipped today and
the structural fix (offset + aggregates) wasn't applied, so a
larger cap would just make the same latent silent-failure easier
to hit. (TODO: fold in the same shape when Gainwell starts
shipping TA1s.)
Frontend
- listAcks (api.ts): accept `offset`, surface `aggregates`,
adapt wire `*_count` keys to the in-page
`accepted`/`rejected`/`received` shape so the page can use
`data.aggregates` as a drop-in for the page-local fallback
accumulator.
- useAcks (hooks): pass `offset` through; return type carries
`aggregates`.
- Acks.tsx: add `page` state (PAGE_SIZE=50), use `data.total`
for eyebrow + watermark (not `items.length`), use
`data.aggregates` for the KPI strip (with in-page fallback
accumulator on first paint), render `<Pagination>` when
`totalCount > PAGE_SIZE`. Footer row reads "N rows on file"
instead of "N rows".
- ActivityLog.tsx: bump `limit: 200`→`limit: 500`, eyebrow reads
"Activity · showing N most recent" to make the bounded-window
semantics honest (the endpoint doesn't expose a true total — it
reports events matching the current kind/since filter, capped at
the request limit).
Tests
- test_acks.py: 4 new tests pin the fix:
1. `offset` walks the full set; `has_more` flips at the
boundary.
2. `aggregates` reflects the full row set, not the page (the
silent-failure pin) — and stays stable across page slices.
3. `limit` cap of 5000 is enforced (422 above it).
4. `offset` past the end returns an empty page with stable
aggregates (a stale UI page state across a row count change
must not 500 or zero the KPIs).
Live smoke-verified: /api/acks?limit=2&offset=0 vs ?offset=2 return
the expected row slices, aggregates stable at 15/10/15 for 5 seeded
rows, /api/acks?limit=10000 rejected with 422.
Triage note: the TA1 section (`Ta1AcksSection`, lines 609-616 of
Acks.tsx) has the same latent silent-failure pattern (page-sums
KPIs, no offset on /api/ta1-acks). Left untouched because the
empty-state copy says Gainwell doesn't ship TA1s today and the
larger structural fix belongs in a follow-up.
CycloneStore split precedent (SP21) — lift three helpers from
scheduler.py + api.py into one module. Both callers will switch
to this in Task 6. The new package also defines HandleResult
and the HANDLERS registry; handle_999 / handle_ta1 / handle_277ca
/ handle_835 fill in over Tasks 2-5.
The registry is lazy + best-effort: register_handlers() catches
broad Exception so a partial-modification SyntaxError in one
in-flight handler module can't break scheduler or API import.
11 tests added; existing suite unchanged (1 failed + 2 errors
pre-existing in test_provider_extended_response.py are not
introduced by this commit).
Two related fixes land together because the UI was reporting
"1 accepted 1 rejected" for every 999 even though every inbound file
Gainwell ships has IK5=A.
1. Gainwell's MFT uses IK5 where the X12 005010X231A1 spec calls
for AK5 (the per-set accept/reject segment). The parser only
recognized AK5, so set_responses[0].set_accept_reject.code
defaulted to 'R' and the count summary showed all rejections.
_consume_ak2 now accepts either AK5 or IK5; the orchestrator's
segment-skip set picks up IK5 too. A new fixture
(minimal_999_ik5_gainwell.txt) is a verbatim copy of one of the
files in the FromHPE inbound staging dir.
2. _ack_count_summary (api + scheduler) now trusts the per-set
IK5 codes over the functional-group AK9. Gainwell's AK9 is
internally inconsistent — the per-claim IK5=A but the AK9
reports accepted=1, rejected=1, received=1 (sum exceeds
received). Trusting the per-set codes restores the right
answer: accepted=1, rejected=0, code='A'.
3. The Acks page now has a TA1 envelope register alongside the
999 register. TA1s are the lower-level sibling of the 999
(one row per inbound ISA/IEA). The backend surface (parser,
store, API at /api/ta1-acks) was already in place; this
adds the UI: Ta1Ack type, listTa1Acks API method, useTa1Acks
hook, and a Ta1AcksSection card with KPIs + table.
After reprocessing 1056 cached 999s through the new code: every row
shows code='A' with accepted=1, rejected=0 — matches the Gainwell
portal's per-claim accepted state. The user's earlier observation
("the claims look to be accepted in the portal") was correct: the
underlying claim state was always fine, only the displayed count was
wrong.
- backend/src/cyclone/parsers/parse_999.py | 21 ++-
- backend/src/cyclone/api.py | 11 +-
- backend/src/cyclone/scheduler.py | 13 +-
- backend/tests/test_parse_999.py | 32 ++++
- backend/tests/fixtures/minimal_999_ik5_gainwell.txt
- src/types/index.ts | 32 ++++
- src/lib/api.ts | 62 +++++-
- src/hooks/useTa1Acks.ts | 26 +++ (new)
- src/pages/Acks.tsx | 209 +++++++++++++++++++-
Gainwell's MFT ships every 999 with the same ISA interchange
control number (`000000001`) and one 999 ack covers a whole
batch, not a single claim — so the AK2 set_control_number
(patient_control_number) is the same for the ~96 999s in a
batch. With the old synthetic-id formula (`999-{icn}`), all 385
daily acks collapsed onto a single row the operator couldn't
distinguish.
The new formula is `999-{pcn}-{filename_hash8}` (or
`999-{icn}-{filename_hash8}` for envelope-only 999s without an
AK2). The PCN gives the operator a human-readable handle to the
claim batch; the 8-char hash of the inbound filename guarantees
uniqueness within a batch. Fits in the VARCHAR(32) source_batch_id
column (max 22 chars).
Also surface `patient_control_number` in /api/acks list response
(extracted from raw_json's set_responses[0].set_control_number)
and in the Acks UI as the primary label, with the synthetic id
shown dimmed after a middle dot. The detail endpoint already
exposed raw_json for the full 999 parse tree.
Three changes that unblock the daily inbound pull from Gainwell's
FromHPE MFT path:
1. INBOUND_RE now accepts both 'TP' and 'tp' prefixes via inline
(?i:TP) scoping — case-folding the whole pattern would let
lowercase '837p' / tracking IDs through, which is invalid HCPF.
The rest of the pattern is still case-sensitive.
2. _list_inbound_paramiko skips *_warn.txt entries. Gainwell's MFT
drops ~583 advisory text-format notes in the same inbound dir;
they come first alphabetically and were padding every poll with
~80 min of pointless downloads.
3. New SftpClient.list_inbound_names() + download_inbound() pair
gives a metadata-only listing and on-demand fetch. The
scheduler's existing full-listing path still works (now without
the warn padding); the new path is what the new
/api/admin/scheduler/pull-inbound endpoint and the
'cyclone pull-inbound' CLI use to fast-target a date range
without paying the cost of a full ~6000-file download.
Scheduler.process_inbound_files() runs the same per-file pipeline
as a regular tick on the pre-fetched list, so dedup via
processed_inbound_files still applies.
Tests added in test_filenames.py (lowercase + mixed-case cases),
test_sftp_paramiko.py (warn skip + no-download listing), and
test_scheduler.py (process_inbound_files idempotency).
With this, the daily 385-file pull for 20260624 completes in
seconds via 'docker exec cyclone-backend-1 python -m cyclone
pull-inbound --date 20260624 --block dzinesco' (or the equivalent
POST to /api/admin/scheduler/pull-inbound?date=20260624).
In real (paramiko) mode, `_download_and_parse` called
`client.read_file(f.name)` with a bare filename. paramiko's
`sftp.open(f.name)` opens at the SFTP root, not at `paths.inbound`
(FromHPE) — so the scheduler would fail to download any file in real
mode even with the path swap from the previous commit.
But this round-trip is also unnecessary: `_list_inbound_paramiko`
already downloads each entry into the local cache (cache_path) and
returns it as `InboundFile.local_path` as part of the listing pass.
Reading from disk is faster than re-fetching and avoids the path bug.
Stub mode was already reading from `f.local_path`. Now both modes do,
which is the simpler invariant.
Verified: 36 tests pass (test_scheduler, test_api_scheduler, test_sftp_stub,
test_sftp_paramiko).
The dzinesco SP9 seed had `paths.inbound` and `paths.outbound` mapped to
the wrong Gainwell MFT directories:
- paths.outbound was FromHPE/ (HPE sends files FROM here TO us — inbound)
- paths.inbound was ToHPE/ (we send files TO here — outbound)
So `/api/clearhouse/submit` was writing 837P claims to FromHPE (where
HPE puts acks/835s) and the SP16 scheduler was polling ToHPE (where our
claims go). 999 / TA1 / 835 files in the real FromHPE inbox were
unreachable.
Semantics per operator (2026-06-24):
- FromHPE = HPE/Gainwell → us = 999, TA1, 835 (inbound)
- ToHPE = us → HPE/Gainwell = 837P claims (outbound)
**Runtime code (the actual fix):**
- backend/src/cyclone/store.py — SP9 seed paths flipped
- backend/src/cyclone/edi/filenames.py — docstring corrected
**Test fixtures + assertions (would otherwise fail on the new seed):**
- backend/tests/test_clearhouse_api.py
- backend/tests/test_providers_seed.py
- backend/tests/test_sftp_stub.py — incl. inbound dir paths
- backend/tests/test_sftp_paramiko.py
- backend/tests/test_store_update_clearhouse.py
- backend/tests/test_api_clearhouse_patch.py
- backend/tests/test_scheduler.py
- backend/tests/test_api_scheduler.py
**Docs (text-only — keeps the codebase self-consistent):**
- README.md
- docs/reference/co-medicaid.md
- docs/superpowers/specs/2026-06-20-cyclone-multi-payer-npi-sftp-design.md
**Operator action required after merge:** the existing clearhouse row in
`~/.local/share/cyclone/cyclone.db` was seeded with the old wrong
paths. Easiest recovery:
1. `rm ~/.local/share/cyclone/cyclone.db` and let `ensure_clearhouse_seeded` re-run on next boot, OR
2. PATCH /api/clearhouse with the new `paths` block (the SP25
reconfigure hook picks it up live, including by the running scheduler).
**Verification:**
- 82 tests in the affected files pass (test_clearhouse_api, test_providers_seed,
test_sftp_stub, test_sftp_paramiko, test_store_update_clearhouse,
test_api_clearhouse_patch, test_scheduler, test_api_scheduler, test_filenames).
- Full backend suite: 1029 pass + 36 pre-existing order-dependent flakes
unrelated to this change (verified by running the same tests in isolation).
The backend Dockerfile only installed the [sqlcipher] extra, so
paramiko wasn't on the path inside the container. SP25 + SP26's
real-mode SFTP client needs paramiko to connect to the Gainwell
MFT — the scheduler tick surfaced this as
'No module named paramiko' on the very first /api/admin/scheduler/tick
against the dzinesco clearhouse with stub=false.
Install both extras ([sqlcipher,sftp]) in builder and runtime stages.
Bring the SP25 SFTP Polling Enablement (PATCH /api/clearhouse,
scheduler hot-reload, env-var-first secret lookup) and SP26
SFTP Password File Companion (Docker secret _FILE tier) onto
the v1.0.0 release branch so the production stack can actually
flip the Gainwell MFT poll from stub → real.
Conflict resolved in docker-compose.yml: kept both the SP26
CYCLONE_SFTP_PASSWORD_FILE env var and the new 'Always bind
to 0.0.0.0' comment.
The matrix_gate dependency was added to ~50 routes (clearhouse, all
/api/admin/*, /api/config/*, /api/eligibility/*, /api/parse-999/ta1/277ca,
/api/batches/{id}/export-837, /api/payer-rejected/acknowledge, etc.)
but the PERMISSIONS matrix was only populated for ~25 of them. Default-
deny therefore returned 403 to every authenticated role on the rest,
including the SFTP admin endpoints needed for MFT polling.
Roles:
- Admin-only: /api/clearhouse* (SFTP creds + dzinesco identity), all
/api/admin/* (audit-log, backup, scheduler, db rotate, reload-config,
validate-provider)
- All authenticated: /api/batch-diff, /api/config/*, /api/payers/*
- Write (admin + user, no viewer): /api/parse-999/ta1/277ca, the
/api/batches POST prefix (regenerates X12 from DB rows), and
/api/eligibility/* (270 build / 271 parse)
Unblocks /api/clearhouse, /api/admin/scheduler/*, /api/admin/backup/*,
and the rest of the admin surface so MFT polling and live verification
can proceed.
The PERMISSIONS matrix only listed the POST /api/acks entry (parse-999
ingest). The GET list + detail endpoints for 999 / TA1 / 277CA acks
were missing, so the matrix_gate (default-deny) returned 403 to every
authenticated role, breaking the 999 ACKs / TA1 / 277CA inbox pages.
Add the three GET entries as ALL_ROLES — they're read-only metadata
surfaces that every authenticated operator needs to see. Add a
regression test that logs in as admin via the public route and
exercises the matrix; the existing test_existing_endpoints_require_auth
suite only checks the unauthenticated 401 path, which is why this slipped
through.
Fixes the live-verification 'Couldn't load ACKs from the backend /
forbidden' error reported against the UI.
Reachability is controlled by the host firewall / compose port
publishing, not the bind address. Backend, compose, and docs all
now default to 0.0.0.0 so the API is reachable from the frontend
container, the host, and the LAN without per-env overrides.
Files:
- backend/src/cyclone/__main__.py: default host = 0.0.0.0
- docker-compose.yml: refresh the comment to match the new posture
- CLAUDE.md / README.md / docs/ARCHITECTURE.md / docs/REQUIREMENTS.md:
reframe the bind note accordingly
Add a second tab to the Upload page that surfaces the persisted batch
archive and lets the user re-download any 837P batch as a ZIP without
re-parsing the original file.
- Backend: /api/batches now carries per-row claimIds (837P only).
835 batches return an empty list, which the UI uses as the signal
to hide the Re-export button on those rows. Avoids an extra
round-trip to /api/batches/{id} per row.
- Frontend: BatchSummary.claimIds added to the list-endpoint type.
- Upload page: page body wrapped in Tabs.Root with a History trigger
that mirrors ?tab= in the URL for deep-link round-trip. The
History tab renders UploadHistory → HistoryTable → HistoryRow with
a one-click Re-export ZIP button per 837P row. The button calls
POST /api/batches/{id}/export-837 with the row's claim ids and
downloads the ZIP via downloadBlob. Falls back to the in-memory
parsedBatches store when the backend returns no rows so the tab
stays useful in sample-data mode.
- Backend tests: claimIds present on 837P rows, empty on 835 rows.
- Frontend tests: 13 tests covering tab switching, URL deep-link,
loading/error/empty states, the 837P-vs-835 button visibility
split, the Re-export happy path, and the failure toast.
The dev-server allow-list (VITE_DEV_ORIGINS = localhost:5173 +
127.0.0.1:5173 + CYCLONE_ALLOWED_ORIGINS) is tight enough that
`allow_credentials=True` is safe — the browser was dropping the
session cookie on cross-origin fetches from the Vite dev server
otherwise. Tight allow-list + credentials is the standard setup.
Two test-only fixes after running test_compose_up_brings_up_healthy_stack
on a non-CI host for the first time:
1. The test only used -f docker-compose.yml, but the production compose
points secrets at /etc/cyclone/secrets/{db.key,admin_username,
admin_pw} — which requires sudo to create. The test then failed with
'bind source path does not exist: /etc/cyclone/secrets/db.key' even
though the stack itself was correctly configured.
Fix: if docker-compose.override.yml exists at the repo root, the
test uses `-f compose.yml -f override.yml` so secrets come from
/tmp/cyclone-test-secrets/. Production CI skips the override.
2. The stack publishes host port 8080. If another local service (e.g.
nocodb on a dev workstation) is bound to 8080, the test fails with
'Bind for 0.0.0.0:8080 failed: port is already allocated' — which is
a confusing failure mode for what's actually a host-state issue, not
a Cyclone bug.
Fix: probe 127.0.0.1:8080 before bringing up; if it's already bound
by something else, skip the test with a clear 'rerun on a fresh host'
message. CI workers don't have this conflict.
Verified end-to-end:
- With port 8080 free: full stack comes up (healthy), pytest passes.
- With port 8080 bound: pytest skips cleanly with the message above.
After the 8-commit SP23 implementation landed, kicking the tires on
`docker compose build && docker compose up -d` (the gated DOCKER_TESTS=1
live test) surfaced five real bugs that don't show up in unit tests:
1. Backend wheel was built from the stub `__init__.py`, not the real
source. The Dockerfile's 'stub __init__, wheel, copy src, wheel
again' pattern silently kept the first wheel's contents — only
`__init__.py` got re-stubbed. The installed package had an empty
`__init__.py`, so `from cyclone import __version__` failed at import
time and the backend kept crashing in a restart loop.
Fix: single `COPY src/` + single `pip wheel`. Comment explains
why the stub trick is gone for good.
2. Backend binds to 127.0.0.1 (intentional — local-only by design,
see CLAUDE.md). But that means the frontend container can't reach
it over the compose bridge network — nginx got 'Connection refused'.
Fix: `CYCLONE_HOST` env var, defaults to 127.0.0.1 (preserves local
posture for non-Docker runs), set to 0.0.0.0 by the docker-compose
backend service. Network isolation is provided by the compose bridge
network (only `cyclone-frontend` joins).
3. Healthcheck probed `/api/healthz` (404 — the route is `/api/health`).
Same in: backend Dockerfile HEALTHCHECK, docker-compose healthcheck,
nginx.conf doesn't have one (frontend proxies through), RUNBOOK.md,
scripts/post-deploy.sh, scripts/smoke.sh.
Fix: `/api/healthz` → `/api/health` everywhere SP23 owns.
4. The auth matrix in `cyclone.auth.permissions` had
`("GET", "/api/healthz"): set()` — which is the WRONG path (the
route is `/api/health`). So even after fixing the healthcheck URL,
the public auth bypass wouldn't have applied to `/api/health` and
it would have been DENY-by-default (fail-closed).
Fix: matrix entry updated to `/api/health`.
5. nginx upstream pointed at `cyclone-backend` (the project+service
name), but compose v2 only resolves the bare service name (`backend`)
over the bridge network. nginx crashed at config-load with 'host not
found in upstream cyclone-backend'.
Fix: `cyclone-backend:8000` → `backend:8000` in nginx.conf + spec
+ plan.
6. Frontend HEALTHCHECK used `http://localhost:8080/`. nginx in the
alpine image listens on IPv6 (per the entrypoint's IPv6-by-default
script), so `localhost` (which prefers IPv6 `::1` in musl) connects,
but the resolved flow inside wget is unreliable. `127.0.0.1` works.
Fix: HEALTHCHECK uses `http://127.0.0.1:8080/`.
Also moves `frontend/Dockerfile` → `Dockerfile.frontend` and
`frontend/nginx.conf` → `nginx.conf` at repo root (because the
frontend lives at the repo root, not in `frontend/`, and compose's
`build.context: .` needs them at the same root as compose.yml).
The frontend's pre-existing `.dockerignore` was empty/unused, so it's
dropped — the root `.dockerignore` covers it.
Adds `docker-compose.override.yml` for local bring-up testing on a
host without sudo. Production uses `/etc/cyclone/secrets/` directly.
Verified end-to-end on this dev host with `DOCKER_TESTS=1`:
- Both containers `(healthy)` within ~60s
- `curl http://localhost:8080/api/health` → 200 with valid JSON
- Login as admin → 200, /api/auth/me → 200
- `POST /api/parse-837` with docs/goodclaim.x12 → 200, batch created
- Full backend test suite: 1014 passed, 9 skipped (prodfiles gitignored)
Eight tests: compose file exists, docker compose config validates, services declare required volumes/restart/healthcheck/ports/depends_on, secrets and volumes tables include the names the operator expects, backend wires CYCLONE_BACKUP_AUTOSTART=1, both Dockerfiles pass . The live bring-up test (compose up + wait for healthy) is gated on DOCKER_TESTS=1 so it skips on bare CI. PyYAML was already a hard dep.
Add _read_secret() helper that prefers *_FILE env vars (the standard Docker-secret pattern) over bare env vars. Strips trailing whitespace from file contents so printf/echo newlines don't break bcrypt verify. Existing CYCLONE_ADMIN_USERNAME + CYCLONE_ADMIN_PASSWORD still work for non-Docker deployments. Add tests/test_auth_bootstrap_file.py covering: file-path read, file-overrides-bare, bare-fallback, whitespace-stripping, missing-file raises.
Align CLAUDE.md + REQUIREMENTS.md + ARCHITECTURE.md with the auth work
that landed on main on 2026-06-23; emit an AUTH_DISABLED WARNING at boot
when AUTH_DISABLED is true so misconfigured production deploys fail loudly.
Closes requirements R-1 (was Open; auth shipped via the origin/main merge,
SP24 reconciles the docs).
* CLAUDE.md — track in git for the first time
* docs/REQUIREMENTS.md + docs/ARCHITECTURE.md — replace every stale
'no auth' / 'no authentication' / 'no second party to authenticate'
claim with the v1 posture (bcrypt + HttpOnly session cookie; first
admin bootstrapped from CYCLONE_ADMIN_USERNAME + CYCLONE_ADMIN_PASSWORD)
* backend/src/cyclone/__main__.py — boot-time WARNING when AUTH_DISABLED
* ARCHITECTURE §4.2 module map — add cyclone.auth.* package
* ARCHITECTURE §6.1 migrations table — list 0013 + 0014 with renumbering note
* .superpowers/skills/{cyclone-tests,cyclone-api-router,cyclone-spec} —
add AUTH_DISABLED / matrix_gate / auth-aware threat-model guidance
Spec: docs/superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md
Plan: docs/superpowers/plans/2026-06-23-cyclone-auth-posture-alignment.md
- test_migration_latest_idempotent_on_fresh_db now uses the pytest
tmp_path fixture instead of a literal /tmp path that did not exist,
so sqlite can create the test DB.
- test_drop_claims_unique_constraint_migration now inserts the NOT
NULL batches.kind / input_filename / parsed_at columns when seeding
the parent row for the two same-PCN claims.
Adds migration 0015_drop_claims_unique_constraint.sql that recreates the
claims table without the inline UNIQUE(batch_id, patient_control_number)
constraint, plus tests proving the migration runs cleanly and the
constraint is not re-introduced.
Replaces every stale 'no auth' / 'no authentication' / 'no second party to
authenticate' claim in the three top-level docs with the v1 posture: the
auth boundary is HTTP (bcrypt + HttpOnly session cookie; first admin
bootstrapped from CYCLONE_ADMIN_USERNAME + CYCLONE_ADMIN_PASSWORD env
vars); the file-system posture (SQLCipher at rest, macOS Keychain) is
unchanged; the threat model is still a stolen/imaged drive.
Closes requirements §11 R-1 (was Open; now Closed by SP24 — auth shipped
via the origin/main merge on 2026-06-23, SP24 reconciled the docs).
Also:
- backend/src/cyclone/__main__.py — emit WARNING when AUTH_DISABLED is
True at boot so a misconfigured production deploy fails loudly
- §6.1 migrations table in ARCHITECTURE.md — list 0013 (auth_users_and_
sessions) and 0014 (audit_log_user_id) with the renumbering note
- §4.2 module map — add the cyclone.auth.* package
- Three skills addenda: cyclone-tests (AUTH_DISABLED conftest bypass is
mandatory context), cyclone-api-router (every router needs
Depends(matrix_gate)), cyclone-spec (spec template threat-model is now
auth-aware)
This also tracks CLAUDE.md in git for the first time (was previously
untracked; the SP24 doc updates are in scope for the increment).
Spec: docs/superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md
Plan: docs/superpowers/plans/2026-06-23-cyclone-auth-posture-alignment.md
Two bugs surfaced as 403/500 storms in the live UI:
1. PERMISSIONS matrix only had '/api/reconcile' as a prefix but the
client hits '/api/reconciliation/*' (dashboard reconciliation tab,
manual match endpoint). Every call returned 403 even for admin.
Add ('GET', '/api/reconciliation') and ('POST', '/api/reconciliation')
alongside the existing /api/reconcile entries.
2. The seed CLI built billing_provider.address as a flat string but
to_ui_claim_detail._address_to_ui expects {line1,line2,city,state,zip},
so opening any seeded claim in the detail drawer threw
'str object has no attribute get' → 500. Build the dict from the
provider fields instead.
Before this change, every claim — matched or not — came back from
`/api/claims` with `receivedAmount: 0.0`. `to_ui_claim_from_orm`
hardcoded the value, so the Dashboard's 'Received' KPI and the
claim detail drawer's paid amount always read $0 regardless of
whether the claim had been paired with a paying remittance.
The fix threads the real value through the read path:
- `to_ui_claim_from_orm` accepts a new `received_total: float`
kwarg (default 0.0 for callers that don't have a remittance in
scope).
- `iter_claims` bulk-loads `Remittance.total_paid` for every
matched claim id in the result set (single SQL roundtrip via
`IN` filter — no N+1) and stamps the sum onto each claim dict.
- `manual_match` passes `float(remit.total_paid)` from the remit
it's already holding.
- `manual_unmatch` reads `paired_remit.total_paid` before clearing
the FK so the response still reports what was paid pre-unpair.
- `add` (write path) and `list_unmatched` (filters matched-remit
is NULL) pass `0.0` explicitly for readability.
The dev seed CLI now also generates matched Remittance rows for
every PAID/PARTIAL claim — `matched_remittance_id` is set on the
claim, and the remittance's `total_paid` is derived from the
billed amount using the same ratios the frontend's old sample
fixtures used (60–100% for PAID, 20–50% for PARTIAL). That gives
the Dashboard a non-zero 'Received' KPI on a fresh dev DB.
New tests:
- tests/test_iter_claims_received.py — 4 cases covering matched,
unmatched, orphan FK, and bulk-load paths. Catches regressions
if anyone re-introduces the hardcoded 0.0 or breaks the bulk
query.
Now that Dashboard reads live API hooks instead of the in-memory
sample store, a fresh dev DB renders as $0 KPIs and 'No activity
yet.' for a long time. `python -m cyclone seed` inserts a
deterministic batch of 96 claims + 28 activity events so the
Dashboard / Claims / Activity Log pages have something to render
without going through the Inbox / EDI parser pipeline.
python -m cyclone seed # insert (no-op if seeded)
python -m cyclone seed --reset # wipe + re-insert
python -m cyclone seed --count 10
python -m cyclone seed --status # counts only
The seed mirrors the frontend `sampleData.ts` fixtures (3 TX
providers, 7 payers, 8 CPTs, 96 claims spread across the last 200
days) so dev dashboards look identical to the previous in-memory
fixture mode. `raw_json` and `payload_json` are populated in the
shape that `store.to_ui_claim_from_orm` / `recent_activity` parse
back into the UI wire format — wire parity, not shortcuts.
`--reset` cleans by id prefix (`SEED-` / `CLM-S`) rather than
relying on SQLite's `ON DELETE CASCADE`, which the dev session
doesn't enforce (no `PRAGMA foreign_keys = ON` at the SQLAlchemy
session layer). Safer than silent orphans.
Verified end-to-end: 986 backend tests pass, dashboard renders
$75,833 / 96 claims / 11.5% denial rate / 10 activity events /
top-3 providers ranked by claim count.
Two bootability gaps caught by running the live stack:
1. cyclone.auth.bootstrap.run() called SessionLocal() before db.init_db()
had run on a fresh DB, so 'python -m cyclone serve' / 'users create'
/ '--help' all crashed with 'db.init_db() has not been called'.
Add init_db() inside bootstrap (idempotent — no-op once the schema
is current) so first-boot works without manual prep.
2. PERMISSIONS matrix didn't register the inbox endpoints that were
added in SP10 (export.csv) and SP14 (candidates/dismiss,
rejected/resubmit, payer-rejected/acknowledge). The matrix's
longest-prefix match meant /api/inbox/candidates/dismiss matched
the '/api/inbox' prefix but only because that single entry
existed; adding the dedicated write-prefix entries makes the
intent explicit and keeps viewer role out of every inbox write
even if the broader /api/inbox prefix is later broadened.
Verified live:
- admin can login, GET /api/inbox/lanes (200), GET
/api/inbox/export.csv (200), POST /api/inbox/candidates/dismiss
(200).
- viewer can login, GET /api/inbox/lanes (200), GET
/api/inbox/export.csv (200), but POST /api/inbox/candidates/dismiss
and POST /api/inbox/rejected/resubmit both return 403.
- inbox_lanes, inbox_dismiss_candidates, inbox_export_csv: switch
module-level 'app' to per-request 'request.app' so per-endpoint
state stays consistent with the test's TestClient target across
importlib.reload() (test_api.py::test_cors_extra_origins_via_env
reloads the api module to mutate CORS allow-lists; pre-reload
endpoints then mutate the wrong app instance).
- _http_exc_handler: when HTTPException.detail is a dict, wrap under
'detail' so the standard envelope stays stable and callers can
branch on body['detail']['error'].
- conftest._auto_init_db: re-resolve cyclone.api.app each fixture
invocation (instead of caching at module load) so the reload pattern
doesn't leave event_bus set on a stale app.
- test_acks.test_migration_latest_idempotent_on_fresh_db: bump
user_version assertion from 12 to 14 after auth migration renumber
(0013 users+sessions, 0014 audit_log.user_id).
Also installs sqlcipher3 + paramiko into the backend venv so the
capability tests can run; both modules were optional deps that
test_db_crypto and test_sftp_paramiko assume are present.
Backend test results: 1008 passed, 9 skipped (gitignored prodfile
fixtures), 0 failed.
Nora
Tyler Martinez
tyler
tyler
Cyclone
Tyler