Commit Graph

466 Commits

Author SHA1 Message Date
Nora d81b6ed4fc fix(sp27): /api/claims + /api/remittances total counts the full population
Before this fix, the list endpoints computed `total` via
`len(list(store.iter_*(**common)))`. Both `iter_claims` and
`iter_remittances` default to `limit=100`, so the reported total
silently capped at 100 even when the DB held 60k claims or 835
remits. The frontend rendered a `data.total` of 100 in the KPI
tile and a 100-row table, the page looked complete, and the bug
stayed hidden — exactly the Dashboard silent-failure pattern that
59c3275 (server-aggregated KPIs) was meant to retire.

The Remittances page symptom reported on Jun 29 ('REMITS 100',
empty CLAIM column on 100 rows) was this bug. The empty CLAIM is
a separate matching concern (those remits have `claim_id = NULL`
in the DB); the count fix addresses the population-size lie.

Fix:
* `CycloneStore.count_claims` + `count_remittances` reuse the
  iter's filter pipeline (DB filters + in-memory payer/date
  filters) with an effectively-unbounded limit so the count
  reflects the true DB population.
* `list_claims` + `list_remittances` call the new count
  helpers instead of slicing `iter_*` twice.
* 13 new tests in `test_list_endpoint_counts.py` cover the
  empty/filter/per-dimension/cardinality cases plus the HTTP
  regression (seed 150/120, assert total matches).

Bonus fix:
* `conftest._reset_rate_limit_buckets` walks the middleware
  stack and clears `RateLimitMiddleware._buckets` between
  tests. Without this, the full suite tripped the 300 req/60s
  rate limiter mid-run and 9 later tests (4 of mine, 5
  pre-existing in test_inbox_endpoints / test_payer_summary)
  got 429s. All 1167 tests now pass.

Follow-ups noted but out of scope:
* `/api/admin/audit-log`, `/api/admin/backup/list`,
  `/api/admin/scheduler/processed-files` use the same
  `len(rows)`-after-`.limit(limit)` pattern. Admin-only,
  no `has_more` consumption, so impact is bounded.
* `count_*` could short-circuit to `func.count()` to skip
  the UI-dict build, but iter already calls `q.all()` so
  the win is modest.
2026-06-29 13:22:42 -06:00
Nora 59c3275adf feat(sp27): server-aggregate Dashboard KPIs so 100-row sample doesn't lie
The Dashboard was hardcoded to useClaims({ limit: 100 }) and reduce
KPIs client-side. With 60k+ claims in production, every tile
(Billed $940K, Received $59K, Denial rate, Pending AR, monthly
sparkline, top providers, recent denials) was computed from a
0.16% sample of the dataset — silently wrong numbers on the
operator's primary view.

Fix: add GET /api/dashboard/kpis that aggregates server-side in
one read over the entire claim population. The new useDashboardKpis
hook consumes it and polls every 60s. Dashboard.tsx drops
useClaims({limit:100}) + useProviders() + the client-side
buildMonthly reduce.

Backend:
- store.py: dashboard_kpis() — one Claim query (selectinload on
  batch to avoid N+1) + one bulk Remittance lookup, Python reduce
  over the full population. Zero-filled response for empty DB.
- api.py: GET /api/dashboard/kpis behind matrix_gate, query-param
  clamps (1..24 months, 0..50 top_n_*).

Frontend:
- api.ts: DashboardKpis types + getDashboardKpis() wrapper.
- useDashboardKpis.ts: TanStack Query hook, 60s refetchInterval,
  bypass to data:undefined when not configured.
- Dashboard.tsx: switched to useDashboardKpis, extracted
  ZERO_TOTALS constant, dropped the buildMonthly helper.

Tests:
- backend/tests/test_dashboard_kpis.py: 12 tests covering empty DB,
  matched-remit math, pending-state semantics, monthly binning,
  top-providers/top-denials sort + cap, orphan-claim defensive
  guard, HTTP wiring + param validation.
- src/hooks/useDashboardKpis.test.ts: 3 tests for the hook
  contract (configured path, unconfigured fallback, param
  passthrough).
- src/pages/Dashboard.test.tsx: wrapped renders in
  QueryClientProvider + stubbed useAuth + isConfigured=false. This
  fixes 3 pre-existing Dashboard test failures (the page never had
  a QueryClient set up because useClaims/useProviders were the
  first useQuery hooks in the page).

Reviewer fixes (same commit):
1. topDenials sort placed empty submissionDate claims first under
   reverse-lex. Drop them at append time.
2. r.batch lazy-load → N+1 on 60k rows. selectinload(Claim.batch).
3. pending_states rebuilt per call as a mutable set — moved to
   module-level _DASHBOARD_PENDING_STATES frozenset.
4. Module-level ProviderORM import inconsistent with the
   "local import inside the function" pattern — moved inline.
2026-06-29 12:51:28 -06:00
Nora f5d119fbe7 feat(sp27): pin Claim↔Remit matched-pair invariant + startup drift audit
Add check_matched_pair_drift() at module level in store.py — read-only
audit of the Claim.matched_remittance_id ↔ Remittance.claim_id FK pair.
Logs WARNING on drift with up to 5 examples of each of two cases
(claim-side and remit-side), returns the count of drifted rows. Wired
into api.py::lifespan right after db.init_db() and ensure_clearhouse_seeded(),
wrapped in try/except so a query failure logs but doesn't crash boot.

The symmetric-write + symmetric-clear invariants on
manual_match / manual_unmatch were already correct; this commit pins
them with focused regression tests in test_store_match_invariant.py
(8 tests covering both directions, both happy-path, rollback pin, and
drift-check unit behavior). PCN asymmetry in the fixture prevents the
auto-match inside CycloneStore.add (Task 10) from pre-pairing, so
manual_match is the only writer.

Migration 0016 adds the missing ix_claims_matched_remittance_id index —
the drift check scans WHERE matched_remittance_id IS NOT NULL on every
boot and would become a full-table scan past ~10k claims. Symmetric
with ix_remittances_claim_id (added in 0007). Migration tests bumped
from head=15 to head=16.
2026-06-29 12:21:34 -06:00
Nora d5f95b4f3c feat(sp27): unify 835 ingest + reconciliation in handle_835 (atomic)
Move 'reconcile.run(s, record.id)' inside CycloneStore.add()'s ingest
session, before s.commit(). The placeholder adjustment_amount set by
_remittance_835_row is overwritten by reconcile's CAS-aggregate pass
in the same transaction — readers never see a half-reconciled
Remittance row. If reconcile raises, the entire 835 ingest rolls back
via the session's __exit__.

The previous flow committed batch + remittance rows in session-1,
then opened session-2 to run reconcile.fail-soft. Two visible
problems closed: a race window where readers fetched the placeholder
adjustment_amount, and a half-reconciled state left visible if
reconcile crashed.

Deviation from the plan (N4 in autoreview): the reconcile call lives
in cycl_store.add() rather than handle_835 calling a new
reconcile.run_now(batch_id) helper after add(). Same end state, one
fewer module surface, handler stays a thin wrapper over the store.
2026-06-29 12:08:47 -06:00
Nora 44a6fb031a feat(sp27): surface consecutive_failures + last_error in Scheduler.status() 2026-06-29 11:53:56 -06:00
Nora cf1a0a80d8 feat(sp27): wrap SFTP list_inbound in asyncio.wait_for with CYCLONE_SFTP_OP_TIMEOUT_SECONDS
The 06/25 silent hang was a hung ``sftp.listdir_attr()`` on the
worker thread — paramiko TCP-acked then went silent, the scheduler's
``asyncio.to_thread`` waited forever, and the operator had no
signal that polling had stalled. Every poll cycle since was
suspected of the same failure mode.

``backend/src/cyclone/clearhouse/__init__.``
- New ``_op_timeout_seconds()`` helper reads
  ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` (default 30s). Rejects
  unparseable, zero, and negative values — ``asyncio.wait_for(timeout=0)``
  raises immediately and ``timeout<0`` is undefined per the asyncio
  docs, so a typo would silently turn every SFTP call into an
  instant failure. Logs a WARNING and falls back to the default.
- New ``async_list_inbound()`` async wrapper applies
  ``asyncio.wait_for(asyncio.to_thread(self.list_inbound),
  timeout=N)``. ``wait_for`` cancels the awaiter after N
  seconds but the worker thread keeps running until paramiko returns
  on its own (paramiko is not asyncio-aware, can't be cancelled
  cleanly). ``asyncio.TimeoutError`` propagates so the scheduler
  can surface it as a transient SFTP error.

``backend/src/cyclone/scheduler.py``
- ``_tick_impl`` now calls ``client.async_list_inbound()``
  instead of ``asyncio.to_thread(self._list_inbound)``. Catches
  ``asyncio.TimeoutError`` explicitly with a clear error message
  ("list_inbound: timeout") — separate from the generic
  ``Exception`` catch-all so the operator's tick-result error
  reads as the actual cause.
- Deleted the now-dead ``Scheduler._list_inbound()`` shim.

``backend/tests/test_sftp_op_timeout.py`` (new, 6 tests):
1. Default timeout is 30s when the env var is unset.
2. Operator can override via env var without restart-rebuild.
3. Unparseable value (e.g. "30s") falls back to default.
4. Zero and negative values fall back to default.
5. A hanging list_inbound raises ``asyncio.TimeoutError`` within
   the configured bound (the 06/25 hang pin).
6. A non-hanging call returns the stub's empty list normally
   (sanity check that the wrapper doesn't break the happy path).

Scope note: only ``async_list_inbound`` is wired up. The other
SFTP methods (``list_inbound_names``, ``download_inbound``,
``read_file``, ``write_file``) are called from
operator-triggered paths (admin endpoints, CLI, claim submission)
where the operator can Ctrl-C the request, so a hang is at least
visible. Wrapping them would require making the FastAPI handlers
async, which is out of scope for Task 8. Tracked as a follow-up —
worth folding into the same shape when those endpoints are next
touched.
2026-06-29 11:43:58 -06:00
Nora 34542d1d34 feat(sp27): loosen INBOUND_RE to accept suffix-less inbound filenames
Gainwell's production filer has shipped at least two inbound filename
shapes — the spec form with a trailing `_{file_type}.x12` suffix and
a shorter suffix-less form where the disambiguator token (between
`-` and `_M`) doubles as both orig_tx and file_type. The 6/15–6/19
835 batch arrived in the suffix-less form, and the strict
`INBOUND_RE` rejected them outright — the scheduler silently
dropped 5 days of production data without logging an error.

`backend/src/cyclone/edi/filenames.py`
- New `INBOUND_RE_LOOSE` regex: same prefix/tpid/tracking/ts/seq
  rules as `INBOUND_RE`, but the disambiguator token is required
  to be 3–5 uppercase alnum (covers 999, TA1, 835, 277CA, ENCR; the
  5-char cap stops the engine from over-eating the next `_M` token
  in a degenerate input).
- `parse_inbound_filename` now tries `INBOUND_RE` first
  (preserves historical behavior for every existing caller) and
  falls back to `INBOUND_RE_LOOSE`. In the loose form, orig_tx is
  set to the disambiguator token so the parsed shape matches what
  the strict form produces when orig_tx == file_type. The
  `ALLOWED_FILE_TYPES` check is enforced in both branches.
- `is_inbound_filename` is loosened in the same shape so the two
  never disagree — a refactor that pre-filters a directory listing
  with `is_inbound_filename` then re-parses with
  `parse_inbound_filename` would otherwise see the suffix-less
  files rejected twice.

`backend/tests/test_inbound_filename_loose.py` (new, 10 tests):
1. Spec form with explicit `_835.x12` suffix still wins (strict
   path unchanged).
2-5. Suffix-less 835 / 999 / 277CA / ENCR all parse correctly.
6. Suffix-less `.txt` is still rejected (the `.x12` ext check
   applies to both forms).
7. Suffix-less unknown type (4-char `ABCD`) is rejected by
   ALLOWED_FILE_TYPES — the loose regex's `{3,5}` shape would
   otherwise let it through.
8. Suffix-less 6-char token (`999XX6`) is rejected by the
   5-char cap, preventing the engine from swallowing the next
   `_M` token.
9. Strict form takes precedence over the loose form when both
   match — pinning the parser's branch order so a refactor can't
   silently change the parsed shape.
10. `is_inbound_filename` accepts the loose form, the spec form,
    and rejects garbage.

Live smoke: 5/5 suffix-less Gainwell patterns now parse correctly
(were ValueError before); spec form unchanged; 4 invalid forms
still rejected. Full backend suite: 1085/1121 — the 36 pre-existing
failures (test_serialize_837, test_api_stream_live,
test_inbox_endpoints) are unrelated and confirmed pre-existing by
running them on stashed pre-change code.
2026-06-29 11:30:00 -06:00
Nora 4c05c6527b hotfix(acks): paginate /api/acks and surface server-side aggregates
The Acks page silently capped at 100 of 1056 rows in the operator's
DB: the eyebrow read `${items.length} on file` (not the server's
`total`), the KPI strip summed from the 100 visible items, and the
endpoint accepted no `offset` — so the user had no signal that 956
more acks existed. Same class of bug on the Activity page (cap=200,
no 'X of Y' hint).

Backend
- /api/acks (acks.py): add `offset`, bump `le=1000`→`le=5000`,
  slice `rows[offset:offset+limit]`, return server-side
  `aggregates` (accepted/rejected/received summed over the full
  row set, not the page) so the KPI strip reflects every persisted
  999 instead of just the visible 50.
- /api/activity list + stream (api.py): bump `le=500`→`le=5000` so
  the page can ask for a denser snapshot.
- /api/277ca-acks (api.py): bump `le=1000`→`le=5000` for
  consistency.
- /api/ta1-acks: left at `le=1000` — TA1s aren't shipped today and
  the structural fix (offset + aggregates) wasn't applied, so a
  larger cap would just make the same latent silent-failure easier
  to hit. (TODO: fold in the same shape when Gainwell starts
  shipping TA1s.)

Frontend
- listAcks (api.ts): accept `offset`, surface `aggregates`,
  adapt wire `*_count` keys to the in-page
  `accepted`/`rejected`/`received` shape so the page can use
  `data.aggregates` as a drop-in for the page-local fallback
  accumulator.
- useAcks (hooks): pass `offset` through; return type carries
  `aggregates`.
- Acks.tsx: add `page` state (PAGE_SIZE=50), use `data.total`
  for eyebrow + watermark (not `items.length`), use
  `data.aggregates` for the KPI strip (with in-page fallback
  accumulator on first paint), render `<Pagination>` when
  `totalCount > PAGE_SIZE`. Footer row reads "N rows on file"
  instead of "N rows".
- ActivityLog.tsx: bump `limit: 200`→`limit: 500`, eyebrow reads
  "Activity · showing N most recent" to make the bounded-window
  semantics honest (the endpoint doesn't expose a true total — it
  reports events matching the current kind/since filter, capped at
  the request limit).

Tests
- test_acks.py: 4 new tests pin the fix:
  1. `offset` walks the full set; `has_more` flips at the
     boundary.
  2. `aggregates` reflects the full row set, not the page (the
     silent-failure pin) — and stays stable across page slices.
  3. `limit` cap of 5000 is enforced (422 above it).
  4. `offset` past the end returns an empty page with stable
     aggregates (a stale UI page state across a row count change
     must not 500 or zero the KPIs).

Live smoke-verified: /api/acks?limit=2&offset=0 vs ?offset=2 return
the expected row slices, aggregates stable at 15/10/15 for 5 seeded
rows, /api/acks?limit=10000 rejected with 422.

Triage note: the TA1 section (`Ta1AcksSection`, lines 609-616 of
Acks.tsx) has the same latent silent-failure pattern (page-sums
KPIs, no offset on /api/ta1-acks). Left untouched because the
empty-state copy says Gainwell doesn't ship TA1s today and the
larger structural fix belongs in a follow-up.
2026-06-29 11:19:10 -06:00
Nora 4592bca372 feat(sp27): dedup ack ID helpers — one copy in handlers/_ack_id.py 2026-06-29 11:00:37 -06:00
Nora 79fa30d018 feat(sp27): extract handle_835 from scheduler.py into handlers/ 2026-06-29 10:52:07 -06:00
Nora 35730fcf14 feat(sp27): extract handle_277ca from scheduler.py into handlers/ 2026-06-29 10:46:29 -06:00
Nora 30e1add8a2 feat(sp27): extract handle_ta1 from scheduler.py into handlers/ 2026-06-29 10:38:32 -06:00
Nora d248a5f282 feat(sp27): extract handle_999 from scheduler.py into handlers/ 2026-06-29 10:31:40 -06:00
Nora 0f1e609888 feat(sp27): create handlers/ package skeleton + dedup ack ID helpers
CycloneStore split precedent (SP21) — lift three helpers from
scheduler.py + api.py into one module. Both callers will switch
to this in Task 6. The new package also defines HandleResult
and the HANDLERS registry; handle_999 / handle_ta1 / handle_277ca
/ handle_835 fill in over Tasks 2-5.

The registry is lazy + best-effort: register_handlers() catches
broad Exception so a partial-modification SyntaxError in one
in-flight handler module can't break scheduler or API import.

11 tests added; existing suite unchanged (1 failed + 2 errors
pre-existing in test_provider_extended_response.py are not
introduced by this commit).
2026-06-29 10:09:10 -06:00
Nora 9d8f83d111 docs(plan): implementation plan for SP27 remittances architecture refactor
17 tasks: 1 preflight + 4 handler extracts (999/TA1/277CA/835) +
helpers dedup + INBOUND_RE loosen + SFTP timeouts + status surface +
atomic 835/reconcile + match invariants + chain endpoint +
claim.rejected_after_remit + frontend chain UI + final verify + merge.

Each task ends with live-test + autoreview + commit. Spec:
docs/superpowers/specs/2026-06-29-cyclone-remittances-architecture-refactor-design.md
2026-06-29 09:59:53 -06:00
Nora 454c3598b1 docs(spec): design for SP27 remittances architecture refactor
Tier 1: split scheduler.py into handlers/ subpackage, dedup helpers,
loosen INBOUND_RE, add SFTP operation timeouts, surface SFTP errors
in Scheduler.status().

Tier 2: unify 835 ingest + reconciliation into one critical section,
add GET /api/claims/{id}/chain, guard matched_remittance_id ↔
Remittance.claim_id invariant, emit claim.rejected_after_remit audit
when a 277CA rejection hits a matched claim.

Status: Draft, awaiting user sign-off.
2026-06-29 09:54:33 -06:00
Nora f57f1d2875 merge: fix 999 IK5 and TA1 UI into Version-1.0.0 2026-06-29 09:52:19 -06:00
Nora 315fbfec42 fix(docker): wire CYCLONE_SFTP_PASSWORD via docker secret in compose override
Mirrors the existing admin_username / admin_pw pattern so the dev
compose loads SFTP creds from /tmp/cyclone-test-secrets/sftp_password
when present. Falls back to the CYCLONE_SFTP_PASSWORD env var if the
file is missing. Pre-existing env-var support from SP25+26 unchanged.
2026-06-29 09:51:09 -06:00
Nora 6507a8c874 fix(acks): accept IK5 from Gainwell, trust set-level codes over bogus AK9, surface TA1 in UI
Two related fixes land together because the UI was reporting
"1 accepted 1 rejected" for every 999 even though every inbound file
Gainwell ships has IK5=A.

  1. Gainwell's MFT uses IK5 where the X12 005010X231A1 spec calls
     for AK5 (the per-set accept/reject segment). The parser only
     recognized AK5, so set_responses[0].set_accept_reject.code
     defaulted to 'R' and the count summary showed all rejections.
     _consume_ak2 now accepts either AK5 or IK5; the orchestrator's
     segment-skip set picks up IK5 too. A new fixture
     (minimal_999_ik5_gainwell.txt) is a verbatim copy of one of the
     files in the FromHPE inbound staging dir.

  2. _ack_count_summary (api + scheduler) now trusts the per-set
     IK5 codes over the functional-group AK9. Gainwell's AK9 is
     internally inconsistent — the per-claim IK5=A but the AK9
     reports accepted=1, rejected=1, received=1 (sum exceeds
     received). Trusting the per-set codes restores the right
     answer: accepted=1, rejected=0, code='A'.

  3. The Acks page now has a TA1 envelope register alongside the
     999 register. TA1s are the lower-level sibling of the 999
     (one row per inbound ISA/IEA). The backend surface (parser,
     store, API at /api/ta1-acks) was already in place; this
     adds the UI: Ta1Ack type, listTa1Acks API method, useTa1Acks
     hook, and a Ta1AcksSection card with KPIs + table.

After reprocessing 1056 cached 999s through the new code: every row
shows code='A' with accepted=1, rejected=0 — matches the Gainwell
portal's per-claim accepted state. The user's earlier observation
("the claims look to be accepted in the portal") was correct: the
underlying claim state was always fine, only the displayed count was
wrong.

  - backend/src/cyclone/parsers/parse_999.py | 21 ++-
  - backend/src/cyclone/api.py               | 11 +-
  - backend/src/cyclone/scheduler.py         | 13 +-
  - backend/tests/test_parse_999.py          | 32 ++++
  - backend/tests/fixtures/minimal_999_ik5_gainwell.txt
  - src/types/index.ts                       | 32 ++++
  - src/lib/api.ts                           | 62 +++++-
  - src/hooks/useTa1Acks.ts                  | 26 +++ (new)
  - src/pages/Acks.tsx                       | 209 +++++++++++++++++++-
2026-06-25 00:26:13 -06:00
tyler 1381a7652d fix(acks): make 999 source_batch_id unique per file + surface PCN
Gainwell's MFT ships every 999 with the same ISA interchange
control number (`000000001`) and one 999 ack covers a whole
batch, not a single claim — so the AK2 set_control_number
(patient_control_number) is the same for the ~96 999s in a
batch. With the old synthetic-id formula (`999-{icn}`), all 385
daily acks collapsed onto a single row the operator couldn't
distinguish.

The new formula is `999-{pcn}-{filename_hash8}` (or
`999-{icn}-{filename_hash8}` for envelope-only 999s without an
AK2). The PCN gives the operator a human-readable handle to the
claim batch; the 8-char hash of the inbound filename guarantees
uniqueness within a batch. Fits in the VARCHAR(32) source_batch_id
column (max 22 chars).

Also surface `patient_control_number` in /api/acks list response
(extracted from raw_json's set_responses[0].set_control_number)
and in the Acks UI as the primary label, with the synthetic id
shown dimmed after a middle dot. The detail endpoint already
exposed raw_json for the full 999 parse tree.
2026-06-24 23:55:58 -06:00
tyler c3a6c53096 fix(sftp): case-insensitive inbound regex, skip _warn.txt, add targeted pull
Three changes that unblock the daily inbound pull from Gainwell's
FromHPE MFT path:

1. INBOUND_RE now accepts both 'TP' and 'tp' prefixes via inline
   (?i:TP) scoping — case-folding the whole pattern would let
   lowercase '837p' / tracking IDs through, which is invalid HCPF.
   The rest of the pattern is still case-sensitive.

2. _list_inbound_paramiko skips *_warn.txt entries. Gainwell's MFT
   drops ~583 advisory text-format notes in the same inbound dir;
   they come first alphabetically and were padding every poll with
   ~80 min of pointless downloads.

3. New SftpClient.list_inbound_names() + download_inbound() pair
   gives a metadata-only listing and on-demand fetch. The
   scheduler's existing full-listing path still works (now without
   the warn padding); the new path is what the new
   /api/admin/scheduler/pull-inbound endpoint and the
   'cyclone pull-inbound' CLI use to fast-target a date range
   without paying the cost of a full ~6000-file download.

Scheduler.process_inbound_files() runs the same per-file pipeline
as a regular tick on the pre-fetched list, so dedup via
processed_inbound_files still applies.

Tests added in test_filenames.py (lowercase + mixed-case cases),
test_sftp_paramiko.py (warn skip + no-download listing), and
test_scheduler.py (process_inbound_files idempotency).

With this, the daily 385-file pull for 20260624 completes in
seconds via 'docker exec cyclone-backend-1 python -m cyclone
pull-inbound --date 20260624 --block dzinesco' (or the equivalent
POST to /api/admin/scheduler/pull-inbound?date=20260624).
2026-06-24 23:23:46 -06:00
Nora a436538c15 fix(scheduler): read inbound bytes from the cached local_path, not read_file(f.name)
In real (paramiko) mode, `_download_and_parse` called
`client.read_file(f.name)` with a bare filename. paramiko's
`sftp.open(f.name)` opens at the SFTP root, not at `paths.inbound`
(FromHPE) — so the scheduler would fail to download any file in real
mode even with the path swap from the previous commit.

But this round-trip is also unnecessary: `_list_inbound_paramiko`
already downloads each entry into the local cache (cache_path) and
returns it as `InboundFile.local_path` as part of the listing pass.
Reading from disk is faster than re-fetching and avoids the path bug.

Stub mode was already reading from `f.local_path`. Now both modes do,
which is the simpler invariant.

Verified: 36 tests pass (test_scheduler, test_api_scheduler, test_sftp_stub,
test_sftp_paramiko).
2026-06-24 22:15:52 -06:00
Nora dd7da18279 fix(sftp): swap inbound/outbound paths — FromHPE is inbound, ToHPE is outbound
The dzinesco SP9 seed had `paths.inbound` and `paths.outbound` mapped to
the wrong Gainwell MFT directories:

  - paths.outbound  was  FromHPE/   (HPE sends files FROM here TO us — inbound)
  - paths.inbound   was  ToHPE/     (we send files TO here — outbound)

So `/api/clearhouse/submit` was writing 837P claims to FromHPE (where
HPE puts acks/835s) and the SP16 scheduler was polling ToHPE (where our
claims go). 999 / TA1 / 835 files in the real FromHPE inbox were
unreachable.

Semantics per operator (2026-06-24):
  - FromHPE = HPE/Gainwell → us  = 999, TA1, 835  (inbound)
  - ToHPE   = us → HPE/Gainwell = 837P claims    (outbound)

**Runtime code (the actual fix):**
  - backend/src/cyclone/store.py            — SP9 seed paths flipped
  - backend/src/cyclone/edi/filenames.py    — docstring corrected

**Test fixtures + assertions (would otherwise fail on the new seed):**
  - backend/tests/test_clearhouse_api.py
  - backend/tests/test_providers_seed.py
  - backend/tests/test_sftp_stub.py        — incl. inbound dir paths
  - backend/tests/test_sftp_paramiko.py
  - backend/tests/test_store_update_clearhouse.py
  - backend/tests/test_api_clearhouse_patch.py
  - backend/tests/test_scheduler.py
  - backend/tests/test_api_scheduler.py

**Docs (text-only — keeps the codebase self-consistent):**
  - README.md
  - docs/reference/co-medicaid.md
  - docs/superpowers/specs/2026-06-20-cyclone-multi-payer-npi-sftp-design.md

**Operator action required after merge:** the existing clearhouse row in
`~/.local/share/cyclone/cyclone.db` was seeded with the old wrong
paths. Easiest recovery:
  1. `rm ~/.local/share/cyclone/cyclone.db` and let `ensure_clearhouse_seeded` re-run on next boot, OR
  2. PATCH /api/clearhouse with the new `paths` block (the SP25
     reconfigure hook picks it up live, including by the running scheduler).

**Verification:**
  - 82 tests in the affected files pass (test_clearhouse_api, test_providers_seed,
    test_sftp_stub, test_sftp_paramiko, test_store_update_clearhouse,
    test_api_clearhouse_patch, test_scheduler, test_api_scheduler, test_filenames).
  - Full backend suite: 1029 pass + 36 pre-existing order-dependent flakes
    unrelated to this change (verified by running the same tests in isolation).
2026-06-24 22:05:55 -06:00
tyler 9cb0311544 fix(docker): install [sftp] extra so paramiko is in the runtime image
The backend Dockerfile only installed the [sqlcipher] extra, so
paramiko wasn't on the path inside the container. SP25 + SP26's
real-mode SFTP client needs paramiko to connect to the Gainwell
MFT — the scheduler tick surfaced this as
'No module named paramiko' on the very first /api/admin/scheduler/tick
against the dzinesco clearhouse with stub=false.

Install both extras ([sqlcipher,sftp]) in builder and runtime stages.
2026-06-24 18:20:22 -06:00
tyler b7be9f38bc merge: SP25 + SP26 SFTP polling + secret-file lookup into Version-1.0.0
Bring the SP25 SFTP Polling Enablement (PATCH /api/clearhouse,
scheduler hot-reload, env-var-first secret lookup) and SP26
SFTP Password File Companion (Docker secret _FILE tier) onto
the v1.0.0 release branch so the production stack can actually
flip the Gainwell MFT poll from stub → real.

Conflict resolved in docker-compose.yml: kept both the SP26
CYCLONE_SFTP_PASSWORD_FILE env var and the new 'Always bind
to 0.0.0.0' comment.
2026-06-24 18:09:32 -06:00
tyler 1b74af4d3a fix(permissions): populate matrix for all gated endpoints
The matrix_gate dependency was added to ~50 routes (clearhouse, all
/api/admin/*, /api/config/*, /api/eligibility/*, /api/parse-999/ta1/277ca,
/api/batches/{id}/export-837, /api/payer-rejected/acknowledge, etc.)
but the PERMISSIONS matrix was only populated for ~25 of them. Default-
deny therefore returned 403 to every authenticated role on the rest,
including the SFTP admin endpoints needed for MFT polling.

Roles:
- Admin-only: /api/clearhouse* (SFTP creds + dzinesco identity), all
  /api/admin/* (audit-log, backup, scheduler, db rotate, reload-config,
  validate-provider)
- All authenticated: /api/batch-diff, /api/config/*, /api/payers/*
- Write (admin + user, no viewer): /api/parse-999/ta1/277ca, the
  /api/batches POST prefix (regenerates X12 from DB rows), and
  /api/eligibility/* (270 build / 271 parse)

Unblocks /api/clearhouse, /api/admin/scheduler/*, /api/admin/backup/*,
and the rest of the admin surface so MFT polling and live verification
can proceed.
2026-06-24 17:58:31 -06:00
tyler b80e40e7e9 fix(permissions): expose GET /api/acks, /api/ta1-acks, /api/277ca-acks
The PERMISSIONS matrix only listed the POST /api/acks entry (parse-999
ingest). The GET list + detail endpoints for 999 / TA1 / 277CA acks
were missing, so the matrix_gate (default-deny) returned 403 to every
authenticated role, breaking the 999 ACKs / TA1 / 277CA inbox pages.

Add the three GET entries as ALL_ROLES — they're read-only metadata
surfaces that every authenticated operator needs to see. Add a
regression test that logs in as admin via the public route and
exercises the matrix; the existing test_existing_endpoints_require_auth
suite only checks the unauthenticated 401 path, which is why this slipped
through.

Fixes the live-verification 'Couldn't load ACKs from the backend /
forbidden' error reported against the UI.
2026-06-24 17:51:44 -06:00
tyler edca04868d feat: always bind to 0.0.0.0
Reachability is controlled by the host firewall / compose port
publishing, not the bind address. Backend, compose, and docs all
now default to 0.0.0.0 so the API is reachable from the frontend
container, the host, and the LAN without per-env overrides.

Files:
- backend/src/cyclone/__main__.py: default host = 0.0.0.0
- docker-compose.yml: refresh the comment to match the new posture
- CLAUDE.md / README.md / docs/ARCHITECTURE.md / docs/REQUIREMENTS.md:
  reframe the bind note accordingly
2026-06-24 17:40:18 -06:00
Nora 74aa64f995 merge: SP26 SFTP Password File Companion into main 2026-06-24 16:11:21 -06:00
Nora 4ef09f9416 test(sp26): assert CYCLONE_SFTP_PASSWORD_FILE + cyclone_sftp_password in compose 2026-06-24 16:09:19 -06:00
Nora 3c907d17e9 docs(sp26): RUNBOOK — CYCLONE_SFTP_PASSWORD_FILE row + Docker variant 2026-06-24 16:08:12 -06:00
Nora 675faf9844 feat(sp26): wire cyclone_sftp_password secret into docker-compose.yml 2026-06-24 16:07:32 -06:00
Nora 1e8217ff84 feat(sp26): secrets.get_secret() _FILE-tier lookup for Docker secrets 2026-06-24 16:06:54 -06:00
Nora 3075955826 docs(plan): SP26 SFTP Password File Companion — implementation plan 2026-06-24 16:03:51 -06:00
Nora 5b4c1513b6 docs(spec): SP26 SFTP Password File Companion — initial design 2026-06-24 15:52:19 -06:00
Nora 4e580ffe0e merge: SP25 SFTP Polling Enablement into main 2026-06-24 15:42:59 -06:00
Nora 13a3e7c334 docs(sp25): RUNBOOK.md — enable SFTP polling for real Gainwell MFT 2026-06-24 15:38:35 -06:00
Nora 439a3c3d4b feat(sp25): PATCH /api/clearhouse with hot-reload 2026-06-24 15:38:14 -06:00
Nora 5db764d50e feat(sp25): scheduler.reconfigure_scheduler() hot-reload helper 2026-06-24 15:36:19 -06:00
Nora 2d3667738d feat(sp25): store.update_clearhouse() for PATCH endpoint 2026-06-24 15:30:53 -06:00
Nora b2d88d13d3 feat(sp25): secrets.get_secret() env-var-first lookup 2026-06-24 15:29:52 -06:00
Nora 9fda46b891 docs(plan): SP25 SFTP Polling Enablement — implementation plan 2026-06-24 15:28:54 -06:00
Nora d401904412 docs(plan): SP25 SFTP Polling Enablement — implementation plan 2026-06-24 15:25:37 -06:00
Nora 7d2000ec31 docs(spec): SP25 SFTP Polling Enablement — initial design 2026-06-24 15:25:20 -06:00
Nora 1860782ad6 chore(release): bump to 1.0.0
Marks v1.0.0 launch. Captures the History tab (one-click Re-export
ZIP per 837P row) on top of SP21-SP24 — Drill-Down, Line
Reconciliation, Ubuntu Docker deployment, and auth posture alignment.
v1.0.0
2026-06-24 14:02:39 -06:00
Nora ce37c10c06 merge: History tab on Upload page into main 2026-06-24 13:57:16 -06:00
Nora f4bafc1c94 feat: History tab on Upload page with one-click Re-export ZIP
Add a second tab to the Upload page that surfaces the persisted batch
archive and lets the user re-download any 837P batch as a ZIP without
re-parsing the original file.

- Backend: /api/batches now carries per-row claimIds (837P only).
  835 batches return an empty list, which the UI uses as the signal
  to hide the Re-export button on those rows. Avoids an extra
  round-trip to /api/batches/{id} per row.
- Frontend: BatchSummary.claimIds added to the list-endpoint type.
- Upload page: page body wrapped in Tabs.Root with a History trigger
  that mirrors ?tab= in the URL for deep-link round-trip. The
  History tab renders UploadHistory → HistoryTable → HistoryRow with
  a one-click Re-export ZIP button per 837P row. The button calls
  POST /api/batches/{id}/export-837 with the row's claim ids and
  downloads the ZIP via downloadBlob. Falls back to the in-memory
  parsedBatches store when the backend returns no rows so the tab
  stays useful in sample-data mode.
- Backend tests: claimIds present on 837P rows, empty on 835 rows.
- Frontend tests: 13 tests covering tab switching, URL deep-link,
  loading/error/empty states, the 837P-vs-835 button visibility
  split, the Re-export happy path, and the failure toast.
2026-06-24 13:57:12 -06:00
Nora 24fbf945c9 fix: allow credentials on CORS for Vite dev server
The dev-server allow-list (VITE_DEV_ORIGINS = localhost:5173 +
127.0.0.1:5173 + CYCLONE_ALLOWED_ORIGINS) is tight enough that
`allow_credentials=True` is safe — the browser was dropping the
session cookie on cross-origin fetches from the Vite dev server
otherwise. Tight allow-list + credentials is the standard setup.
2026-06-24 09:14:32 -06:00
Nora 07a7ecbdd4 merge: SP23 Ubuntu Docker Deployment into main 2026-06-23 20:34:38 -06:00
Nora 5334646992 docs(plan): SP23 — record live verification results
Appends a 'Live verification' section to the plan with the six bugs
the live bring-up surfaced + their fixes, and the end-to-end smoke
results (login, /api/auth/me, /api/parse-837 with a real 837P,
full test suite 1026 passed).
2026-06-23 17:55:54 -06:00