20 pre-existing pytest failures were caused by tests/test_api.py
calling importlib.reload(cyclone.api) mid-suite. Reload creates a
NEW FastAPI app with a NEW RateLimitMiddleware whose private
_buckets dict is independent of the OLD instance. Tests that
imported 'from cyclone.api import app' at module load kept
referencing the OLD app, so their requests accumulated in the
orphaned bucket which the conftest reset never cleared. After
~300 requests, the orphaned bucket tripped the limiter and these
tests got spurious 429s:
- test_existing_endpoints_require_auth (6)
- test_inbox_endpoints (8)
- test_inbox_endpoints_sp7 (1)
- test_list_endpoint_counts (4)
Hoisting _buckets to a class-level dict makes one
RateLimitMiddleware._buckets.clear() (in conftest reset) reach
every instance — current, stale, post-reload — eliminating the
orphaned-bucket leak. Per-instance _lock stays per-instance
since it guards mutation of the shared dict.
Verified stable: 3 consecutive full-suite runs all pass with
1435 passed, 10 skipped, 0 failed in ~82s each.
Three pr-reviewer followups from the 2026-07-07 review of commit ad14b56:
1. BUG: find_ack_orphans refactor routed 277ca/ta1 through
_ack_control_number which only knew 999 — restored per-kind source
(999 reads raw_json.envelope.control_number, 277ca/ta1 read the ORM
control_number column). Added regression test pinning all three
kinds.
2. DOCSTRING DRIFT: reconcile_orphan_st02s said 'remaining columns
take their defaults' but explicitly passes parsed_at and
transaction_set_control_number — added both to the explicit list
and clarified the rest take schema defaults.
3. WASTED SORT: _iter_orphan_999_st02s yielded sorted() but the
caller re-sorts by (-ack_count, st02) — yielding unsorted now.
Plus three cleanups the reviewer flagged:
* Hoisted 'json' / 'uuid' / 'datetime' imports to module top of
store/__init__.py (replaced in-method imports).
* Added two missing tests: sentinel grep-discoverability via
LIKE '<synthetic:%>' + 999-walk tolerance for non-dict raw_json
(None / list shapes — bytes is unreachable through the ORM).
* Aligned spec/plan exit codes to the cyclone-cli convention
(exit 1 on DB error, not 2 — matches the existing CLI
sys.exit(1) and the cyclone-cli skill documentation).
36/36 SP38 tests pass.
Implements tasks 4-5 of
docs/superpowers/plans/2026-07-07-cyclone-orphan-ack-housekeeping.md.
Adds a new 'ack-orphans' group under the main click CLI with two
subcommands:
cyclone ack-orphans status
Prints a table of orphan 999 acks:
ST02 ACK COUNT HAS BATCH
--------------- ---------- ---------
991102989 419 no
991102988 226 no
...
TOTAL 804
Empty-DB case prints 'no orphans' instead of a blank table.
cyclone ack-orphans reconcile [--dry-run]
One-shot synthetic-batch seeder. --dry-run prints the plan
without writing. Real call inserts one synthetic batches row
per orphan ST02 lacking one, marked with the
'<synthetic:orphan-reconcile>' sentinel so they're trivially
distinguishable in queries.
Exit codes per cyclone-cli convention: 0 = success, 1 = DB error.
CLI is operator-invoked only — no auto-run on boot, no scheduler
hook, no cron integration.
7 new tests (test_ack_orphans_cli.py) use click.testing.CliRunner
(matching the SP37-followup #5 pattern that replaced subprocess.run
with the in-process runner). Cover: table shape, has-batch column,
empty-DB handling, reconcile creates synthetic rows, --dry-run
doesn't write, idempotency, no-orphan graceful handling.
Live-verified against ~/.local/share/cyclone/cyclone.db: status
prints the 4 distinct ST02s (419/226/106/53 = 804 total). Reconcile
inserts 4 synthetic batches. Subsequent status shows all 4 marked
has_batch=yes.
Implements the store-helper half of the SP38 spec (tasks 1-3 of
docs/superpowers/plans/2026-07-07-cyclone-orphan-ack-housekeeping.md).
Two new CycloneStore methods + one extraction helper in claim_acks.py:
- find_ack_orphan_st02_summary() -> list[dict]
Per-ST02 breakdown of orphan 999 acks. Returns one row per
distinct (st02, ack_count, has_batch, batch_id). Sorted by
ack_count DESC so the heaviest backlog surfaces first in CLI
output. 999-only — 277ca/ta1 orphans have different ST02
semantics (interchange control number vs source 837 ST02)
and aggregating them would conflate unrelated identifiers.
- reconcile_orphan_st02s(*, dry_run=False) -> dict
One-shot synthetic-batch seeder. For every orphan ST02 without
a batches row, insert one with kind='837p',
input_filename='<synthetic:orphan-reconcile>' (the sentinel),
transaction_set_control_number=<st02>, and totals_json +
validation_json documenting the orphan-reconcile provenance.
Idempotent: re-running after a successful pass is a no-op.
dry_run=True returns the plan shape without writing.
- _iter_orphan_999_st02s() -> Iterator[(st02, count)]
Extracted walk helper. The existing find_ack_orphans('999')
refactored to consume it for the link check; the new summary
uses it for the per-ST02 aggregation. Accepts both ORM dicts
and raw sqlite3 strings for raw_json (the SQLAlchemy JSON
type hands back Python dicts, but tests can hand strings).
11 new tests (test_ack_orphan_summary.py) cover:
- per-ST02 counts with duplicates
- has_batch=True when a batches row covers the ST02
- sort order (heaviest first)
- exclusion of 999s with a claim_acks link
- skip of malformed/empty raw_json
- empty-DB case
- reconcile creates synthetic rows for missing ST02s only
- reconcile skips ST02s with existing batches
- idempotency
- --dry-run flag
- ack_count preserved in totals_json
Live-verified against ~/.local/share/cyclone/cyclone.db: 4 distinct
orphan ST02s, 804 total orphan rows. After reconcile, all 4 marked
has_batch=yes.
Audit on 2026-07-07 found three production bugs and six stale matrix
entries:
Three registered routes were DENIED in production (fail-closed):
- DELETE /api/acks/{kind}/{ack_id}/match-claim/{claim_id}
(Inbox unlink-a-wrong-match action; wired from the frontend)
- GET /api/dashboard/kpis (dashboard summary cards)
- GET /api/inbox/ack-orphans (Inbox 'Ack orphans' lane; already
wired into src/hooks/useAckOrphans.ts + AckOrphansLane.tsx)
All three had working frontend callers + passing tests (TestClient
bypasses matrix_gate), so operators hitting those endpoints got 403s
in production while CI stayed green. Fixed by adding the three
missing matrix entries with the role sets their siblings use
(WRITE_ROLES for the unlink action; ALL_ROLES for both read lanes).
Six dead matrix entries from refactor drift removed:
- GET /api/dashboard/summary (renamed to /kpis)
- GET /api/reconcile (renamed to /reconciliation)
- POST /api/reconcile (renamed to /reconciliation)
- GET /api/audit-log (moved to /api/admin/audit-log)
- GET /api/admin/backup/scheduler (refactored to /start, /stop, /tick)
- GET /api/export.csv (renamed to /api/inbox/export.csv)
Plus a one-line doc fix at api_routers/claim_acks.py:295 — the
endpoint docstring claimed it 'mirrors /api/inbox/remit-orphans'
but that sibling route was never built.
Guard test: tests/test_routes_have_auth_coverage.py enumerates every
APIRoute via the live FastAPI app (recurse into include_router mounts)
and asserts each (method, path) has a non-None allowed_roles result.
Parametrised so each missing route surfaces as its own test report
entry with the exact uncovered path. 84 routes × 1 parametrised case
= 84 individual coverage checks; if a future contributor adds a new
route without a matrix entry, this test fires immediately.
The SftpClient wrapper previously exposed write_file, list_inbound,
list_inbound_names, download_inbound, and read_file — but NO stat().
The canonical cyclone.submission.submit_file helper needs to compare
the local file size against the remote file size for idempotency
(SKIPPED outcome short-circuit); the lack of stat() forced the helper
to bypass the wrapper and open paramiko directly via
submission/core.py:_default_sftp_factory.
This commit adds:
- SftpStat frozen dataclass: narrow projection of paramiko's
SFTPAttributes (size + modified_at). Callers don't need paramiko.
- SftpClient.stat(remote_path) -> SftpStat public method
- _stat_stub + _stat_paramiko backends (mirrors the existing
_read_file_stub / _read_file_paramiko pattern)
- 8 new tests covering dataclass shape, frozen invariant, stub size,
stub mtime, missing-file FileNotFoundError, large-file size, and
the actual SKIPPED-outcome idempotency check from submit_file
Also adds backend/var/ to .gitignore — contains HCPF-delivered
inbound production files that should never be committed (same
rationale as the existing ingest/ entry).
Followup #4 from the SP37 final-state tracker. The previous
BACKFILL_SQL constant in test_migration_0020.py was a hand-copied
duplicate of the migration's UPDATE statement. A future contributor
could edit one without the other and the test would silently
replay a different SQL than production — defeating the regression.
Fix: tests now load the migration file at test time and extract
its UPDATE via the same splitter db_migrate.run() uses (strip
'--' comments, split on ';'). The test can never disagree with
what production runs.
Changes:
* test_migration_0020.py:
- Remove the hand-copied BACKFILL_SQL constant
- Add _migration_0020_path(), _extract_update_statements(),
and _load_migration_0020_backfill_sql() helpers
- Replace 3 BACKFILL_SQL references with helper calls
* test_migration_0020_no_drift.py (new, 3 tests):
- test_migration_0020_backfill_sql_uses_migration_file
(asserts the extracted SQL targets the right column + path)
- test_migration_0020_backfill_sql_is_non_empty_single_statement
- test_migration_0020_has_exactly_one_update (guardrail against
future contributors adding a second UPDATE — the extraction
fails loudly so the test author can decide which is the
backfill)
Tests: 43/43 pass in 1.46s (full SP37 followup chain).
Imports across test files match the existing pattern (test_store.py
imports from test_store_reconcile.py).
Followup #3 from the SP37 final-state tracker. The router's
per-file try/except previously coerced every unexpected exception
to SubmitOutcome.SFTP_FAILED — misleading because a RuntimeError
from cycl_store.add has nothing to do with SFTP.
Changes:
* submission/result.py — add UNEXPECTED_ERROR = 'unexpected_error'
enum value. Docstring distinguishes it from the typed SFTP_FAILED.
* api_routers/submission.py — exception handler now returns
UNEXPECTED_ERROR (was SFTP_FAILED).
* test_api_submit_batch.py::test_submit_batch_unexpected_exception_recorded_in_results
— updated assertion (was sftp_failed, now unexpected_error).
* test_unexpected_error_outcome.py (new, 4 tests):
- enum value exists
- distinct from typed failure values
- API surfaces 'unexpected_error' for uncaught exceptions
- typed SFTP_FAILED path still surfaces 'sftp_failed' (regression lock)
Tests: 40/40 pass in 1.41s (full SP37 chain + new file).
Live curl: POST /api/submit-batch in stub mode → HTTP 409 (unchanged
behavior, no regression on the config-level guards).
Followup #2 from the SP37 final-state tracker. Replaces the
subprocess.run-based CLI help test with click.testing.CliRunner
(matches the pattern already used in test_cli.py).
Speedup: CliRunner doesn't spawn a subprocess, so the test runs in
~0.5s vs ~0.2s+ for subprocess (and the subprocess version was
susceptible to fork overhead, environment leakage between tests,
and PATH/CWD surprises on different hosts).
Same assertions:
* result.exit_code == 0 (matches subprocess.returncode == 0)
* '--ingest-dir' in result.output (matches stdout check)
Test passes in isolation and as part of the full SP37 chain
(36 tests in 1.29s — same speedup as the single test).
Followup #1 from the SP37 final-state tracker. The pre-existing
('POST', '/api/resubmit'): WRITE_ROLES entry only matched the exact
path /api/resubmit (no children — the prefix-match logic requires
the trailing slash for sub-paths). No route is registered at that
path; the actual resubmit lives at /api/inbox/rejected/resubmit
(which has its own entry).
Removing the dead entry is fail-closed: any future request to
/api/resubmit (or its descendants) returns None from allowed_roles,
which means deny. Without this cleanup, the matrix silently granted
WRITE_ROLES to a non-existent route.
Tests (test_auth_permissions_matrix.py, 5 cases):
* Exact path /api/resubmit not in PERMISSIONS
* allowed_roles('POST', '/api/resubmit') is None
* /api/resubmit/, /api/resubmit/anything all deny
Pin the invariant so a future contributor can't re-add a dead prefix.
TDD: tests written first, watched fail (5 fail with the dead entry),
removed the line, tests pass (5 pass). Full SP37 chain (36 tests)
passes with the change.
Code review of f1dc06e flagged 3 consistency outliers vs every other
gated router in api_routers/:
- Drop prefix='/api' from APIRouter. Every sibling (clearhouse,
parse, claims, batches, inbox, acks, ta1_acks, claim_acks,
remittances, providers, reconciliation, activity, eligibility,
dashboard, admin, config, payers, acks) declares the full path
in the decorator instead. Submission was the only one with
prefix='/api' + '/submit-batch'.
- Rename handler post_submit_batch -> submit_batch. Siblings use
verb_noun (submit_to_clearhouse, export_batch_837, etc.). The
'post_' prefix was unique to this file.
- Move 'from cyclone.store import store as cycl_store' from inside
the handler to the module top. Sibling routers all import store
at module top — the lazy-import rationale in the inline comment
was wrong; pulling store at module import time is fine (the rest
of the package already does it).
Tests: tests/test_api_submit_batch.py + tests/test_submission.py +
tests/test_batch_txn_set_cn.py + tests/test_batch_envelope_index_sp37.py
= 26 passed in 1.37s (no behavior change, just naming).
Thin HTTP wrapper around `cyclone.submission.submit_file` — the
HTTP twin of the `cyclone submit-batch` CLI (SP37 Task 5). Same
walker pattern, same `._*` AppleDouble skip, same `limit`
semantics, same per-file outcomes; both surfaces now agree on DB +
SFTP state byte-for-byte.
Body: `{ingest_dir, validate?, actor?, limit?}`. Response:
`{submitted, skipped, failed, results:[{file, outcome, batch_id,
error}]}`. Status codes:
- 200 on completed runs (per-file failures live in the body)
- 401 unauthenticated (matrix_gate)
- 404 no clearhouse seeded
- 409 clearhouse SFTP block in stub mode
- 422 ingest_dir missing on disk or Pydantic body error
Does NOT inject `sftp_client_factory` — submit_file uses its
default paramiko factory so SKIPPED is reachable in production.
Tests monkey-patch submit_file itself.
Permissions entry: `POST /api/submit-batch` → WRITE_ROLES (admin +
user), matching `POST /api/resubmit` posture.
Tests (11): happy path, AppleDouble skip, limit, empty ingest_dir
422, mixed-outcome 200, unexpected-exception-as-failed,
Pydantic-missing-422, on-disk-missing-422, no-clearhouse-404,
stub-mode-409, auth-gate-401.
Walks batch-*-claims/*.x12 under --ingest-dir, calls submit_file per
file, prints submitted/skipped/failed counts. Exits 0 even on
per-file failures (details in stdout); exits 2 on config-level
errors (no clearhouse, stub mode, missing dir).
This is the canonical outbound path; resubmit-rejected-claims
remains for one-off cases where no DB row is wanted.
SftpClient wrapper lacks a stat() method, so the previous default
factory produced a client whose stat() raised AttributeError — making
the SKIPPED outcome unreachable in production and silently
misclassifying all uploads as SFTP_FAILED. The helper now opens a
paramiko SFTP session directly (same pattern as
resubmit-rejected-claims in cli.py:620-650), so stat() and write_file()
both work end-to-end.
Also: tighten typing (SftpBlock instead of Any), add PAYER_MISMATCH
test, drop duplicate _db fixture, rename misleading test, add class
docstrings to SubmitResult/SubmitOutcome.
Owns the parse → DB write → SFTP upload sequence in one place. CLI
and HTTP thin-call it. DB-first invariant: if add_record raises, no
SFTP call is made. Idempotency: add_record dedupes via s.get(Claim,
claim_id); SFTP layer dedupes via stat().st_size match.
- test_acks.py:test_migration_latest_idempotent_on_fresh_db was still
asserting user_version == 19; migration 0020 (Task 1) bumped it to 20.
Without this fix, the full pytest suite fails the moment SP37 ships.
- claim_acks.py module docstring + store/__init__.py facade docstring
still advertised the index as single-key {envelope.control_number:
batch.id}. SP37 Task 3 made it dual-key (ISA13 + ST02); the docs now
match the implementation.
Each 837p batch row contributes up to two entries to the join-key
index (envelope.control_number + transaction_set_control_number). One
idx.get(set_control_number) call resolves either, so 999 acks whose
AK201 echoes the source 837's ST02 now hit Pass 1 where they
previously fell through to Pass 2 (and to the orphan log).
Backward compat preserved: rows with transaction_set_control_number
NULL (pre-SP37 batches) still resolve by ISA13.
The parser now captures the source 837's ST02 (transaction set control
number) on the Envelope model as 'transaction_set_control_number'.
add_record reads it off the envelope and stores it on the Batch row,
mirroring how 'envelope.control_number' (ISA13) was already handled.
Unlocks the SP37 join-key update so 999 set_control_number (AK201)
values resolve back to the right batch via Pass 1.
Migration 0020 adds the column additively (nullable, no default) and
backfills from raw_result_json.envelope.transaction_set_control_number
for any existing batch rows that already carry it. Required for the
SP37 join-key update so 999 acks can resolve by ST02 (the source 837's
transaction set control number) instead of just ISA13.
Task 15.
api_routers/claims.py (new, 530 lines):
- GET /api/claims (paginated list + NDJSON)
- GET /api/claims/stream (NDJSON live-tail on claim_written)
- GET /api/claims/{claim_id} (drawer detail + SP28 ack_links)
- GET /api/claims/{claim_id}/serialize-837 (regenerate X12 837P)
- GET /api/claims/{claim_id}/line-reconciliation (837 vs 835 side-by-side)
- 3 single-router helpers stay in-file per D4:
_compact_ack_links_for_claim, _claim_line_dict, _svc_to_dict
- All inline imports inside handlers preserved verbatim
(select, LineReconciliation/ServiceLinePayment/CasAdjustment,
json as _json, Decimal)
Slicing: 1 contiguous cut (drop 1-indexed 1274-1688 = the empty
section divider + the entire claims block). After the cut, the
next thing is the app-level auth_router import at api.py:1274+.
api.py: 1720 -> 1305 LOC (-415; 5 routes + 3 helpers extracted).
api_routers/__init__.py: registry extended (alphabetical) with
claims. 17 routers in registry.
Added 1 backward-compat shim for test_api_stream_live.py:
from cyclone.api_routers.claims import claims_stream
The test does 'from cyclone.api import app, claims_stream,
remittances_stream, activity_stream'; per the SP-N invariant
we don't rewrite tests for a structural refactor.
Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors.
Live-tested via curl on the running container:
GET /api/claims -> 401
GET /api/claims/stream -> 401
GET /api/claims/<id> -> 401
GET /api/claims/<id>/serialize-837 -> 401
GET /api/claims/<id>/line-reconciliation-> 401
POST /api/claims -> 405
pr-reviewer: skipped (Tasks 13-16 batch — folded into Task 17).
Task 14.
api_routers/batches.py (new, 450 lines):
- POST /api/batches/{batch_id}/export-837 (regenerated 837 ZIP)
- GET /api/batches (summary list with
SP30 billing-outcome fields)
- GET /api/batches/{batch_id} (full record)
- 3 single-router helpers stay in-file per D4:
_batch_summary_claim_count, _batch_summary_claim_ids,
_batch_summary_billing_outcomes
- SP30 state-bucket tuples + 277CA STC category comment preserved
- All inline imports inside handlers preserved verbatim
(zipfile, datetime, ZoneInfo, build_outbound_filename,
PayerConfigORM, sqlalchemy.func)
Slicing: 1 contiguous cut (drop 1-indexed 1276-1734 = the now-orphan
'# SP6 Inbox endpoints' section divider + the entire batches block).
After the cut, the next route is /api/claims at api.py:1278 with
the standard 3-blank separator.
Import fix: BatchRecord lives in cyclone.store, not cyclone.db.
Initial draft imported it from cyclone.db which crashed the
import; corrected to 'from cyclone.store import BatchRecord, store'.
api.py: 2179 -> 1720 LOC (-459; 3 routes + 3 helpers extracted).
api_routers/__init__.py: registry extended (alphabetical) with
batches. 16 routers in registry.
Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors.
Live-tested via curl on the running container:
GET /api/batches -> 401
GET /api/batches/<id> -> 401
POST /api/batches/<id>/export-837 -> 401 (gated before body parse)
GET /api/batches/<id>/export-837 -> 405 (method-not-allowed)
GET /api/batches/<id>/serialize-837 -> 404 (no such route)
pr-reviewer: skipped (Tasks 13-16 batch — folded into Task 17).
Task 13.
api_routers/inbox.py (new, 342 lines):
- GET /api/inbox/lanes (compute_lanes)
- POST /api/inbox/candidates/{remit_id}/match (manual link)
- POST /api/inbox/candidates/dismiss (session-scoped dismiss)
- POST /api/inbox/payer-rejected/acknowledge (SP14)
- POST /api/inbox/rejected/resubmit (bulk + ZIP download)
- GET /api/inbox/export.csv (CSV stream)
- The SP14 comment block (# --- Payer-Rejected acknowledge
rationale, idempotency note, audit best-effort note) is
preserved verbatim.
Slicing note: the 6 inbox routes are non-contiguous in api.py —
5 of them are in api.py:1280-1524 and the 6th (export.csv) is
in api.py:1734-1776, with /api/batches/{batch_id}/export-837
sandwiched in between. Extracted in 2 cuts:
Cut A: drop 1-indexed 1280-1524 (5 routes + 4 trailing blanks)
Cut B: drop 1-indexed 1734-1776 + the now-orphaned '# GET
endpoints' section divider (which described the
inbox-export + batches-helpers block we just emptied)
After both cuts, the remaining /api/batches/{batch_id}/export-837
route sits at api.py:1280+ in the new file, and the
_batch_summary_* helpers follow as before. api.py: 2470 -> 2179
LOC (-291; 6 routes extracted).
api_routers/__init__.py: registry extended (alphabetical) with
inbox. 15 routers in registry.
Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors. (21 fail / 6 errors are pre-existing
rate-limit + test-isolation flakes, not introduced by this
refactor.)
Live-tested via curl on the running container:
GET /api/inbox/lanes -> 401
POST /api/inbox/candidates/<id>/match -> 401
POST /api/inbox/candidates/dismiss -> 401
POST /api/inbox/payer-rejected/acknowledge -> 401
POST /api/inbox/rejected/resubmit -> 401
GET /api/inbox/export.csv -> 401
GET /api/inbox/export.csv?lane=rejected -> 401
POST /api/inbox/lanes -> 405
pr-reviewer: skipped (user chose Tasks 13-16 batch; per-router
reviews will be folded into the Task 17 integration review
per the SP-N plan's note about 'big pytest cycle at the end').
Tasks 11 + 12 combined per user direction (one commit for the
non-streaming singleton block).
api_routers/clearhouse.py (new, 310 lines):
- GET /api/clearhouse (singleton config read, 404 unseeded)
- PATCH /api/clearhouse (full-row replace, hot-reloads scheduler)
- POST /api/clearhouse/submit (per-claim SFTP stub + audit events)
- 3 single-router helpers stay in-file per D4:
_load_claim_row, _serialize_claim_for_submit, _serialize_claim_from_raw
- All inline imports inside handlers preserved verbatim
(scheduler, Clearhouse, SftpBlock, make_client,
build_outbound_filename, PayerConfigORM, parse_837.parse).
api_routers/providers.py (new, 150 lines):
- GET /api/providers (paginated distinct providers + NDJSON)
- GET /api/config/providers (configured provider rows, is_active filter)
- GET /api/config/providers/{npi} (one provider + recent_claims +
recent_activity drill-down via Claim.id outer-join Remittance.claim_id)
- Three URL prefixes, one router per spec.
api_routers/_shared.py:
- Early promotion: _actor_user_id moved here from api.py.
The clearhouse router needs it; leaving it in api.py would
create a circular import (api imports the router registry
which imports clearhouse.py, which would import api for
_actor_user_id). Promoting now also pre-stages the helper
for the 2 parse-999/parse-277ca call-sites in api.py that
Task 16 (parse router) will extract. The function body is
verbatim — only the location moved. Docstring documents
the early-promotion rationale.
api.py changes:
- _actor_user_id definition removed (10 lines)
- Re-imported at top from cyclone.api_routers._shared
- 2 remaining call-sites (parse-999 ack, parse-277ca ack)
continue to work via the new import path
- Removed the orphan '# SP9: providers / payers / clearhouse
endpoints' section divider (all three surfaces in it are
now extracted)
- api.py: 2858 -> 2468 LOC (-390; 6 routes extracted)
api_routers/__init__.py: registry extended (alphabetical) with
clearhouse + providers. 14 routers in registry.
Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors. (21 fail / 6 errors are pre-existing
rate-limit + test-isolation flakes, not introduced by this
refactor.)
Live-tested via curl on the running container:
GET /api/clearhouse -> 401 (matrix_gate fires)
POST /api/clearhouse/submit -> 401 (gated before body parse)
GET /api/providers -> 401
GET /api/config/providers -> 401
GET /api/config/providers/<npi> -> 401
POST /api/providers -> 405 (method-not-allowed)
POST /api/config/providers -> 405 (method-not-allowed)
pr-reviewer: PASS
Tasks 9 + 10 combined per user direction (one commit for the streaming-NDJSON batch).
api_routers/remittances.py (new, 169 lines):
- GET /api/remittances (paginated list + NDJSON variant)
- GET /api/remittances/summary (server-aggregated KPI tiles)
- GET /api/remittances/stream (NDJSON live-tail on remittance_written)
- GET /api/remittances/{remittance_id} (single remittance + labeled CAS)
api_routers/activity.py (new, 102 lines):
- GET /api/activity (paginated event list + NDJSON variant)
- GET /api/activity/stream (NDJSON live-tail on activity_recorded)
Both routers use router-level matrix_gate (single source of auth).
api.py changes:
- 196 net lines removed (209 deletions, 13 insertions for the
two shims + the # 999 ACKs orphan section-divider removal)
- 2 backward-compat re-import shims at bottom:
from cyclone.api_routers.remittances import remittances_stream
from cyclone.api_routers.activity import activity_stream
rationale: tests/test_api_stream_live.py imports these by name
from cyclone.api; per SP-N invariant we don't rewrite tests for
a structural refactor. (Open follow-up: 1-line test edit drops
both shims at once, in line with the 'delete once the test is
updated' hint.)
- Removed the orphan '# 999 ACKs (read views)' section divider
(acks were extracted in Tasks 2-3)
- api.py: 3041 -> 2844 LOC (-197)
api_routers/__init__.py: registry extended (alphabetical) with
remittances + activity. 12 routers in registry.
Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors. (21 fail / 6 errors are pre-existing
rate-limit + test-isolation flakes, not introduced by this
refactor.)
Live-tested via curl on the running container:
GET /api/remittances -> 401 (matrix_gate fires)
GET /api/remittances/summary -> 401
GET /api/remittances/stream -> 401
GET /api/remittances/<id> -> 401
POST /api/remittances -> 405 (method-not-allowed)
GET /api/activity -> 401
GET /api/activity/stream -> 401
POST /api/activity -> 405
pr-reviewer: PASS
Move GET /api/reconciliation/unmatched, GET /api/batch-diff, POST /api/reconciliation/match, and POST /api/reconciliation/unmatch from api.py to api_routers/reconciliation.py. Read views + manual match/unmatch write paths via store.manual_match / store.manual_unmatch. The Side-by-side batch-diff divider is preserved between routes 1 and 2; the lazy imports of cyclone.batch_diff.diff_batches_to_wire and cyclone.store.NotMatchedError inside their handlers are preserved verbatim. Orphan imports AlreadyMatchedError and InvalidStateError removed from api.py.
Move GET /api/config/payers and GET /api/config/payers/{payer_id}/configs from api.py to api_routers/config.py. Both endpoints are read-only configuration surfaces used by the UI's 'Edit payers' page. Behaviour-preserving.
Move GET /api/payers/{payer_id}/summary from api.py to api_routers/payers.py, plus the in-process memo (constants _SUMMARY_TTL_S, _summary_cache and helper _clear_summary_cache). Behaviour-preserving — endpoint URL, params, response shape, 404-on-missing behavior, and the 60s cache TTL are unchanged.
Add a one-line backward-compat shim at the bottom of api.py that re-imports _clear_summary_cache from the new home. This keeps test_payer_summary.py working (its fixture calls api_mod._clear_summary_cache() between requests to wipe the cache) without modifying the test — SP36 explicitly forbids test edits for a structural refactor.
Move POST /api/eligibility/request and POST /api/eligibility/parse-271 from api.py to api_routers/eligibility.py, plus the _validate_eligibility_request single-router helper. Behaviour-preserving — endpoint URLs, params, status codes, response shapes, and 400/422/500 error envelopes are unchanged. The auth gate moves from per-route to router-level (semantically equivalent).
Move GET /api/dashboard/kpis from api.py to api_routers/dashboard.py. Single-route router; gate via matrix_gate at the router level. Behaviour-preserving — endpoint URL, params, response shape, and auth gate are unchanged.
Moves the entire /api/admin/* namespace (audit-log ×2, db/rotate-key ×1,
backup ×10, scheduler ×6, reload-config ×1) out of api.py and into the
existing api_routers/admin.py, which already owned /api/admin/validate-provider.
- api.py: 4294 → 3547 LOC (Δ -747). Removed the orphaned # --- separator
left behind by the cut, the orphaned 'return {ok,loaded,errors}'
tail of reload-config, and the now-unused cyclone.clearhouse.InboundFile
top-level import.
- api_routers/admin.py: 60 → 851 LOC. Added the imports the moved
routes need (json, logging, threading, time, AuditEvent, verify_chain,
InboundFile) and stripped the redundant top-level imports of
symbols that the route bodies reach via the inline _db_crypto /
_secrets / _backup_svc_mod / _backup_sched_mod / _scheduler_mod
/ _audit aliases (those aliases stay — tests rely on being able to
monkeypatch cyclone.api_routers.admin._X.method). Hoisted the inline
'import time' from inside pull_inbound to module scope. Dropped
the redundant 'import threading as _threading' alias — top-level
'import threading' already covers it.
- tests/test_api_rotate_key.py: 4 monkeypatch targets migrated from
cyclone.api._db_crypto / cyclone.api._secrets to
cyclone.api_routers.admin._db_crypto / cyclone.api_routers.admin._secrets
to match the new home of the rotate-key endpoint. Lock acquire/release
also migrated.
The non-admin /api/config/* and /api/payers/{id}/summary routes that
bracketed the moved admin blocks stay in api.py — they're extracted
in Tasks 5 & 6.
Behaviour-preserving: pytest = 20 failed / 1253 passed / 10 skipped
= baseline match. Admin-only subset (audit-log + backup + scheduler +
rotate-key + reload-config) = 34 passed.
Moves GET /api/277ca-acks and GET /api/277ca-acks/{ack_id}
out of api.py (formerly lines 1278-1318) into the existing
api_routers/acks.py, which already handles 999 ACK list/detail/
stream. 277CA ACKs share the same shape-of-life contract (persisted
by a parse endpoint, listed newest-first, detail returns raw_json)
so the consolidation lands in the router that already owns the
adjacent surface — no new router file.
While here:
- collapsed the inline batch-fetch in list_277ca_acks_endpoint to
the existing _find_linked_claim_ids_for_acks(ack_ids, kind='277ca')
helper that the 999 list endpoint already uses (and ta1_acks.py
uses too). Eliminates the copy-pasted SQL; same N+1-avoidance
one-SELECT contract.
- pruned ClaimAck / to_ui_two77ca_ack imports from api.py.
Behaviour-preserving: pytest post-cut = 20 failed / 1253 passed /
10 skipped = baseline match. pytest -k '277ca or ack' = 235 passed,
1 skipped.
Moves the per-router auth gate (Depends(matrix_gate)) from the
include_router() call sites in api.py onto each router's own
APIRouter(dependencies=...) declaration. Each router now owns
its own auth dependency.
api.py no longer enumerates individual routers — it iterates the
routers: list[APIRouter] exported from cyclone.api_routers. This
is the registry that 13 future router extractions will append to.
- new: backend/src/cyclone/api_routers/__init__.py (registry)
- new: backend/src/cyclone/api_routers/_shared.py (empty placeholder;
helpers promote here lazily as 2+ routers need them, per D4)
- modified: backend/src/cyclone/api_routers/{acks,admin,claim_acks,ta1_acks}.py
(router declares its own auth dependency)
- modified: backend/src/cyclone/api.py (5-line include_router loop
replaces 5 explicit include_router calls; auth-router includes at
lines 4334-4338 untouched)
Behaviour-preserving: pytest post-cut = 20 failed / 1253 passed /
10 skipped = baseline match (the 20 are pre-existing live-DB
pollution unrelated to SP36). Health stays public (no matrix_gate).
The 999, 277CA, and TA1 parsers already enforce envelope correctness at
the parser level (parse_999.py line 290 raises 'No AK9 segment found';
parse_277ca.py line 298 raises 'Expected ST*277 or ST*277CA'; parse_ta1.py
line 111 raises 'Expected TA1, got <other>'). These tests lock the HTTP
surface contract: a wrong-kind file POSTed to those endpoints must come
back as 400, never as 200 or 500.
Tests added:
- test_api_999.py: rejects_837_input, rejects_835_input
- test_api_277ca.py: rejects_835_input, rejects_837_input
- test_api_ta1.py: rejects_837_input, rejects_835_input
If a future PR relaxes any of those parser-level guards, the
corresponding regression lock fires immediately.
Mirror of the parse-837 SP35 guards. Same defense-in-depth shape:
tokenize first, reject anything whose ST01 doesn't start with '835'
(400 'Mismatched file kind'), and after parse refuse to persist a
batch with zero CLP segments (400 'No claims parsed').
Reuses the _transaction_set_id_from_segments helper added by the
parse-837 commit.
New tests in tests/test_api_835.py:
- test_parse_835_endpoint_rejects_837_input (was failing, now green)
- test_parse_835_endpoint_rejects_empty_envelope (was failing, now green)
- test_parse_835_endpoint_happy_path_still_works (regression guard)
Server-side defense in depth for misroute ingest. Before SP35, posting
an 835 file (or any other X12 with a parseable ISA envelope) to
/api/parse-837 silently produced a BatchRecord with claims=[] and a
bogus row on the History tab. The 837 parser only required an ISA
envelope; it didn't check the ST transaction-set id.
Two new guards run before persistence:
1. Envelope check: tokenize first, read ST01, reject anything that
doesn't start with '837'. 400 with error='Mismatched file kind',
expected='837p', detected_st=<actual>. Catches an 835/999/270/etc
routed to the wrong endpoint.
2. Empty-claims check: even with the right envelope, if the parser
produces zero CLM segments, return 400 'No claims parsed' and do
NOT persist.
New tests in tests/test_api.py:
- test_parse_837_endpoint_rejects_835_input (was failing, now green)
- test_parse_837_endpoint_rejects_empty_envelope (was failing, now green)
- test_parse_837_endpoint_happy_path_still_works (regression guard)
Helper _transaction_set_id_from_segments reused by the 835 mirror.
Finishes the in-tree was_skipped draft: size-matched remote files no
longer emit clearhouse.submitted events or increment uploaded, and the
periodic reconnect now triggers only after real uploads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- payer-mismatch files are now skipped instead of uploaded
- SFTP session persists across files (was reopened and leaked per file)
- --reconnect-every now counts successful uploads, after increment
- drop unused SftpClient instantiation; read each file once
- apply_999_rejections docstring: R/E/X, not R/E
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Adds `cyclone resubmit-rejected-claims` to push corrected single-claim
837 files to the Gainwell ToHPE SFTP dir. Idempotent (stat-then-skip by
byte size). One persistent paramiko session per batch with
reconnect-every=50 to dodge MOVEit's silent per-session file cap
(~200 puts/session, no exception).
- Validates each file via `parse_837` before upload and rejects any
whose payer_id is not `CO_TXIX` (catches a bad byte-fix early).
- Refreshes test fixtures (`minimal_837p.txt`, `co_medicaid_837p.txt`,
`co_medicaid_837p_with_renderer.txt`) and the corresponding test
assertions (`test_payer.py`, `test_payer_summary.py`,
`test_co_medicaid_fixture.py`, `test_parse_837.py`) from the old
`SKCO0`/`COHCPF` payer IDs to `CO_TXIX`, matching
PayerConfig.co_medicaid() and the HCPF 837P Companion Guide.
- Adds `ingest/` to .gitignore — local scratch / production-data
staging only.
Add the 'cyclone backfill-999-rejections' subcommand to replay the
cascade fix in apply_999_rejections for any 999 acks already in the
DB. Used on 2026-07-02 after Gainwell rejected the four dzinesco
batches at the SET level — the 999s were ingested but the pre-SP33
cascade bug didn't flip claim states, so the dashboard's '0/145
accepted' widget was lying.
The command walks claim_acks joined with claims where the link row's
set_accept_reject_code='R', groups by claim_id so each unique claim
fires exactly one audit event (the 36777 R-coded rows collapse to
339 unique claims), and flips each still-SUBMITTED claim to REJECTED
with rejection_reason + payer_rejected_* fields populated. Claims
already in REJECTED are skipped.
Also moves 'if __name__ == "__main__": main()' to the bottom of the
file. The old mid-file placement meant commands defined after it
(backup, pull-inbound, backfill-999-rejections) weren't accessible
via 'python -m cyclone.cli <sub>' — only the 'cyclone' console script
worked. Latent bug since SP17; surfaced when SP33 added another
post-block command.
The 999 handler's rejection pass was looking up claims by patient
control number, but Gainwell rejects at the SET level (ST envelope)
when the whole batch fails the NM109=CO_TXIX rule. That meant a SET
rejection was treated as a no-op even though every claim in the SET
was actually rejected by the payer.
Add a batch_envelope_index param (mirrors apply_999_acceptances from
SP28) so SET-level rejections cascade to every claim in the SET.
Falls back to the legacy PCN lookup when the index has no entry.
Also tightens test_payer.py: PayerConfig.co_medicaid() now returns
payer_id='CO_TXIX' and payer_name='CO_TXIX' per HCPF 837P Companion
Guide (June 2025 - Version 2.5).
Nora
Cyclone SP37
sp37 bot