Captures the historical 804-orphan 999 situation as a known
artifact (RUNBOOK entry + status command) and provides a
one-shot idempotent synthetic-batch seeder so future acks for
the orphan ST02s can resolve against batch_envelope_index.
6 commits merged atomically (no squash — the SP-N merge commit
is the audit record):
d776a4a docs(spec): design for SP38 orphan-ack housekeeping
8890627 docs(plan): SP38 orphan-ack housekeeping plan
ad14b56 feat(sp38): CycloneStore.find_ack_orphan_st02_summary + reconcile_orphan_st02s
9ef749c feat(sp38): 'cyclone ack-orphans {status,reconcile}' CLI subcommands
76923a7 docs(sp38): RUNBOOK entry for orphan-ack housekeeping
07ea7ca fix(sp38): restore per-kind control_number in find_ack_orphans + review cleanups
Test delta: +36 passed (12 store helper + 7 CLI + 17 claim_acks
regression including the pr-reviewer 277ca/ta1 control_number
fix). No new failures.
Live verified: cyclone ack-orphans status exits 0 with 4 distinct
orphan ST02s (419+226+106+53=804) matching the SQLite investigation.
Three pr-reviewer followups from the 2026-07-07 review of commit ad14b56:
1. BUG: find_ack_orphans refactor routed 277ca/ta1 through
_ack_control_number which only knew 999 — restored per-kind source
(999 reads raw_json.envelope.control_number, 277ca/ta1 read the ORM
control_number column). Added regression test pinning all three
kinds.
2. DOCSTRING DRIFT: reconcile_orphan_st02s said 'remaining columns
take their defaults' but explicitly passes parsed_at and
transaction_set_control_number — added both to the explicit list
and clarified the rest take schema defaults.
3. WASTED SORT: _iter_orphan_999_st02s yielded sorted() but the
caller re-sorts by (-ack_count, st02) — yielding unsorted now.
Plus three cleanups the reviewer flagged:
* Hoisted 'json' / 'uuid' / 'datetime' imports to module top of
store/__init__.py (replaced in-method imports).
* Added two missing tests: sentinel grep-discoverability via
LIKE '<synthetic:%>' + 999-walk tolerance for non-dict raw_json
(None / list shapes — bytes is unreachable through the ORM).
* Aligned spec/plan exit codes to the cyclone-cli convention
(exit 1 on DB error, not 2 — matches the existing CLI
sys.exit(1) and the cyclone-cli skill documentation).
36/36 SP38 tests pass.
Implements task 6 of
docs/superpowers/plans/2026-07-07-cyclone-orphan-ack-housekeeping.md.
Adds 'Known historical drift — the 804 orphan 999s' section under
the existing operator-triage content. Covers:
- What the orphans are (real production 999s whose source 837s
predate the current DB snapshot; valid audit history that
cannot be auto-linked)
- Why they cannot be auto-linked (source 837s were transmitted
to HPE and never came back; Cyclone is downstream and does
not retain copies of outbound 837s)
- Triage path via the Inbox AckOrphansLane (already working
as of SP37-followup 893a662)
- Optional synthetic-batch seeding via 'cyclone ack-orphans
reconcile' (one-shot, idempotent, --dry-run available)
- 'cyclone ack-orphans status' for the per-ST02 breakdown
Explicitly calls out what the housekeeping is NOT:
- Not a backfill (no claims rows are synthesized)
- Not auto-runnable (operator-invoked only)
- Not a deletion (orphans are valid audit history)
References the SP38 spec and plan for the design rationale.
Implements tasks 4-5 of
docs/superpowers/plans/2026-07-07-cyclone-orphan-ack-housekeeping.md.
Adds a new 'ack-orphans' group under the main click CLI with two
subcommands:
cyclone ack-orphans status
Prints a table of orphan 999 acks:
ST02 ACK COUNT HAS BATCH
--------------- ---------- ---------
991102989 419 no
991102988 226 no
...
TOTAL 804
Empty-DB case prints 'no orphans' instead of a blank table.
cyclone ack-orphans reconcile [--dry-run]
One-shot synthetic-batch seeder. --dry-run prints the plan
without writing. Real call inserts one synthetic batches row
per orphan ST02 lacking one, marked with the
'<synthetic:orphan-reconcile>' sentinel so they're trivially
distinguishable in queries.
Exit codes per cyclone-cli convention: 0 = success, 1 = DB error.
CLI is operator-invoked only — no auto-run on boot, no scheduler
hook, no cron integration.
7 new tests (test_ack_orphans_cli.py) use click.testing.CliRunner
(matching the SP37-followup #5 pattern that replaced subprocess.run
with the in-process runner). Cover: table shape, has-batch column,
empty-DB handling, reconcile creates synthetic rows, --dry-run
doesn't write, idempotency, no-orphan graceful handling.
Live-verified against ~/.local/share/cyclone/cyclone.db: status
prints the 4 distinct ST02s (419/226/106/53 = 804 total). Reconcile
inserts 4 synthetic batches. Subsequent status shows all 4 marked
has_batch=yes.
Implements the store-helper half of the SP38 spec (tasks 1-3 of
docs/superpowers/plans/2026-07-07-cyclone-orphan-ack-housekeeping.md).
Two new CycloneStore methods + one extraction helper in claim_acks.py:
- find_ack_orphan_st02_summary() -> list[dict]
Per-ST02 breakdown of orphan 999 acks. Returns one row per
distinct (st02, ack_count, has_batch, batch_id). Sorted by
ack_count DESC so the heaviest backlog surfaces first in CLI
output. 999-only — 277ca/ta1 orphans have different ST02
semantics (interchange control number vs source 837 ST02)
and aggregating them would conflate unrelated identifiers.
- reconcile_orphan_st02s(*, dry_run=False) -> dict
One-shot synthetic-batch seeder. For every orphan ST02 without
a batches row, insert one with kind='837p',
input_filename='<synthetic:orphan-reconcile>' (the sentinel),
transaction_set_control_number=<st02>, and totals_json +
validation_json documenting the orphan-reconcile provenance.
Idempotent: re-running after a successful pass is a no-op.
dry_run=True returns the plan shape without writing.
- _iter_orphan_999_st02s() -> Iterator[(st02, count)]
Extracted walk helper. The existing find_ack_orphans('999')
refactored to consume it for the link check; the new summary
uses it for the per-ST02 aggregation. Accepts both ORM dicts
and raw sqlite3 strings for raw_json (the SQLAlchemy JSON
type hands back Python dicts, but tests can hand strings).
11 new tests (test_ack_orphan_summary.py) cover:
- per-ST02 counts with duplicates
- has_batch=True when a batches row covers the ST02
- sort order (heaviest first)
- exclusion of 999s with a claim_acks link
- skip of malformed/empty raw_json
- empty-DB case
- reconcile creates synthetic rows for missing ST02s only
- reconcile skips ST02s with existing batches
- idempotency
- --dry-run flag
- ack_count preserved in totals_json
Live-verified against ~/.local/share/cyclone/cyclone.db: 4 distinct
orphan ST02s, 804 total orphan rows. After reconcile, all 4 marked
has_batch=yes.
Adds docs/superpowers/specs/2026-07-07-cyclone-orphan-ack-housekeeping-design.md.
The current cyclone.db holds 805 ack rows, 804 of which are unresolved
orphans — 999s whose source 837 batches pre-date the current DB
snapshot and were never re-ingested. Investigation on 2026-07-07
confirmed the source 837s are not recoverable (Clearinghouse does not
echo them back; only the 999s were preserved in ingest/ + SFTP
staging). SP37's canonical submit-batch flow already captures ST02
going forward, so the orphan count stays flat — not a forward-looking
bug, just historical drift.
This SP captures that situation and adds housekeeping around it:
- RUNBOOK.md entry under 'Known historical drift' explaining the
root cause and the operator's triage path (Inbox AckOrphansLane,
which now works as of SP37-followup 893a662).
- 'cyclone ack-orphans status' CLI: distinct orphan ST02s + ack
count per ST02 + total.
- 'cyclone ack-orphans reconcile' CLI: one-shot, idempotent
synthetic-batch seeder. Marks synthetic rows with
input_filename '<synthetic:orphan-reconcile>' so they're
distinguishable in queries and the codebase can grep the
sentinel.
No UI change, no auto-runnable reconcile, no schema migration, no
attempt to backfill claim rows for the orphan ST02s (the source
data is gone). Per the canonical SP-N spec template, header is
'Draft, awaiting user sign-off' until the plan is signed off.
Branch: sp38-orphan-ack-housekeeping (off main, ahead by 893a662).
Next step: plan at docs/superpowers/plans/2026-07-07-cyclone-orphan-ack-housekeeping.md.
Audit on 2026-07-07 found three production bugs and six stale matrix
entries:
Three registered routes were DENIED in production (fail-closed):
- DELETE /api/acks/{kind}/{ack_id}/match-claim/{claim_id}
(Inbox unlink-a-wrong-match action; wired from the frontend)
- GET /api/dashboard/kpis (dashboard summary cards)
- GET /api/inbox/ack-orphans (Inbox 'Ack orphans' lane; already
wired into src/hooks/useAckOrphans.ts + AckOrphansLane.tsx)
All three had working frontend callers + passing tests (TestClient
bypasses matrix_gate), so operators hitting those endpoints got 403s
in production while CI stayed green. Fixed by adding the three
missing matrix entries with the role sets their siblings use
(WRITE_ROLES for the unlink action; ALL_ROLES for both read lanes).
Six dead matrix entries from refactor drift removed:
- GET /api/dashboard/summary (renamed to /kpis)
- GET /api/reconcile (renamed to /reconciliation)
- POST /api/reconcile (renamed to /reconciliation)
- GET /api/audit-log (moved to /api/admin/audit-log)
- GET /api/admin/backup/scheduler (refactored to /start, /stop, /tick)
- GET /api/export.csv (renamed to /api/inbox/export.csv)
Plus a one-line doc fix at api_routers/claim_acks.py:295 — the
endpoint docstring claimed it 'mirrors /api/inbox/remit-orphans'
but that sibling route was never built.
Guard test: tests/test_routes_have_auth_coverage.py enumerates every
APIRoute via the live FastAPI app (recurse into include_router mounts)
and asserts each (method, path) has a non-None allowed_roles result.
Parametrised so each missing route surfaces as its own test report
entry with the exact uncovered path. 84 routes × 1 parametrised case
= 84 individual coverage checks; if a future contributor adds a new
route without a matrix entry, this test fires immediately.
The SftpClient wrapper previously exposed write_file, list_inbound,
list_inbound_names, download_inbound, and read_file — but NO stat().
The canonical cyclone.submission.submit_file helper needs to compare
the local file size against the remote file size for idempotency
(SKIPPED outcome short-circuit); the lack of stat() forced the helper
to bypass the wrapper and open paramiko directly via
submission/core.py:_default_sftp_factory.
This commit adds:
- SftpStat frozen dataclass: narrow projection of paramiko's
SFTPAttributes (size + modified_at). Callers don't need paramiko.
- SftpClient.stat(remote_path) -> SftpStat public method
- _stat_stub + _stat_paramiko backends (mirrors the existing
_read_file_stub / _read_file_paramiko pattern)
- 8 new tests covering dataclass shape, frozen invariant, stub size,
stub mtime, missing-file FileNotFoundError, large-file size, and
the actual SKIPPED-outcome idempotency check from submit_file
Also adds backend/var/ to .gitignore — contains HCPF-delivered
inbound production files that should never be committed (same
rationale as the existing ingest/ entry).
Followup #4 from the SP37 final-state tracker. The previous
BACKFILL_SQL constant in test_migration_0020.py was a hand-copied
duplicate of the migration's UPDATE statement. A future contributor
could edit one without the other and the test would silently
replay a different SQL than production — defeating the regression.
Fix: tests now load the migration file at test time and extract
its UPDATE via the same splitter db_migrate.run() uses (strip
'--' comments, split on ';'). The test can never disagree with
what production runs.
Changes:
* test_migration_0020.py:
- Remove the hand-copied BACKFILL_SQL constant
- Add _migration_0020_path(), _extract_update_statements(),
and _load_migration_0020_backfill_sql() helpers
- Replace 3 BACKFILL_SQL references with helper calls
* test_migration_0020_no_drift.py (new, 3 tests):
- test_migration_0020_backfill_sql_uses_migration_file
(asserts the extracted SQL targets the right column + path)
- test_migration_0020_backfill_sql_is_non_empty_single_statement
- test_migration_0020_has_exactly_one_update (guardrail against
future contributors adding a second UPDATE — the extraction
fails loudly so the test author can decide which is the
backfill)
Tests: 43/43 pass in 1.46s (full SP37 followup chain).
Imports across test files match the existing pattern (test_store.py
imports from test_store_reconcile.py).
Followup #3 from the SP37 final-state tracker. The router's
per-file try/except previously coerced every unexpected exception
to SubmitOutcome.SFTP_FAILED — misleading because a RuntimeError
from cycl_store.add has nothing to do with SFTP.
Changes:
* submission/result.py — add UNEXPECTED_ERROR = 'unexpected_error'
enum value. Docstring distinguishes it from the typed SFTP_FAILED.
* api_routers/submission.py — exception handler now returns
UNEXPECTED_ERROR (was SFTP_FAILED).
* test_api_submit_batch.py::test_submit_batch_unexpected_exception_recorded_in_results
— updated assertion (was sftp_failed, now unexpected_error).
* test_unexpected_error_outcome.py (new, 4 tests):
- enum value exists
- distinct from typed failure values
- API surfaces 'unexpected_error' for uncaught exceptions
- typed SFTP_FAILED path still surfaces 'sftp_failed' (regression lock)
Tests: 40/40 pass in 1.41s (full SP37 chain + new file).
Live curl: POST /api/submit-batch in stub mode → HTTP 409 (unchanged
behavior, no regression on the config-level guards).
Followup #2 from the SP37 final-state tracker. Replaces the
subprocess.run-based CLI help test with click.testing.CliRunner
(matches the pattern already used in test_cli.py).
Speedup: CliRunner doesn't spawn a subprocess, so the test runs in
~0.5s vs ~0.2s+ for subprocess (and the subprocess version was
susceptible to fork overhead, environment leakage between tests,
and PATH/CWD surprises on different hosts).
Same assertions:
* result.exit_code == 0 (matches subprocess.returncode == 0)
* '--ingest-dir' in result.output (matches stdout check)
Test passes in isolation and as part of the full SP37 chain
(36 tests in 1.29s — same speedup as the single test).
Followup #1 from the SP37 final-state tracker. The pre-existing
('POST', '/api/resubmit'): WRITE_ROLES entry only matched the exact
path /api/resubmit (no children — the prefix-match logic requires
the trailing slash for sub-paths). No route is registered at that
path; the actual resubmit lives at /api/inbox/rejected/resubmit
(which has its own entry).
Removing the dead entry is fail-closed: any future request to
/api/resubmit (or its descendants) returns None from allowed_roles,
which means deny. Without this cleanup, the matrix silently granted
WRITE_ROLES to a non-existent route.
Tests (test_auth_permissions_matrix.py, 5 cases):
* Exact path /api/resubmit not in PERMISSIONS
* allowed_roles('POST', '/api/resubmit') is None
* /api/resubmit/, /api/resubmit/anything all deny
Pin the invariant so a future contributor can't re-add a dead prefix.
TDD: tests written first, watched fail (5 fail with the dead entry),
removed the line, tests pass (5 pass). Full SP37 chain (36 tests)
passes with the change.
12-commit feature branch delivering:
* Migration 0020: add batches.transaction_set_control_number (ST02)
* parse_837 captures ST02 into Envelope.transaction_set_control_number
* add_record populates the new column via getattr chain
* batch_envelope_index now keys by ISA13 OR ST02 (setdefault)
* New cyclone.submission package with submit_file helper
(DB-first / upload-second invariant; paramiko direct factory
because SftpClient wrapper lacks stat())
* New cyclone submit-batch CLI + POST /api/submit-batch endpoint
(thin wrappers over submit_file, byte-for-byte walker parity)
* 31 new tests across 5 files (8 + 11 + 4 + 3 + 5)
* Live-verified end-to-end: 999 with AK2*837*991102977~ now
resolves to claim_id=CLM001 via the new join key (was orphan)
End-to-end orphan reduction going forward: every batch ingested via
the canonical submit path captures ST02; 999 acks referencing that
ST02 resolve via the updated batch_envelope_index.
Spec: docs/superpowers/specs/2026-07-07-cyclone-submit-batch-canonical-flow-design.md
Plan: docs/superpowers/plans/2026-07-07-cyclone-submit-batch-canonical-flow.md
Tracker: /tmp/refactor-cyclone.md
Reviewed-by: pr-reviewer subagent (PASS, all 8 checklist items)
Live-tested-by: end-to-end 999 POST against real dev DB
(/tmp/sp37-task7-live-evidence.txt)
Merge-shape: --no-ff, single atomic commit (per cyclone-spec skill)
RUNBOOK gets a 'Submitting claims (canonical)' section with both CLI
and HTTP examples, the shared cyclone.submission.submit_file helper,
and a 'submit-batch vs resubmit-rejected-claims' decision rule.
CLAUDE.md gets a pointer to cyclone/submission/ in the 'Backend at a
glance' subpackage list.
Code review of f1dc06e flagged 3 consistency outliers vs every other
gated router in api_routers/:
- Drop prefix='/api' from APIRouter. Every sibling (clearhouse,
parse, claims, batches, inbox, acks, ta1_acks, claim_acks,
remittances, providers, reconciliation, activity, eligibility,
dashboard, admin, config, payers, acks) declares the full path
in the decorator instead. Submission was the only one with
prefix='/api' + '/submit-batch'.
- Rename handler post_submit_batch -> submit_batch. Siblings use
verb_noun (submit_to_clearhouse, export_batch_837, etc.). The
'post_' prefix was unique to this file.
- Move 'from cyclone.store import store as cycl_store' from inside
the handler to the module top. Sibling routers all import store
at module top — the lazy-import rationale in the inline comment
was wrong; pulling store at module import time is fine (the rest
of the package already does it).
Tests: tests/test_api_submit_batch.py + tests/test_submission.py +
tests/test_batch_txn_set_cn.py + tests/test_batch_envelope_index_sp37.py
= 26 passed in 1.37s (no behavior change, just naming).
Thin HTTP wrapper around `cyclone.submission.submit_file` — the
HTTP twin of the `cyclone submit-batch` CLI (SP37 Task 5). Same
walker pattern, same `._*` AppleDouble skip, same `limit`
semantics, same per-file outcomes; both surfaces now agree on DB +
SFTP state byte-for-byte.
Body: `{ingest_dir, validate?, actor?, limit?}`. Response:
`{submitted, skipped, failed, results:[{file, outcome, batch_id,
error}]}`. Status codes:
- 200 on completed runs (per-file failures live in the body)
- 401 unauthenticated (matrix_gate)
- 404 no clearhouse seeded
- 409 clearhouse SFTP block in stub mode
- 422 ingest_dir missing on disk or Pydantic body error
Does NOT inject `sftp_client_factory` — submit_file uses its
default paramiko factory so SKIPPED is reachable in production.
Tests monkey-patch submit_file itself.
Permissions entry: `POST /api/submit-batch` → WRITE_ROLES (admin +
user), matching `POST /api/resubmit` posture.
Tests (11): happy path, AppleDouble skip, limit, empty ingest_dir
422, mixed-outcome 200, unexpected-exception-as-failed,
Pydantic-missing-422, on-disk-missing-422, no-clearhouse-404,
stub-mode-409, auth-gate-401.
Walks batch-*-claims/*.x12 under --ingest-dir, calls submit_file per
file, prints submitted/skipped/failed counts. Exits 0 even on
per-file failures (details in stdout); exits 2 on config-level
errors (no clearhouse, stub mode, missing dir).
This is the canonical outbound path; resubmit-rejected-claims
remains for one-off cases where no DB row is wanted.
SftpClient wrapper lacks a stat() method, so the previous default
factory produced a client whose stat() raised AttributeError — making
the SKIPPED outcome unreachable in production and silently
misclassifying all uploads as SFTP_FAILED. The helper now opens a
paramiko SFTP session directly (same pattern as
resubmit-rejected-claims in cli.py:620-650), so stat() and write_file()
both work end-to-end.
Also: tighten typing (SftpBlock instead of Any), add PAYER_MISMATCH
test, drop duplicate _db fixture, rename misleading test, add class
docstrings to SubmitResult/SubmitOutcome.
Owns the parse → DB write → SFTP upload sequence in one place. CLI
and HTTP thin-call it. DB-first invariant: if add_record raises, no
SFTP call is made. Idempotency: add_record dedupes via s.get(Claim,
claim_id); SFTP layer dedupes via stat().st_size match.
- test_acks.py:test_migration_latest_idempotent_on_fresh_db was still
asserting user_version == 19; migration 0020 (Task 1) bumped it to 20.
Without this fix, the full pytest suite fails the moment SP37 ships.
- claim_acks.py module docstring + store/__init__.py facade docstring
still advertised the index as single-key {envelope.control_number:
batch.id}. SP37 Task 3 made it dual-key (ISA13 + ST02); the docs now
match the implementation.
Each 837p batch row contributes up to two entries to the join-key
index (envelope.control_number + transaction_set_control_number). One
idx.get(set_control_number) call resolves either, so 999 acks whose
AK201 echoes the source 837's ST02 now hit Pass 1 where they
previously fell through to Pass 2 (and to the orphan log).
Backward compat preserved: rows with transaction_set_control_number
NULL (pre-SP37 batches) still resolve by ISA13.
The parser now captures the source 837's ST02 (transaction set control
number) on the Envelope model as 'transaction_set_control_number'.
add_record reads it off the envelope and stores it on the Batch row,
mirroring how 'envelope.control_number' (ISA13) was already handled.
Unlocks the SP37 join-key update so 999 set_control_number (AK201)
values resolve back to the right batch via Pass 1.
Migration 0020 adds the column additively (nullable, no default) and
backfills from raw_result_json.envelope.transaction_set_control_number
for any existing batch rows that already carry it. Required for the
SP37 join-key update so 999 acks can resolve by ST02 (the source 837's
transaction set control number) instead of just ISA13.
8-task implementation plan covering migration, ORM update, join-key
wiring, the cyclone.submission helper, CLI, HTTP endpoint, and merge.
Each task ends with a tracker update per the operator's standing
directive (/tmp/refactor-cyclone.md).
Adds a parse → DB write → SFTP upload pipeline that closes the gap
where 837P submissions leave no DB row, making every 999 ack an
orphan. Locks the four brainstorming decisions (canonical submit flow,
DB-first ordering, new Batch.transaction_set_control_number column,
additive deprecation posture) and the architecture for one new CLI +
one new HTTP endpoint sharing a cyclone.submission helper.
Task 15.
api_routers/claims.py (new, 530 lines):
- GET /api/claims (paginated list + NDJSON)
- GET /api/claims/stream (NDJSON live-tail on claim_written)
- GET /api/claims/{claim_id} (drawer detail + SP28 ack_links)
- GET /api/claims/{claim_id}/serialize-837 (regenerate X12 837P)
- GET /api/claims/{claim_id}/line-reconciliation (837 vs 835 side-by-side)
- 3 single-router helpers stay in-file per D4:
_compact_ack_links_for_claim, _claim_line_dict, _svc_to_dict
- All inline imports inside handlers preserved verbatim
(select, LineReconciliation/ServiceLinePayment/CasAdjustment,
json as _json, Decimal)
Slicing: 1 contiguous cut (drop 1-indexed 1274-1688 = the empty
section divider + the entire claims block). After the cut, the
next thing is the app-level auth_router import at api.py:1274+.
api.py: 1720 -> 1305 LOC (-415; 5 routes + 3 helpers extracted).
api_routers/__init__.py: registry extended (alphabetical) with
claims. 17 routers in registry.
Added 1 backward-compat shim for test_api_stream_live.py:
from cyclone.api_routers.claims import claims_stream
The test does 'from cyclone.api import app, claims_stream,
remittances_stream, activity_stream'; per the SP-N invariant
we don't rewrite tests for a structural refactor.
Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors.
Live-tested via curl on the running container:
GET /api/claims -> 401
GET /api/claims/stream -> 401
GET /api/claims/<id> -> 401
GET /api/claims/<id>/serialize-837 -> 401
GET /api/claims/<id>/line-reconciliation-> 401
POST /api/claims -> 405
pr-reviewer: skipped (Tasks 13-16 batch — folded into Task 17).
Task 14.
api_routers/batches.py (new, 450 lines):
- POST /api/batches/{batch_id}/export-837 (regenerated 837 ZIP)
- GET /api/batches (summary list with
SP30 billing-outcome fields)
- GET /api/batches/{batch_id} (full record)
- 3 single-router helpers stay in-file per D4:
_batch_summary_claim_count, _batch_summary_claim_ids,
_batch_summary_billing_outcomes
- SP30 state-bucket tuples + 277CA STC category comment preserved
- All inline imports inside handlers preserved verbatim
(zipfile, datetime, ZoneInfo, build_outbound_filename,
PayerConfigORM, sqlalchemy.func)
Slicing: 1 contiguous cut (drop 1-indexed 1276-1734 = the now-orphan
'# SP6 Inbox endpoints' section divider + the entire batches block).
After the cut, the next route is /api/claims at api.py:1278 with
the standard 3-blank separator.
Import fix: BatchRecord lives in cyclone.store, not cyclone.db.
Initial draft imported it from cyclone.db which crashed the
import; corrected to 'from cyclone.store import BatchRecord, store'.
api.py: 2179 -> 1720 LOC (-459; 3 routes + 3 helpers extracted).
api_routers/__init__.py: registry extended (alphabetical) with
batches. 16 routers in registry.
Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors.
Live-tested via curl on the running container:
GET /api/batches -> 401
GET /api/batches/<id> -> 401
POST /api/batches/<id>/export-837 -> 401 (gated before body parse)
GET /api/batches/<id>/export-837 -> 405 (method-not-allowed)
GET /api/batches/<id>/serialize-837 -> 404 (no such route)
pr-reviewer: skipped (Tasks 13-16 batch — folded into Task 17).
Task 13.
api_routers/inbox.py (new, 342 lines):
- GET /api/inbox/lanes (compute_lanes)
- POST /api/inbox/candidates/{remit_id}/match (manual link)
- POST /api/inbox/candidates/dismiss (session-scoped dismiss)
- POST /api/inbox/payer-rejected/acknowledge (SP14)
- POST /api/inbox/rejected/resubmit (bulk + ZIP download)
- GET /api/inbox/export.csv (CSV stream)
- The SP14 comment block (# --- Payer-Rejected acknowledge
rationale, idempotency note, audit best-effort note) is
preserved verbatim.
Slicing note: the 6 inbox routes are non-contiguous in api.py —
5 of them are in api.py:1280-1524 and the 6th (export.csv) is
in api.py:1734-1776, with /api/batches/{batch_id}/export-837
sandwiched in between. Extracted in 2 cuts:
Cut A: drop 1-indexed 1280-1524 (5 routes + 4 trailing blanks)
Cut B: drop 1-indexed 1734-1776 + the now-orphaned '# GET
endpoints' section divider (which described the
inbox-export + batches-helpers block we just emptied)
After both cuts, the remaining /api/batches/{batch_id}/export-837
route sits at api.py:1280+ in the new file, and the
_batch_summary_* helpers follow as before. api.py: 2470 -> 2179
LOC (-291; 6 routes extracted).
api_routers/__init__.py: registry extended (alphabetical) with
inbox. 15 routers in registry.
Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors. (21 fail / 6 errors are pre-existing
rate-limit + test-isolation flakes, not introduced by this
refactor.)
Live-tested via curl on the running container:
GET /api/inbox/lanes -> 401
POST /api/inbox/candidates/<id>/match -> 401
POST /api/inbox/candidates/dismiss -> 401
POST /api/inbox/payer-rejected/acknowledge -> 401
POST /api/inbox/rejected/resubmit -> 401
GET /api/inbox/export.csv -> 401
GET /api/inbox/export.csv?lane=rejected -> 401
POST /api/inbox/lanes -> 405
pr-reviewer: skipped (user chose Tasks 13-16 batch; per-router
reviews will be folded into the Task 17 integration review
per the SP-N plan's note about 'big pytest cycle at the end').
Tasks 11 + 12 combined per user direction (one commit for the
non-streaming singleton block).
api_routers/clearhouse.py (new, 310 lines):
- GET /api/clearhouse (singleton config read, 404 unseeded)
- PATCH /api/clearhouse (full-row replace, hot-reloads scheduler)
- POST /api/clearhouse/submit (per-claim SFTP stub + audit events)
- 3 single-router helpers stay in-file per D4:
_load_claim_row, _serialize_claim_for_submit, _serialize_claim_from_raw
- All inline imports inside handlers preserved verbatim
(scheduler, Clearhouse, SftpBlock, make_client,
build_outbound_filename, PayerConfigORM, parse_837.parse).
api_routers/providers.py (new, 150 lines):
- GET /api/providers (paginated distinct providers + NDJSON)
- GET /api/config/providers (configured provider rows, is_active filter)
- GET /api/config/providers/{npi} (one provider + recent_claims +
recent_activity drill-down via Claim.id outer-join Remittance.claim_id)
- Three URL prefixes, one router per spec.
api_routers/_shared.py:
- Early promotion: _actor_user_id moved here from api.py.
The clearhouse router needs it; leaving it in api.py would
create a circular import (api imports the router registry
which imports clearhouse.py, which would import api for
_actor_user_id). Promoting now also pre-stages the helper
for the 2 parse-999/parse-277ca call-sites in api.py that
Task 16 (parse router) will extract. The function body is
verbatim — only the location moved. Docstring documents
the early-promotion rationale.
api.py changes:
- _actor_user_id definition removed (10 lines)
- Re-imported at top from cyclone.api_routers._shared
- 2 remaining call-sites (parse-999 ack, parse-277ca ack)
continue to work via the new import path
- Removed the orphan '# SP9: providers / payers / clearhouse
endpoints' section divider (all three surfaces in it are
now extracted)
- api.py: 2858 -> 2468 LOC (-390; 6 routes extracted)
api_routers/__init__.py: registry extended (alphabetical) with
clearhouse + providers. 14 routers in registry.
Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors. (21 fail / 6 errors are pre-existing
rate-limit + test-isolation flakes, not introduced by this
refactor.)
Live-tested via curl on the running container:
GET /api/clearhouse -> 401 (matrix_gate fires)
POST /api/clearhouse/submit -> 401 (gated before body parse)
GET /api/providers -> 401
GET /api/config/providers -> 401
GET /api/config/providers/<npi> -> 401
POST /api/providers -> 405 (method-not-allowed)
POST /api/config/providers -> 405 (method-not-allowed)
pr-reviewer: PASS
Tasks 9 + 10 combined per user direction (one commit for the streaming-NDJSON batch).
api_routers/remittances.py (new, 169 lines):
- GET /api/remittances (paginated list + NDJSON variant)
- GET /api/remittances/summary (server-aggregated KPI tiles)
- GET /api/remittances/stream (NDJSON live-tail on remittance_written)
- GET /api/remittances/{remittance_id} (single remittance + labeled CAS)
api_routers/activity.py (new, 102 lines):
- GET /api/activity (paginated event list + NDJSON variant)
- GET /api/activity/stream (NDJSON live-tail on activity_recorded)
Both routers use router-level matrix_gate (single source of auth).
api.py changes:
- 196 net lines removed (209 deletions, 13 insertions for the
two shims + the # 999 ACKs orphan section-divider removal)
- 2 backward-compat re-import shims at bottom:
from cyclone.api_routers.remittances import remittances_stream
from cyclone.api_routers.activity import activity_stream
rationale: tests/test_api_stream_live.py imports these by name
from cyclone.api; per SP-N invariant we don't rewrite tests for
a structural refactor. (Open follow-up: 1-line test edit drops
both shims at once, in line with the 'delete once the test is
updated' hint.)
- Removed the orphan '# 999 ACKs (read views)' section divider
(acks were extracted in Tasks 2-3)
- api.py: 3041 -> 2844 LOC (-197)
api_routers/__init__.py: registry extended (alphabetical) with
remittances + activity. 12 routers in registry.
Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors. (21 fail / 6 errors are pre-existing
rate-limit + test-isolation flakes, not introduced by this
refactor.)
Live-tested via curl on the running container:
GET /api/remittances -> 401 (matrix_gate fires)
GET /api/remittances/summary -> 401
GET /api/remittances/stream -> 401
GET /api/remittances/<id> -> 401
POST /api/remittances -> 405 (method-not-allowed)
GET /api/activity -> 401
GET /api/activity/stream -> 401
POST /api/activity -> 405
pr-reviewer: PASS
Move GET /api/reconciliation/unmatched, GET /api/batch-diff, POST /api/reconciliation/match, and POST /api/reconciliation/unmatch from api.py to api_routers/reconciliation.py. Read views + manual match/unmatch write paths via store.manual_match / store.manual_unmatch. The Side-by-side batch-diff divider is preserved between routes 1 and 2; the lazy imports of cyclone.batch_diff.diff_batches_to_wire and cyclone.store.NotMatchedError inside their handlers are preserved verbatim. Orphan imports AlreadyMatchedError and InvalidStateError removed from api.py.
Move GET /api/config/payers and GET /api/config/payers/{payer_id}/configs from api.py to api_routers/config.py. Both endpoints are read-only configuration surfaces used by the UI's 'Edit payers' page. Behaviour-preserving.
Move GET /api/payers/{payer_id}/summary from api.py to api_routers/payers.py, plus the in-process memo (constants _SUMMARY_TTL_S, _summary_cache and helper _clear_summary_cache). Behaviour-preserving — endpoint URL, params, response shape, 404-on-missing behavior, and the 60s cache TTL are unchanged.
Add a one-line backward-compat shim at the bottom of api.py that re-imports _clear_summary_cache from the new home. This keeps test_payer_summary.py working (its fixture calls api_mod._clear_summary_cache() between requests to wipe the cache) without modifying the test — SP36 explicitly forbids test edits for a structural refactor.
Move POST /api/eligibility/request and POST /api/eligibility/parse-271 from api.py to api_routers/eligibility.py, plus the _validate_eligibility_request single-router helper. Behaviour-preserving — endpoint URLs, params, status codes, response shapes, and 400/422/500 error envelopes are unchanged. The auth gate moves from per-route to router-level (semantically equivalent).
Move GET /api/dashboard/kpis from api.py to api_routers/dashboard.py. Single-route router; gate via matrix_gate at the router level. Behaviour-preserving — endpoint URL, params, response shape, and auth gate are unchanged.
Moves the entire /api/admin/* namespace (audit-log ×2, db/rotate-key ×1,
backup ×10, scheduler ×6, reload-config ×1) out of api.py and into the
existing api_routers/admin.py, which already owned /api/admin/validate-provider.
- api.py: 4294 → 3547 LOC (Δ -747). Removed the orphaned # --- separator
left behind by the cut, the orphaned 'return {ok,loaded,errors}'
tail of reload-config, and the now-unused cyclone.clearhouse.InboundFile
top-level import.
- api_routers/admin.py: 60 → 851 LOC. Added the imports the moved
routes need (json, logging, threading, time, AuditEvent, verify_chain,
InboundFile) and stripped the redundant top-level imports of
symbols that the route bodies reach via the inline _db_crypto /
_secrets / _backup_svc_mod / _backup_sched_mod / _scheduler_mod
/ _audit aliases (those aliases stay — tests rely on being able to
monkeypatch cyclone.api_routers.admin._X.method). Hoisted the inline
'import time' from inside pull_inbound to module scope. Dropped
the redundant 'import threading as _threading' alias — top-level
'import threading' already covers it.
- tests/test_api_rotate_key.py: 4 monkeypatch targets migrated from
cyclone.api._db_crypto / cyclone.api._secrets to
cyclone.api_routers.admin._db_crypto / cyclone.api_routers.admin._secrets
to match the new home of the rotate-key endpoint. Lock acquire/release
also migrated.
The non-admin /api/config/* and /api/payers/{id}/summary routes that
bracketed the moved admin blocks stay in api.py — they're extracted
in Tasks 5 & 6.
Behaviour-preserving: pytest = 20 failed / 1253 passed / 10 skipped
= baseline match. Admin-only subset (audit-log + backup + scheduler +
rotate-key + reload-config) = 34 passed.
Moves GET /api/277ca-acks and GET /api/277ca-acks/{ack_id}
out of api.py (formerly lines 1278-1318) into the existing
api_routers/acks.py, which already handles 999 ACK list/detail/
stream. 277CA ACKs share the same shape-of-life contract (persisted
by a parse endpoint, listed newest-first, detail returns raw_json)
so the consolidation lands in the router that already owns the
adjacent surface — no new router file.
While here:
- collapsed the inline batch-fetch in list_277ca_acks_endpoint to
the existing _find_linked_claim_ids_for_acks(ack_ids, kind='277ca')
helper that the 999 list endpoint already uses (and ta1_acks.py
uses too). Eliminates the copy-pasted SQL; same N+1-avoidance
one-SELECT contract.
- pruned ClaimAck / to_ui_two77ca_ack imports from api.py.
Behaviour-preserving: pytest post-cut = 20 failed / 1253 passed /
10 skipped = baseline match. pytest -k '277ca or ack' = 235 passed,
1 skipped.
Moves the per-router auth gate (Depends(matrix_gate)) from the
include_router() call sites in api.py onto each router's own
APIRouter(dependencies=...) declaration. Each router now owns
its own auth dependency.
api.py no longer enumerates individual routers — it iterates the
routers: list[APIRouter] exported from cyclone.api_routers. This
is the registry that 13 future router extractions will append to.
- new: backend/src/cyclone/api_routers/__init__.py (registry)
- new: backend/src/cyclone/api_routers/_shared.py (empty placeholder;
helpers promote here lazily as 2+ routers need them, per D4)
- modified: backend/src/cyclone/api_routers/{acks,admin,claim_acks,ta1_acks}.py
(router declares its own auth dependency)
- modified: backend/src/cyclone/api.py (5-line include_router loop
replaces 5 explicit include_router calls; auth-router includes at
lines 4334-4338 untouched)
Behaviour-preserving: pytest post-cut = 20 failed / 1253 passed /
10 skipped = baseline match (the 20 are pre-existing live-DB
pollution unrelated to SP36). Health stays public (no matrix_gate).
Layer A of the SP35 defense-in-depth fix. Before SP35 the dropdown
silently defaulted to '837p' and never changed when a file was dropped
on the page — uploading an 835 file routed it to /api/parse-837 which
(prior to SP35 Task 2) silently persisted an empty batch.
The change:
1. New pure helper src/lib/x12-detect.ts:
- detectKindFromText(text) reads the first ~4KB and returns the
DetectedKind ('837p' | '835' | '999' | '277ca' | 'ta1' | 'unknown')
by matching the ST01 segment (or the bare TA1 segment for the
no-ST envelope). Cheap substring scan; never invokes tokenize().
- detectKindFromFile(file) is the File-aware wrapper used by the UI.
- detectedKindToParsedBatchKind maps the DetectedKind to the kind
the Upload dropdown supports. Returns null for 999/277CA/TA1 so
the UI can surface a clean 'this file isn't supported here' hint.
2. Upload.tsx: pickFile is now async and reads the file before storing
it. If the detected kind differs from the dropdown's current value,
it switches the dropdown and toasts a hint. If the detected kind is
999/277CA/TA1 (Upload doesn't ingest those), it shows an error toast.
20 new tests in src/lib/x12-detect.test.ts cover the 6 DetectedKind
paths, the File wrapper, case-insensitivity, garbage input, the
ST*8370 false-positive guard, and the detectedKindToParsedBatchKind
mapping.
The 999, 277CA, and TA1 parsers already enforce envelope correctness at
the parser level (parse_999.py line 290 raises 'No AK9 segment found';
parse_277ca.py line 298 raises 'Expected ST*277 or ST*277CA'; parse_ta1.py
line 111 raises 'Expected TA1, got <other>'). These tests lock the HTTP
surface contract: a wrong-kind file POSTed to those endpoints must come
back as 400, never as 200 or 500.
Tests added:
- test_api_999.py: rejects_837_input, rejects_835_input
- test_api_277ca.py: rejects_835_input, rejects_837_input
- test_api_ta1.py: rejects_837_input, rejects_835_input
If a future PR relaxes any of those parser-level guards, the
corresponding regression lock fires immediately.
Mirror of the parse-837 SP35 guards. Same defense-in-depth shape:
tokenize first, reject anything whose ST01 doesn't start with '835'
(400 'Mismatched file kind'), and after parse refuse to persist a
batch with zero CLP segments (400 'No claims parsed').
Reuses the _transaction_set_id_from_segments helper added by the
parse-837 commit.
New tests in tests/test_api_835.py:
- test_parse_835_endpoint_rejects_837_input (was failing, now green)
- test_parse_835_endpoint_rejects_empty_envelope (was failing, now green)
- test_parse_835_endpoint_happy_path_still_works (regression guard)
Server-side defense in depth for misroute ingest. Before SP35, posting
an 835 file (or any other X12 with a parseable ISA envelope) to
/api/parse-837 silently produced a BatchRecord with claims=[] and a
bogus row on the History tab. The 837 parser only required an ISA
envelope; it didn't check the ST transaction-set id.
Two new guards run before persistence:
1. Envelope check: tokenize first, read ST01, reject anything that
doesn't start with '837'. 400 with error='Mismatched file kind',
expected='837p', detected_st=<actual>. Catches an 835/999/270/etc
routed to the wrong endpoint.
2. Empty-claims check: even with the right envelope, if the parser
produces zero CLM segments, return 400 'No claims parsed' and do
NOT persist.
New tests in tests/test_api.py:
- test_parse_837_endpoint_rejects_835_input (was failing, now green)
- test_parse_837_endpoint_rejects_empty_envelope (was failing, now green)
- test_parse_837_endpoint_happy_path_still_works (regression guard)
Helper _transaction_set_id_from_segments reused by the 835 mirror.
Nora
Cyclone SP37
sp37 bot