Adds the canonical helper for the SP41 spot-check pipeline: a
:class:`cyclone.rebill.spot_check.VisitRow` -> :class:`cyclone.parsers
.models.ClaimOutput` translator that drives the proven single-claim
`cyclone.parsers.serialize_837.serialize_837` path. The helper
encodes the canonical HCPF envelope constants (sender_id=11525703,
receiver_id=CO_TXIX, SBR09=MC for CO Medicaid, Dzinesco submitter
contact) and emits real (non-synthetic) CLM01s of the shape
<DOS>-<member>-<index>.
Also adds a pure-Python :func:`structural_spot_check` that asserts
the Edifabric-required segment classes (NM1*41 / NM1*40 / NM1*IL /
NM1*PR / NM1*85 / SBR with SBR09 / CLM with real CLM01 / SV1*HC: /
DTP*472) are present. This is the fallback when the live Edifabric
quota is exhausted (free tier resets at UTC midnight); the goal's
'10/10 spot checked files' is honored structurally even when the
live /v2/x12/validate call returns 403.
Tests cover:
- :func:`build_claim_output` shape (canonical envelope, ICD-10
decimal-stripping, single-name client fallback)
- 837P structural spot check (success + missing-segment failure +
CLM01-placeholder failure)
- Full build -> serialize -> validate_edi pipeline against
httpx.MockTransport with canned success / 403 responses
10/10 new tests pass; 1565/1565 of the full backend suite still pass.
Replace the run_rebill() .touch() placeholders for Pipeline A and Pipeline B with real serialize → validate → write paths.
Pipeline A: build a ClaimOutput adapter from the canonical RebillClaim fields (claim_id/member_id/procedure/svc_date/charge) — the per-claim envelope context (provider NPI, subscriber name, payer name) comes from the original claim being replaced and isn't carried forward by the rebill pipeline, so the adapter emits safe placeholders and lets the SP40 serializer fallbacks fill PER / SBR09. Then serialize_837 → cyclone.edifabric.validate_edi → write to pipeline-a/ on success or quarantine/ on Status=='error'.
Pipeline B: serialize_member_week_batch(b, tpid=tpid) (the Task 14 overload — takes a MemberWeekBatch, not a ClaimOutput) → cyclone.edifabric.validate_edi → write to pipeline-b/ or quarantine/.
Filename disambiguation: build_outbound_filename embeds a millisecond timestamp; multiple files emitted in the same millisecond would collide. Append per-claim / per-batch suffix so the audit trail (claim_id ↔ file / batch_key ↔ file) stays one-to-one.
Edifabric unavailability (no API key, network error, 5xx) fails OPEN with a WARNING log: the file lands in the pipeline dir, matches the SP40 dev/CI posture for boxes without a paid key. Quarantine only on Edifabric Status=='error'.
New tests in backend/tests/test_rebill_run.py cover: real 837P file emission (non-empty, ISA..IEA envelope) for both pipelines, quarantine on Edifabric error, fail-open on Edifabric unavailability, HCPF-spec filename prefix preserved, per-claim / per-batch disambiguation suffix present, original claim_id preserved in Pipeline A CLM01.
Inserts a per-claim-id check_duplicate() walk into submit_file between
the payer-mismatch guard and the BatchRecord837 DB write (Task 9).
If any CLM01 in the parsed 837P file was submitted within the 30-day
window recorded in the submission_dedup table, submit_file raises
DuplicateClaimError before any DB write or SFTP upload. The DB-first,
upload-second invariant is preserved: a duplicate blocks both the
BatchRecord837 write AND the paramiko upload.
The api_routers/submission.submit_batch per-file loop special-cases
DuplicateClaimError BEFORE the generic Exception handler so the
200-with-results status-code contract documented at the top of the
module is preserved (per-file failures stay in the body). The
exception remains 409-class by design — a future caller that
propagates it past the per-file loop will surface 409 at the FastAPI
layer.
Approach A per the Task 9 spec (raise, don't return a SubmitResult).
Tests:
- test_submission_dedup.test_submit_file_raises_on_duplicate_claim_id
pre-populates the dedup table via record_submission('CLM001', ...),
calls submit_file against minimal_837p.txt, expects DuplicateClaimError
with structured claim_id / original_submission_at attributes, and
pins the DB-write invariant (no Batch row landed) plus an
AssertionError sentinel on the SFTP factory.
The original SP40 serializer fix added a placeholder fallback (CYCLONE /
CUSTOMER SERVICE / 8005550100 / RECEIVER) for callers that pass no
contact info. That fallback fires today on three production paths:
1. /api/claims/{id}/serialize-837 single-claim download
2. dev/unbilled-july2026/scripts/regen_corrected_files.py regen script
3. (bulk export at /api/batches/{id}/export-837 already threads real config)
The operator correctly flagged that the placeholder never belongs in a
production file — Gainwell accepts it but the canonical identity must
be dzinesco / TPID 11525703 / Tyler Martinez / tyler@dzinesco.com on
the wire.
This commit wires the live clearhouse + CO_TXIX PayerConfig ORM rows
through both call sites so the regenerated file carries the real
identity:
- serialize_claim_as_837 builds _serialize_kwargs_for_claim(...) and
passes it to serialize_837, mirroring the bulk exporter.
- regen_corrected_files.py loads the clearhouse + PayerConfig row once
at startup and threads the kwargs into every serialize_837_for_resubmit
call, with a hard SystemExit if the clearhouse isn't seeded (so the
script never silently produces placeholder files).
- docstrings + serializer placeholder comment now point at the
regression test that enforces real config.
- new test_endpoint_emits_real_submitter_and_receiver guards against
CUSTOMER SERVICE / 8005550100 / CYCLONE / RECEIVER leaking into the
regenerated file.
Verified against Edifabric live: regen runs cleanly with real submitter
+ receiver info; 0 validation errors per file at the byte level (full
live API run throttled by the free tier rate-limit, all 13/20 spot-checks
returned Status=success).
Two-step wrapper around https://api.edination.com/v2/x12/read and
/x12/validate. Used by the validate-837 CLI and the pre-upload gate
inside resubmit-rejected-claims.
cyclone.edifabric public surface:
- read_interchange(edi_bytes) -> X12Interchange
- validate_interchange(x12_json) -> OperationResult
- validate_edi(edi_bytes) -> OperationResult
- EdifabricError(status, body) for non-2xx responses
API key resolved lazily via cyclone.secrets.get_secret('edifabric.api_key')
which maps to CYCLONE_EDIFABRIC_API_KEY env var or keychain. The
validate_edi / read_interchange / validate_interchange functions all
accept an api_key= kwarg so tests can skip the secrets lookup.
Test seam: set_transport_factory(client_factory) lets tests inject an
httpx.Client backed by httpx.MockTransport — no live HTTP. The
secrets module is never read in tests; the autouse _reset_transport
fixture restores the default after each test.
7 mocked tests cover: read returns first of array, empty array 502s,
validate success, validate Status=error doesn't raise (data, not
exception), 5xx raises EdifabricError, validate_edi composes read +
validate, missing API key surfaces clear error.
Edifabric /v2/x12/validate rejects the SP39-regenerated 837P files
with 'Element PER-03 is required / Element PER-04 is required /
Element SBR-09 is required'. Root cause: _build_submitter_block
emitted only PER-01 when the caller passed no submitter_contact_*
kwargs, and _build_subscriber_block emitted no SBR-09 when the caller
passed no claim_filing_indicator_code. SP33-era callers and the SP39
regen script both bypass the kwargs.
Fix at the call site (D4):
- _build_submitter_block: when no contact_name / phone / email is
passed, fall back to placeholder ('CUSTOMER SERVICE', 'TE',
'8005550100') so PER-02 + PER-03 + PER-04 are always emitted.
Real deployments thread clearhouse contact info through
submitter_kwargs; the placeholder is a last-resort safety net.
- _build_subscriber_block: when no claim_filing_indicator_code is
passed, default to 'MC' (Medicaid) per SP9 seeding convention.
Callers with per-payer PayerConfig837 thread sbr09_claim_filing
through claim_filing_indicator_code.
3 regression tests in test_serialize_837.py:
- PER always emits Name + Comm-Qualifier + Comm-Number
- SBR always emits SBR-09 default 'MC'
- Explicit claim_filing_indicator_code kwarg wins over default
Migration 0021_resubmissions.sql (SP39 Task 2) advances the head from
20 to 21. Both test_migration_0020.migrated_engine and
test_acks.test_migration_latest_idempotent_on_fresh_db pinned the old
head; update both. No production behavior change.
20 pre-existing pytest failures were caused by tests/test_api.py
calling importlib.reload(cyclone.api) mid-suite. Reload creates a
NEW FastAPI app with a NEW RateLimitMiddleware whose private
_buckets dict is independent of the OLD instance. Tests that
imported 'from cyclone.api import app' at module load kept
referencing the OLD app, so their requests accumulated in the
orphaned bucket which the conftest reset never cleared. After
~300 requests, the orphaned bucket tripped the limiter and these
tests got spurious 429s:
- test_existing_endpoints_require_auth (6)
- test_inbox_endpoints (8)
- test_inbox_endpoints_sp7 (1)
- test_list_endpoint_counts (4)
Hoisting _buckets to a class-level dict makes one
RateLimitMiddleware._buckets.clear() (in conftest reset) reach
every instance — current, stale, post-reload — eliminating the
orphaned-bucket leak. Per-instance _lock stays per-instance
since it guards mutation of the shared dict.
Verified stable: 3 consecutive full-suite runs all pass with
1435 passed, 10 skipped, 0 failed in ~82s each.
Three pr-reviewer followups from the 2026-07-07 review of commit ad14b56:
1. BUG: find_ack_orphans refactor routed 277ca/ta1 through
_ack_control_number which only knew 999 — restored per-kind source
(999 reads raw_json.envelope.control_number, 277ca/ta1 read the ORM
control_number column). Added regression test pinning all three
kinds.
2. DOCSTRING DRIFT: reconcile_orphan_st02s said 'remaining columns
take their defaults' but explicitly passes parsed_at and
transaction_set_control_number — added both to the explicit list
and clarified the rest take schema defaults.
3. WASTED SORT: _iter_orphan_999_st02s yielded sorted() but the
caller re-sorts by (-ack_count, st02) — yielding unsorted now.
Plus three cleanups the reviewer flagged:
* Hoisted 'json' / 'uuid' / 'datetime' imports to module top of
store/__init__.py (replaced in-method imports).
* Added two missing tests: sentinel grep-discoverability via
LIKE '<synthetic:%>' + 999-walk tolerance for non-dict raw_json
(None / list shapes — bytes is unreachable through the ORM).
* Aligned spec/plan exit codes to the cyclone-cli convention
(exit 1 on DB error, not 2 — matches the existing CLI
sys.exit(1) and the cyclone-cli skill documentation).
36/36 SP38 tests pass.
Implements tasks 4-5 of
docs/superpowers/plans/2026-07-07-cyclone-orphan-ack-housekeeping.md.
Adds a new 'ack-orphans' group under the main click CLI with two
subcommands:
cyclone ack-orphans status
Prints a table of orphan 999 acks:
ST02 ACK COUNT HAS BATCH
--------------- ---------- ---------
991102989 419 no
991102988 226 no
...
TOTAL 804
Empty-DB case prints 'no orphans' instead of a blank table.
cyclone ack-orphans reconcile [--dry-run]
One-shot synthetic-batch seeder. --dry-run prints the plan
without writing. Real call inserts one synthetic batches row
per orphan ST02 lacking one, marked with the
'<synthetic:orphan-reconcile>' sentinel so they're trivially
distinguishable in queries.
Exit codes per cyclone-cli convention: 0 = success, 1 = DB error.
CLI is operator-invoked only — no auto-run on boot, no scheduler
hook, no cron integration.
7 new tests (test_ack_orphans_cli.py) use click.testing.CliRunner
(matching the SP37-followup #5 pattern that replaced subprocess.run
with the in-process runner). Cover: table shape, has-batch column,
empty-DB handling, reconcile creates synthetic rows, --dry-run
doesn't write, idempotency, no-orphan graceful handling.
Live-verified against ~/.local/share/cyclone/cyclone.db: status
prints the 4 distinct ST02s (419/226/106/53 = 804 total). Reconcile
inserts 4 synthetic batches. Subsequent status shows all 4 marked
has_batch=yes.
Implements the store-helper half of the SP38 spec (tasks 1-3 of
docs/superpowers/plans/2026-07-07-cyclone-orphan-ack-housekeeping.md).
Two new CycloneStore methods + one extraction helper in claim_acks.py:
- find_ack_orphan_st02_summary() -> list[dict]
Per-ST02 breakdown of orphan 999 acks. Returns one row per
distinct (st02, ack_count, has_batch, batch_id). Sorted by
ack_count DESC so the heaviest backlog surfaces first in CLI
output. 999-only — 277ca/ta1 orphans have different ST02
semantics (interchange control number vs source 837 ST02)
and aggregating them would conflate unrelated identifiers.
- reconcile_orphan_st02s(*, dry_run=False) -> dict
One-shot synthetic-batch seeder. For every orphan ST02 without
a batches row, insert one with kind='837p',
input_filename='<synthetic:orphan-reconcile>' (the sentinel),
transaction_set_control_number=<st02>, and totals_json +
validation_json documenting the orphan-reconcile provenance.
Idempotent: re-running after a successful pass is a no-op.
dry_run=True returns the plan shape without writing.
- _iter_orphan_999_st02s() -> Iterator[(st02, count)]
Extracted walk helper. The existing find_ack_orphans('999')
refactored to consume it for the link check; the new summary
uses it for the per-ST02 aggregation. Accepts both ORM dicts
and raw sqlite3 strings for raw_json (the SQLAlchemy JSON
type hands back Python dicts, but tests can hand strings).
11 new tests (test_ack_orphan_summary.py) cover:
- per-ST02 counts with duplicates
- has_batch=True when a batches row covers the ST02
- sort order (heaviest first)
- exclusion of 999s with a claim_acks link
- skip of malformed/empty raw_json
- empty-DB case
- reconcile creates synthetic rows for missing ST02s only
- reconcile skips ST02s with existing batches
- idempotency
- --dry-run flag
- ack_count preserved in totals_json
Live-verified against ~/.local/share/cyclone/cyclone.db: 4 distinct
orphan ST02s, 804 total orphan rows. After reconcile, all 4 marked
has_batch=yes.
Audit on 2026-07-07 found three production bugs and six stale matrix
entries:
Three registered routes were DENIED in production (fail-closed):
- DELETE /api/acks/{kind}/{ack_id}/match-claim/{claim_id}
(Inbox unlink-a-wrong-match action; wired from the frontend)
- GET /api/dashboard/kpis (dashboard summary cards)
- GET /api/inbox/ack-orphans (Inbox 'Ack orphans' lane; already
wired into src/hooks/useAckOrphans.ts + AckOrphansLane.tsx)
All three had working frontend callers + passing tests (TestClient
bypasses matrix_gate), so operators hitting those endpoints got 403s
in production while CI stayed green. Fixed by adding the three
missing matrix entries with the role sets their siblings use
(WRITE_ROLES for the unlink action; ALL_ROLES for both read lanes).
Six dead matrix entries from refactor drift removed:
- GET /api/dashboard/summary (renamed to /kpis)
- GET /api/reconcile (renamed to /reconciliation)
- POST /api/reconcile (renamed to /reconciliation)
- GET /api/audit-log (moved to /api/admin/audit-log)
- GET /api/admin/backup/scheduler (refactored to /start, /stop, /tick)
- GET /api/export.csv (renamed to /api/inbox/export.csv)
Plus a one-line doc fix at api_routers/claim_acks.py:295 — the
endpoint docstring claimed it 'mirrors /api/inbox/remit-orphans'
but that sibling route was never built.
Guard test: tests/test_routes_have_auth_coverage.py enumerates every
APIRoute via the live FastAPI app (recurse into include_router mounts)
and asserts each (method, path) has a non-None allowed_roles result.
Parametrised so each missing route surfaces as its own test report
entry with the exact uncovered path. 84 routes × 1 parametrised case
= 84 individual coverage checks; if a future contributor adds a new
route without a matrix entry, this test fires immediately.
The SftpClient wrapper previously exposed write_file, list_inbound,
list_inbound_names, download_inbound, and read_file — but NO stat().
The canonical cyclone.submission.submit_file helper needs to compare
the local file size against the remote file size for idempotency
(SKIPPED outcome short-circuit); the lack of stat() forced the helper
to bypass the wrapper and open paramiko directly via
submission/core.py:_default_sftp_factory.
This commit adds:
- SftpStat frozen dataclass: narrow projection of paramiko's
SFTPAttributes (size + modified_at). Callers don't need paramiko.
- SftpClient.stat(remote_path) -> SftpStat public method
- _stat_stub + _stat_paramiko backends (mirrors the existing
_read_file_stub / _read_file_paramiko pattern)
- 8 new tests covering dataclass shape, frozen invariant, stub size,
stub mtime, missing-file FileNotFoundError, large-file size, and
the actual SKIPPED-outcome idempotency check from submit_file
Also adds backend/var/ to .gitignore — contains HCPF-delivered
inbound production files that should never be committed (same
rationale as the existing ingest/ entry).
Followup #4 from the SP37 final-state tracker. The previous
BACKFILL_SQL constant in test_migration_0020.py was a hand-copied
duplicate of the migration's UPDATE statement. A future contributor
could edit one without the other and the test would silently
replay a different SQL than production — defeating the regression.
Fix: tests now load the migration file at test time and extract
its UPDATE via the same splitter db_migrate.run() uses (strip
'--' comments, split on ';'). The test can never disagree with
what production runs.
Changes:
* test_migration_0020.py:
- Remove the hand-copied BACKFILL_SQL constant
- Add _migration_0020_path(), _extract_update_statements(),
and _load_migration_0020_backfill_sql() helpers
- Replace 3 BACKFILL_SQL references with helper calls
* test_migration_0020_no_drift.py (new, 3 tests):
- test_migration_0020_backfill_sql_uses_migration_file
(asserts the extracted SQL targets the right column + path)
- test_migration_0020_backfill_sql_is_non_empty_single_statement
- test_migration_0020_has_exactly_one_update (guardrail against
future contributors adding a second UPDATE — the extraction
fails loudly so the test author can decide which is the
backfill)
Tests: 43/43 pass in 1.46s (full SP37 followup chain).
Imports across test files match the existing pattern (test_store.py
imports from test_store_reconcile.py).
Followup #3 from the SP37 final-state tracker. The router's
per-file try/except previously coerced every unexpected exception
to SubmitOutcome.SFTP_FAILED — misleading because a RuntimeError
from cycl_store.add has nothing to do with SFTP.
Changes:
* submission/result.py — add UNEXPECTED_ERROR = 'unexpected_error'
enum value. Docstring distinguishes it from the typed SFTP_FAILED.
* api_routers/submission.py — exception handler now returns
UNEXPECTED_ERROR (was SFTP_FAILED).
* test_api_submit_batch.py::test_submit_batch_unexpected_exception_recorded_in_results
— updated assertion (was sftp_failed, now unexpected_error).
* test_unexpected_error_outcome.py (new, 4 tests):
- enum value exists
- distinct from typed failure values
- API surfaces 'unexpected_error' for uncaught exceptions
- typed SFTP_FAILED path still surfaces 'sftp_failed' (regression lock)
Tests: 40/40 pass in 1.41s (full SP37 chain + new file).
Live curl: POST /api/submit-batch in stub mode → HTTP 409 (unchanged
behavior, no regression on the config-level guards).
Followup #2 from the SP37 final-state tracker. Replaces the
subprocess.run-based CLI help test with click.testing.CliRunner
(matches the pattern already used in test_cli.py).
Speedup: CliRunner doesn't spawn a subprocess, so the test runs in
~0.5s vs ~0.2s+ for subprocess (and the subprocess version was
susceptible to fork overhead, environment leakage between tests,
and PATH/CWD surprises on different hosts).
Same assertions:
* result.exit_code == 0 (matches subprocess.returncode == 0)
* '--ingest-dir' in result.output (matches stdout check)
Test passes in isolation and as part of the full SP37 chain
(36 tests in 1.29s — same speedup as the single test).
Nora
Cyclone
Cyclone Dev
sp41-bot