Inserts a per-claim-id check_duplicate() walk into submit_file between
the payer-mismatch guard and the BatchRecord837 DB write (Task 9).
If any CLM01 in the parsed 837P file was submitted within the 30-day
window recorded in the submission_dedup table, submit_file raises
DuplicateClaimError before any DB write or SFTP upload. The DB-first,
upload-second invariant is preserved: a duplicate blocks both the
BatchRecord837 write AND the paramiko upload.
The api_routers/submission.submit_batch per-file loop special-cases
DuplicateClaimError BEFORE the generic Exception handler so the
200-with-results status-code contract documented at the top of the
module is preserved (per-file failures stay in the body). The
exception remains 409-class by design — a future caller that
propagates it past the per-file loop will surface 409 at the FastAPI
layer.
Approach A per the Task 9 spec (raise, don't return a SubmitResult).
Tests:
- test_submission_dedup.test_submit_file_raises_on_duplicate_claim_id
pre-populates the dedup table via record_submission('CLM001', ...),
calls submit_file against minimal_837p.txt, expects DuplicateClaimError
with structured claim_id / original_submission_at attributes, and
pins the DB-write invariant (no Batch row landed) plus an
AssertionError sentinel on the SFTP factory.
The regen script was emitting filenames like 'tp-cyc-837P-20260707183154271-0271-1of1.x12'
which violate the HCPF X12 File Naming Standards Quick Guide. The spec requires:
tp{tpid}-{transaction_type}-yyyymmddhhmmssSSS-1of1.x12
— no "-cyc-" placeholder segment, no trailing 4-digit counter ("-0271-"),
and the 17-digit timestamp must come from a single millisecond-precision
instant (the trailing counter was a separate monotonically-increasing integer).
Route filename generation through the canonical
cyclone.edi.filenames.build_outbound_filename() helper so:
* the format is guaranteed by the existing OUTBOUND_RE regex
* the 1ms stride between files yields unique millisecond slots
for all 363 outputs in a single regen run
* a final is_outbound_filename() guard fails a file with a clear
'HCPF OUTBOUND_RE (SPEC VIOLATION)' reason if anything downstream
bypasses the builder
Verified clean: 363/363 generated with new builder, 0/0 filename
validation failures, 96/96 SP40 backend tests pass, 1/1 regen-script
unit test passes. Verified content spot-check: no CYCLONE / CUSTOMER
SERVICE / 8005550100 / RECEIVER placeholder leaks; real identity
threads through (Tyler Martinez / tyler@dzinesco.com / COLORADO MEDICAL
ASSISTANCE PROGRAM / COMEDASSISTPROG).
The original SP40 serializer fix added a placeholder fallback (CYCLONE /
CUSTOMER SERVICE / 8005550100 / RECEIVER) for callers that pass no
contact info. That fallback fires today on three production paths:
1. /api/claims/{id}/serialize-837 single-claim download
2. dev/unbilled-july2026/scripts/regen_corrected_files.py regen script
3. (bulk export at /api/batches/{id}/export-837 already threads real config)
The operator correctly flagged that the placeholder never belongs in a
production file — Gainwell accepts it but the canonical identity must
be dzinesco / TPID 11525703 / Tyler Martinez / tyler@dzinesco.com on
the wire.
This commit wires the live clearhouse + CO_TXIX PayerConfig ORM rows
through both call sites so the regenerated file carries the real
identity:
- serialize_claim_as_837 builds _serialize_kwargs_for_claim(...) and
passes it to serialize_837, mirroring the bulk exporter.
- regen_corrected_files.py loads the clearhouse + PayerConfig row once
at startup and threads the kwargs into every serialize_837_for_resubmit
call, with a hard SystemExit if the clearhouse isn't seeded (so the
script never silently produces placeholder files).
- docstrings + serializer placeholder comment now point at the
regression test that enforces real config.
- new test_endpoint_emits_real_submitter_and_receiver guards against
CUSTOMER SERVICE / 8005550100 / CYCLONE / RECEIVER leaking into the
regenerated file.
Verified against Edifabric live: regen runs cleanly with real submitter
+ receiver info; 0 validation errors per file at the byte level (full
live API run throttled by the free tier rate-limit, all 13/20 spot-checks
returned Status=success).
Two-step wrapper around https://api.edination.com/v2/x12/read and
/x12/validate. Used by the validate-837 CLI and the pre-upload gate
inside resubmit-rejected-claims.
cyclone.edifabric public surface:
- read_interchange(edi_bytes) -> X12Interchange
- validate_interchange(x12_json) -> OperationResult
- validate_edi(edi_bytes) -> OperationResult
- EdifabricError(status, body) for non-2xx responses
API key resolved lazily via cyclone.secrets.get_secret('edifabric.api_key')
which maps to CYCLONE_EDIFABRIC_API_KEY env var or keychain. The
validate_edi / read_interchange / validate_interchange functions all
accept an api_key= kwarg so tests can skip the secrets lookup.
Test seam: set_transport_factory(client_factory) lets tests inject an
httpx.Client backed by httpx.MockTransport — no live HTTP. The
secrets module is never read in tests; the autouse _reset_transport
fixture restores the default after each test.
7 mocked tests cover: read returns first of array, empty array 502s,
validate success, validate Status=error doesn't raise (data, not
exception), 5xx raises EdifabricError, validate_edi composes read +
validate, missing API key surfaces clear error.
Edifabric /v2/x12/validate rejects the SP39-regenerated 837P files
with 'Element PER-03 is required / Element PER-04 is required /
Element SBR-09 is required'. Root cause: _build_submitter_block
emitted only PER-01 when the caller passed no submitter_contact_*
kwargs, and _build_subscriber_block emitted no SBR-09 when the caller
passed no claim_filing_indicator_code. SP33-era callers and the SP39
regen script both bypass the kwargs.
Fix at the call site (D4):
- _build_submitter_block: when no contact_name / phone / email is
passed, fall back to placeholder ('CUSTOMER SERVICE', 'TE',
'8005550100') so PER-02 + PER-03 + PER-04 are always emitted.
Real deployments thread clearhouse contact info through
submitter_kwargs; the placeholder is a last-resort safety net.
- _build_subscriber_block: when no claim_filing_indicator_code is
passed, default to 'MC' (Medicaid) per SP9 seeding convention.
Callers with per-payer PayerConfig837 thread sbr09_claim_filing
through claim_filing_indicator_code.
3 regression tests in test_serialize_837.py:
- PER always emits Name + Comm-Qualifier + Comm-Number
- SBR always emits SBR-09 default 'MC'
- Explicit claim_filing_indicator_code kwarg wins over default
Covers three things:
1. _build_per / _build_sbr call-site fixes so the serializer always
emits PER-02/03/04 + SBR-09 (Edifabric rejects the current shape).
2. New cyclone.edifabric HTTP client wrapping the two-step
/x12/read -> /x12/validate flow against api.edination.com.
3. Pre-upload gate inside resubmit-rejected-claims so an
Edifabric-invalid file can't reach Gainwell.
Public eval API key 3ecf6b1c5cf34bd797a5f4c57951a1cf for dev.
Production is operator-supplied via cyclone secrets set
edifabric.api_key <paid-key>.
Migration 0021_resubmissions.sql (SP39 Task 2) advances the head from
20 to 21. Both test_migration_0020.migrated_engine and
test_acks.test_migration_latest_idempotent_on_fresh_db pinned the old
head; update both. No production behavior change.
Re-serialize the 363 corrected 837P files through the SP39-fixed
serializer, emitting NM1*PR*2*CO_TXIX*****PI*CO_TXIX in 2010BB and
writing to ingest/corrected-v2/v2-batch-<batch-id>-<N>-claims/.
Lives under dev/unbilled-july2026/scripts/ because the unbilled-july2026
project at /home/tyler/dev/unbilled-july2026/ is NOT a git repo of its
own (the operator's working tree only). Mirroring the physical path
keeps the script's import path (sys.path.insert ../../cyclone/backend/src)
working without per-run shimming.
Verified end-to-end: ok=363 failed=0, 5 random spot-checks all show
NM1*PR*2*CO_TXIX*****PI*CO_TXIX, zero SKCO0/COHCPF fallbacks in any
of the 363 output files.
20 pre-existing pytest failures were caused by tests/test_api.py
calling importlib.reload(cyclone.api) mid-suite. Reload creates a
NEW FastAPI app with a NEW RateLimitMiddleware whose private
_buckets dict is independent of the OLD instance. Tests that
imported 'from cyclone.api import app' at module load kept
referencing the OLD app, so their requests accumulated in the
orphaned bucket which the conftest reset never cleared. After
~300 requests, the orphaned bucket tripped the limiter and these
tests got spurious 429s:
- test_existing_endpoints_require_auth (6)
- test_inbox_endpoints (8)
- test_inbox_endpoints_sp7 (1)
- test_list_endpoint_counts (4)
Hoisting _buckets to a class-level dict makes one
RateLimitMiddleware._buckets.clear() (in conftest reset) reach
every instance — current, stale, post-reload — eliminating the
orphaned-bucket leak. Per-instance _lock stays per-instance
since it guards mutation of the shared dict.
Verified stable: 3 consecutive full-suite runs all pass with
1435 passed, 10 skipped, 0 failed in ~82s each.
Captures the historical 804-orphan 999 situation as a known
artifact (RUNBOOK entry + status command) and provides a
one-shot idempotent synthetic-batch seeder so future acks for
the orphan ST02s can resolve against batch_envelope_index.
6 commits merged atomically (no squash — the SP-N merge commit
is the audit record):
d776a4a docs(spec): design for SP38 orphan-ack housekeeping
8890627 docs(plan): SP38 orphan-ack housekeeping plan
ad14b56 feat(sp38): CycloneStore.find_ack_orphan_st02_summary + reconcile_orphan_st02s
9ef749c feat(sp38): 'cyclone ack-orphans {status,reconcile}' CLI subcommands
76923a7 docs(sp38): RUNBOOK entry for orphan-ack housekeeping
07ea7ca fix(sp38): restore per-kind control_number in find_ack_orphans + review cleanups
Test delta: +36 passed (12 store helper + 7 CLI + 17 claim_acks
regression including the pr-reviewer 277ca/ta1 control_number
fix). No new failures.
Live verified: cyclone ack-orphans status exits 0 with 4 distinct
orphan ST02s (419+226+106+53=804) matching the SQLite investigation.
Three pr-reviewer followups from the 2026-07-07 review of commit ad14b56:
1. BUG: find_ack_orphans refactor routed 277ca/ta1 through
_ack_control_number which only knew 999 — restored per-kind source
(999 reads raw_json.envelope.control_number, 277ca/ta1 read the ORM
control_number column). Added regression test pinning all three
kinds.
2. DOCSTRING DRIFT: reconcile_orphan_st02s said 'remaining columns
take their defaults' but explicitly passes parsed_at and
transaction_set_control_number — added both to the explicit list
and clarified the rest take schema defaults.
3. WASTED SORT: _iter_orphan_999_st02s yielded sorted() but the
caller re-sorts by (-ack_count, st02) — yielding unsorted now.
Plus three cleanups the reviewer flagged:
* Hoisted 'json' / 'uuid' / 'datetime' imports to module top of
store/__init__.py (replaced in-method imports).
* Added two missing tests: sentinel grep-discoverability via
LIKE '<synthetic:%>' + 999-walk tolerance for non-dict raw_json
(None / list shapes — bytes is unreachable through the ORM).
* Aligned spec/plan exit codes to the cyclone-cli convention
(exit 1 on DB error, not 2 — matches the existing CLI
sys.exit(1) and the cyclone-cli skill documentation).
36/36 SP38 tests pass.
Nora
sp41-bot