The regen script was emitting filenames like 'tp-cyc-837P-20260707183154271-0271-1of1.x12'
which violate the HCPF X12 File Naming Standards Quick Guide. The spec requires:
tp{tpid}-{transaction_type}-yyyymmddhhmmssSSS-1of1.x12
— no "-cyc-" placeholder segment, no trailing 4-digit counter ("-0271-"),
and the 17-digit timestamp must come from a single millisecond-precision
instant (the trailing counter was a separate monotonically-increasing integer).
Route filename generation through the canonical
cyclone.edi.filenames.build_outbound_filename() helper so:
* the format is guaranteed by the existing OUTBOUND_RE regex
* the 1ms stride between files yields unique millisecond slots
for all 363 outputs in a single regen run
* a final is_outbound_filename() guard fails a file with a clear
'HCPF OUTBOUND_RE (SPEC VIOLATION)' reason if anything downstream
bypasses the builder
Verified clean: 363/363 generated with new builder, 0/0 filename
validation failures, 96/96 SP40 backend tests pass, 1/1 regen-script
unit test passes. Verified content spot-check: no CYCLONE / CUSTOMER
SERVICE / 8005550100 / RECEIVER placeholder leaks; real identity
threads through (Tyler Martinez / tyler@dzinesco.com / COLORADO MEDICAL
ASSISTANCE PROGRAM / COMEDASSISTPROG).
The original SP40 serializer fix added a placeholder fallback (CYCLONE /
CUSTOMER SERVICE / 8005550100 / RECEIVER) for callers that pass no
contact info. That fallback fires today on three production paths:
1. /api/claims/{id}/serialize-837 single-claim download
2. dev/unbilled-july2026/scripts/regen_corrected_files.py regen script
3. (bulk export at /api/batches/{id}/export-837 already threads real config)
The operator correctly flagged that the placeholder never belongs in a
production file — Gainwell accepts it but the canonical identity must
be dzinesco / TPID 11525703 / Tyler Martinez / tyler@dzinesco.com on
the wire.
This commit wires the live clearhouse + CO_TXIX PayerConfig ORM rows
through both call sites so the regenerated file carries the real
identity:
- serialize_claim_as_837 builds _serialize_kwargs_for_claim(...) and
passes it to serialize_837, mirroring the bulk exporter.
- regen_corrected_files.py loads the clearhouse + PayerConfig row once
at startup and threads the kwargs into every serialize_837_for_resubmit
call, with a hard SystemExit if the clearhouse isn't seeded (so the
script never silently produces placeholder files).
- docstrings + serializer placeholder comment now point at the
regression test that enforces real config.
- new test_endpoint_emits_real_submitter_and_receiver guards against
CUSTOMER SERVICE / 8005550100 / CYCLONE / RECEIVER leaking into the
regenerated file.
Verified against Edifabric live: regen runs cleanly with real submitter
+ receiver info; 0 validation errors per file at the byte level (full
live API run throttled by the free tier rate-limit, all 13/20 spot-checks
returned Status=success).
Two-step wrapper around https://api.edination.com/v2/x12/read and
/x12/validate. Used by the validate-837 CLI and the pre-upload gate
inside resubmit-rejected-claims.
cyclone.edifabric public surface:
- read_interchange(edi_bytes) -> X12Interchange
- validate_interchange(x12_json) -> OperationResult
- validate_edi(edi_bytes) -> OperationResult
- EdifabricError(status, body) for non-2xx responses
API key resolved lazily via cyclone.secrets.get_secret('edifabric.api_key')
which maps to CYCLONE_EDIFABRIC_API_KEY env var or keychain. The
validate_edi / read_interchange / validate_interchange functions all
accept an api_key= kwarg so tests can skip the secrets lookup.
Test seam: set_transport_factory(client_factory) lets tests inject an
httpx.Client backed by httpx.MockTransport — no live HTTP. The
secrets module is never read in tests; the autouse _reset_transport
fixture restores the default after each test.
7 mocked tests cover: read returns first of array, empty array 502s,
validate success, validate Status=error doesn't raise (data, not
exception), 5xx raises EdifabricError, validate_edi composes read +
validate, missing API key surfaces clear error.
Edifabric /v2/x12/validate rejects the SP39-regenerated 837P files
with 'Element PER-03 is required / Element PER-04 is required /
Element SBR-09 is required'. Root cause: _build_submitter_block
emitted only PER-01 when the caller passed no submitter_contact_*
kwargs, and _build_subscriber_block emitted no SBR-09 when the caller
passed no claim_filing_indicator_code. SP33-era callers and the SP39
regen script both bypass the kwargs.
Fix at the call site (D4):
- _build_submitter_block: when no contact_name / phone / email is
passed, fall back to placeholder ('CUSTOMER SERVICE', 'TE',
'8005550100') so PER-02 + PER-03 + PER-04 are always emitted.
Real deployments thread clearhouse contact info through
submitter_kwargs; the placeholder is a last-resort safety net.
- _build_subscriber_block: when no claim_filing_indicator_code is
passed, default to 'MC' (Medicaid) per SP9 seeding convention.
Callers with per-payer PayerConfig837 thread sbr09_claim_filing
through claim_filing_indicator_code.
3 regression tests in test_serialize_837.py:
- PER always emits Name + Comm-Qualifier + Comm-Number
- SBR always emits SBR-09 default 'MC'
- Explicit claim_filing_indicator_code kwarg wins over default
Covers three things:
1. _build_per / _build_sbr call-site fixes so the serializer always
emits PER-02/03/04 + SBR-09 (Edifabric rejects the current shape).
2. New cyclone.edifabric HTTP client wrapping the two-step
/x12/read -> /x12/validate flow against api.edination.com.
3. Pre-upload gate inside resubmit-rejected-claims so an
Edifabric-invalid file can't reach Gainwell.
Public eval API key 3ecf6b1c5cf34bd797a5f4c57951a1cf for dev.
Production is operator-supplied via cyclone secrets set
edifabric.api_key <paid-key>.
Migration 0021_resubmissions.sql (SP39 Task 2) advances the head from
20 to 21. Both test_migration_0020.migrated_engine and
test_acks.test_migration_latest_idempotent_on_fresh_db pinned the old
head; update both. No production behavior change.
Re-serialize the 363 corrected 837P files through the SP39-fixed
serializer, emitting NM1*PR*2*CO_TXIX*****PI*CO_TXIX in 2010BB and
writing to ingest/corrected-v2/v2-batch-<batch-id>-<N>-claims/.
Lives under dev/unbilled-july2026/scripts/ because the unbilled-july2026
project at /home/tyler/dev/unbilled-july2026/ is NOT a git repo of its
own (the operator's working tree only). Mirroring the physical path
keeps the script's import path (sys.path.insert ../../cyclone/backend/src)
working without per-run shimming.
Verified end-to-end: ok=363 failed=0, 5 random spot-checks all show
NM1*PR*2*CO_TXIX*****PI*CO_TXIX, zero SKCO0/COHCPF fallbacks in any
of the 363 output files.
20 pre-existing pytest failures were caused by tests/test_api.py
calling importlib.reload(cyclone.api) mid-suite. Reload creates a
NEW FastAPI app with a NEW RateLimitMiddleware whose private
_buckets dict is independent of the OLD instance. Tests that
imported 'from cyclone.api import app' at module load kept
referencing the OLD app, so their requests accumulated in the
orphaned bucket which the conftest reset never cleared. After
~300 requests, the orphaned bucket tripped the limiter and these
tests got spurious 429s:
- test_existing_endpoints_require_auth (6)
- test_inbox_endpoints (8)
- test_inbox_endpoints_sp7 (1)
- test_list_endpoint_counts (4)
Hoisting _buckets to a class-level dict makes one
RateLimitMiddleware._buckets.clear() (in conftest reset) reach
every instance — current, stale, post-reload — eliminating the
orphaned-bucket leak. Per-instance _lock stays per-instance
since it guards mutation of the shared dict.
Verified stable: 3 consecutive full-suite runs all pass with
1435 passed, 10 skipped, 0 failed in ~82s each.
Captures the historical 804-orphan 999 situation as a known
artifact (RUNBOOK entry + status command) and provides a
one-shot idempotent synthetic-batch seeder so future acks for
the orphan ST02s can resolve against batch_envelope_index.
6 commits merged atomically (no squash — the SP-N merge commit
is the audit record):
d776a4a docs(spec): design for SP38 orphan-ack housekeeping
8890627 docs(plan): SP38 orphan-ack housekeeping plan
ad14b56 feat(sp38): CycloneStore.find_ack_orphan_st02_summary + reconcile_orphan_st02s
9ef749c feat(sp38): 'cyclone ack-orphans {status,reconcile}' CLI subcommands
76923a7 docs(sp38): RUNBOOK entry for orphan-ack housekeeping
07ea7ca fix(sp38): restore per-kind control_number in find_ack_orphans + review cleanups
Test delta: +36 passed (12 store helper + 7 CLI + 17 claim_acks
regression including the pr-reviewer 277ca/ta1 control_number
fix). No new failures.
Live verified: cyclone ack-orphans status exits 0 with 4 distinct
orphan ST02s (419+226+106+53=804) matching the SQLite investigation.
Three pr-reviewer followups from the 2026-07-07 review of commit ad14b56:
1. BUG: find_ack_orphans refactor routed 277ca/ta1 through
_ack_control_number which only knew 999 — restored per-kind source
(999 reads raw_json.envelope.control_number, 277ca/ta1 read the ORM
control_number column). Added regression test pinning all three
kinds.
2. DOCSTRING DRIFT: reconcile_orphan_st02s said 'remaining columns
take their defaults' but explicitly passes parsed_at and
transaction_set_control_number — added both to the explicit list
and clarified the rest take schema defaults.
3. WASTED SORT: _iter_orphan_999_st02s yielded sorted() but the
caller re-sorts by (-ack_count, st02) — yielding unsorted now.
Plus three cleanups the reviewer flagged:
* Hoisted 'json' / 'uuid' / 'datetime' imports to module top of
store/__init__.py (replaced in-method imports).
* Added two missing tests: sentinel grep-discoverability via
LIKE '<synthetic:%>' + 999-walk tolerance for non-dict raw_json
(None / list shapes — bytes is unreachable through the ORM).
* Aligned spec/plan exit codes to the cyclone-cli convention
(exit 1 on DB error, not 2 — matches the existing CLI
sys.exit(1) and the cyclone-cli skill documentation).
36/36 SP38 tests pass.
Implements task 6 of
docs/superpowers/plans/2026-07-07-cyclone-orphan-ack-housekeeping.md.
Adds 'Known historical drift — the 804 orphan 999s' section under
the existing operator-triage content. Covers:
- What the orphans are (real production 999s whose source 837s
predate the current DB snapshot; valid audit history that
cannot be auto-linked)
- Why they cannot be auto-linked (source 837s were transmitted
to HPE and never came back; Cyclone is downstream and does
not retain copies of outbound 837s)
- Triage path via the Inbox AckOrphansLane (already working
as of SP37-followup 893a662)
- Optional synthetic-batch seeding via 'cyclone ack-orphans
reconcile' (one-shot, idempotent, --dry-run available)
- 'cyclone ack-orphans status' for the per-ST02 breakdown
Explicitly calls out what the housekeeping is NOT:
- Not a backfill (no claims rows are synthesized)
- Not auto-runnable (operator-invoked only)
- Not a deletion (orphans are valid audit history)
References the SP38 spec and plan for the design rationale.
Implements tasks 4-5 of
docs/superpowers/plans/2026-07-07-cyclone-orphan-ack-housekeeping.md.
Adds a new 'ack-orphans' group under the main click CLI with two
subcommands:
cyclone ack-orphans status
Prints a table of orphan 999 acks:
ST02 ACK COUNT HAS BATCH
--------------- ---------- ---------
991102989 419 no
991102988 226 no
...
TOTAL 804
Empty-DB case prints 'no orphans' instead of a blank table.
cyclone ack-orphans reconcile [--dry-run]
One-shot synthetic-batch seeder. --dry-run prints the plan
without writing. Real call inserts one synthetic batches row
per orphan ST02 lacking one, marked with the
'<synthetic:orphan-reconcile>' sentinel so they're trivially
distinguishable in queries.
Exit codes per cyclone-cli convention: 0 = success, 1 = DB error.
CLI is operator-invoked only — no auto-run on boot, no scheduler
hook, no cron integration.
7 new tests (test_ack_orphans_cli.py) use click.testing.CliRunner
(matching the SP37-followup #5 pattern that replaced subprocess.run
with the in-process runner). Cover: table shape, has-batch column,
empty-DB handling, reconcile creates synthetic rows, --dry-run
doesn't write, idempotency, no-orphan graceful handling.
Live-verified against ~/.local/share/cyclone/cyclone.db: status
prints the 4 distinct ST02s (419/226/106/53 = 804 total). Reconcile
inserts 4 synthetic batches. Subsequent status shows all 4 marked
has_batch=yes.
Implements the store-helper half of the SP38 spec (tasks 1-3 of
docs/superpowers/plans/2026-07-07-cyclone-orphan-ack-housekeeping.md).
Two new CycloneStore methods + one extraction helper in claim_acks.py:
- find_ack_orphan_st02_summary() -> list[dict]
Per-ST02 breakdown of orphan 999 acks. Returns one row per
distinct (st02, ack_count, has_batch, batch_id). Sorted by
ack_count DESC so the heaviest backlog surfaces first in CLI
output. 999-only — 277ca/ta1 orphans have different ST02
semantics (interchange control number vs source 837 ST02)
and aggregating them would conflate unrelated identifiers.
- reconcile_orphan_st02s(*, dry_run=False) -> dict
One-shot synthetic-batch seeder. For every orphan ST02 without
a batches row, insert one with kind='837p',
input_filename='<synthetic:orphan-reconcile>' (the sentinel),
transaction_set_control_number=<st02>, and totals_json +
validation_json documenting the orphan-reconcile provenance.
Idempotent: re-running after a successful pass is a no-op.
dry_run=True returns the plan shape without writing.
- _iter_orphan_999_st02s() -> Iterator[(st02, count)]
Extracted walk helper. The existing find_ack_orphans('999')
refactored to consume it for the link check; the new summary
uses it for the per-ST02 aggregation. Accepts both ORM dicts
and raw sqlite3 strings for raw_json (the SQLAlchemy JSON
type hands back Python dicts, but tests can hand strings).
11 new tests (test_ack_orphan_summary.py) cover:
- per-ST02 counts with duplicates
- has_batch=True when a batches row covers the ST02
- sort order (heaviest first)
- exclusion of 999s with a claim_acks link
- skip of malformed/empty raw_json
- empty-DB case
- reconcile creates synthetic rows for missing ST02s only
- reconcile skips ST02s with existing batches
- idempotency
- --dry-run flag
- ack_count preserved in totals_json
Live-verified against ~/.local/share/cyclone/cyclone.db: 4 distinct
orphan ST02s, 804 total orphan rows. After reconcile, all 4 marked
has_batch=yes.
Adds docs/superpowers/specs/2026-07-07-cyclone-orphan-ack-housekeeping-design.md.
The current cyclone.db holds 805 ack rows, 804 of which are unresolved
orphans — 999s whose source 837 batches pre-date the current DB
snapshot and were never re-ingested. Investigation on 2026-07-07
confirmed the source 837s are not recoverable (Clearinghouse does not
echo them back; only the 999s were preserved in ingest/ + SFTP
staging). SP37's canonical submit-batch flow already captures ST02
going forward, so the orphan count stays flat — not a forward-looking
bug, just historical drift.
This SP captures that situation and adds housekeeping around it:
- RUNBOOK.md entry under 'Known historical drift' explaining the
root cause and the operator's triage path (Inbox AckOrphansLane,
which now works as of SP37-followup 893a662).
- 'cyclone ack-orphans status' CLI: distinct orphan ST02s + ack
count per ST02 + total.
- 'cyclone ack-orphans reconcile' CLI: one-shot, idempotent
synthetic-batch seeder. Marks synthetic rows with
input_filename '<synthetic:orphan-reconcile>' so they're
distinguishable in queries and the codebase can grep the
sentinel.
No UI change, no auto-runnable reconcile, no schema migration, no
attempt to backfill claim rows for the orphan ST02s (the source
data is gone). Per the canonical SP-N spec template, header is
'Draft, awaiting user sign-off' until the plan is signed off.
Branch: sp38-orphan-ack-housekeeping (off main, ahead by 893a662).
Next step: plan at docs/superpowers/plans/2026-07-07-cyclone-orphan-ack-housekeeping.md.
Nora
sp41-bot