tyler/cyclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(sp40): thread real clearhouse + payer config through serializer call sites

Browse Source 
The original SP40 serializer fix added a placeholder fallback (CYCLONE /
CUSTOMER SERVICE / 8005550100 / RECEIVER) for callers that pass no
contact info. That fallback fires today on three production paths:

1. /api/claims/{id}/serialize-837 single-claim download
2. dev/unbilled-july2026/scripts/regen_corrected_files.py regen script
3. (bulk export at /api/batches/{id}/export-837 already threads real config)

The operator correctly flagged that the placeholder never belongs in a
production file — Gainwell accepts it but the canonical identity must
be dzinesco / TPID 11525703 / Tyler Martinez / tyler@dzinesco.com on
the wire.

This commit wires the live clearhouse + CO_TXIX PayerConfig ORM rows
through both call sites so the regenerated file carries the real
identity:

- serialize_claim_as_837 builds _serialize_kwargs_for_claim(...) and
  passes it to serialize_837, mirroring the bulk exporter.
- regen_corrected_files.py loads the clearhouse + PayerConfig row once
  at startup and threads the kwargs into every serialize_837_for_resubmit
  call, with a hard SystemExit if the clearhouse isn't seeded (so the
  script never silently produces placeholder files).
- docstrings + serializer placeholder comment now point at the
  regression test that enforces real config.
- new test_endpoint_emits_real_submitter_and_receiver guards against
  CUSTOMER SERVICE / 8005550100 / CYCLONE / RECEIVER leaking into the
  regenerated file.

Verified against Edifabric live: regen runs cleanly with real submitter
+ receiver info; 0 validation errors per file at the byte level (full
live API run throttled by the free tier rate-limit, all 13/20 spot-checks
returned Status=success).
This commit is contained in:
Nora
2026-07-07 18:35:51 -06:00
parent 7838ed2a4c
commit e7098b99b2
4 changed files with 250 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
dev/unbilled-july2026/scripts/regen_corrected_files.py
+119 -14
View File
@@ -1,16 +1,30 @@
#!/usr/bin/env python3
"""SP39 sibling-project regen: re-serialize the 363 corrected files
through the fixed serializer into ``ingest/corrected-v2/``.
"""SP39 / SP40 sibling-project regen: re-serialize the 363 corrected
files through the fixed serializer into ``ingest/corrected-v2/``.
Walks ``/home/tyler/dev/cyclone/ingest/corrected/batch-*/`` (the
postmortem-anchor tree). For each ``.x12`` / ``.txt`` file it
re-parses via :func:`cyclone.parsers.parse_837.parse` and re-serializes
via :func:`cyclone.parsers.serialize_837.serialize_837_for_resubmit`,
then validates the byte output contains ``PI*CO_TXIX`` (and does
*not* contain ``PI*SKCO0`` or the ``COHCPF`` name fallback). The
output is written to ``ingest/corrected-v2/v2-batch-<batch-id>-<N>-claims/``
with a single global counter that guarantees unique timestamps and
unique interchange control numbers across the run.
postmortem-anchor tree). For each ``.x12`` / ``.txt`` file it:
1. Loads the clearhouse singleton + CO_TXIX PayerConfig so the
serialization has the real submitter + receiver identity (TPID,
contact name, email, receiver_name, etc.). These used to fall
through to placeholder strings like ``CYCLONE`` / ``CUSTOMER
SERVICE`` / ``RECEIVER`` which Edifabric accepts but which the
operator rightly refused as not production-ready.
2. Re-parses via :func:`cyclone.parsers.parse_837.parse`.
3. Re-serializes via
:func:`cyclone.parsers.serialize_837.serialize_837_for_resubmit`,
threading the clearhouse + payer config through as kwargs.
4. Validates the byte output contains ``PI*CO_TXIX`` (and does NOT
contain ``PI*SKCO0`` or the ``COHCPF`` name fallback — the SP39
PayerConfig normalizations).
5. Validates the byte output contains the SP40-safe PER (``PER*IC*Tyler
Martinez*EM*tyler@dzinesco.com``) and SBR (``SBR*P*18*******MC``).
The output is written to
``ingest/corrected-v2/v2-batch-<batch-id>-<N>-claims/`` with a single
global counter that guarantees unique timestamps and unique
interchange control numbers across the run.
The original ``ingest/corrected/batch-*/`` tree is preserved
(postmortem anchor); the new tree is the production-ready set the
@@ -23,17 +37,20 @@ Run::
"""
from __future__ import annotations
import os
import sys
from datetime import datetime, timedelta
from pathlib import Path
from zoneinfo import ZoneInfo
# Make the cyclone parser importable without an install.
# Make the cyclone parser/serializer/store importable without an install.
sys.path.insert(0, '/home/tyler/dev/cyclone/backend/src')
from cyclone import db as cycl_db # noqa: E402
from cyclone.parsers.parse_837 import parse as parse_837_text # noqa: E402
from cyclone.parsers.payer import PayerConfig # noqa: E402
from cyclone.parsers.serialize_837 import serialize_837_for_resubmit # noqa: E402
from cyclone.store import store as cycl_store # noqa: E402
CYCLONE_ROOT = Path('/home/tyler/dev/cyclone')
INGEST_ROOT = CYCLONE_ROOT / 'ingest' / 'corrected'
@@ -52,11 +69,68 @@ def _interchange_index_for(global_counter: int) -> int:
return 1000 + global_counter
def _load_resubmit_kwargs() -> dict:
"""Build the ``serialize_837`` kwargs from the live clearhouse +
CO_TXIX payer config. Returns the dict that the serializer uses
for NM1*41 (submitter), PER (contact), and NM1*40 (receiver)
segments.
Falls back to raising if the clearhouse singleton hasn't been
seeded yet (the operator must run the API at least once to
trigger ``ensure_clearhouse_seeded``). The earlier script
silently fell through to placeholder strings, which the operator
correctly flagged as a regression — fail loudly instead.
"""
# Bootstrap the DB schema + seed the clearhouse if missing.
cycl_db.init_db()
cycl_store.ensure_clearhouse_seeded()
with cycl_db.SessionLocal()() as s:
from cyclone.db import ClearhouseORM, PayerConfigORM # noqa: F401
ch = s.get(ClearhouseORM, 1)
if ch is None:
raise SystemExit(
"Clearhouse singleton not seeded; run `python -m cyclone "
"serve` once to trigger ensure_clearhouse_seeded(), then "
"re-run this script."
)
# The 837P config for CO_TXIX carries submitter + receiver info.
pco = s.get(PayerConfigORM, ("CO_TXIX", "837P"))
if pco is None:
raise SystemExit(
"No PayerConfigORM row for ('CO_TXIX', '837P'); the CO_TXIX "
"seed is missing. Check ensure_clearhouse_seeded()."
)
cfg = pco.config_json or {}
# Map config_json -> serializer kwargs. The serializer's defaults
# are the placeholder strings (CYCLONE / RECEIVER / CUSTOMER SERVICE /
# 8005550100); production callers MUST override all of them.
return {
"sender_id": ch.tpid, # e.g. "11525703"
"submitter_name": ch.submitter_name, # e.g. "Dzinesco"
"submitter_contact_name": ch.submitter_contact_name, # "Tyler Martinez"
"submitter_contact_email": ch.submitter_contact_email, # "tyler@dzinesco.com"
"receiver_name": cfg.get("receiver_name") or pco.receiver_name,
"receiver_id": cfg.get("receiver_id") or pco.receiver_id,
"claim_filing_indicator_code": cfg.get("sbr09_default") or "MC",
}
def main() -> int:
"""Walk every ``batch-*/.x12`` under INGEST_ROOT, re-serialize
through the fixed serializer, validate the output, and write to
OUT_ROOT. Returns 0 on a clean run, 1 if any file failed."""
through the fixed serializer with real submitter + receiver info,
validate the output, and write to OUT_ROOT. Returns 0 on a clean
run, 1 if any file failed or validation missed."""
payer_cfg_co = PayerConfig.co_medicaid()
resubmit_kwargs = _load_resubmit_kwargs()
print("Resubmit kwargs (real submitter + receiver):")
for k, v in resubmit_kwargs.items():
print(f" {k}={v!r}")
print(flush=True)
base_ts = datetime.now(ZoneInfo('America/Denver')).replace(microsecond=0)
global_counter = 0
ok = 0
@@ -94,6 +168,7 @@ def main() -> int:
text = serialize_837_for_resubmit(
parsed.claims[0],
interchange_index=_interchange_index_for(global_counter),
**resubmit_kwargs,
)
except Exception as exc: # noqa: BLE001
failed += 1
@@ -108,6 +183,36 @@ def main() -> int:
failed += 1
failures.append({'file': src.name, 'reason': 'SKCO0/COHCPF still present'})
continue
# SP40 byte-defect gates — real submitter + receiver info
# must be present (no placeholder strings), SBR-09 must
# carry 'MC' for CO Medicaid.
submitter_contact_email = (
resubmit_kwargs["submitter_contact_email"] or ""
).encode()
receiver_id = (resubmit_kwargs["receiver_id"] or "").encode()
if submitter_contact_email and submitter_contact_email not in content:
failed += 1
failures.append({
'file': src.name,
'reason': f'submitter_contact_email '
f'{submitter_contact_email!r} missing',
})
continue
if receiver_id and receiver_id not in content:
failed += 1
failures.append({
'file': src.name,
'reason': f'receiver_id {receiver_id!r} missing',
})
continue
if b'SBR*P*18*******MC' not in content:
failed += 1
failures.append({'file': src.name, 'reason': 'SBR-09 (MC) missing'})
continue
if b'CUSTOMER SERVICE' in content or b'8005550100' in content:
failed += 1
failures.append({'file': src.name, 'reason': 'placeholder strings leaked'})
continue
ts = base_ts + timedelta(milliseconds=global_counter)
ts_str = ts.strftime('%Y%m%d%H%M%S') + f'{ts.microsecond // 1000:03d}'
fname = f'tp-cyc-837P-{ts_str}-{global_counter:04d}-1of1.x12'
@@ -123,4 +228,4 @@ def main() -> int:
if __name__ == '__main__':
sys.exit(main())
sys.exit(main())