docs(spec): SP39 2010BB NM109 byte defect fix + resubmission tracking design
This commit is contained in:
Nora
@@ -0,0 +1,334 @@
|
||||
# Sub-project 39 — 2010BB NM109 byte defect fix: Design Spec
|
||||
|
||||
**Date:** 2026-07-07
|
||||
**Status:** Draft, awaiting user sign-off
|
||||
**Branch:** `sp39-2010bb-nm109-fix`
|
||||
**Aesthetic direction:** No new UI; one serializer normalization helper + regression tests + regen script.
|
||||
|
||||
## 1. Scope
|
||||
|
||||
On 2026-07-07, an audit of Cyclone's outbound 837P files surfaced a byte-level
|
||||
defect in loop 2010BB (payer): the `NM1*PR*2*<name>*****PI*<id>` segment was
|
||||
being emitted with `NM109 = "SKCO0"` (and `NM103 = "COHCPF"`) for CO Medicaid
|
||||
claims, where Gainwell's MOVEit Transfer SFTP requires `NM109 = "CO_TXIX"`
|
||||
or `"CO_BHA"`. The payer rejected four batches at the SET level with
|
||||
`"2010BB NM109 must equal CO_TXIX or CO_BHA"`. SP33 already corrected the
|
||||
canonical `PayerConfig.co_medicaid()` payer id going forward, but the
|
||||
serializer's `_build_payer_block` still emits whatever `claim.payer.id` it
|
||||
is handed — so any pre-SP33 `ClaimOutput` rows in the DB whose `raw_json`
|
||||
captured `payer.id = "SKCO0"` will round-trip the defect on re-serialize.
|
||||
|
||||
A 363-file pre-correction set already exists at
|
||||
`ingest/corrected/batch-*/` (regenerated earlier with an upstream
|
||||
`raw['payer']['id']` workaround in `regen_837s.py`), plus four original
|
||||
outbound 837s at `ingest/tp11525703-837P-20260701*.txt` that retain the
|
||||
defect. The SP-N lands a defense-in-depth fix inside the serializer
|
||||
itself so the workaround can be removed and so any future caller that
|
||||
hands the serializer a stale `SKCO0` (or empty) payer id still emits a
|
||||
byte that Gainwell accepts.
|
||||
|
||||
**In scope:**
|
||||
|
||||
- A new private helper `_normalize_payer_id(payer)` in
|
||||
`backend/src/cyclone/parsers/serialize_837.py` that substitutes
|
||||
`"CO_TXIX"` for `payer.id` whenever the value is empty,
|
||||
`"SKCO0"`, or `"CO_BHA"`. The substitution also aligns
|
||||
`payer.name` to `"CO_TXIX"` when the original `payer.name` was
|
||||
the legacy `"COHCPF"`, `"CO_BHA"`, or empty, so the regenerated
|
||||
`NM103` stays consistent with the substituted `NM109`. Foreign
|
||||
payer IDs (anything not in the substitution set) are emitted
|
||||
verbatim — the helper only normalizes CO Medicaid-shape values.
|
||||
- A WARNING log emitted by `_build_payer_block` whenever the
|
||||
helper substitutes a value, carrying the substituted-from and
|
||||
substituted-to strings (one log per affected call; the
|
||||
serializer is invoked once per claim so the volume is bounded
|
||||
by batch size).
|
||||
- Hardening `_build_payer_block` to call the helper before
|
||||
delegating to `_build_nm1`.
|
||||
- A regression test in `backend/tests/test_serialize_837.py`
|
||||
asserting: `SKCO0` → `CO_TXIX` in NM109 + NM103; empty payer id
|
||||
→ `CO_TXIX` in NM109; `CO_BHA` → `CO_TXIX` in NM109 + NM103;
|
||||
a foreign-payer id (e.g. `"OTHER_PAYER"`) preserved unchanged.
|
||||
- An idempotent regen script
|
||||
`unbilled-july2026/scripts/regen_corrected_files.py` (alongside
|
||||
the analysis scripts in the sibling project folder) that walks
|
||||
the 363 corrected `ingest/corrected/batch-*/` files,
|
||||
re-parses each one through `parse_837.parse_837_text`, and
|
||||
re-serializes through `serialize_837_for_resubmit`. Output goes
|
||||
to `ingest/corrected-v2/<batch-id>-<N>-claims/` with a single
|
||||
global counter for unique timestamps. Each emitted file is
|
||||
re-validated with `parse_837_text` + a content check that
|
||||
asserts `PI*CO_TXIX` is present and neither `PI*SKCO0` nor
|
||||
`*COHCPF****` appear.
|
||||
- A new `resubmissions` table
|
||||
(`claim_id`, `batch_id`, `resubmitted_at`, `source_corrected_path`,
|
||||
`interchange_control_number`, `group_control_number`) plus a
|
||||
matching SQLAlchemy migration under `backend/migrations/0013_*.py`.
|
||||
The table is populated by the existing `cyclone resubmit-rejected-claims`
|
||||
CLI when the operator pushes a corrected file (one row per claim
|
||||
per resubmission; idempotent on `(claim_id, interchange_control_number)`).
|
||||
- A new CLI `cyclone resubmissions status [--batch-id=<id>]`
|
||||
that joins `resubmissions` → `claims` → `claim_acks` (via the
|
||||
existing SP28/31 ack-claim auto-link) → `remittances` (via the
|
||||
existing CLP→claim auto-link) and prints a per-claim row:
|
||||
`claim_id | patient | resubmitted_at | 999_status | 277ca_status |
|
||||
payment_status`. Statuses are derived as: `pending_999` if no
|
||||
ack row exists yet, `999_accepted` if a 999 with accept code
|
||||
exists, `999_rejected` if a 999 with reject code exists,
|
||||
`277ca_accepted` / `277ca_rejected` once 277CA arrives,
|
||||
`paid` once a remittance CLP links, `denied_again` if a 277CA
|
||||
reject exists post-resubmission.
|
||||
- A note in `docs/RUNBOOK.md` under "Manual SFTP mode" pointing
|
||||
the operator at the new `corrected-v2/` tree + the new
|
||||
`cyclone resubmissions status` CLI after this SP merges, and
|
||||
an inline TODO comment in
|
||||
`submission/core.py:EXPECTED_PAYER_ID` block referencing the
|
||||
future follow-up that will trace where `ClaimOutput.payer.id`
|
||||
gets set to `"SKCO0"` upstream.
|
||||
|
||||
**Out of scope:**
|
||||
|
||||
- Tracing and patching the upstream setter that populates
|
||||
`ClaimOutput.payer.id = "SKCO0"` (a separate future ticket; this
|
||||
SP only defense-in-depths the serializer). The
|
||||
`_normalize_payer_id` helper masks the upstream bug for any path
|
||||
that goes through `serialize_837` / `serialize_837_for_resubmit`,
|
||||
but does not touch the `claims` table.
|
||||
- Re-ingesting the four rejected `ingest/tp11525703-837P-*.txt`
|
||||
files. They are historical artifacts of the rejected
|
||||
submission and remain in `ingest/` for audit only; the operator
|
||||
does not resubmit them. The 363 corrected files in
|
||||
`ingest/corrected/batch-*/` are the resubmission payload, not
|
||||
these four.
|
||||
- Auto-resubmitting `ingest/corrected-v2/` via
|
||||
`cyclone resubmit-rejected-claims`. That CLI is the operator's
|
||||
workflow (per RUNBOOK §"Manual SFTP mode") and remains
|
||||
operator-invoked only. This SP only regenerates the corrected
|
||||
files; the operator chooses when to push them.
|
||||
- Any change to `PayerConfig.co_medicaid()` (SP33 already
|
||||
canonicalized it) or to the `submission/core.py` validation
|
||||
gate at `EXPECTED_PAYER_ID`. The gate continues to enforce
|
||||
CO_TXIX at submit time and is not relaxed by this SP.
|
||||
- A frontend surface for resubmission status. The CLI report is
|
||||
the only operator-facing view; a future UI increment (likely a
|
||||
new Inbox lane or Dashboard KPI) can render the same joined
|
||||
data once the workflow is proven.
|
||||
- Touching the existing 999/277CA/remit auto-link code paths.
|
||||
The resubmission status CLI consumes the same joined tables
|
||||
the auto-link already populates; this SP does not modify the
|
||||
auto-link logic itself.
|
||||
|
||||
## 2. Decisions (locked during brainstorming)
|
||||
|
||||
**D1: Substitute both legacy values — `SKCO0` and `CO_BHA` —
|
||||
plus empty — to `CO_TXIX`.**
|
||||
|
||||
The operator's policy is "every claim is CO_TXIX". The companion
|
||||
guide technically accepts `CO_BHA` for behavioral-health claims,
|
||||
but Cyclone is not configured to distinguish BHA claims at
|
||||
submission time and the operator's existing manual workflow
|
||||
treats `CO_TXIX` as the canonical value for all CO Medicaid
|
||||
submits. `CO_BHA` therefore gets normalized the same way `SKCO0`
|
||||
does. Foreign payer IDs (any value not in `{empty, "SKCO0",
|
||||
"CO_BHA"}`) are emitted verbatim — the helper only normalizes
|
||||
CO Medicaid-shape values and must not corrupt a non-CO submit.
|
||||
|
||||
**D2: The helper also fixes `payer.name` to match.**
|
||||
|
||||
When the helper substitutes `payer.id` to `CO_TXIX`, it also
|
||||
substitutes `payer.name` from `"COHCPF"`, `"CO_BHA"`, or empty
|
||||
to `"CO_TXIX"`. This keeps NM103 consistent with NM109 and
|
||||
matches the byte the operator's manual-mode workflow already
|
||||
uses in the corrected files.
|
||||
|
||||
**D3: WARNING log per substitution, not per batch.**
|
||||
|
||||
`_build_payer_block` is invoked once per claim. One log line per
|
||||
substitution is bounded by batch size (≤ 145 lines for the Jun 24
|
||||
batches) and is the minimum signal the operator needs to detect
|
||||
"this batch still has upstream contamination". Per-batch
|
||||
aggregation would hide multi-payer batches (if any are ever
|
||||
supported).
|
||||
|
||||
**D4: Regression tests live in `test_serialize_837.py`, not a
|
||||
new file.**
|
||||
|
||||
The existing file already covers NM1 segment shape; appending
|
||||
three tests for the helper is the minimal-surface change.
|
||||
Matching the `cyclone-tests` convention (sibling test file per
|
||||
module).
|
||||
|
||||
**D5: Regen script lives in the unbilled-july2026 sibling
|
||||
project.**
|
||||
|
||||
The 363 files + the four `ingest/tp11525703-837P-*.txt` originals
|
||||
are part of the unbilled-july2026 recovery work, not the Cyclone
|
||||
core codebase. The sibling project folder already houses the
|
||||
analysis scripts; the regen script follows the same convention.
|
||||
The script reads `ingest/corrected/batch-*/` (relative to
|
||||
Cyclone's repo root, since `ingest/` is shared with the
|
||||
production SFTP workflow) and writes to `ingest/corrected-v2/`.
|
||||
|
||||
**D6: Re-run into `ingest/corrected-v2/`, not clobber the
|
||||
original `batch-*/` tree.**
|
||||
|
||||
The 363 corrected files are the postmortem anchor for SP33-era
|
||||
SKCO0 contamination. Preserving them under `batch-*/` lets a
|
||||
future contributor diff pre-SP39 and post-SP39 outputs and
|
||||
confirms the helper is a byte-faithful transformation on every
|
||||
input that was already correct. The `corrected-v2/` tree is the
|
||||
new submission payload.
|
||||
|
||||
**D7: File a follow-up note (in RUNBOOK), don't patch the
|
||||
upstream `ClaimOutput.payer.id` setter in this SP.**
|
||||
|
||||
The operator picked hard-fix the serializer (Approach 1) over
|
||||
trace to root cause (Approach 3). The follow-up ticket belongs
|
||||
in a future SP; this SP captures the need via a TODO comment in
|
||||
`submission/core.py` and a RUNBOOK entry pointing at the new
|
||||
corrected-v2/ tree.
|
||||
|
||||
**D8: `resubmissions` is a write-once audit table, not a
|
||||
state-machine.**
|
||||
|
||||
The table records "this claim was pushed to SFTP at this time
|
||||
from this local file" — one row per claim per push. The CLI
|
||||
derives status (`pending_999` / `999_accepted` / `paid` / etc.)
|
||||
at read-time by joining against `claim_acks` + `remittances`
|
||||
+ the existing auto-link tables. We do not denormalize status
|
||||
onto the `resubmissions` row, because that would create a
|
||||
write-coordination problem between the SFTP push and the
|
||||
inbound 999/277CA ingestion.
|
||||
|
||||
**D9: Post-submission tracking joins reuse the existing
|
||||
auto-link data, no new matching logic.**
|
||||
|
||||
The 999 ack auto-link (SP28/31) already maps an inbound CLP
|
||||
segment back to a `claim_id` via CLP01 + charge + service date.
|
||||
Because the corrected-v2 regen preserves the original `claim_id`
|
||||
from `raw_json.claim.claim_id`, the 999 ack for a resubmitted
|
||||
file auto-links to the original claim row — and the new status
|
||||
CLI just joins `resubmissions.claim_id` against `claim_acks.claim_id`
|
||||
to surface the result. No new matching code is needed; this SP
|
||||
adds the read-side join and the CLI surface, not a new matcher.
|
||||
|
||||
**D10: The `resubmit-rejected-claims` CLI inserts `resubmissions`
|
||||
rows; we do not add a new push CLI.**
|
||||
|
||||
The existing `cyclone resubmit-rejected-claims` (per
|
||||
`backend/src/cyclone/cli.py` and the SP33 followup work) is
|
||||
the operator's only entry point for pushing corrected files via
|
||||
SFTP. We instrument it to insert one `resubmissions` row per
|
||||
claim per push (matching the same idempotency key as the SFTP
|
||||
upload). No new CLI command for the push side.
|
||||
|
||||
## 3. Open questions
|
||||
|
||||
None. The operator confirmed the design via the brainstorming
|
||||
Q&A on 2026-07-07: hard-fix the serializer with empty/SKCO0/CO_BHA
|
||||
substitution to CO_TXIX (no CO_BHA preservation), regen into
|
||||
corrected-v2/, add a post-submission tracking table + status CLI
|
||||
(joined against existing auto-link data, no new matcher), file
|
||||
a follow-up note for upstream tracing, no UI, no auto-resubmit.
|
||||
|
||||
## 4. Test impact
|
||||
|
||||
Per `cyclone-tests` (autouse conftest at `backend/tests/conftest.py`):
|
||||
|
||||
- `backend/tests/test_serialize_837.py` — append three tests
|
||||
targeting `_build_payer_block` via the public
|
||||
`serialize_837_for_resubmit` entry point. Each test constructs
|
||||
a minimal `ClaimOutput` whose `payer.id` is set to the value
|
||||
under test, calls `serialize_837_for_resubmit`, and asserts the
|
||||
emitted `NM1*PR` segment. Tests:
|
||||
- `test_2010bb_normalizes_skco0_to_co_txix` — asserts
|
||||
`SKCO0` → `CO_TXIX` in NM109 and `COHCPF` → `CO_TXIX` in
|
||||
NM103.
|
||||
- `test_2010bb_normalizes_empty_payer_id_to_co_txix` —
|
||||
asserts empty `payer.id` → `CO_TXIX` and empty `payer.name`
|
||||
→ `CO_TXIX`.
|
||||
- `test_2010bb_normalizes_co_bha_to_co_txix` — asserts
|
||||
`CO_BHA` (with name `CO_BHA`) → `CO_TXIX` in both NM109
|
||||
and NM103. Plus a fourth assertion that a foreign payer id
|
||||
(e.g. `"OTHER_PAYER"` with name `"OTHER PAYER NAME"`) is
|
||||
emitted verbatim — the helper must not corrupt non-CO
|
||||
submits.
|
||||
- `backend/tests/test_resubmissions_cli.py` — new, tests the
|
||||
`cyclone resubmissions status` CLI via `click.testing.CliRunner`.
|
||||
Seeds the `resubmissions` table + `claim_acks` (via the
|
||||
existing fixture pattern) and asserts:
|
||||
- `pending_999` when no ack exists yet.
|
||||
- `999_accepted` after a 999 ack with accept code is linked.
|
||||
- `paid` after a remittance CLP links.
|
||||
- Exit code 0 with empty `resubmissions` table prints
|
||||
"no resubmissions recorded" and exits 0.
|
||||
- `backend/tests/test_resubmissions_table.py` — new, tests the
|
||||
SQLAlchemy model + idempotency of `(claim_id, interchange_control_number)`.
|
||||
Asserts schema columns, asserts a second insert with the same
|
||||
key is a no-op (raises IntegrityError or ON CONFLICT DO NOTHING,
|
||||
matching the convention in the rest of `db.py`).
|
||||
- `unbilled-july2026/scripts/test_regen_corrected_files.py` —
|
||||
new, sibling test in the unbilled-july2026 project. Uses a
|
||||
fixture of three fake-corrected files (one already correct,
|
||||
one with SKCO0 from a degenerate raw_json, one with CO_BHA)
|
||||
and asserts the regen script emits all three with the expected
|
||||
normalization.
|
||||
- Migration test impact: append to
|
||||
`backend/tests/test_db_migrate.py` (or equivalent) a check
|
||||
that migration 0013 creates the `resubmissions` table with
|
||||
the documented columns + idempotency unique constraint.
|
||||
- No frontend test impact.
|
||||
|
||||
## 5. Files expected to change
|
||||
|
||||
- `backend/src/cyclone/parsers/serialize_837.py` — add
|
||||
`_normalize_payer_id(payer)`, call it from `_build_payer_block`,
|
||||
emit WARNING log on substitution.
|
||||
- `backend/src/cyclone/db.py` — add `Resubmission` SQLAlchemy
|
||||
model.
|
||||
- `backend/migrations/0013_resubmissions.py` — new migration
|
||||
creating the `resubmissions` table.
|
||||
- `backend/src/cyclone/cli.py` — instrument
|
||||
`resubmit-rejected-claims` to insert one `Resubmission` row
|
||||
per claim per push (idempotent on
|
||||
`(claim_id, interchange_control_number)`); add the
|
||||
`cyclone resubmissions status` subcommand.
|
||||
- `backend/src/cyclone/store/__init__.py` — add
|
||||
`CycloneStore.record_resubmission(...)` and
|
||||
`CycloneStore.find_resubmission_status(...)` helpers; re-export
|
||||
through the facade.
|
||||
- `backend/tests/test_serialize_837.py` — append the three
|
||||
regression tests.
|
||||
- `backend/tests/test_resubmissions_cli.py` — new CLI tests.
|
||||
- `backend/tests/test_resubmissions_table.py` — new model +
|
||||
idempotency tests.
|
||||
- `backend/tests/test_db_migrate.py` — append migration test
|
||||
for 0013.
|
||||
- `unbilled-july2026/scripts/regen_corrected_files.py` — new
|
||||
sibling-project regen script.
|
||||
- `unbilled-july2026/scripts/test_regen_corrected_files.py` —
|
||||
new sibling-project test.
|
||||
- `docs/RUNBOOK.md` — append "After SP39 lands" entry under
|
||||
"Manual SFTP mode" pointing the operator at `corrected-v2/`,
|
||||
the new `cyclone resubmissions status` CLI, and the workflow
|
||||
to push via `cyclone resubmit-rejected-claims`.
|
||||
- `backend/src/cyclone/submission/core.py` — append TODO comment
|
||||
above `EXPECTED_PAYER_ID` referencing the future upstream-
|
||||
trace ticket.
|
||||
- `docs/superpowers/plans/2026-07-07-cyclone-2010bb-nm109-fix.md`
|
||||
— the implementation plan, written after this spec is signed
|
||||
off.
|
||||
|
||||
## 6. Auth boundary
|
||||
|
||||
The auth boundary is HTTP (login required, bcrypt + HttpOnly
|
||||
session cookie); file-system threats remain the local-only threat
|
||||
model (SQLCipher at rest, macOS Keychain). The serializer helper
|
||||
is invoked server-side by both `serialize_837` (tested via
|
||||
`cyclone submit-batch` / `POST /api/submit-batch`, both behind
|
||||
HTTP auth) and `serialize_837_for_resubmit` (called by the
|
||||
operator's regen workflow and by `/api/inbox/rejected/resubmit`,
|
||||
also behind HTTP auth). The sibling-project regen script is
|
||||
operator-invoked only and runs against a local file tree; it does
|
||||
not touch the HTTP surface or the DB. No change to the threat
|
||||
model.
|
||||
Reference in New Issue
Block a user