From 8516b90601095522199e1c66477932a4183b77d2 Mon Sep 17 00:00:00 2001 From: Nora Date: Tue, 7 Jul 2026 17:18:00 -0600 Subject: [PATCH] docs(spec): SP39 2010BB NM109 byte defect fix + resubmission tracking design --- ...6-07-07-cyclone-2010bb-nm109-fix-design.md | 334 ++++++++++++++++++ 1 file changed, 334 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-07-cyclone-2010bb-nm109-fix-design.md diff --git a/docs/superpowers/specs/2026-07-07-cyclone-2010bb-nm109-fix-design.md b/docs/superpowers/specs/2026-07-07-cyclone-2010bb-nm109-fix-design.md new file mode 100644 index 0000000..c1adaa9 --- /dev/null +++ b/docs/superpowers/specs/2026-07-07-cyclone-2010bb-nm109-fix-design.md @@ -0,0 +1,334 @@ +# Sub-project 39 — 2010BB NM109 byte defect fix: Design Spec + +**Date:** 2026-07-07 +**Status:** Draft, awaiting user sign-off +**Branch:** `sp39-2010bb-nm109-fix` +**Aesthetic direction:** No new UI; one serializer normalization helper + regression tests + regen script. + +## 1. Scope + +On 2026-07-07, an audit of Cyclone's outbound 837P files surfaced a byte-level +defect in loop 2010BB (payer): the `NM1*PR*2******PI*` segment was +being emitted with `NM109 = "SKCO0"` (and `NM103 = "COHCPF"`) for CO Medicaid +claims, where Gainwell's MOVEit Transfer SFTP requires `NM109 = "CO_TXIX"` +or `"CO_BHA"`. The payer rejected four batches at the SET level with +`"2010BB NM109 must equal CO_TXIX or CO_BHA"`. SP33 already corrected the +canonical `PayerConfig.co_medicaid()` payer id going forward, but the +serializer's `_build_payer_block` still emits whatever `claim.payer.id` it +is handed — so any pre-SP33 `ClaimOutput` rows in the DB whose `raw_json` +captured `payer.id = "SKCO0"` will round-trip the defect on re-serialize. + +A 363-file pre-correction set already exists at +`ingest/corrected/batch-*/` (regenerated earlier with an upstream +`raw['payer']['id']` workaround in `regen_837s.py`), plus four original +outbound 837s at `ingest/tp11525703-837P-20260701*.txt` that retain the +defect. The SP-N lands a defense-in-depth fix inside the serializer +itself so the workaround can be removed and so any future caller that +hands the serializer a stale `SKCO0` (or empty) payer id still emits a +byte that Gainwell accepts. + +**In scope:** + +- A new private helper `_normalize_payer_id(payer)` in + `backend/src/cyclone/parsers/serialize_837.py` that substitutes + `"CO_TXIX"` for `payer.id` whenever the value is empty, + `"SKCO0"`, or `"CO_BHA"`. The substitution also aligns + `payer.name` to `"CO_TXIX"` when the original `payer.name` was + the legacy `"COHCPF"`, `"CO_BHA"`, or empty, so the regenerated + `NM103` stays consistent with the substituted `NM109`. Foreign + payer IDs (anything not in the substitution set) are emitted + verbatim — the helper only normalizes CO Medicaid-shape values. +- A WARNING log emitted by `_build_payer_block` whenever the + helper substitutes a value, carrying the substituted-from and + substituted-to strings (one log per affected call; the + serializer is invoked once per claim so the volume is bounded + by batch size). +- Hardening `_build_payer_block` to call the helper before + delegating to `_build_nm1`. +- A regression test in `backend/tests/test_serialize_837.py` + asserting: `SKCO0` → `CO_TXIX` in NM109 + NM103; empty payer id + → `CO_TXIX` in NM109; `CO_BHA` → `CO_TXIX` in NM109 + NM103; + a foreign-payer id (e.g. `"OTHER_PAYER"`) preserved unchanged. +- An idempotent regen script + `unbilled-july2026/scripts/regen_corrected_files.py` (alongside + the analysis scripts in the sibling project folder) that walks + the 363 corrected `ingest/corrected/batch-*/` files, + re-parses each one through `parse_837.parse_837_text`, and + re-serializes through `serialize_837_for_resubmit`. Output goes + to `ingest/corrected-v2/--claims/` with a single + global counter for unique timestamps. Each emitted file is + re-validated with `parse_837_text` + a content check that + asserts `PI*CO_TXIX` is present and neither `PI*SKCO0` nor + `*COHCPF****` appear. +- A new `resubmissions` table + (`claim_id`, `batch_id`, `resubmitted_at`, `source_corrected_path`, + `interchange_control_number`, `group_control_number`) plus a + matching SQLAlchemy migration under `backend/migrations/0013_*.py`. + The table is populated by the existing `cyclone resubmit-rejected-claims` + CLI when the operator pushes a corrected file (one row per claim + per resubmission; idempotent on `(claim_id, interchange_control_number)`). +- A new CLI `cyclone resubmissions status [--batch-id=]` + that joins `resubmissions` → `claims` → `claim_acks` (via the + existing SP28/31 ack-claim auto-link) → `remittances` (via the + existing CLP→claim auto-link) and prints a per-claim row: + `claim_id | patient | resubmitted_at | 999_status | 277ca_status | + payment_status`. Statuses are derived as: `pending_999` if no + ack row exists yet, `999_accepted` if a 999 with accept code + exists, `999_rejected` if a 999 with reject code exists, + `277ca_accepted` / `277ca_rejected` once 277CA arrives, + `paid` once a remittance CLP links, `denied_again` if a 277CA + reject exists post-resubmission. +- A note in `docs/RUNBOOK.md` under "Manual SFTP mode" pointing + the operator at the new `corrected-v2/` tree + the new + `cyclone resubmissions status` CLI after this SP merges, and + an inline TODO comment in + `submission/core.py:EXPECTED_PAYER_ID` block referencing the + future follow-up that will trace where `ClaimOutput.payer.id` + gets set to `"SKCO0"` upstream. + +**Out of scope:** + +- Tracing and patching the upstream setter that populates + `ClaimOutput.payer.id = "SKCO0"` (a separate future ticket; this + SP only defense-in-depths the serializer). The + `_normalize_payer_id` helper masks the upstream bug for any path + that goes through `serialize_837` / `serialize_837_for_resubmit`, + but does not touch the `claims` table. +- Re-ingesting the four rejected `ingest/tp11525703-837P-*.txt` + files. They are historical artifacts of the rejected + submission and remain in `ingest/` for audit only; the operator + does not resubmit them. The 363 corrected files in + `ingest/corrected/batch-*/` are the resubmission payload, not + these four. +- Auto-resubmitting `ingest/corrected-v2/` via + `cyclone resubmit-rejected-claims`. That CLI is the operator's + workflow (per RUNBOOK §"Manual SFTP mode") and remains + operator-invoked only. This SP only regenerates the corrected + files; the operator chooses when to push them. +- Any change to `PayerConfig.co_medicaid()` (SP33 already + canonicalized it) or to the `submission/core.py` validation + gate at `EXPECTED_PAYER_ID`. The gate continues to enforce + CO_TXIX at submit time and is not relaxed by this SP. +- A frontend surface for resubmission status. The CLI report is + the only operator-facing view; a future UI increment (likely a + new Inbox lane or Dashboard KPI) can render the same joined + data once the workflow is proven. +- Touching the existing 999/277CA/remit auto-link code paths. + The resubmission status CLI consumes the same joined tables + the auto-link already populates; this SP does not modify the + auto-link logic itself. + +## 2. Decisions (locked during brainstorming) + +**D1: Substitute both legacy values — `SKCO0` and `CO_BHA` — +plus empty — to `CO_TXIX`.** + +The operator's policy is "every claim is CO_TXIX". The companion +guide technically accepts `CO_BHA` for behavioral-health claims, +but Cyclone is not configured to distinguish BHA claims at +submission time and the operator's existing manual workflow +treats `CO_TXIX` as the canonical value for all CO Medicaid +submits. `CO_BHA` therefore gets normalized the same way `SKCO0` +does. Foreign payer IDs (any value not in `{empty, "SKCO0", +"CO_BHA"}`) are emitted verbatim — the helper only normalizes +CO Medicaid-shape values and must not corrupt a non-CO submit. + +**D2: The helper also fixes `payer.name` to match.** + +When the helper substitutes `payer.id` to `CO_TXIX`, it also +substitutes `payer.name` from `"COHCPF"`, `"CO_BHA"`, or empty +to `"CO_TXIX"`. This keeps NM103 consistent with NM109 and +matches the byte the operator's manual-mode workflow already +uses in the corrected files. + +**D3: WARNING log per substitution, not per batch.** + +`_build_payer_block` is invoked once per claim. One log line per +substitution is bounded by batch size (≤ 145 lines for the Jun 24 +batches) and is the minimum signal the operator needs to detect +"this batch still has upstream contamination". Per-batch +aggregation would hide multi-payer batches (if any are ever +supported). + +**D4: Regression tests live in `test_serialize_837.py`, not a +new file.** + +The existing file already covers NM1 segment shape; appending +three tests for the helper is the minimal-surface change. +Matching the `cyclone-tests` convention (sibling test file per +module). + +**D5: Regen script lives in the unbilled-july2026 sibling +project.** + +The 363 files + the four `ingest/tp11525703-837P-*.txt` originals +are part of the unbilled-july2026 recovery work, not the Cyclone +core codebase. The sibling project folder already houses the +analysis scripts; the regen script follows the same convention. +The script reads `ingest/corrected/batch-*/` (relative to +Cyclone's repo root, since `ingest/` is shared with the +production SFTP workflow) and writes to `ingest/corrected-v2/`. + +**D6: Re-run into `ingest/corrected-v2/`, not clobber the +original `batch-*/` tree.** + +The 363 corrected files are the postmortem anchor for SP33-era +SKCO0 contamination. Preserving them under `batch-*/` lets a +future contributor diff pre-SP39 and post-SP39 outputs and +confirms the helper is a byte-faithful transformation on every +input that was already correct. The `corrected-v2/` tree is the +new submission payload. + +**D7: File a follow-up note (in RUNBOOK), don't patch the +upstream `ClaimOutput.payer.id` setter in this SP.** + +The operator picked hard-fix the serializer (Approach 1) over +trace to root cause (Approach 3). The follow-up ticket belongs +in a future SP; this SP captures the need via a TODO comment in +`submission/core.py` and a RUNBOOK entry pointing at the new +corrected-v2/ tree. + +**D8: `resubmissions` is a write-once audit table, not a +state-machine.** + +The table records "this claim was pushed to SFTP at this time +from this local file" — one row per claim per push. The CLI +derives status (`pending_999` / `999_accepted` / `paid` / etc.) +at read-time by joining against `claim_acks` + `remittances` ++ the existing auto-link tables. We do not denormalize status +onto the `resubmissions` row, because that would create a +write-coordination problem between the SFTP push and the +inbound 999/277CA ingestion. + +**D9: Post-submission tracking joins reuse the existing +auto-link data, no new matching logic.** + +The 999 ack auto-link (SP28/31) already maps an inbound CLP +segment back to a `claim_id` via CLP01 + charge + service date. +Because the corrected-v2 regen preserves the original `claim_id` +from `raw_json.claim.claim_id`, the 999 ack for a resubmitted +file auto-links to the original claim row — and the new status +CLI just joins `resubmissions.claim_id` against `claim_acks.claim_id` +to surface the result. No new matching code is needed; this SP +adds the read-side join and the CLI surface, not a new matcher. + +**D10: The `resubmit-rejected-claims` CLI inserts `resubmissions` +rows; we do not add a new push CLI.** + +The existing `cyclone resubmit-rejected-claims` (per +`backend/src/cyclone/cli.py` and the SP33 followup work) is +the operator's only entry point for pushing corrected files via +SFTP. We instrument it to insert one `resubmissions` row per +claim per push (matching the same idempotency key as the SFTP +upload). No new CLI command for the push side. + +## 3. Open questions + +None. The operator confirmed the design via the brainstorming +Q&A on 2026-07-07: hard-fix the serializer with empty/SKCO0/CO_BHA +substitution to CO_TXIX (no CO_BHA preservation), regen into +corrected-v2/, add a post-submission tracking table + status CLI +(joined against existing auto-link data, no new matcher), file +a follow-up note for upstream tracing, no UI, no auto-resubmit. + +## 4. Test impact + +Per `cyclone-tests` (autouse conftest at `backend/tests/conftest.py`): + +- `backend/tests/test_serialize_837.py` — append three tests + targeting `_build_payer_block` via the public + `serialize_837_for_resubmit` entry point. Each test constructs + a minimal `ClaimOutput` whose `payer.id` is set to the value + under test, calls `serialize_837_for_resubmit`, and asserts the + emitted `NM1*PR` segment. Tests: + - `test_2010bb_normalizes_skco0_to_co_txix` — asserts + `SKCO0` → `CO_TXIX` in NM109 and `COHCPF` → `CO_TXIX` in + NM103. + - `test_2010bb_normalizes_empty_payer_id_to_co_txix` — + asserts empty `payer.id` → `CO_TXIX` and empty `payer.name` + → `CO_TXIX`. + - `test_2010bb_normalizes_co_bha_to_co_txix` — asserts + `CO_BHA` (with name `CO_BHA`) → `CO_TXIX` in both NM109 + and NM103. Plus a fourth assertion that a foreign payer id + (e.g. `"OTHER_PAYER"` with name `"OTHER PAYER NAME"`) is + emitted verbatim — the helper must not corrupt non-CO + submits. +- `backend/tests/test_resubmissions_cli.py` — new, tests the + `cyclone resubmissions status` CLI via `click.testing.CliRunner`. + Seeds the `resubmissions` table + `claim_acks` (via the + existing fixture pattern) and asserts: + - `pending_999` when no ack exists yet. + - `999_accepted` after a 999 ack with accept code is linked. + - `paid` after a remittance CLP links. + - Exit code 0 with empty `resubmissions` table prints + "no resubmissions recorded" and exits 0. +- `backend/tests/test_resubmissions_table.py` — new, tests the + SQLAlchemy model + idempotency of `(claim_id, interchange_control_number)`. + Asserts schema columns, asserts a second insert with the same + key is a no-op (raises IntegrityError or ON CONFLICT DO NOTHING, + matching the convention in the rest of `db.py`). +- `unbilled-july2026/scripts/test_regen_corrected_files.py` — + new, sibling test in the unbilled-july2026 project. Uses a + fixture of three fake-corrected files (one already correct, + one with SKCO0 from a degenerate raw_json, one with CO_BHA) + and asserts the regen script emits all three with the expected + normalization. +- Migration test impact: append to + `backend/tests/test_db_migrate.py` (or equivalent) a check + that migration 0013 creates the `resubmissions` table with + the documented columns + idempotency unique constraint. +- No frontend test impact. + +## 5. Files expected to change + +- `backend/src/cyclone/parsers/serialize_837.py` — add + `_normalize_payer_id(payer)`, call it from `_build_payer_block`, + emit WARNING log on substitution. +- `backend/src/cyclone/db.py` — add `Resubmission` SQLAlchemy + model. +- `backend/migrations/0013_resubmissions.py` — new migration + creating the `resubmissions` table. +- `backend/src/cyclone/cli.py` — instrument + `resubmit-rejected-claims` to insert one `Resubmission` row + per claim per push (idempotent on + `(claim_id, interchange_control_number)`); add the + `cyclone resubmissions status` subcommand. +- `backend/src/cyclone/store/__init__.py` — add + `CycloneStore.record_resubmission(...)` and + `CycloneStore.find_resubmission_status(...)` helpers; re-export + through the facade. +- `backend/tests/test_serialize_837.py` — append the three + regression tests. +- `backend/tests/test_resubmissions_cli.py` — new CLI tests. +- `backend/tests/test_resubmissions_table.py` — new model + + idempotency tests. +- `backend/tests/test_db_migrate.py` — append migration test + for 0013. +- `unbilled-july2026/scripts/regen_corrected_files.py` — new + sibling-project regen script. +- `unbilled-july2026/scripts/test_regen_corrected_files.py` — + new sibling-project test. +- `docs/RUNBOOK.md` — append "After SP39 lands" entry under + "Manual SFTP mode" pointing the operator at `corrected-v2/`, + the new `cyclone resubmissions status` CLI, and the workflow + to push via `cyclone resubmit-rejected-claims`. +- `backend/src/cyclone/submission/core.py` — append TODO comment + above `EXPECTED_PAYER_ID` referencing the future upstream- + trace ticket. +- `docs/superpowers/plans/2026-07-07-cyclone-2010bb-nm109-fix.md` + — the implementation plan, written after this spec is signed + off. + +## 6. Auth boundary + +The auth boundary is HTTP (login required, bcrypt + HttpOnly +session cookie); file-system threats remain the local-only threat +model (SQLCipher at rest, macOS Keychain). The serializer helper +is invoked server-side by both `serialize_837` (tested via +`cyclone submit-batch` / `POST /api/submit-batch`, both behind +HTTP auth) and `serialize_837_for_resubmit` (called by the +operator's regen workflow and by `/api/inbox/rejected/resubmit`, +also behind HTTP auth). The sibling-project regen script is +operator-invoked only and runs against a local file tree; it does +not touch the HTTP surface or the DB. No change to the threat +model. \ No newline at end of file