docs(spec): SP32 rendering & service-provider NPI extraction
This commit is contained in:
Nora
@@ -0,0 +1,153 @@
|
||||
Status: Draft, awaiting user sign-off.
|
||||
|
||||
# SP32 — 835/837P rendering & service-provider NPI extraction
|
||||
|
||||
## Scope
|
||||
|
||||
### In scope
|
||||
|
||||
- **Extract** `NM1*82` (rendering provider, Loop 2420A) from 837p files into a typed field on `Claim`.
|
||||
- **Extract** `NM1*1P` (service provider, Loop 2100) from 835 files into a typed field on `ClaimPayment` and an aggregated column on `Remittance`.
|
||||
- **Persist** all three new fields in their typed columns AND mirror to `raw_json` for audit and fallback reads.
|
||||
- **Backfill** existing rows by re-parsing on-disk files via a new CLI subcommand.
|
||||
- **Re-reconcile** open claim/remit pairs after backfill so the NPI arm of SP31's 2-of-3 content-match rule can actually fire on historical data.
|
||||
- **Widen** `_content_keys_match` in `reconcile.py` to read the new NPI sources (typed-column primary, `raw_json` fallback) and to treat a comma-separated `Remittance.rendering_provider_npi` as a set.
|
||||
|
||||
### Out of scope
|
||||
|
||||
- NPI Luhn validation or any other validation of format/length. Plain string, like the existing `Claim.provider_npi`.
|
||||
- UI surface — no changes to drawer/kpi/activity widgets. The new columns are exposed via the existing `raw_json` API paths; rendering them in the UI is a separate SP.
|
||||
- Multi-payer `payer_id` filtering — still deferred (SP31 documented deviation stands).
|
||||
- Spec-stricter lineage fields (CLP09 frequency, NM1*82 last-name vs organization-name semantics) beyond what the spec requires for matching.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1 — Both sides extracted (apples-to-apples)
|
||||
|
||||
The 837p parser today only captures the **billing** provider NPI (`NM1*85`). The 835 parser today discards `NM1*1P` entirely. SP32 extracts both: 837p `NM1*82` → rendering NPI; 835 `NM1*1P` → service provider NPI. They are compared against each other in the matcher's NPI arm, not against `Claim.provider_npi` (billing).
|
||||
|
||||
### D2 — Per-segment typed field on `ClaimPayment`
|
||||
|
||||
Add `ClaimPayment.service_provider_npi: str | None` to `backend/src/cyclone/parsers/models_835.py`. Length-10 string (NPI is 10 digits; no Luhn enforced). Empty/missing stored as `None`, not `""`.
|
||||
|
||||
### D3 — Per-Claim typed column on `Claim` (837p)
|
||||
|
||||
Add `Claim.rendering_provider_npi: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)` to `backend/src/cyclone/db.py`. Mirrors the existing `Claim.provider_npi` column. Schema migration `0016_add_rendering_and_service_provider_npis.sql` adds both this and the next column.
|
||||
|
||||
### D4 — Per-Remit aggregated column for the matcher
|
||||
|
||||
Add `Remittance.rendering_provider_npi: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)` (length-64 accommodates comma-separated lists up to ~6 NPI strings with separators). For multi-CLP remits, the writer concatenates each `ClaimPayment.service_provider_npi` with `","` as the separator (e.g. `"1234567890,0987654321"`). Single-CLP remits store the single NPI as-is. `None` means no CLP in the remit had a service-provider NPI.
|
||||
|
||||
### D5 — `raw_json` mirrors
|
||||
|
||||
`ClaimPayment` is a parser wire-format model only — there is no `claim_payments` SQL table. The typed per-segment NPI persists through these mirrors:
|
||||
|
||||
- `Remittance.raw_json["service_provider_npis"]`: list of every `ClaimPayment.service_provider_npi`, written verbatim at parse time. Source of truth for audit.
|
||||
- `Claim.raw_json["rendering_provider_npi"]`: mirror of the typed column.
|
||||
|
||||
These keep the existing `raw_json` fallback chain in `_content_keys_match` functional and provide a structured audit trail.
|
||||
|
||||
### D6 — `_content_keys_match` NPI arm
|
||||
|
||||
Extend the existing matcher's NPI branch (currently a raw_json fallback chain) with a typed-column primary path:
|
||||
|
||||
1. Read `Claim.rendering_provider_npi` (typed) and `Remittance.rendering_provider_npi` (typed).
|
||||
2. Split `Remittance.rendering_provider_npi` on `","` and treat as a set (single-CLP produces a one-element set; multi-CLP produces N elements).
|
||||
3. If the claim's rendering NPI is present in that set, count `rendering_provider_npi` as a matched key.
|
||||
4. Fall back to the `raw_json` chain only when the typed columns are `None` or empty.
|
||||
|
||||
Effective algorithm unchanged: 2-of-3 `{PCN, charge, NPI}` still fires when any two keys agree. The NPI arm now actually contributes when the data is present.
|
||||
|
||||
### D7 — Backfill via CLI
|
||||
|
||||
New subcommand: `python -m cyclone.cli backfill-rendering-npi`.
|
||||
|
||||
Behavior:
|
||||
1. For every `claim_batches.inbound_path`, re-parse the 837p file with the new parser; for every claim in the batch, write `Claim.rendering_provider_npi = parsed_value` **only when the column is currently `NULL`** (idempotent).
|
||||
2. For every `remittances.inbound_path`, re-parse the 835 file with the new parser; for every remit, write both the per-segment NPIs (`ClaimPayment.service_provider_npi`) and the aggregated `Remittance.rendering_provider_npi`.
|
||||
3. Run `reconcile.run()` once over all open pairs at the end so the NPI arm can fire retroactively.
|
||||
4. Log per-batch counts (parsed, updated, skipped).
|
||||
5. Exit code 0 on success, 2 on file-level failure (no batch updated), 1 on unexpected exception.
|
||||
|
||||
Idempotent: re-running on a fully-populated DB is a no-op (typed columns not overwritten; reconcile already matched pairs skipped via `Claim.matched_remittance_id IS NOT NULL`).
|
||||
|
||||
### D8 — Backfill event emission
|
||||
|
||||
Each new `score-auto` match produced during post-backfill reconcile emits an `auto_matched_835` ActivityEvent with payload `{new_state, strategy, keys_matched, candidate_count}` — same shape as SP31 D8. The audit trail explicitly captures backfill work; nothing is silent.
|
||||
|
||||
### D9 — Migration is forward-only, additive
|
||||
|
||||
Migration `0019_add_rendering_and_service_provider_npis.sql` adds:
|
||||
|
||||
```sql
|
||||
ALTER TABLE claims ADD COLUMN rendering_provider_npi TEXT;
|
||||
ALTER TABLE remittances ADD COLUMN rendering_provider_npi TEXT;
|
||||
```
|
||||
|
||||
`ClaimPayment.service_provider_npi` has no SQL column because `claim_payments` has no SQL table — the typed value lives in `Remittance.raw_json["service_provider_npis"]`.
|
||||
|
||||
No `UPDATE` statements in the migration — backfill is a separate, deliberate step via the CLI. No indexes — both columns are nullable, used for set-equality at match time, not range queries. Scale (Colorado Medicaid single-payer) doesn't justify index write cost.
|
||||
|
||||
### D10 — File paths and code shape
|
||||
|
||||
- `backend/src/cyclone/parsers/parse_835.py` — extend `_consume_claim_payment` to handle the `NM1` child segment: parse `NM1*1P`, capture `NM109` as `service_provider_npi` for the enclosing CLP. Other `NM1` qualifiers (`QC`, `IL`, `PR`, `85`, `87`) within the CLP loop are kept in `raw_segments` (unchanged behavior).
|
||||
- `backend/src/cyclone/parsers/parse_837.py` — add a new `_consume_rendering_provider` (or analogous) function for Loop 2420A; wire into the existing claim assembly path so the rendering NPI lands on `Claim`.
|
||||
- `backend/src/cyclone/parsers/models_835.py` — extend `ClaimPayment` with `service_provider_npi`.
|
||||
- `backend/src/cyclone/db.py` — `Claim.rendering_provider_npi`, `Remittance.rendering_provider_npi`. No claim_payments ORM — service_provider_npi persists only via `Remittance.raw_json["service_provider_npis"]` (D5).
|
||||
- `backend/src/cyclone/writer.py` / `writer_835.py` — set the new typed columns and the `raw_json` mirrors at write time. `writer_835.py` aggregates per-CLPs into `Remittance.rendering_provider_npi` (comma-separated, D4) and emits the `service_provider_npis` list into `raw_json` (D5).
|
||||
- `backend/src/cyclone/reconcile.py` — extend `_content_keys_match` (D6).
|
||||
- `backend/src/cyclone/cli.py` — new `backfill-rendering-npi` subcommand.
|
||||
- `backend/src/cyclone/migrations/0019_add_rendering_and_service_provider_npis.sql` — D9.
|
||||
- `backend/tests/fixtures/` — new 837p fixture(s) with `NM1*82` and new 835 fixture(s) with `NM1*1P`.
|
||||
- `backend/tests/test_reconcile.py` — new tests for D6 (set-membership, comma-split, fallback chain).
|
||||
- `backend/tests/test_parse_835.py` — new test for `_consume_claim_payment` `NM1*1P` extraction.
|
||||
- `backend/tests/test_parse_837.py` — new test for rendering-provider loop extraction.
|
||||
|
||||
## Edge cases
|
||||
|
||||
| Case | Behavior |
|
||||
|---|---|
|
||||
| `NM1*1P` missing or `1P` qualifier absent in a CLP segment | `ClaimPayment.service_provider_npi = None`. Matcher's NPI arm simply doesn't fire for that key. |
|
||||
| `NM1*82` missing or empty in an 837p claim | `Claim.rendering_provider_npi = None`. Same graceful degrade. |
|
||||
| `NM1*1P` with `NM109` empty | Treat as missing — store `None`, never empty string. |
|
||||
| Multi-CLP remit with mixed NPIs | Writer aggregates `"<npi1>,<npi2>,..."` on `Remittance.rendering_provider_npi`. Per-segment NPIs still on `ClaimPayment.service_provider_npi`. |
|
||||
| 837p rendering NPI == billing NPI (solo practice) | Both columns populated independently. Matcher compares against the service-provider NPI; identity is coincidence in this case. |
|
||||
| Backfill on a deleted-on-disk file | CLI logs a WARNING, skips the batch. Typed columns stay NULL for unmatched rows. Existing match state untouched. |
|
||||
| Re-reconcile creates new `score-auto` matches | Each emits an `auto_matched_835` ActivityEvent per SP31 D8 (D8). Audit trail is explicit. |
|
||||
| Idempotent re-run | Typed columns written only when currently NULL; reconcile skips already-matched pairs. `backfill-rendering-npi` exit code 0. |
|
||||
| Existing 115 SP31 tests | All pass; widened matcher signature preserves the boolean-return path used today. |
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit tests (parser-level)
|
||||
|
||||
- `test_parse_835.py`: `NM1*1P` extraction — claim with `NM1*1P*2*...*XX*1234567890` populates `ClaimPayment.service_provider_npi == "1234567890"`. Multi-CLP with mixed NPIs produces per-segment values correctly.
|
||||
- `test_parse_835.py`: `NM1*1P` absent — `service_provider_npi is None`, no exception.
|
||||
- `test_parse_837.py`: `NM1*82` extraction — Loop 2420A rendering provider populates `Claim.rendering_provider_npi`.
|
||||
- `test_parse_837.py`: 837p without `NM1*82` — column `None`.
|
||||
|
||||
### Unit tests (matcher level)
|
||||
|
||||
- `test_reconcile.py`: comma-list set membership — single-CLP-remit with `Remittance.rendering_provider_npi = "1234567890,0987654321"` and `Claim.rendering_provider_npi = "0987654321"` counts NPI as a matched key.
|
||||
- `test_reconcile.py`: multi-key fallback — `NM1*1P` NPI absent (`None`) on one side, present on the other → NPI arm does not fire, PCN+charge still matches (1-of-2 effective, but per SP31 the rule is 2-of-3 of {PCN, charge, NPI}, so PCN+charge alone does NOT auto-link — must stay 2 of 3 with NPI also matching or charge differing).
|
||||
- `test_reconcile.py`: typed-column primary path beats `raw_json` fallback — fixture sets both, matcher uses typed column.
|
||||
- `test_reconcile.py`: existing 22+ SP31 tests still pass without modification.
|
||||
|
||||
### Integration tests
|
||||
|
||||
- `test_backfill_cli.py`: synthetic batch + remit in DB, fixture files on disk, run `backfill-rendering-npi` — assert columns populated, exit code 0, ActivityEvents emitted for any new matches.
|
||||
- `test_backfill_idempotent.py`: run twice, second run is no-op.
|
||||
- `test_backfill_missing_file.py`: deleted file path → WARNING log, no crash, other batches still processed.
|
||||
|
||||
### Regression
|
||||
|
||||
- Full backend `pytest` (target: 115 existing + ~8 new = ~123 total, all passing).
|
||||
- Frontend `typecheck`: no new errors (no new types; matcher's typed return is internal).
|
||||
- Manual smoke: ingest a real 835 via CLI, query the DB, confirm `Remittance.rendering_provider_npi` populated and any retroactive `score-auto` Match row exists.
|
||||
|
||||
## Out of scope (deferred to follow-up SPs)
|
||||
|
||||
- **NPI Luhn validation** — separately enforced if/when claim submission rejects malformed NPIs upstream.
|
||||
- **UI rendering** of rendering/service-provider NPI in ClaimDrawer / RemitDrawer — separate SP.
|
||||
- **Multi-payer `payer_id` filter restoration** — SP31 documented deviation; needs schema migration or raw_json read.
|
||||
- **Per-segment NPI display in RemitDrawer's CLP rows** — separate SP.
|
||||
Reference in New Issue
Block a user