tyler/cyclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d44948cb90389708c2ba5c28a44106a0ce1defbc
cyclone/docs/superpowers/specs/2026-07-02-cyclone-rendering-npi-extraction-design.md
T

12 KiB
Raw Blame History

Status: Draft, awaiting user sign-off.

SP32 — 835/837P rendering & service-provider NPI extraction

Scope

In scope

  • Extract NM1*82 (rendering provider, Loop 2420A) from 837p files into a typed field on Claim.
  • Extract NM1*1P (service provider, Loop 2100) from 835 files into a typed field on ClaimPayment and an aggregated column on Remittance.
  • Persist all three new fields in their typed columns AND mirror to raw_json for audit and fallback reads.
  • Backfill existing rows by re-parsing on-disk files via a new CLI subcommand.
  • Re-reconcile open claim/remit pairs after backfill so the NPI arm of SP31's 2-of-3 content-match rule can actually fire on historical data.
  • Widen _content_keys_match in reconcile.py to read the new NPI sources (typed-column primary, raw_json fallback).

Out of scope

  • NPI Luhn validation or any other validation of format/length. Plain string, like the existing Claim.provider_npi.
  • UI surface — no changes to drawer/kpi/activity widgets. The new columns are exposed via the existing raw_json API paths; rendering them in the UI is a separate SP.
  • Multi-payer payer_id filtering — still deferred (SP31 documented deviation stands).
  • Spec-stricter lineage fields (CLP09 frequency, NM1*82 last-name vs organization-name semantics) beyond what the spec requires for matching.

Decisions

D1 — Both sides extracted (apples-to-apples)

The 837p parser today only captures the billing provider NPI (NM1*85). The 835 parser today discards NM1*1P entirely. SP32 extracts both: 837p NM1*82 → rendering NPI; 835 NM1*1P → service provider NPI. They are compared against each other in the matcher's NPI arm, not against Claim.provider_npi (billing).

D2 — Per-segment typed field on ClaimPayment

Add ClaimPayment.service_provider_npi: str | None to backend/src/cyclone/parsers/models_835.py. Length-10 string (NPI is 10 digits; no Luhn enforced). Empty/missing stored as None, not "".

D3 — Per-Claim typed column on Claim (837p)

Add Claim.rendering_provider_npi: Mapped[Optional[str]] = mapped_column(String(16), nullable=True) to backend/src/cyclone/db.py. Mirrors the existing Claim.provider_npi column. Schema migration 0016_add_rendering_and_service_provider_npis.sql adds both this and the next column.

D4 — Per-Remit column for the matcher

Add Remittance.rendering_provider_npi: Mapped[Optional[str]] = mapped_column(String(16), nullable=True). Today, each Remittance row corresponds to exactly one ClaimPayment (the write layer keys Remittance by cp.payer_claim_control_number; see store/write.py:134), so the column holds the single NPI for that segment (or None). The column is left unaggregated to keep the schema simple — if multi-CLP-per-Remit support is added in a future SP, the schema is large enough (String(16)) to upgrade to comma-joined storage without re-migration. The matcher's NPI arm reads this column directly.

D5 — raw_json mirrors

ClaimPayment is a parser wire-format model only — there is no claim_payments SQL table. The typed per-segment NPI persists through these mirrors:

  • Remittance.raw_json["service_provider_npis"]: list of every ClaimPayment.service_provider_npi, written verbatim at parse time. Source of truth for audit.
  • Claim.raw_json["rendering_provider_npi"]: mirror of the typed column.

These keep the existing raw_json fallback chain in _content_keys_match functional and provide a structured audit trail.

D6 — _content_keys_match NPI arm

Extend the existing matcher's NPI branch (currently a raw_json fallback chain) with a typed-column primary path:

  1. Read Claim.rendering_provider_npi (typed) and Remittance.rendering_provider_npi (typed).
  2. Split Remittance.rendering_provider_npi on "," and treat as a set (single-CLP produces a one-element set; multi-CLP produces N elements).
  3. If the claim's rendering NPI is present in that set, count rendering_provider_npi as a matched key.
  4. Fall back to the raw_json chain only when the typed columns are None or empty.

Effective algorithm unchanged: 2-of-3 {PCN, charge, NPI} still fires when any two keys agree. The NPI arm now actually contributes when the data is present.

D7 — Backfill via CLI

New subcommand: python -m cyclone.cli backfill-rendering-npi.

Behavior:

  1. For every claim_batches.inbound_path, re-parse the 837p file with the new parser; for every claim in the batch, write Claim.rendering_provider_npi = parsed_value only when the column is currently NULL (idempotent).
  2. For every remittances.inbound_path, re-parse the 835 file with the new parser; for every remit, write both the per-segment NPIs (ClaimPayment.service_provider_npi) and the aggregated Remittance.rendering_provider_npi.
  3. Run reconcile.run() once over all open pairs at the end so the NPI arm can fire retroactively.
  4. Log per-batch counts (parsed, updated, skipped).
  5. Exit code 0 on success, 2 on file-level failure (no batch updated), 1 on unexpected exception.

Idempotent: re-running on a fully-populated DB is a no-op (typed columns not overwritten; reconcile already matched pairs skipped via Claim.matched_remittance_id IS NOT NULL).

D8 — Backfill event emission

Each new score-auto match produced during post-backfill reconcile emits an auto_matched_835 ActivityEvent with payload {new_state, strategy, keys_matched, candidate_count} — same shape as SP31 D8. The audit trail explicitly captures backfill work; nothing is silent.

D9 — Migration is forward-only, additive

Migration 0019_add_rendering_and_service_provider_npis.sql adds:

ALTER TABLE claims ADD COLUMN rendering_provider_npi TEXT;
ALTER TABLE remittances ADD COLUMN rendering_provider_npi TEXT;

ClaimPayment.service_provider_npi has no SQL column because claim_payments has no SQL table — the typed value lives in Remittance.raw_json["service_provider_npis"].

No UPDATE statements in the migration — backfill is a separate, deliberate step via the CLI. No indexes — both columns are nullable, used for set-equality at match time, not range queries. Scale (Colorado Medicaid single-payer) doesn't justify index write cost.

D10 — File paths and code shape

  • backend/src/cyclone/parsers/parse_835.py — extend _consume_claim_payment to handle the NM1 child segment: parse NM1*1P, capture NM109 as service_provider_npi for the enclosing CLP. Other NM1 qualifiers (QC, IL, PR, 85, 87) within the CLP loop are kept in raw_segments (unchanged behavior).
  • backend/src/cyclone/parsers/parse_837.py — add a new _consume_rendering_provider (or analogous) function for Loop 2420A; wire into the existing claim assembly path so the rendering NPI lands on Claim.
  • backend/src/cyclone/parsers/models_835.py — extend ClaimPayment with service_provider_npi.
  • backend/src/cyclone/db.py — Claim.rendering_provider_npi, Remittance.rendering_provider_npi. No claim_payments ORM — service_provider_npi persists only via Remittance.raw_json["service_provider_npis"] (D5).
  • backend/src/cyclone/writer.py / writer_835.py — set the new typed columns and the raw_json mirrors at write time. writer_835.py writes the per-CLP service_provider_npi into Remittance.rendering_provider_npi (D4, single value) and into raw_json["service_provider_npis"] (D5).
  • backend/src/cyclone/reconcile.py — extend _content_keys_match (D6).
  • backend/src/cyclone/cli.py — new backfill-rendering-npi subcommand.
  • backend/src/cyclone/migrations/0019_add_rendering_and_service_provider_npis.sql — D9.
  • backend/tests/fixtures/ — new 837p fixture(s) with NM1*82 and new 835 fixture(s) with NM1*1P.
  • backend/tests/test_reconcile.py — new tests for D6 (set-membership, comma-split, fallback chain).
  • backend/tests/test_parse_835.py — new test for _consume_claim_payment NM1*1P extraction.
  • backend/tests/test_parse_837.py — new test for rendering-provider loop extraction.

Edge cases

Case Behavior
NM1*1P missing or 1P qualifier absent in a CLP segment ClaimPayment.service_provider_npi = None. Matcher's NPI arm simply doesn't fire for that key.
NM1*82 missing or empty in an 837p claim Claim.rendering_provider_npi = None. Same graceful degrade.
NM1*1P with NM109 empty Treat as missing — store None, never empty string.
Multi-CLP per Remit in a future SP Schema-size forward note (D4). Today, each Remit = one CLP = single NPI, no aggregation needed.
837p rendering NPI == billing NPI (solo practice) Both columns populated independently. Matcher compares against the service-provider NPI; identity is coincidence in this case.
Backfill on a deleted-on-disk file CLI logs a WARNING, skips the batch. Typed columns stay NULL for unmatched rows. Existing match state untouched.
Re-reconcile creates new score-auto matches Each emits an auto_matched_835 ActivityEvent per SP31 D8 (D8). Audit trail is explicit.
Idempotent re-run Typed columns written only when currently NULL; reconcile skips already-matched pairs. backfill-rendering-npi exit code 0.
Existing 115 SP31 tests All pass; widened matcher signature preserves the boolean-return path used today.

Testing

Unit tests (parser-level)

  • test_parse_835.py: NM1*1P extraction — claim with NM1*1P*2*...*XX*1234567890 populates ClaimPayment.service_provider_npi == "1234567890". Multi-CLP with mixed NPIs produces per-segment values correctly.
  • test_parse_835.py: NM1*1P absent — service_provider_npi is None, no exception.
  • test_parse_837.py: NM1*82 extraction — Loop 2420A rendering provider populates Claim.rendering_provider_npi.
  • test_parse_837.py: 837p without NM1*82 — column None.

Unit tests (matcher level)

  • test_reconcile.py: comma-list set membership — single-CLP-remit with Remittance.rendering_provider_npi = "1234567890,0987654321" and Claim.rendering_provider_npi = "0987654321" counts NPI as a matched key.
  • test_reconcile.py: multi-key fallback — NM1*1P NPI absent (None) on one side, present on the other → NPI arm does not fire, PCN+charge still matches (1-of-2 effective, but per SP31 the rule is 2-of-3 of {PCN, charge, NPI}, so PCN+charge alone does NOT auto-link — must stay 2 of 3 with NPI also matching or charge differing).
  • test_reconcile.py: typed-column primary path beats raw_json fallback — fixture sets both, matcher uses typed column.
  • test_reconcile.py: existing 22+ SP31 tests still pass without modification.

Integration tests

  • test_backfill_cli.py: synthetic batch + remit in DB, fixture files on disk, run backfill-rendering-npi — assert columns populated, exit code 0, ActivityEvents emitted for any new matches.
  • test_backfill_idempotent.py: run twice, second run is no-op.
  • test_backfill_missing_file.py: deleted file path → WARNING log, no crash, other batches still processed.

Regression

  • Full backend pytest (target: 115 existing + ~8 new = ~123 total, all passing).
  • Frontend typecheck: no new errors (no new types; matcher's typed return is internal).
  • Manual smoke: ingest a real 835 via CLI, query the DB, confirm Remittance.rendering_provider_npi populated and any retroactive score-auto Match row exists.

Out of scope (deferred to follow-up SPs)

  • NPI Luhn validation — separately enforced if/when claim submission rejects malformed NPIs upstream.
  • UI rendering of rendering/service-provider NPI in ClaimDrawer / RemitDrawer — separate SP.
  • Multi-payer payer_id filter restoration — SP31 documented deviation; needs schema migration or raw_json read.
  • Per-segment NPI display in RemitDrawer's CLP rows — separate SP.
Reference in New Issue View Git Blame Copy Permalink