docs(spec): SP28 D10 — two-pass join via Batch.envelope.control_number
The original SP28 spec assumed `Claim.patient_control_number == 999.set_control_number`. Empirically that's wrong: Gainwell's 999 echoes the source 837's ST02, not its CLM01, and TOC's billing software fills them differently. Measured against prod on 2026-07-02: the PCN join matches 0 / 1,398 acks. Fix: two-pass join. Primary is `Batch.envelope.control_number (== 837 ST02) → claims via batch_id`; fallback is `Claim.patient_control_number` (for senders that fill CLM01 == ST02). Coverage after fix: 727 / 1,398 (52%); the remaining 671 are real orphans (ST02=0001 placeholder, no matching 837 batch in our DB). Spec adds D10 + a critical-correction paragraph at the top of §1. Plan gains the `lookup_claims_for_ack_set_response` pure helper, the `batch_envelope_index()` store method, and three new tests covering the two-pass join + the false-positive guard + the one-ack-to-many case.
This commit is contained in:
@@ -7,7 +7,9 @@
|
|||||||
|
|
||||||
**Goal:** Persist a durable link row (`claim_acks`) for every inbound 999 / 277CA / TA1 acknowledgment at parse time, so the operator can answer "which claims does this ack acknowledge?" and "which acks does this claim have?" without manual correlation, and surface those links in `ClaimDrawer` + `AckDrawer` + a new Inbox orphan lane.
|
**Goal:** Persist a durable link row (`claim_acks`) for every inbound 999 / 277CA / TA1 acknowledgment at parse time, so the operator can answer "which claims does this ack acknowledge?" and "which acks does this claim have?" without manual correlation, and surface those links in `ClaimDrawer` + `AckDrawer` + a new Inbox orphan lane.
|
||||||
|
|
||||||
**Architecture:** A new `claim_acks` join table (one row per AK2 set-response for 999, per ClaimStatus for 277CA, per TA1 envelope) is the durable record. Three pure helpers (`apply_999_acceptances`, `apply_277ca_acks`, `apply_ta1_envelope_link`) run inside the existing handler transactions alongside the existing rejection logic. The orchestrator (`apply_claim_ack_links`) is called from the same point in `handle_999.handle` / the 277CA handler / the TA1 handler. A new `claim_ack_written` pubsub event drives two new stream endpoints (`/api/claims/{id}/acks/stream`, `/api/acks/{kind}/{id}/claims/stream`) so both drawers update in real time. Frontend mirrors the SP25 acks/ta1_acks live-tail extension pattern (`useTailStore` slice + `useTailStream` + `useMergedTail`) plus a manual-match dropdown for the orphan case.
|
**Architecture:** A new `claim_acks` join table (one row per AK2 set-response for 999, per ClaimStatus for 277CA, per TA1 envelope) is the durable record. Three pure helpers (`apply_999_acceptances`, `apply_277ca_acks`, `apply_ta1_envelope_link`) run inside the existing handler transactions alongside the existing rejection logic. The orchestrator (`apply_claim_ack_links`) is called from the same point in `handle_999.handle` / the 277CA handler / the TA1 handler. The auto-linker's per-AK2 join uses a two-pass lookup (D10 in the spec): primary is `Batch.envelope.control_number (== 837 ST02) → claims in batch via batch_id`; fallback is `Claim.patient_control_number`. A new `claim_ack_written` pubsub event drives two new stream endpoints (`/api/claims/{id}/acks/stream`, `/api/acks/{kind}/{id}/claims/stream`) so both drawers update in real time. Frontend mirrors the SP25 acks/ta1_acks live-tail extension pattern (`useTailStore` slice + `useTailStream` + `useMergedTail`) plus a manual-match dropdown for the orphan case.
|
||||||
|
|
||||||
|
**Critical-path correction (added 2026-07-02):** the original plan's `_lookup` closure queried `Claim.patient_control_number == set_control_number`. That join is empirically 0-effective on this codebase (1,398 acks, 0 matched) because Gainwell's 999 echoes the 837's `ST02` not its `CLM01`. The corrected plan uses `Batch.envelope.control_number` (which holds the 837's `ST02`) as the primary join key, with `Claim.patient_control_number` as a fallback for the rare case where the sender filled `CLM01 == ST02`. The corrected join recovers 727 / 1,398 acks automatically (52%); the remaining 671 have ST02=`0001` (no matching 837 batch) and remain real orphans surfaced by the Inbox ack-orphans lane.
|
||||||
|
|
||||||
**Tech Stack:** Python 3.11+, FastAPI, SQLAlchemy 2.x, SQLite (encrypted via SQLCipher), React 18 + TypeScript + Vite, TanStack Query, Zustand, Radix UI primitives.
|
**Tech Stack:** Python 3.11+, FastAPI, SQLAlchemy 2.x, SQLite (encrypted via SQLCipher), React 18 + TypeScript + Vite, TanStack Query, Zustand, Radix UI primitives.
|
||||||
|
|
||||||
@@ -164,7 +166,7 @@ src/
|
|||||||
|
|
||||||
- [ ] **Step 2.1: Create `backend/src/cyclone/claim_acks.py` with the public API.**
|
- [ ] **Step 2.1: Create `backend/src/cyclone/claim_acks.py` with the public API.**
|
||||||
|
|
||||||
Three public functions + one dataclass result. The module is pure (no DB session ownership — callers pass in the session, just like `apply_999_rejections`).
|
Four public functions + one dataclass result. The module is pure (no DB session ownership — callers pass in the session, just like `apply_999_rejections`). The fourth function is the two-pass join helper from D10.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
"""SP28: per-ACK auto-linker.
|
"""SP28: per-ACK auto-linker.
|
||||||
@@ -191,12 +193,41 @@ src/
|
|||||||
# set_control_numbers we couldn't resolve to a claim
|
# set_control_numbers we couldn't resolve to a claim
|
||||||
|
|
||||||
|
|
||||||
|
def lookup_claims_for_ack_set_response(
|
||||||
|
session: Session,
|
||||||
|
set_control_number: str,
|
||||||
|
*,
|
||||||
|
batch_envelope_index: Callable[[str], str | None] | None = None,
|
||||||
|
) -> list[Claim]:
|
||||||
|
"""D10: two-pass join for a single AK2 set_response / 277CA claim_status.
|
||||||
|
|
||||||
|
Returns 0..N matching claims (one-ack-to-many when one 837 batch
|
||||||
|
shipped multiple claims under one ST02).
|
||||||
|
|
||||||
|
Pass 1 (primary): ``Batch.envelope.control_number == set_control_number``.
|
||||||
|
If the caller supplies a pre-built ``batch_envelope_index`` (a callable
|
||||||
|
mapping ST02 -> batch.id, built once at the start of the ingest),
|
||||||
|
we skip the per-set-response ``Batch`` scan entirely. If no index
|
||||||
|
is supplied we fall back to a session.query() against ``Batch``.
|
||||||
|
|
||||||
|
Pass 2 (fallback): ``Claim.patient_control_number == set_control_number``.
|
||||||
|
Catches the case where the sender filled CLM01 == ST02 (rare;
|
||||||
|
Gainwell's TOC batches do not).
|
||||||
|
|
||||||
|
Pass 1 wins. The two paths cannot both fire — if Pass 1 returns
|
||||||
|
one or more claims, Pass 2 is skipped. This is the false-positive
|
||||||
|
guard from spec §7.
|
||||||
|
"""
|
||||||
|
...
|
||||||
|
|
||||||
|
|
||||||
def apply_999_acceptances(
|
def apply_999_acceptances(
|
||||||
session: Session,
|
session: Session,
|
||||||
parsed_999,
|
parsed_999,
|
||||||
*,
|
*,
|
||||||
ack_id: int,
|
ack_id: int,
|
||||||
claim_lookup: Callable[[str], Claim | None],
|
batch_envelope_index: Callable[[str], str | None] | None = None,
|
||||||
|
pc_claim_lookup: Callable[[str], Claim | None] | None = None,
|
||||||
) -> ClaimAckLinkResult:
|
) -> ClaimAckLinkResult:
|
||||||
"""For every AK2 set-response, create a claim_acks row.
|
"""For every AK2 set-response, create a claim_acks row.
|
||||||
|
|
||||||
@@ -204,6 +235,10 @@ src/
|
|||||||
show the rejection inline). Orphans are returned but not linked.
|
show the rejection inline). Orphans are returned but not linked.
|
||||||
Idempotent: if (claim_id, '999', ack_id, ak2_index) already exists,
|
Idempotent: if (claim_id, '999', ack_id, ak2_index) already exists,
|
||||||
skip silently.
|
skip silently.
|
||||||
|
|
||||||
|
The two lookup arguments replace the single ``claim_lookup`` from
|
||||||
|
the original spec — D10 split the join into a primary (ST02 via
|
||||||
|
batch envelope index) and a fallback (PCN match). See spec §D10.
|
||||||
"""
|
"""
|
||||||
...
|
...
|
||||||
|
|
||||||
@@ -212,11 +247,14 @@ src/
|
|||||||
parsed_277ca,
|
parsed_277ca,
|
||||||
*,
|
*,
|
||||||
ack_id: int,
|
ack_id: int,
|
||||||
claim_lookup: Callable[[str], Claim | None],
|
batch_envelope_index: Callable[[str], str | None] | None = None,
|
||||||
|
pc_claim_lookup: Callable[[str], Claim | None] | None = None,
|
||||||
) -> ClaimAckLinkResult:
|
) -> ClaimAckLinkResult:
|
||||||
"""For every ClaimStatus with a payer_claim_control_number,
|
"""For every ClaimStatus with a payer_claim_control_number,
|
||||||
create a claim_acks row. Accepted AND rejected both link.
|
create a claim_acks row. Accepted AND rejected both link.
|
||||||
Idempotent: (claim_id, '277ca', ack_id, NULL) is the dedup key.
|
Idempotent: (claim_id, '277ca', ack_id, NULL) is the dedup key.
|
||||||
|
|
||||||
|
Same two-pass lookup as ``apply_999_acceptances`` (D10).
|
||||||
"""
|
"""
|
||||||
...
|
...
|
||||||
|
|
||||||
@@ -239,22 +277,23 @@ src/
|
|||||||
- [ ] **Step 2.2: Implement `apply_999_acceptances`.**
|
- [ ] **Step 2.2: Implement `apply_999_acceptances`.**
|
||||||
|
|
||||||
Walk `parsed_999.set_responses` (each has `.set_control_number` + `.set_accept_reject.code`). For each:
|
Walk `parsed_999.set_responses` (each has `.set_control_number` + `.set_accept_reject.code`). For each:
|
||||||
1. Call `claim_lookup(set_control_number)`. If None → `result.orphans.append(set_control_number)`, continue.
|
1. Call `lookup_claims_for_ack_set_response(session, sr.set_control_number, batch_envelope_index=batch_envelope_index)` (D10 — two-pass join: ST02-via-batch primary, PCN fallback). If `[]` → `result.orphans.append(sr.set_control_number)`, continue.
|
||||||
2. Check for existing `claim_acks` row with `(claim_id, '999', ack_id, ak2_index=i)`. If exists → skip silently (idempotent).
|
2. For each matched `claim`:
|
||||||
3. Insert `ClaimAck(claim_id=claim.id, ack_id=ack_id, ack_kind='999', ak2_index=i, set_control_number=pcn, set_accept_reject_code=code, linked_at=utcnow(), linked_by='auto')`.
|
- Check for existing `claim_acks` row with `(claim.id, '999', ack_id, ak2_index=i)`. If exists → skip silently (idempotent).
|
||||||
4. Append `(claim.id, i)` to `result.linked`.
|
- Insert `ClaimAck(claim_id=claim.id, ack_id=ack_id, ack_kind='999', ak2_index=i, set_control_number=sr.set_control_number, set_accept_reject_code=sr.set_accept_reject.code, linked_at=utcnow(), linked_by='auto')`.
|
||||||
5. `session.flush()` at the end (do NOT commit — the caller owns the transaction).
|
- Append `(claim.id, i)` to `result.linked`.
|
||||||
|
3. `session.flush()` at the end (do NOT commit — the caller owns the transaction).
|
||||||
|
|
||||||
Insert order matters: walk set_responses in order, capture `i = enumerate index`.
|
Insert order matters: walk set_responses in order, capture `i = enumerate index`. One AK2 can produce **multiple** `claim_acks` rows when the 837 batch carried more than one claim under a shared ST02 (rare on this codebase but supported by the schema and the join).
|
||||||
|
|
||||||
- [ ] **Step 2.3: Implement `apply_277ca_acks`.**
|
- [ ] **Step 2.3: Implement `apply_277ca_acks`.**
|
||||||
|
|
||||||
Walk `parsed_277ca.claim_statuses`. For each:
|
Walk `parsed_277ca.claim_statuses`. For each:
|
||||||
1. Get `pcn = status.payer_claim_control_number`. If empty/None → orphan with `status.status_code or ""`.
|
1. Get `pcn = status.payer_claim_control_number`. If empty/None → orphan with `status.status_code or ""`.
|
||||||
2. Call `claim_lookup(pcn)`. If None → orphan.
|
2. Call `lookup_claims_for_ack_set_response(session, pcn, batch_envelope_index=...)` (D10 — same two-pass join as 999, except the lookup key is `payer_claim_control_number` from the 277CA instead of `set_control_number` from the 999).
|
||||||
3. Check dedup: `(claim_id, '277ca', ack_id, NULL)`.
|
3. For each matched `claim`: check dedup `(claim.id, '277ca', ack_id, NULL)`, insert `ClaimAck`, append to result.
|
||||||
4. Insert `ClaimAck(claim_id=claim.id, ack_id=ack_id, ack_kind='277ca', ak2_index=NULL, set_control_number=pcn, set_accept_reject_code=status.status_code, linked_at=utcnow(), linked_by='auto')`.
|
4. If `[]` from the join → orphan.
|
||||||
5. Append to result. Flush.
|
5. Flush.
|
||||||
|
|
||||||
- [ ] **Step 2.4: Implement `apply_ta1_envelope_link`.**
|
- [ ] **Step 2.4: Implement `apply_ta1_envelope_link`.**
|
||||||
|
|
||||||
@@ -273,6 +312,28 @@ src/
|
|||||||
# Build a stub parsed_999 with 2 set_responses, mock claim_lookup
|
# Build a stub parsed_999 with 2 set_responses, mock claim_lookup
|
||||||
# that returns a claim for the first PCN and None for the second.
|
# that returns a claim for the first PCN and None for the second.
|
||||||
# Assert: result.linked == [(claim.id, 0)], result.orphans == ['<second-pcn>'].
|
# Assert: result.linked == [(claim.id, 0)], result.orphans == ['<second-pcn>'].
|
||||||
|
|
||||||
|
def test_lookup_claims_two_pass_join():
|
||||||
|
# Seed two Claims, one with batch.envelope.control_number='991102989'
|
||||||
|
# and patient_control_number='t991102989o1c120d', and a separate
|
||||||
|
# claim with patient_control_number='991102987'.
|
||||||
|
# Call lookup_claims_for_ack_set_response(session, '991102989')
|
||||||
|
# — assert the first claim is returned via Pass 1 (batch).
|
||||||
|
# Call with '991102987' — assert the second claim is returned via Pass 2 (PCN).
|
||||||
|
# Call with '0001' (no match) — assert [] returned.
|
||||||
|
|
||||||
|
def test_lookup_claims_pass1_wins_over_pass2():
|
||||||
|
# If a ST02 collides with a PCN from a different batch, Pass 1
|
||||||
|
# must win and Pass 2 must NOT also fire. Seed:
|
||||||
|
# Claim A: batch_id=B1, patient_control_number='991102989'
|
||||||
|
# Claim B: batch_id=B2 (with envelope.control_number='OTHER'),
|
||||||
|
# patient_control_number='991102989'
|
||||||
|
# Call lookup('991102989') — assert [Claim A] only (Pass 1 wins).
|
||||||
|
|
||||||
|
def test_999_linker_emits_one_row_per_claim_in_multi_claim_batch():
|
||||||
|
# Seed Batch B with ST02='991102990' and TWO claims (batch_id=B).
|
||||||
|
# Submit a 999 with one AK2 set_control_number='991102990'.
|
||||||
|
# Assert result.linked == [(claim1.id, 0), (claim2.id, 0)].
|
||||||
```
|
```
|
||||||
|
|
||||||
Run `pytest backend/tests/test_apply_claim_ack_links.py -v` — should now PASS.
|
Run `pytest backend/tests/test_apply_claim_ack_links.py -v` — should now PASS.
|
||||||
@@ -285,7 +346,9 @@ src/
|
|||||||
|
|
||||||
- [ ] **Step 3.1: Create `backend/src/cyclone/store/claim_acks.py` with the persistence layer.**
|
- [ ] **Step 3.1: Create `backend/src/cyclone/store/claim_acks.py` with the persistence layer.**
|
||||||
|
|
||||||
Mirrors `store/acks.py` shape: `add_claim_ack`, `list_acks_for_claim`, `list_claims_for_ack`, `find_ack_orphans`, `remove_claim_ack`. Each opens its own `db.SessionLocal()()`. The `add_claim_ack` publishes `claim_ack_written` via the existing `_safe_publish` pattern; `remove_claim_ack` publishes `claim_ack_dropped`.
|
Mirrors `store/acks.py` shape: `add_claim_ack`, `list_acks_for_claim`, `list_claims_for_ack`, `find_ack_orphans`, `remove_claim_ack`, **plus** `batch_envelope_index()` (the in-memory map for D10's primary join). Each mutating method opens its own `db.SessionLocal()()`. The `add_claim_ack` publishes `claim_ack_written` via the existing `_safe_publish` pattern; `remove_claim_ack` publishes `claim_ack_dropped`.
|
||||||
|
|
||||||
|
The `batch_envelope_index()` method walks `Batch` once and returns a `dict[str, str]` mapping `envelope.control_number → batch.id`. Called once per ingest (current max 16 batches; trivial cost). Invalidated at module-import time (so a fresh ingest sees fresh state) AND explicitly after any `Batch` write that touches `raw_result_json`.
|
||||||
|
|
||||||
- [ ] **Step 3.2: Add `to_ui_claim_ack` serializer to `backend/src/cyclone/store/ui.py`.**
|
- [ ] **Step 3.2: Add `to_ui_claim_ack` serializer to `backend/src/cyclone/store/ui.py`.**
|
||||||
|
|
||||||
@@ -309,13 +372,14 @@ src/
|
|||||||
|
|
||||||
- [ ] **Step 3.3: Extend `CycloneStore` facade in `backend/src/cyclone/store/__init__.py`.**
|
- [ ] **Step 3.3: Extend `CycloneStore` facade in `backend/src/cyclone/store/__init__.py`.**
|
||||||
|
|
||||||
Add five methods that delegate to `store.claim_acks`:
|
Add six methods that delegate to `store.claim_acks`:
|
||||||
```python
|
```python
|
||||||
def add_claim_ack(self, *, claim_id, batch_id, ack_id, ack_kind, ak2_index, set_control_number, set_accept_reject_code, linked_by, event_bus=None): ...
|
def add_claim_ack(self, *, claim_id, batch_id, ack_id, ack_kind, ak2_index, set_control_number, set_accept_reject_code, linked_by, event_bus=None): ...
|
||||||
def list_acks_for_claim(self, claim_id: str) -> list[db.ClaimAck]: ...
|
def list_acks_for_claim(self, claim_id: str) -> list[db.ClaimAck]: ...
|
||||||
def list_claims_for_ack(self, kind: str, ack_id: int) -> list[db.ClaimAck]: ...
|
def list_claims_for_ack(self, kind: str, ack_id: int) -> list[db.ClaimAck]: ...
|
||||||
def find_ack_orphans(self, kind: str) -> list[dict]: ...
|
def find_ack_orphans(self, kind: str) -> list[dict]: ...
|
||||||
def remove_claim_ack(self, link_id: int, event_bus=None) -> None: ...
|
def remove_claim_ack(self, link_id: int, event_bus=None) -> None: ...
|
||||||
|
def batch_envelope_index(self) -> dict[str, str]: ... # D10 — ST02 -> batch.id
|
||||||
```
|
```
|
||||||
|
|
||||||
- [ ] **Step 3.4: Verify facade wiring with a quick test.**
|
- [ ] **Step 3.4: Verify facade wiring with a quick test.**
|
||||||
@@ -341,17 +405,35 @@ src/
|
|||||||
|
|
||||||
After the existing `apply_999_rejections` call (at `handlers/handle_999.py:79-92`), within the same `with db.SessionLocal()() as session:` block, call:
|
After the existing `apply_999_rejections` call (at `handlers/handle_999.py:79-92`), within the same `with db.SessionLocal()() as session:` block, call:
|
||||||
```python
|
```python
|
||||||
|
# Build the D10 batch envelope index once per ingest (cheap — ~16 batches today)
|
||||||
|
batch_index = cycl_store.batch_envelope_index()
|
||||||
|
|
||||||
|
def _pcn_lookup(pcn: str):
|
||||||
|
# Fallback path (D10 Pass 2). Only fires when the ST02 lookup misses.
|
||||||
|
return session.query(Claim).filter_by(patient_control_number=pcn).first()
|
||||||
|
|
||||||
link_result = apply_999_acceptances(
|
link_result = apply_999_acceptances(
|
||||||
session, result, ack_id=row.id, claim_lookup=_lookup,
|
session, result, ack_id=row.id,
|
||||||
|
batch_envelope_index=batch_index,
|
||||||
|
pc_claim_lookup=_pcn_lookup,
|
||||||
)
|
)
|
||||||
```
|
```
|
||||||
The `row.id` is the just-inserted `Ack` row from `cycl_store.add_ack(...)`. The `_lookup` closure is already defined. The `link_result.orphans` should be logged but not surfaced as a rejection (orphan acks are normal — the 837 batch may not have been ingested yet).
|
The `row.id` is the just-inserted `Ack` row from `cycl_store.add_ack(...)`. The `_pcn_lookup` closure is the Pass-2 fallback (rarely hits). The `link_result.orphans` should be logged but not surfaced as a rejection — orphan acks are normal (the 837 batch may not have been ingested yet, or the ST02 may be a `0001` placeholder from before the Gainwell MFT convention landed).
|
||||||
|
|
||||||
- [ ] **Step 4.2: Same change for the 277CA handler.**
|
- [ ] **Step 4.2: Same change for the 277CA handler.**
|
||||||
|
|
||||||
Locate `handlers/handle_277ca.py` (analogous shape to `handle_999.py`). After the existing `apply_277ca_rejections` call, add:
|
Locate `handlers/handle_277ca.py` (analogous shape to `handle_999.py`). After the existing `apply_277ca_rejections` call, add:
|
||||||
```python
|
```python
|
||||||
link_result = apply_277ca_acks(session, parsed_277ca, ack_id=row.id, claim_lookup=_lookup)
|
batch_index = cycl_store.batch_envelope_index()
|
||||||
|
|
||||||
|
def _pcn_lookup(pcn: str):
|
||||||
|
return session.query(Claim).filter_by(patient_control_number=pcn).first()
|
||||||
|
|
||||||
|
link_result = apply_277ca_acks(
|
||||||
|
session, parsed_277ca, ack_id=row.id,
|
||||||
|
batch_envelope_index=batch_index,
|
||||||
|
pc_claim_lookup=_pcn_lookup,
|
||||||
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
- [ ] **Step 4.3: Same change for the TA1 handler.**
|
- [ ] **Step 4.3: Same change for the TA1 handler.**
|
||||||
@@ -381,14 +463,25 @@ src/
|
|||||||
|
|
||||||
In `backend/tests/test_e2e_999_to_claim_drawer.py`:
|
In `backend/tests/test_e2e_999_to_claim_drawer.py`:
|
||||||
```python
|
```python
|
||||||
def test_ingest_999_creates_link_rows(client, store):
|
def test_ingest_999_creates_link_rows_via_st02_join(client, store):
|
||||||
# Seed a claim with patient_control_number='991102986'
|
# Seed a Batch with raw_result_json.envelope.control_number='991102986'
|
||||||
# POST /api/parse-999 with the 999 that has AK2*837*991102986*...
|
# and a Claim whose batch_id points to that Batch (the claim's
|
||||||
|
# patient_control_number is set to the long form 't991102986o...'
|
||||||
|
# to mirror the real-world t-prefix pattern from Gainwell/TOC).
|
||||||
|
# POST /api/parse-999 with a 999 carrying AK2*837*991102986*...
|
||||||
# Assert: GET /api/claims/{id}/acks returns 1 row.
|
# Assert: GET /api/claims/{id}/acks returns 1 row.
|
||||||
# Assert: GET /api/acks/{id}/claims returns 1 row.
|
# Assert: GET /api/acks/999/{id}/claims returns 1 row.
|
||||||
|
# Assert: the link row's set_control_number == '991102986'
|
||||||
|
# (i.e. the AK2-2 value, not the claim's CLM01).
|
||||||
# Assert: the two views agree on the link.
|
# Assert: the two views agree on the link.
|
||||||
|
|
||||||
|
def test_ingest_999_creates_link_rows_via_pcn_fallback(client, store):
|
||||||
|
# Seed a Claim with patient_control_number='991102986' (no matching
|
||||||
|
# Batch envelope — Pass 2 fallback path).
|
||||||
|
# POST /api/parse-999 with AK2*837*991102986*...
|
||||||
|
# Assert: GET /api/claims/{id}/acks returns 1 row (Pass 2 hit).
|
||||||
```
|
```
|
||||||
Run the new test — should PASS.
|
Run both tests — should PASS.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -603,4 +696,14 @@ src/
|
|||||||
|
|
||||||
## Deviations log
|
## Deviations log
|
||||||
|
|
||||||
(Add any deviations from this plan to the merge commit body. Pre-existing baseline is 1 backend pollution + 10 frontend failures; document any new pre-existing or changed-baseline entries here.)
|
### 2026-07-02 — D10 added, two-pass join replaces single PCN lookup
|
||||||
|
|
||||||
|
**Finding:** Empirically measured against prod (1,398 acks, 530 claims), the join `Claim.patient_control_number == 999.set_control_number` matched **0 acks**. Gainwell's 999 AK2-2 echoes the source 837's `ST02` (batch control number), not its `CLM01` (patient control number). TOC's billing software fills CLM01 with the full claim id (`t991102989o1c120d`) and ST02 with the batch control number (`991102989`), so the two values never collide.
|
||||||
|
|
||||||
|
**Fix:** Auto-linker join rewritten as a two-pass lookup — primary is `Batch.envelope.control_number (== 837 ST02) → claims via batch_id`, fallback is `Claim.patient_control_number`. New pure helper `lookup_claims_for_ack_set_response` encapsulates both passes. New store method `batch_envelope_index()` builds an in-memory `ST02 → batch.id` map at the start of each ingest.
|
||||||
|
|
||||||
|
**Coverage after fix:** 727 of 1,398 acks auto-link (52%); 671 stay orphans because their ST02 is `0001` (placeholder) and no 837 batch with ST02=`0001` exists in our DB — these are real orphans that surface in the Inbox `ack-orphans` lane.
|
||||||
|
|
||||||
|
**Spec change:** New decision D10 in spec. Plan changes: Step 2.1 grew by one helper; Step 2.2 rewritten to walk a list of matched claims per AK2 (one-ack-to-many); Step 2.5 grew by three new tests (`test_lookup_claims_two_pass_join`, `test_lookup_claims_pass1_wins_over_pass2`, `test_999_linker_emits_one_row_per_claim_in_multi_claim_batch`); Step 3.1 grew by `batch_envelope_index()`; Step 3.3 grew by one facade method; Step 4.1 / 4.2 rewritten to pass the index; Step 4.5 grew by one extra test (`test_ingest_999_creates_link_rows_via_pcn_fallback`).
|
||||||
|
|
||||||
|
(Add any further deviations from this plan to the merge commit body. Pre-existing baseline is 1 backend pollution + 10 frontend failures; document any new pre-existing or changed-baseline entries here.)
|
||||||
@@ -11,6 +11,8 @@
|
|||||||
|
|
||||||
Today every inbound 999 / 277CA / TA1 ACK is **persisted** but **not linked to any claim**. The 999 rejection path mutates `Claim.state` (because it carries an AK2 set-control-number that matches `Claim.patient_control_number`) but does not persist a durable link row, so the operator cannot answer "which 999 acknowledged this claim?" or "which claims does this 999 acknowledge?". The 277CA path has the same gap. The TA1 path (envelope-level ack) has no per-claim granularity.
|
Today every inbound 999 / 277CA / TA1 ACK is **persisted** but **not linked to any claim**. The 999 rejection path mutates `Claim.state` (because it carries an AK2 set-control-number that matches `Claim.patient_control_number`) but does not persist a durable link row, so the operator cannot answer "which 999 acknowledged this claim?" or "which claims does this 999 acknowledge?". The 277CA path has the same gap. The TA1 path (envelope-level ack) has no per-claim granularity.
|
||||||
|
|
||||||
|
**Critical correction (added 2026-07-02 in response to the t991102989o1c120d investigation):** The opening sentence above is wrong about the PCN join. Empirically measured against production on 2026-07-02 against 1,398 acks and 530 claims: the join `Claim.patient_control_number == 999.set_control_number` matches **0 acks**. The 999 AK2-2 `set_control_number` is Gainwell's echo of the source 837's **`ST02`** (transaction-set control number), not the 837's **`CLM01`** (patient control number). TOC's billing software fills CLM01 with the full claim id (`t991102989o1c120d`) and ST02 with the batch control number (`991102989`). The correct primary join is `999.set_control_number → Batch.envelope.control_number (== 837 ST02) → claims in that batch via claim.batch_id`. SP28's join logic is rewritten in D10 below. **Coverage after fix:** 727 of 1,398 acks auto-link (52%); 671 stay orphans because their ST02 is `0001` (placeholder) and no 837 batch with ST02=0001 exists in our DB — these are real orphans for the Inbox lane.
|
||||||
|
|
||||||
SP28 closes that gap: every ACK carries enough structural metadata to be tied back to the claim(s) it acknowledges, and the system **auto-creates the link row at parse time** so the operator never has to. A new `claim_acks` join table is the durable record of the link; the existing `Claim.matched_remittance_id` / `Remittance.claim_id` pair is the precedent.
|
SP28 closes that gap: every ACK carries enough structural metadata to be tied back to the claim(s) it acknowledges, and the system **auto-creates the link row at parse time** so the operator never has to. A new `claim_acks` join table is the durable record of the link; the existing `Claim.matched_remittance_id` / `Remittance.claim_id` pair is the precedent.
|
||||||
|
|
||||||
**In scope:**
|
**In scope:**
|
||||||
@@ -73,6 +75,21 @@ So the `ClaimDrawer` panel can show "Rejected (999 AK5 R)" inline without re-par
|
|||||||
|
|
||||||
Every new endpoint goes through the same `request.state.user` check as `/api/claims` and `/api/acks`. The `POST /api/acks/{kind}/{id}/match-claim` endpoint is **any logged-in user** (D5) — differs from `POST /api/inbox/remit-orphans/{id}/match-claim` which is admin-only. All read endpoints (`GET /api/claims/{id}/acks`, `GET /api/acks/{kind}/{id}/claims`, the two stream endpoints, `GET /api/inbox/ack-orphans`) are also any logged-in user. The auth boundary is unchanged.
|
Every new endpoint goes through the same `request.state.user` check as `/api/claims` and `/api/acks`. The `POST /api/acks/{kind}/{id}/match-claim` endpoint is **any logged-in user** (D5) — differs from `POST /api/inbox/remit-orphans/{id}/match-claim` which is admin-only. All read endpoints (`GET /api/claims/{id}/acks`, `GET /api/acks/{kind}/{id}/claims`, the two stream endpoints, `GET /api/inbox/ack-orphans`) are also any logged-in user. The auth boundary is unchanged.
|
||||||
|
|
||||||
|
### D10. Primary join is `999.set_control_number → Batch.envelope.control_number (== 837 ST02)`, not `Claim.patient_control_number`
|
||||||
|
|
||||||
|
The original spec opening assumed `999.AK2-2 set_control_number == Claim.patient_control_number`. Empirically that's wrong on this codebase (TOC's 837s fill CLM01 with a unique id-per-claim and ST02 with the batch control number; Gainwell's 999 echoes ST02). The fix is a two-pass join with `Batch.envelope.control_number` as the primary key and `Claim.patient_control_number` as a fallback:
|
||||||
|
|
||||||
|
1. **Primary (per-AK2, for 999/277CA):** for each set-response's `set_control_number`, look up the `Batch` whose `raw_result_json.envelope.control_number` matches. If found, link the row to **every claim whose `batch_id == batch.id`** (a single 999 ack can carry one AK2 per original 837 CLM, and many of TOC's 837 batches ship one claim per file — but a batch can carry multiple claims, so the join is one-ack-to-many-claims when applicable). Per-AK2 granularity (D1) is preserved by recording each link row with its `ak2_index` so the operator can correlate which AK2 acknowledged which claim.
|
||||||
|
2. **Fallback (per-AK2, for 999/277CA):** if no batch matches the set_control_number, look up `Claim.patient_control_number == set_control_number`. This catches the case where the sender filled CLM01 == ST02 (some billing software does). If found, link to that single claim.
|
||||||
|
3. **Orphan:** neither pass matched → no `claim_acks` row; the ack surfaces in the Inbox `ack-orphans` lane (D7).
|
||||||
|
4. **TA1 (envelope-level, D4 unchanged):** joins on `Batch.envelope.sender_id` + `receiver_id`, no set_control_number involved.
|
||||||
|
|
||||||
|
**Implementation:** a new pure helper `cyclone.claim_acks.lookup_claims_for_ack_set_response(sr) → list[Claim]` encapsulates both passes; `apply_999_acceptances` calls it inside the `for sr in set_responses:` loop and emits one `claim_acks` row per matched claim. The `set_control_number` is **still persisted on the link row** (per the existing `set_control_number STRING(64)` column) for orphan traceability — the row records the value the 999 actually carried, regardless of which join path resolved it.
|
||||||
|
|
||||||
|
**Performance:** the primary join reads `Batch.raw_result_json` (a JSON blob) on every set-response, which is O(N acks × M batches) without an index. Two mitigations: (1) cache the `Batch.envelope.control_number → batch.id` map in memory at the start of each ingest (max 16 batches today, so 16 lookups for any size ack); (2) once the 837 ingest path is touched up to denormalize the ST02 onto a dedicated column (out of scope for SP28; tracked as a follow-up if the row count grows), the lookup becomes a single indexed `SELECT`. Today's 530-claim workload makes (1) sufficient.
|
||||||
|
|
||||||
|
**Test impact:** the existing fixture `minimal_999.txt` uses ST02 matching the seeded `Batch`; no fixture rewrite needed. The two-AK2 fixture (`minimal_999_2ak2.txt`) gets one extra assertion that both AK2s land in `claim_acks` even when their `set_control_number`s don't equal any `Claim.patient_control_number` (proves the ST02 path is exercised, not the fallback).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 3. Data model
|
## 3. Data model
|
||||||
@@ -88,7 +105,7 @@ Columns:
|
|||||||
- `ack_id INTEGER NOT NULL` — discriminated union with `ack_kind` (no FK constraint — `acks` / `ta1_acks` / `two77ca_acks` are three separate tables).
|
- `ack_id INTEGER NOT NULL` — discriminated union with `ack_kind` (no FK constraint — `acks` / `ta1_acks` / `two77ca_acks` are three separate tables).
|
||||||
- `ack_kind STRING(8) NOT NULL` — one of `"999"`, `"277ca"`, `"ta1"`.
|
- `ack_kind STRING(8) NOT NULL` — one of `"999"`, `"277ca"`, `"ta1"`.
|
||||||
- `ak2_index INTEGER NULL` — for 999 only; the AK2 set-response index that produced this link (NULL for 277CA and TA1).
|
- `ak2_index INTEGER NULL` — for 999 only; the AK2 set-response index that produced this link (NULL for 277CA and TA1).
|
||||||
- `set_control_number STRING(64) NULL` — for 999/277CA only; the patient_control_number used for the lookup. Preserved for orphan traceability.
|
- `set_control_number STRING(64) NULL` — for 999/277CA only; the `AK2-2 set_control_number` carried by the source 999/277CA (== source 837's `ST02` for Gainwell batches). Preserved on the link row for orphan traceability — it's the value the upstream ack actually carried, regardless of which join path (D10) resolved the link.
|
||||||
- `set_accept_reject_code STRING(8) NULL` — for 999 (AK5 code: A/E/R/X); for 277CA (STC category: A1/A2/A3/A4/A6/A7 etc.).
|
- `set_accept_reject_code STRING(8) NULL` — for 999 (AK5 code: A/E/R/X); for 277CA (STC category: A1/A2/A3/A4/A6/A7 etc.).
|
||||||
- `linked_at DATETIME NOT NULL` — set by the server.
|
- `linked_at DATETIME NOT NULL` — set by the server.
|
||||||
- `linked_by STRING(16) NOT NULL` — `"auto"` or `"manual"` (D5).
|
- `linked_by STRING(16) NOT NULL` — `"auto"` or `"manual"` (D5).
|
||||||
@@ -215,9 +232,12 @@ Add `claimAcks: Record<string, ClaimAck> + claimAckOrder: number[]` slice. Mirro
|
|||||||
|
|
||||||
- **Risk:** Migration `0018_claim_acks.sql` runs against a production DB with thousands of existing 999 rows. The migration adds a new table; it does NOT backfill historical links. Mitigation: a one-shot backfill script (`scripts/backfill_claim_acks.py`) lives outside the SP-N flow and is run manually after the deploy. Documented in the runbook delta.
|
- **Risk:** Migration `0018_claim_acks.sql` runs against a production DB with thousands of existing 999 rows. The migration adds a new table; it does NOT backfill historical links. Mitigation: a one-shot backfill script (`scripts/backfill_claim_acks.py`) lives outside the SP-N flow and is run manually after the deploy. Documented in the runbook delta.
|
||||||
- **Risk:** `apply_999_acceptances` running alongside `apply_999_rejections` double-touches the same DB session. Mitigation: both helpers share a single session; the link row is created inside the same `for sr in set_responses:` loop as the rejection mutation, so there's exactly one transaction.
|
- **Risk:** `apply_999_acceptances` running alongside `apply_999_rejections` double-touches the same DB session. Mitigation: both helpers share a single session; the link row is created inside the same `for sr in set_responses:` loop as the rejection mutation, so there's exactly one transaction.
|
||||||
|
- **Risk:** The two-pass join (D10) — `Batch.envelope.control_number` primary + `Claim.patient_control_number` fallback — can produce a false-positive link if a batch ST02 collides with a claim PCN from a different batch. Mitigation: the primary pass only fires when the batch lookup hits; the fallback only fires when the batch lookup misses AND the PCN lookup hits. The two paths cannot both fire for the same set_response, so the link is at most one row per claim per AK2. The `ux_claim_acks_dedup` unique index (3.1) catches the pathological re-ingest case.
|
||||||
|
- **Risk:** Future 837 batches may carry multiple claims sharing one ST02 (a real possibility once billing software starts batching properly). Mitigation: the join is one-ack-to-many-claims by design (D10 implementation note); the `claim_acks` table allows the same `(ack_kind, ack_id, ak2_index)` to map to multiple `claim_id` rows. Tests cover the 1-to-N case explicitly.
|
||||||
|
- **Risk:** 671 historical 999 acks with ST02=`0001` remain orphans after SP28 ships. Mitigation: D7's Inbox lane surfaces them; the operator can dismiss or manually link via the existing 999 detail drawer. No data loss — the acks remain visible on the Acks page; they just don't auto-link to a claim.
|
||||||
- **Risk:** The `ux_claim_acks_dedup` unique index rejects re-ingests of the same file with a `sqlite3.IntegrityError`. Mitigation: the orchestrator catches this specifically (mirrors the existing `apply_999_rejections` idempotency check at line 60) and treats it as success — re-ingest of an identical file is a no-op.
|
- **Risk:** The `ux_claim_acks_dedup` unique index rejects re-ingests of the same file with a `sqlite3.IntegrityError`. Mitigation: the orchestrator catches this specifically (mirrors the existing `apply_999_rejections` idempotency check at line 60) and treats it as success — re-ingest of an identical file is a no-op.
|
||||||
- **Risk:** TA1 batch-level links (`claim_id IS NULL`) confuse the `ClaimDrawer` panel. Mitigation: the new `/api/claims/{id}/acks` endpoint explicitly filters them out; the `to_ui_claim_ack` serializer tags them with `claim_state="n/a"` so they're visually distinct even if a future endpoint leaks them through.
|
- **Risk:** TA1 batch-level links (`claim_id IS NULL`) confuse the `ClaimDrawer` panel. Mitigation: the new `/api/claims/{id}/acks` endpoint explicitly filters them out; the `to_ui_claim_ack` serializer tags them with `claim_state="n/a"` so they're visually distinct even if a future endpoint leaks them through.
|
||||||
- **Risk:** Manual-match endpoint opens a window for the operator to attach an ack to the wrong claim. Mitigation: the endpoint requires admin role + the claim must currently be in `{SUBMITTED, RECEIVED, REJECTED, PAID, PARTIAL, DENIED}` (i.e., not in a reversed/closed state). Returns 409 if the claim is in a terminal state.
|
- **Risk:** Manual-match endpoint opens a window for the operator to attach an ack to the wrong claim. Mitigation: the endpoint accepts any logged-in user (D5/D9 — explicitly differs from the admin-only remit-orphans posture), but the claim must currently be in `{SUBMITTED, RECEIVED, REJECTED, PAID, PARTIAL, DENIED}` (i.e., not in a reversed/closed state). Returns 409 if the claim is in a terminal state. The `linked_by="manual"` audit column preserves who matched what.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -226,12 +246,14 @@ Add `claimAcks: Record<string, ClaimAck> + claimAckOrder: number[]` slice. Mirro
|
|||||||
The work breaks into ~14 tasks across 6 phases. The full plan lives at `docs/superpowers/plans/2026-07-02-cyclone-ack-claim-auto-link.md` (TBD).
|
The work breaks into ~14 tasks across 6 phases. The full plan lives at `docs/superpowers/plans/2026-07-02-cyclone-ack-claim-auto-link.md` (TBD).
|
||||||
|
|
||||||
1. **Migration** — `0018_claim_acks.sql` + `ClaimAck` ORM model + `db._reset_for_tests()` updates.
|
1. **Migration** — `0018_claim_acks.sql` + `ClaimAck` ORM model + `db._reset_for_tests()` updates.
|
||||||
2. **Pure helpers** — `cyclone.claim_acks.py` module with `apply_999_acceptances`, `apply_277ca_acks`, `apply_ta1_envelope_link`, `link_manual`.
|
2. **Pure helpers** — `cyclone.claim_acks.py` module with `apply_999_acceptances`, `apply_277ca_acks`, `apply_ta1_envelope_link`, `link_manual`, and `lookup_claims_for_ack_set_response` (the two-pass join from D10 — Batch.envelope.control_number primary, Claim.patient_control_number fallback).
|
||||||
3. **Store facade** — extend `CycloneStore` with `add_claim_ack`, `list_acks_for_claim`, `list_claims_for_ack`, `find_ack_orphans`. Add the `claim_ack_written` / `claim_ack_dropped` event publish.
|
3. **Store facade** — extend `CycloneStore` with `add_claim_ack`, `list_acks_for_claim`, `list_claims_for_ack`, `find_ack_orphans`, and `batch_envelope_index()` (cached in-memory map of `envelope.control_number → batch.id`, invalidated on every new `Batch` write). Add the `claim_ack_written` / `claim_ack_dropped` event publish.
|
||||||
4. **Handler integration** — wire `apply_claim_ack_links` into `handle_999.handle`, the 277CA handler, and the TA1 handler. Mirror the existing `apply_999_rejections` call site at `handlers/handle_999.py:79-92`.
|
4. **Handler integration** — wire `apply_claim_ack_links` into `handle_999.handle`, the 277CA handler, and the TA1 handler. Mirror the existing `apply_999_rejections` call site at `handlers/handle_999.py:79-92`. Inside `apply_999_acceptances`, the per-AK2 loop calls `lookup_claims_for_ack_set_response(sr)` and emits one `claim_acks` row per matched claim (one-ack-to-many-claims when the 837 batch shipped multiple claims).
|
||||||
5. **API endpoints** — new `api_routers/claim_acks.py` (mirrors `api_routers/acks.py` shape) + register the router. Extend the existing `/api/claims/{id}` and `/api/acks` serializers.
|
5. **API endpoints** — new `api_routers/claim_acks.py` (mirrors `api_routers/acks.py` shape) + register the router. Extend the existing `/api/claims/{id}` and `/api/acks` serializers.
|
||||||
6. **Frontend** — `useClaimAcks` + `useAckClaims` hooks, `ClaimDrawer` panel, `AckDrawer` panel, Acks page column, Inbox lane, `useTailStore` slice.
|
6. **Frontend** — `useClaimAcks` + `useAckClaims` hooks, `ClaimDrawer` panel, `AckDrawer` panel, Acks page column, Inbox lane, `useTailStore` slice.
|
||||||
|
|
||||||
|
**Critical-path change vs. the original plan:** Task 2 grows by one helper (`lookup_claims_for_ack_set_response`); Task 3 grows by one method (`batch_envelope_index`). Task 4's per-AK2 loop becomes a `for claim in lookup_claims_for_ack_set_response(sr):` instead of a single `s.query(Claim).filter_by(patient_control_number=...).first()`. No other tasks change shape.
|
||||||
|
|
||||||
Estimated lines: ~1,200 backend + ~800 frontend + ~600 tests. Mirrors SP25's footprint.
|
Estimated lines: ~1,200 backend + ~800 frontend + ~600 tests. Mirrors SP25's footprint.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
Reference in New Issue
Block a user