tyler/cyclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(spec): SP28 D10 — two-pass join via Batch.envelope.control_number

Browse Source 
The original SP28 spec assumed `Claim.patient_control_number == 999.set_control_number`.
Empirically that's wrong: Gainwell's 999 echoes the source 837's ST02, not its
CLM01, and TOC's billing software fills them differently. Measured against
prod on 2026-07-02: the PCN join matches 0 / 1,398 acks.

Fix: two-pass join. Primary is `Batch.envelope.control_number (== 837 ST02)
→ claims via batch_id`; fallback is `Claim.patient_control_number` (for
senders that fill CLM01 == ST02). Coverage after fix: 727 / 1,398 (52%);
the remaining 671 are real orphans (ST02=0001 placeholder, no matching 837
batch in our DB).

Spec adds D10 + a critical-correction paragraph at the top of §1. Plan
gains the `lookup_claims_for_ack_set_response` pure helper, the
`batch_envelope_index()` store method, and three new tests covering the
two-pass join + the false-positive guard + the one-ack-to-many case.
This commit is contained in:
Nora
2026-07-02 11:16:53 -06:00
parent f88a7c7c4f
commit ec87f98f02
2 changed files with 155 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/plans/2026-07-02-cyclone-ack-claim-auto-link.md
+128 -25
View File
@@ -7,7 +7,9 @@
**Goal:** Persist a durable link row (`claim_acks`) for every inbound 999 / 277CA / TA1 acknowledgment at parse time, so the operator can answer "which claims does this ack acknowledge?" and "which acks does this claim have?" without manual correlation, and surface those links in `ClaimDrawer` + `AckDrawer` + a new Inbox orphan lane.
**Architecture:** A new `claim_acks` join table (one row per AK2 set-response for 999, per ClaimStatus for 277CA, per TA1 envelope) is the durable record. Three pure helpers (`apply_999_acceptances`, `apply_277ca_acks`, `apply_ta1_envelope_link`) run inside the existing handler transactions alongside the existing rejection logic. The orchestrator (`apply_claim_ack_links`) is called from the same point in `handle_999.handle` / the 277CA handler / the TA1 handler. A new `claim_ack_written` pubsub event drives two new stream endpoints (`/api/claims/{id}/acks/stream`, `/api/acks/{kind}/{id}/claims/stream`) so both drawers update in real time. Frontend mirrors the SP25 acks/ta1_acks live-tail extension pattern (`useTailStore` slice + `useTailStream` + `useMergedTail`) plus a manual-match dropdown for the orphan case.
**Architecture:** A new `claim_acks` join table (one row per AK2 set-response for 999, per ClaimStatus for 277CA, per TA1 envelope) is the durable record. Three pure helpers (`apply_999_acceptances`, `apply_277ca_acks`, `apply_ta1_envelope_link`) run inside the existing handler transactions alongside the existing rejection logic. The orchestrator (`apply_claim_ack_links`) is called from the same point in `handle_999.handle` / the 277CA handler / the TA1 handler. The auto-linker's per-AK2 join uses a two-pass lookup (D10 in the spec): primary is `Batch.envelope.control_number (== 837 ST02) → claims in batch via batch_id`; fallback is `Claim.patient_control_number`. A new `claim_ack_written` pubsub event drives two new stream endpoints (`/api/claims/{id}/acks/stream`, `/api/acks/{kind}/{id}/claims/stream`) so both drawers update in real time. Frontend mirrors the SP25 acks/ta1_acks live-tail extension pattern (`useTailStore` slice + `useTailStream` + `useMergedTail`) plus a manual-match dropdown for the orphan case.
**Critical-path correction (added 2026-07-02):** the original plan's `_lookup` closure queried `Claim.patient_control_number == set_control_number`. That join is empirically 0-effective on this codebase (1,398 acks, 0 matched) because Gainwell's 999 echoes the 837's `ST02` not its `CLM01`. The corrected plan uses `Batch.envelope.control_number` (which holds the 837's `ST02`) as the primary join key, with `Claim.patient_control_number` as a fallback for the rare case where the sender filled `CLM01 == ST02`. The corrected join recovers 727 / 1,398 acks automatically (52%); the remaining 671 have ST02=`0001` (no matching 837 batch) and remain real orphans surfaced by the Inbox ack-orphans lane.
**Tech Stack:** Python 3.11+, FastAPI, SQLAlchemy 2.x, SQLite (encrypted via SQLCipher), React 18 + TypeScript + Vite, TanStack Query, Zustand, Radix UI primitives.
@@ -164,7 +166,7 @@ src/
- [ ] **Step 2.1: Create `backend/src/cyclone/claim_acks.py` with the public API.**
Three public functions + one dataclass result. The module is pure (no DB session ownership — callers pass in the session, just like `apply_999_rejections`).
Four public functions + one dataclass result. The module is pure (no DB session ownership — callers pass in the session, just like `apply_999_rejections`). The fourth function is the two-pass join helper from D10.
```python
"""SP28: per-ACK auto-linker.
@@ -191,12 +193,41 @@ src/
# set_control_numbers we couldn't resolve to a claim
def lookup_claims_for_ack_set_response(
session: Session,
set_control_number: str,
*,
batch_envelope_index: Callable[[str], str | None] | None = None,
) -> list[Claim]:
"""D10: two-pass join for a single AK2 set_response / 277CA claim_status.
Returns 0..N matching claims (one-ack-to-many when one 837 batch
shipped multiple claims under one ST02).
Pass 1 (primary): ``Batch.envelope.control_number == set_control_number``.
If the caller supplies a pre-built ``batch_envelope_index`` (a callable
mapping ST02 -> batch.id, built once at the start of the ingest),
we skip the per-set-response ``Batch`` scan entirely. If no index
is supplied we fall back to a session.query() against ``Batch``.
Pass 2 (fallback): ``Claim.patient_control_number == set_control_number``.
Catches the case where the sender filled CLM01 == ST02 (rare;
Gainwell's TOC batches do not).
Pass 1 wins. The two paths cannot both fire — if Pass 1 returns
one or more claims, Pass 2 is skipped. This is the false-positive
guard from spec §7.
"""
...
def apply_999_acceptances(
session: Session,
parsed_999,
*,
ack_id: int,
claim_lookup: Callable[[str], Claim | None],
batch_envelope_index: Callable[[str], str | None] | None = None,
pc_claim_lookup: Callable[[str], Claim | None] | None = None,
) -> ClaimAckLinkResult:
"""For every AK2 set-response, create a claim_acks row.
@@ -204,6 +235,10 @@ src/
show the rejection inline). Orphans are returned but not linked.
Idempotent: if (claim_id, '999', ack_id, ak2_index) already exists,
skip silently.
The two lookup arguments replace the single ``claim_lookup`` from
the original spec — D10 split the join into a primary (ST02 via
batch envelope index) and a fallback (PCN match). See spec §D10.
"""
...
@@ -212,11 +247,14 @@ src/
parsed_277ca,
*,
ack_id: int,
claim_lookup: Callable[[str], Claim | None],
batch_envelope_index: Callable[[str], str | None] | None = None,
pc_claim_lookup: Callable[[str], Claim | None] | None = None,
) -> ClaimAckLinkResult:
"""For every ClaimStatus with a payer_claim_control_number,
create a claim_acks row. Accepted AND rejected both link.
Idempotent: (claim_id, '277ca', ack_id, NULL) is the dedup key.
Same two-pass lookup as ``apply_999_acceptances`` (D10).
"""
...
@@ -239,22 +277,23 @@ src/
- [ ] **Step 2.2: Implement `apply_999_acceptances`.**
Walk `parsed_999.set_responses` (each has `.set_control_number` + `.set_accept_reject.code`). For each:
1. Call `claim_lookup(set_control_number)`. If None → `result.orphans.append(set_control_number)`, continue.
2. Check for existing `claim_acks` row with `(claim_id, '999', ack_id, ak2_index=i)`. If exists → skip silently (idempotent).
3. Insert `ClaimAck(claim_id=claim.id, ack_id=ack_id, ack_kind='999', ak2_index=i, set_control_number=pcn, set_accept_reject_code=code, linked_at=utcnow(), linked_by='auto')`.
4. Append `(claim.id, i)` to `result.linked`.
5. `session.flush()` at the end (do NOT commit — the caller owns the transaction).
1. Call `lookup_claims_for_ack_set_response(session, sr.set_control_number, batch_envelope_index=batch_envelope_index)` (D10 — two-pass join: ST02-via-batch primary, PCN fallback). If `[]` → `result.orphans.append(sr.set_control_number)`, continue.
2. For each matched `claim`:
- Check for existing `claim_acks` row with `(claim.id, '999', ack_id, ak2_index=i)`. If exists → skip silently (idempotent).
- Insert `ClaimAck(claim_id=claim.id, ack_id=ack_id, ack_kind='999', ak2_index=i, set_control_number=sr.set_control_number, set_accept_reject_code=sr.set_accept_reject.code, linked_at=utcnow(), linked_by='auto')`.
- Append `(claim.id, i)` to `result.linked`.
3. `session.flush()` at the end (do NOT commit — the caller owns the transaction).
Insert order matters: walk set_responses in order, capture `i = enumerate index`.
Insert order matters: walk set_responses in order, capture `i = enumerate index`. One AK2 can produce **multiple** `claim_acks` rows when the 837 batch carried more than one claim under a shared ST02 (rare on this codebase but supported by the schema and the join).
- [ ] **Step 2.3: Implement `apply_277ca_acks`.**
Walk `parsed_277ca.claim_statuses`. For each:
1. Get `pcn = status.payer_claim_control_number`. If empty/None → orphan with `status.status_code or ""`.
2. Call `claim_lookup(pcn)`. If None → orphan.
3. Check dedup: `(claim_id, '277ca', ack_id, NULL)`.
4. Insert `ClaimAck(claim_id=claim.id, ack_id=ack_id, ack_kind='277ca', ak2_index=NULL, set_control_number=pcn, set_accept_reject_code=status.status_code, linked_at=utcnow(), linked_by='auto')`.
5. Append to result. Flush.
2. Call `lookup_claims_for_ack_set_response(session, pcn, batch_envelope_index=...)` (D10 — same two-pass join as 999, except the lookup key is `payer_claim_control_number` from the 277CA instead of `set_control_number` from the 999).
3. For each matched `claim`: check dedup `(claim.id, '277ca', ack_id, NULL)`, insert `ClaimAck`, append to result.
4. If `[]` from the join → orphan.
5. Flush.
- [ ] **Step 2.4: Implement `apply_ta1_envelope_link`.**
@@ -273,6 +312,28 @@ src/
# Build a stub parsed_999 with 2 set_responses, mock claim_lookup
# that returns a claim for the first PCN and None for the second.
# Assert: result.linked == [(claim.id, 0)], result.orphans == ['<second-pcn>'].
def test_lookup_claims_two_pass_join():
# Seed two Claims, one with batch.envelope.control_number='991102989'
# and patient_control_number='t991102989o1c120d', and a separate
# claim with patient_control_number='991102987'.
# Call lookup_claims_for_ack_set_response(session, '991102989')
# — assert the first claim is returned via Pass 1 (batch).
# Call with '991102987' — assert the second claim is returned via Pass 2 (PCN).
# Call with '0001' (no match) — assert [] returned.
def test_lookup_claims_pass1_wins_over_pass2():
# If a ST02 collides with a PCN from a different batch, Pass 1
# must win and Pass 2 must NOT also fire. Seed:
# Claim A: batch_id=B1, patient_control_number='991102989'
# Claim B: batch_id=B2 (with envelope.control_number='OTHER'),
# patient_control_number='991102989'
# Call lookup('991102989') — assert [Claim A] only (Pass 1 wins).
def test_999_linker_emits_one_row_per_claim_in_multi_claim_batch():
# Seed Batch B with ST02='991102990' and TWO claims (batch_id=B).
# Submit a 999 with one AK2 set_control_number='991102990'.
# Assert result.linked == [(claim1.id, 0), (claim2.id, 0)].
```
Run `pytest backend/tests/test_apply_claim_ack_links.py -v` — should now PASS.
@@ -285,7 +346,9 @@ src/
- [ ] **Step 3.1: Create `backend/src/cyclone/store/claim_acks.py` with the persistence layer.**
Mirrors `store/acks.py` shape: `add_claim_ack`, `list_acks_for_claim`, `list_claims_for_ack`, `find_ack_orphans`, `remove_claim_ack`. Each opens its own `db.SessionLocal()()`. The `add_claim_ack` publishes `claim_ack_written` via the existing `_safe_publish` pattern; `remove_claim_ack` publishes `claim_ack_dropped`.
Mirrors `store/acks.py` shape: `add_claim_ack`, `list_acks_for_claim`, `list_claims_for_ack`, `find_ack_orphans`, `remove_claim_ack`, **plus** `batch_envelope_index()` (the in-memory map for D10's primary join). Each mutating method opens its own `db.SessionLocal()()`. The `add_claim_ack` publishes `claim_ack_written` via the existing `_safe_publish` pattern; `remove_claim_ack` publishes `claim_ack_dropped`.
The `batch_envelope_index()` method walks `Batch` once and returns a `dict[str, str]` mapping `envelope.control_number → batch.id`. Called once per ingest (current max 16 batches; trivial cost). Invalidated at module-import time (so a fresh ingest sees fresh state) AND explicitly after any `Batch` write that touches `raw_result_json`.
- [ ] **Step 3.2: Add `to_ui_claim_ack` serializer to `backend/src/cyclone/store/ui.py`.**
@@ -309,13 +372,14 @@ src/
- [ ] **Step 3.3: Extend `CycloneStore` facade in `backend/src/cyclone/store/__init__.py`.**
Add five methods that delegate to `store.claim_acks`:
Add six methods that delegate to `store.claim_acks`:
```python
def add_claim_ack(self, *, claim_id, batch_id, ack_id, ack_kind, ak2_index, set_control_number, set_accept_reject_code, linked_by, event_bus=None): ...
def list_acks_for_claim(self, claim_id: str) -> list[db.ClaimAck]: ...
def list_claims_for_ack(self, kind: str, ack_id: int) -> list[db.ClaimAck]: ...
def find_ack_orphans(self, kind: str) -> list[dict]: ...
def remove_claim_ack(self, link_id: int, event_bus=None) -> None: ...
def batch_envelope_index(self) -> dict[str, str]: ... # D10 — ST02 -> batch.id
```
- [ ] **Step 3.4: Verify facade wiring with a quick test.**
@@ -341,17 +405,35 @@ src/
After the existing `apply_999_rejections` call (at `handlers/handle_999.py:79-92`), within the same `with db.SessionLocal()() as session:` block, call:
```python
# Build the D10 batch envelope index once per ingest (cheap — ~16 batches today)
batch_index = cycl_store.batch_envelope_index()
def _pcn_lookup(pcn: str):
# Fallback path (D10 Pass 2). Only fires when the ST02 lookup misses.
return session.query(Claim).filter_by(patient_control_number=pcn).first()
link_result = apply_999_acceptances(
session, result, ack_id=row.id, claim_lookup=_lookup,
session, result, ack_id=row.id,
batch_envelope_index=batch_index,
pc_claim_lookup=_pcn_lookup,
)
```
The `row.id` is the just-inserted `Ack` row from `cycl_store.add_ack(...)`. The `_lookup` closure is already defined. The `link_result.orphans` should be logged but not surfaced as a rejection (orphan acks are normal — the 837 batch may not have been ingested yet).
The `row.id` is the just-inserted `Ack` row from `cycl_store.add_ack(...)`. The `_pcn_lookup` closure is the Pass-2 fallback (rarely hits). The `link_result.orphans` should be logged but not surfaced as a rejection — orphan acks are normal (the 837 batch may not have been ingested yet, or the ST02 may be a `0001` placeholder from before the Gainwell MFT convention landed).
- [ ] **Step 4.2: Same change for the 277CA handler.**
Locate `handlers/handle_277ca.py` (analogous shape to `handle_999.py`). After the existing `apply_277ca_rejections` call, add:
```python
link_result = apply_277ca_acks(session, parsed_277ca, ack_id=row.id, claim_lookup=_lookup)
batch_index = cycl_store.batch_envelope_index()
def _pcn_lookup(pcn: str):
return session.query(Claim).filter_by(patient_control_number=pcn).first()
link_result = apply_277ca_acks(
session, parsed_277ca, ack_id=row.id,
batch_envelope_index=batch_index,
pc_claim_lookup=_pcn_lookup,
)
```
- [ ] **Step 4.3: Same change for the TA1 handler.**
@@ -381,14 +463,25 @@ src/
In `backend/tests/test_e2e_999_to_claim_drawer.py`:
```python
def test_ingest_999_creates_link_rows(client, store):
# Seed a claim with patient_control_number='991102986'
# POST /api/parse-999 with the 999 that has AK2*837*991102986*...
def test_ingest_999_creates_link_rows_via_st02_join(client, store):
# Seed a Batch with raw_result_json.envelope.control_number='991102986'
# and a Claim whose batch_id points to that Batch (the claim's
# patient_control_number is set to the long form 't991102986o...'
# to mirror the real-world t-prefix pattern from Gainwell/TOC).
# POST /api/parse-999 with a 999 carrying AK2*837*991102986*...
# Assert: GET /api/claims/{id}/acks returns 1 row.
# Assert: GET /api/acks/{id}/claims returns 1 row.
# Assert: GET /api/acks/999/{id}/claims returns 1 row.
# Assert: the link row's set_control_number == '991102986'
# (i.e. the AK2-2 value, not the claim's CLM01).
# Assert: the two views agree on the link.
def test_ingest_999_creates_link_rows_via_pcn_fallback(client, store):
# Seed a Claim with patient_control_number='991102986' (no matching
# Batch envelope — Pass 2 fallback path).
# POST /api/parse-999 with AK2*837*991102986*...
# Assert: GET /api/claims/{id}/acks returns 1 row (Pass 2 hit).
```
Run the new test — should PASS.
Run both tests — should PASS.
---
@@ -603,4 +696,14 @@ src/
## Deviations log
(Add any deviations from this plan to the merge commit body. Pre-existing baseline is 1 backend pollution + 10 frontend failures; document any new pre-existing or changed-baseline entries here.)
### 2026-07-02 — D10 added, two-pass join replaces single PCN lookup
**Finding:** Empirically measured against prod (1,398 acks, 530 claims), the join `Claim.patient_control_number == 999.set_control_number` matched **0 acks**. Gainwell's 999 AK2-2 echoes the source 837's `ST02` (batch control number), not its `CLM01` (patient control number). TOC's billing software fills CLM01 with the full claim id (`t991102989o1c120d`) and ST02 with the batch control number (`991102989`), so the two values never collide.
**Fix:** Auto-linker join rewritten as a two-pass lookup — primary is `Batch.envelope.control_number (== 837 ST02) → claims via batch_id`, fallback is `Claim.patient_control_number`. New pure helper `lookup_claims_for_ack_set_response` encapsulates both passes. New store method `batch_envelope_index()` builds an in-memory `ST02 → batch.id` map at the start of each ingest.
**Coverage after fix:** 727 of 1,398 acks auto-link (52%); 671 stay orphans because their ST02 is `0001` (placeholder) and no 837 batch with ST02=`0001` exists in our DB — these are real orphans that surface in the Inbox `ack-orphans` lane.
**Spec change:** New decision D10 in spec. Plan changes: Step 2.1 grew by one helper; Step 2.2 rewritten to walk a list of matched claims per AK2 (one-ack-to-many); Step 2.5 grew by three new tests (`test_lookup_claims_two_pass_join`, `test_lookup_claims_pass1_wins_over_pass2`, `test_999_linker_emits_one_row_per_claim_in_multi_claim_batch`); Step 3.1 grew by `batch_envelope_index()`; Step 3.3 grew by one facade method; Step 4.1 / 4.2 rewritten to pass the index; Step 4.5 grew by one extra test (`test_ingest_999_creates_link_rows_via_pcn_fallback`).
(Add any further deviations from this plan to the merge commit body. Pre-existing baseline is 1 backend pollution + 10 frontend failures; document any new pre-existing or changed-baseline entries here.)