Files

T

Nora 53a8c77cb8 docs(spec): SP28 ack-claim auto-link design

2026-07-02 10:30:52 -06:00

24 KiB

Raw Blame History

Sub-project 28 — Ack↔Claim Auto-Link: Design Spec

Date: 2026-07-02 Status: Draft, awaiting user sign-off Branch: sp28-ack-claim-auto-link Aesthetic direction: No new UI library; one new panel in ClaimDrawer + one new panel in AckDrawer + one candidate lane in the Inbox.

1. Scope

Today every inbound 999 / 277CA / TA1 ACK is persisted but not linked to any claim. The 999 rejection path mutates Claim.state (because it carries an AK2 set-control-number that matches Claim.patient_control_number) but does not persist a durable link row, so the operator cannot answer "which 999 acknowledged this claim?" or "which claims does this 999 acknowledge?". The 277CA path has the same gap. The TA1 path (envelope-level ack) has no per-claim granularity.

SP28 closes that gap: every ACK carries enough structural metadata to be tied back to the claim(s) it acknowledges, and the system auto-creates the link row at parse time so the operator never has to. A new claim_acks join table is the durable record of the link; the existing Claim.matched_remittance_id / Remittance.claim_id pair is the precedent.

In scope:

New claim_acks table (one row per AK2 set-response for 999, one row per claim-status for 277CA, one row per TA1 envelope for TA1).
New apply_999_acceptances companion to apply_999_rejections that creates link rows for all AK2 set-responses (both accepted AND rejected).
Refactor apply_277ca_rejections → apply_277ca_acks that creates link rows for every claim-status (accepted / payer-rejected / pending — anything with a payer_claim_control_number).
New apply_ta1_envelope_link that links a TA1 to the most-recent outbound 837 Batch whose ISA sender/receiver matches the TA1's sender_id / receiver_id (with an AckMatched activity event when the match is made).
New apply_claim_ack_links orchestrator that wires the three handlers into the existing 999 / 277CA / TA1 ingest paths.
Three new endpoints: GET /api/claims/{id}/acks, GET /api/acks/{id}/claims, POST /api/acks/{kind}/{id}/match-claim (manual fallback).
One new "Acknowledgments" panel in ClaimDrawer showing the linked 999/277CA/TA1 rows with deep-links to the matching AckDrawer.
One new "Matched claim" panel in AckDrawer showing the linked claim with deep-links to ClaimDrawer (and a "Link to claim…" dropdown for the orphan case).
One new "Ack orphans" candidate lane in the Inbox (sibling of the existing "Payer-rejected" lane), powered by GET /api/inbox/ack-orphans?kind=999.
Per-row live-tail event for claim_ack_written so the ClaimDrawer panel updates without a manual refresh.

Out of scope:

Restructuring the Ack / Ta1Ack / Two77caAck tables themselves (they keep their existing schemas; the link is purely additive).
Bulk re-linking of historical acks (a one-shot backfill SQL script lives outside the SP-N flow; the auto-linker only handles new ACKs).
TA1 per-claim granularity (TA1 has no AK2; envelope-level link to the originating Batch is the deepest granularity X12 supports for TA1).
Changing the existing Claim.state mutation behavior — apply_999_rejections still mutates REJECTED state, apply_277ca_rejections still mutates payer_rejected_* fields; SP28 only adds the link row on top.

2. Decisions (locked during brainstorming)

D1. Link granularity is per-AK2 / per-claim-status, not per-ACK row

A single 999 carries one AK2 per original 837 CLM. Persisting one link row per AK2 (rather than one per 999 row) lets the operator answer "which claims does this 999 acknowledge?" with a single SELECT on claim_acks, and lets the ClaimDrawer panel show per-segment accept/reject status without re-parsing raw_json. Same shape for 277CA (one row per ClaimStatus). TA1 falls back to one row per TA1 envelope (linked to the originating Batch).

D2. The link is stored in a separate `claim_acks` table — not a denormalized FK on `Ack`

Mirrors the existing Claim.matched_remittance_id / Remittance.claim_id precedent: a join table, not a column. Reasons: (1) one 999 row can link to many claims and vice versa; (2) the link row carries its own metadata (ak2_index, set_control_number, set_accept_reject_code, linked_at, linked_by); (3) deleting an Ack cascades cleanly to claim_acks via ON DELETE CASCADE. Mirrors cas_adjustments.remittance_id shape.

D3. Auto-link is eager at parse time — not lazy at read time

apply_999_acceptances / apply_277ca_acks / apply_ta1_envelope_link all run inside the same DB transaction that persists the Ack row. The operator never sees a row without its links (no "I'll figure it out later" stage). This mirrors apply_999_rejections already running at parse time. The trade-off (an extra lookup per set-response) is negligible — patient_control_number is indexed (ix_claims_patient_control_number).

D4. TA1 links to the originating `Batch`, not a `Claim`

TA1 is envelope-level (ISA/IEA). The closest thing to a "source claim" is the outbound 837 Batch that produced this envelope — specifically the most recent Batch whose sender_id + receiver_id matches the TA1's sender_id + receiver_id. The new claim_acks table allows claim_id IS NULL + batch_id (FK to batches.id) for this case; the ClaimDrawer panel filters out batch-level rows so the operator only sees per-claim acks, but the link is preserved for traceability.

D5. Manual fallback: `POST /api/acks/{kind}/{id}/match-claim`

When the auto-linker can't find the claim (orphan PCN — e.g., a 999 arrives before the 837 has been ingested), any logged-in user can manually link via a new endpoint. The endpoint takes a claim_id, looks it up, inserts a claim_acks row with linked_by="manual", and publishes a claim_ack_written event so the drawers refresh. Mirrors the existing POST /api/inbox/remit-orphans/{id}/match-claim endpoint shape (returns the updated ClaimDetail-style payload + a match_id). Auth posture differs from the remit-orphans endpoint: that one is admin-only because a wrong match on a remit can corrupt financial reconciliation; the ack match is metadata-only (no Claim.state mutation, no payment data) so any user can fix orphans. The endpoint rejects (409) when the claim is in a terminal state (REVERSED) or when the link already exists (idempotent 200).

D6. New live-tail event: `claim_ack_written`

Subscribed by both /api/claims/{id}/acks/stream and /api/acks/{id}/claims/stream so a manual match shows up in real time on both drawers. Payload is the new to_ui_claim_ack row shape (id, claim_id, ack_id, ack_kind, ak2_index, set_control_number, set_accept_reject_code, linked_at, linked_by). Adds a new slice to useTailStore + useTailStream + useMergedTail (mirrors SP25's acks/ta1_acks extension).

D7. New Inbox candidate lane: "Ack orphans"

Powered by GET /api/inbox/ack-orphans?kind=999 (and ?kind=277ca / ?kind=ta1). Lists acks where the auto-linker couldn't resolve the claim (orphan PCNs). Each row shows: ack id + kind, PCN, source batch, parsed_at, plus a top-3 candidate list (most-recent claims with the same PCN OR same patient_control_number pattern — relaxed PCN match). Bulk-action dismiss is supported; match is delegated to the existing manual endpoint.

D8. The `set_accept_reject_code` is persisted on the link row

So the ClaimDrawer panel can show "Rejected (999 AK5 R)" inline without re-parsing raw_json. The 999 path passes the per-AK2 code through apply_999_acceptances; the 277CA path passes the STC category code through apply_277ca_acks.

D9. No new auth boundary; existing auth posture covers it

Every new endpoint goes through the same request.state.user check as /api/claims and /api/acks. The POST /api/acks/{kind}/{id}/match-claim endpoint is any logged-in user (D5) — differs from POST /api/inbox/remit-orphans/{id}/match-claim which is admin-only. All read endpoints (GET /api/claims/{id}/acks, GET /api/acks/{kind}/{id}/claims, the two stream endpoints, GET /api/inbox/ack-orphans) are also any logged-in user. The auth boundary is unchanged.

3. Data model

3.1 New table — `claim_acks`

Mirrors cas_adjustments shape: PK + three FK columns + audit columns + a composite unique index.

Columns:

id INTEGER PK autoincrement
claim_id STRING(64) FK claims.id ON DELETE CASCADE NULL — nullable to allow batch-level rows (D4).
batch_id STRING(32) FK batches.id ON DELETE CASCADE NULL — nullable; populated for TA1 envelope-level links.
ack_id INTEGER NOT NULL — discriminated union with ack_kind (no FK constraint — acks / ta1_acks / two77ca_acks are three separate tables).
ack_kind STRING(8) NOT NULL — one of "999", "277ca", "ta1".
ak2_index INTEGER NULL — for 999 only; the AK2 set-response index that produced this link (NULL for 277CA and TA1).
set_control_number STRING(64) NULL — for 999/277CA only; the patient_control_number used for the lookup. Preserved for orphan traceability.
set_accept_reject_code STRING(8) NULL — for 999 (AK5 code: A/E/R/X); for 277CA (STC category: A1/A2/A3/A4/A6/A7 etc.).
linked_at DATETIME NOT NULL — set by the server.
linked_by STRING(16) NOT NULL — "auto" or "manual" (D5).

Indexes:

ix_claim_acks_claim_id(claim_id) — for GET /api/claims/{id}/acks and the live-tail subscription.
ix_claim_acks_ack(ack_kind, ack_id) — for GET /api/acks/{id}/claims and the reverse-direction live-tail subscription.
ix_claim_acks_batch_id(batch_id) — for TA1 batch-level lookups.
ux_claim_acks_dedup — UNIQUE(claim_id, ack_kind, ack_id, ak2_index) WHERE NOT NULL — prevents the auto-linker from creating duplicate rows on a re-ingest of the same file.

Migration: 0018_claim_acks.sql. Mirrors the 0011_processed_inbound_files.sql shape (CREATE TABLE + CREATE INDEX statements).

3.2 ORM model

backend/src/cyclone/db.py:ClaimAck(Base) follows the same pattern as CasAdjustment (line 376) — minimal typed columns + __table_args__ indexes. The claim, batch, and ack_* relationships are declared as Mapped[Optional[...]] but not eagerly loaded (no relationship(back_populates=...)) — the link is read-by-id via the new query helpers, not traversed.

3.3 New `to_ui_claim_ack` serializer

backend/src/cyclone/store/ui.py — mirrors to_ui_ack shape. Returns the row + a derived claim_state (queried in a single join: ClaimAck → Claim.state if claim_id is not NULL) so the drawer panel can render the colored ClaimStateBadge inline. For TA1 rows with claim_id IS NULL, claim_state is "n/a".

4. Wire-level changes

4.1 New endpoints

Method	Path	Auth	Purpose
`GET`	`/api/claims/{claim_id}/acks`	user	List every `claim_acks` row for a claim (per-claim only — filters out batch-level TA1 rows).
`GET`	`/api/acks/{kind}/{ack_id}/claims`	user	List every `claim_acks` row for an ack. `kind` ∈ `999` \| `277ca` \| `ta1`. For 999/277CA, returns one entry per AK2/ClaimStatus.
`GET`	`/api/claims/{claim_id}/acks/stream`	user	NDJSON live-tail of `claim_ack_written` events filtered to `claim_id`.
`GET`	`/api/acks/{kind}/{ack_id}/claims/stream`	user	NDJSON live-tail filtered to `ack_id` + `kind`.
`POST`	`/api/acks/{kind}/{ack_id}/match-claim`	user	Manual link fallback (D5, D9). Body: `{"claim_id": "..."}`. Idempotent — returns 200 with existing row if already linked.
`DELETE`	`/api/acks/{kind}/{ack_id}/match-claim/{claim_id}`	user	Unlink (preserves `Claim.state` mutation; only removes the `claim_acks` row + publishes `claim_ack_dropped`).
`GET`	`/api/inbox/ack-orphans`	user	Lists acks with no resolvable claim. Query: `?kind=999` \| `?kind=277ca` \| `?kind=ta1`. Mirrors the existing `/api/inbox/remit-orphans` shape.

4.2 Modified endpoints

POST /api/parse-999 / POST /api/parse-277ca / POST /api/parse-ta1 — internally call the new apply_claim_ack_links orchestrator. Wire shape unchanged; the response gains a new claim_ack_links: { matched: N, orphans: N, linked: [{claim_id, ack_kind, set_accept_reject_code}, …] } field for observability.
GET /api/acks / GET /api/ta1-acks / GET /api/277ca-acks — each item gains linked_claim_ids: list[str] so the Acks page row can show the "🔗 N claims" badge inline.
GET /api/claims/{id} — the response gains ack_links: list[dict] (compact form: [{ack_id, ack_kind, set_accept_reject_code, parsed_at}]) so the ClaimDrawer can render the panel on initial load.
GET /api/acks/{id}/detail (and the TA1 / 277CA counterparts) — each gains claim_links: list[dict] so the AckDrawer can render the matched-claim panel on initial load.

4.3 New pubsub event

claim_ack_written — payload is the new to_ui_claim_ack row shape (3.3). Subscribed by both /api/claims/{id}/acks/stream and /api/acks/{id}/claims/stream. Mirrors the existing claim_written / remittance_written payload shape.
claim_ack_dropped — fired on DELETE /api/acks/{kind}/{id}/match-claim/{claim_id}. Payload is the dropped row id + the affected claim_id so subscribers can remove the link from their local store.

5. Frontend

5.1 New `useClaimAcks(claim_id)` hook

src/hooks/useClaimAcks.ts — TanStack Query, key ["claim-acks", claimId], calls api.listClaimAcks(claimId). Mirrors useClaims shape. The live-tail subscription (useTailStream("claim-acks") + useMergedTail) lives on the ClaimDrawer (per cyclone-frontend-page convention #3 — subscription on the page, not the hook).

5.2 New `useAckClaims(kind, ackId)` hook

src/hooks/useAckClaims.ts — mirror of useClaimAcks, scoped to a single ack. The AckDrawer mounts the subscription.

5.3 `ClaimDrawer` — new "Acknowledgments" panel

Inserted between the existing "Service lines" and "Reconciliation" panels. Header: "Acknowledgments (N)". Body: a list of compact rows, each showing set_accept_reject_code (color-coded), parsed_at, ack kind badge, and a → link to the matching AckDrawer. If the link list is empty, render the existing <EmptyState> with copy "No acknowledgments received yet for this claim." Subscribes to useClaimAcks(claimId) + useTailStream("claim-acks") so a 999 landing in real time appends a row.

5.4 `AckDrawer` — new "Matched claim" panel

Inserted as the first panel (before "Set responses"). For 999 / 277CA, shows the linked claim(s) — one card per AK2 / ClaimStatus — with patient_control_number, current Claim.state (via the new ClaimStateBadge), charge amount, and a → link to ClaimDrawer. For TA1, shows the originating Batch (batch id + filename + parsed_at) instead. If the link list is empty, render a "Link to claim…" dropdown (calls the manual-match endpoint, then refetches). On a successful match, the panel swaps to the populated card view.

5.5 `Acks` page — new `linkedClaimCount` cell

The Acks table row gains a new column "Claims" between "Accepted" and "Rejected" — value is the count of linked_claim_ids from the list endpoint, rendered as a Badge (e.g. 5 claims). Clicking the badge when count > 0 navigates to the first claim's ClaimDrawer (deep-link round-trips via useDrawerUrlState).

5.6 `Inbox` — new "Ack orphans" lane

Mirrors the existing "Payer-rejected" lane (5th lane). Header: "Ack orphans (N)". Body: a compact table of orphan acks with a per-row "Dismiss" button and a "Link to…" inline dropdown. The lane is empty when all acks auto-linked successfully — same EmptyState shape as the other lanes.

5.7 `useTailStore` extension

Add claimAcks: Record<string, ClaimAck> + claimAckOrder: number[] slice. Mirrors the SP25 acks/ta1_acks slice shape (key-by-id with first-write-wins dedup, FIFO-capped at TAIL_CAP = 10_000).

6. Test impact

Backend — new fixtures

backend/tests/fixtures/minimal_999_2ak2.txt — a 999 with TWO AK2 set-responses (one accepted, one rejected) so the per-AK2 link granularity is exercised. Derived from the existing minimal_999.txt by adding a second AK2/AK5 pair.
backend/tests/fixtures/minimal_277ca_2status.txt — same shape, two ClaimStatus entries.

Backend — new tests

backend/tests/test_apply_claim_ack_links.py:
- test_999_auto_creates_per_ak2_link_rows — submit minimal_999_2ak2.txt against a fixture-seeded claim table, assert claim_acks has 2 rows, one with set_accept_reject_code="A" and one with "R".
- test_999_orphan_does_not_create_link — submit a 999 with an AK2 whose set_control_number matches no claim, assert no claim_acks row, assert claim_ack_links.orphans contains the PCN.
- test_277ca_accepted_creates_link_with_no_state_mutation — submit a 277CA with classification="accepted", assert claim_acks row exists AND Claim.payer_rejected_at IS NULL.
- test_277ca_rejected_creates_link_and_mutates_state — mirror the existing apply_277ca_rejections test, but also assert the link row.
- test_ta1_links_to_most_recent_matching_batch — seed two Batch rows with distinct sender/receiver, submit a TA1 whose sender/receiver matches batch #2, assert the link row's batch_id is batch #2.
- test_reingest_same_999_is_idempotent — submit the same 999 twice, assert exactly one claim_acks row per AK2 (the ux_claim_acks_dedup unique index enforces this).
- test_manual_link_endpoint_idempotent — call POST /api/acks/999/{id}/match-claim twice with the same claim_id, assert one row + 200 OK both times.
- test_manual_link_any_user_succeeds — call the endpoint as a non-admin user (D9), assert 200 + the link row created. Confirms the explicit deviation from the admin-only remit-orphans posture.
- test_manual_link_rejects_terminal_claim — try to link a 999 to a claim in state=REVERSED, assert 409 (D5 / D9 — terminal state guard).
- test_unlink_does_not_revert_claim_state — link a 999 with set_accept_reject_code="R", then unlink it, assert the claim is STILL state=REJECTED (D6: unlinking is purely about the link row).
backend/tests/test_api_claim_acks.py:
- test_claim_detail_includes_ack_links — GET /api/claims/{id} after ingesting a 999, assert ack_links is populated.
- test_acks_list_includes_linked_claim_ids — GET /api/acks after ingesting, assert linked_claim_ids is populated per row.
- test_claim_acks_stream_emits_claim_ack_written — open the stream, manually link via the endpoint, assert one item event arrives within 2s (uses the SP25 test pattern: synthetic Request + iterate body_iterator directly + monkeypatch.setenv("CYCLONE_TAIL_HEARTBEAT_S", "0.2")).

Frontend — new tests

src/hooks/useClaimAcks.test.ts — calls the hook, asserts the API is invoked with the right key.
src/components/ClaimDrawer/ClaimDrawer.test.tsx — extend the existing test to assert the "Acknowledgments" panel renders when ack_links is non-empty, and renders the <EmptyState> when empty.
src/components/AckDrawer/AckDrawer.test.tsx — extend the existing test to assert the "Matched claim" panel renders for a 999 with one AK2 link, and renders the "Link to claim…" dropdown when no link exists.
src/pages/Acks.test.tsx — extend the existing test to assert the "Claims" badge column renders linkedClaimCount.
src/pages/Inbox.test.tsx — add coverage for the new "Ack orphans" lane (this is on the 10 pre-existing failure list — SP28 may incidentally fix some by giving the test a real handler to assert against).

End-to-end smoke

backend/tests/test_e2e_999_to_claim_drawer.py — a single test that ingests a 999 with two AK2s against a seeded claim, then calls GET /api/claims/{id} and GET /api/acks/999/{id}/claims and asserts the two views agree on the link set. (Mirrors the existing test_e2e_837_to_835 smoke.)

7. Risks & mitigations

Risk: Migration 0018_claim_acks.sql runs against a production DB with thousands of existing 999 rows. The migration adds a new table; it does NOT backfill historical links. Mitigation: a one-shot backfill script (scripts/backfill_claim_acks.py) lives outside the SP-N flow and is run manually after the deploy. Documented in the runbook delta.
Risk: apply_999_acceptances running alongside apply_999_rejections double-touches the same DB session. Mitigation: both helpers share a single session; the link row is created inside the same for sr in set_responses: loop as the rejection mutation, so there's exactly one transaction.
Risk: The ux_claim_acks_dedup unique index rejects re-ingests of the same file with a sqlite3.IntegrityError. Mitigation: the orchestrator catches this specifically (mirrors the existing apply_999_rejections idempotency check at line 60) and treats it as success — re-ingest of an identical file is a no-op.
Risk: TA1 batch-level links (claim_id IS NULL) confuse the ClaimDrawer panel. Mitigation: the new /api/claims/{id}/acks endpoint explicitly filters them out; the to_ui_claim_ack serializer tags them with claim_state="n/a" so they're visually distinct even if a future endpoint leaks them through.
Risk: Manual-match endpoint opens a window for the operator to attach an ack to the wrong claim. Mitigation: the endpoint requires admin role + the claim must currently be in {SUBMITTED, RECEIVED, REJECTED, PAID, PARTIAL, DENIED} (i.e., not in a reversed/closed state). Returns 409 if the claim is in a terminal state.

8. Implementation shape (preview)

The work breaks into ~14 tasks across 6 phases. The full plan lives at docs/superpowers/plans/2026-07-02-cyclone-ack-claim-auto-link.md (TBD).

Migration — 0018_claim_acks.sql + ClaimAck ORM model + db._reset_for_tests() updates.
Pure helpers — cyclone.claim_acks.py module with apply_999_acceptances, apply_277ca_acks, apply_ta1_envelope_link, link_manual.
Store facade — extend CycloneStore with add_claim_ack, list_acks_for_claim, list_claims_for_ack, find_ack_orphans. Add the claim_ack_written / claim_ack_dropped event publish.
Handler integration — wire apply_claim_ack_links into handle_999.handle, the 277CA handler, and the TA1 handler. Mirror the existing apply_999_rejections call site at handlers/handle_999.py:79-92.
API endpoints — new api_routers/claim_acks.py (mirrors api_routers/acks.py shape) + register the router. Extend the existing /api/claims/{id} and /api/acks serializers.
Frontend — useClaimAcks + useAckClaims hooks, ClaimDrawer panel, AckDrawer panel, Acks page column, Inbox lane, useTailStore slice.

Estimated lines: ~1,200 backend + ~800 frontend + ~600 tests. Mirrors SP25's footprint.

9. Non-goals

Auto-rejection reconciliation (using 999 AK5 to mutate Claim.state in ways other than REJECTED) — out of scope; that's a separate SP if the user wants it later.
Cross-batch deduplication (a single 999 acknowledging claims from multiple batches because the upstream sender batched them) — out of scope; the link row stores set_control_number and the operator can correlate via that field.
Bulk re-link UI (select 100 orphan acks and link them all in one click) — out of scope; the per-row "Link to claim…" dropdown handles the operator workflow at the volumes we see.
TA1 per-claim granularity — X12 doesn't support it; envelope-level link to Batch is the deepest we can go.
Notification when a long-pending claim receives an ack — out of scope; the live-tail covers the "in-app" case, and an email/SMS notification system doesn't exist yet.

10. Open questions for sign-off

Confirm SP28 is the correct number (the skill text says "next free is SP25"; actual highest merged is SP27 — SP28 is the next free).
Confirm the per-AK2 granularity (D1) is preferred over per-999-row granularity. The trade-off is one extra row per AK2 (vs. collapsing to one per 999 file) — this is what the user implicitly described as "it should be doing itself".
Confirm manual-match requires admin role (D5, D9) — same posture as the existing /api/inbox/remit-orphans/{id}/match-claim.
Confirm the Inbox ack-orphans lane (D7) is desired — alternative is to surface orphans only in a filter on the Acks page, not a new lane.
Confirm the historical backfill is OUT of scope for the SP-N flow (becomes a one-shot script + runbook delta) — alternative is to include it as a Task 0 in the plan.

24 KiB Raw Blame History