6 Commits

Author SHA1 Message Date
Tyler de55003484 chore: stash in-progress test edit
Captures the modified state of test_api_parse_persists.py so the
in-progress test work is preserved on the claims-unique-fix branch
as the sp-prefixed worktrees are cleaned up. The .venv and
.gitignore_local noise in the working tree are intentionally left
untracked.
2026-06-22 11:05:56 -06:00
Grok c055787bd4 refactor(db_migrate): extract _apply_migrations_until helper; add edge-case tests
Issue 1 (refactor): run() and migrate_to_version() duplicated the inner
loop body — every change to the migration loop had to be applied in both
places. Pull the loop into _apply_migrations_until(engine, target_version):
when target_version is None, apply all pending; otherwise stop when the
file's version exceeds target. run() and migrate_to_version() are now
thin wrappers around the helper.

Issue 2 (tests): migrate_to_version() had no direct tests for its
edge-case behavior. Add 4 tests:
- idempotent: calling migrate_to_version(engine, N) twice == once
- lower-than-current: migrate_to_version(engine, 3) after N=13 is a no-op
- zero on fresh DB: migrate_to_version(engine, 0) is a no-op
- beyond-max: migrate_to_version(engine, 999) applies every migration

Tests use a fresh sa.Engine (not db.engine()) so they exercise the
target-version behavior from a clean user_version=0 starting point —
the conftest autouse fixture has already brought the module engine to
the latest version, which would mask the no-op branches.
2026-06-21 19:43:23 -06:00
Tyler ced8d20aed fix(db+tests): data-preservation test for migration 0014; document before_insert events
Address the three concerns raised by the spec reviewer on commit 534130e:

1. The 0014 data-preservation test was degraded: it only verified
   post-0014 INSERTs round-trip, never proving that 0014's INSERT INTO
   claims_new SELECT * FROM claims actually carries pre-existing rows
   through the table recreation. Add migrate_to_version() helper to
   db_migrate.py and rewrite the test to seed at v=13, apply 0014, and
   assert the rows survive. Add a complementary test exercising the
   JOIN-FK tables (matches / cas_adjustments / service_line_payments /
   line_reconciliations) whose batch_id columns get populated by
   0014's JOIN against the parent tables.

2. The four SQLAlchemy ORM before_insert events on Match /
   CasAdjustment / ServiceLinePayment / LineReconciliation are
   undocumented in the codebase. Add a clear module-level docstring
   explaining what the events do, why they exist (composite-FK
   NOT NULL columns require the batch side which most call sites
   don't have readily available), when they fail (parent id not
   findable in session.new / identity_map / DB -> misleading
   'NOT NULL constraint failed' IntegrityError), and the workarounds.

3. The plan Task 1.3 Step 8 said 'do not modify production code to
   make old tests pass'. In practice production code DID need to
   change because composite-PK semantics make single-id lookups
   ambiguous. Amend the plan to acknowledge this and cite the
   s.get(Claim, X) -> s.query(Claim).filter().first() refactor as
   the example. Also document the before_insert events in Step 5.

Brings in docs/superpowers/plans/2026-06-21-cyclone-parse-decide-workflow.md
from main (this branch was created before the plan file was committed)
with Task 1.3 Step 5 and Step 8 amended per the reviewer feedback.
2026-06-21 19:24:13 -06:00
Tyler 534130ee2b feat(db): migration 0014 relaxes claims/remittances PK to (batch_id, id)
Migration 0014 changes the PRIMARY KEYs of claims and remittances from
single-column id to composite (batch_id, id). Enables the spec'd
cross-batch CLM01 / CLP01 collision workflow: same CLM01 in multiple
batches is now representable (resubmits), pre-flight dedup 409 path is
genuinely exercisable, force-insert can skip pre-existing duplicates.

Strategy: PRAGMA defer_foreign_keys = ON + table recreation for every
table whose FKs pointed at the old single-column PK. Tables recreated:
remittances, claims, matches, cas_adjustments, service_line_payments,
line_reconciliations. Each child table gains a batch_id (or
remittance_batch_id) column for the composite FK side; INSERT INTO new
SELECT FROM old JOINs populate it from the already-recreated parent.

Cross-table FKs (remittances.claim_id, claims.matched_remittance_id)
cannot be SQL-enforced with composite PKs (SQLite has no ALTER
CONSTRAINT). Dropped at SQL level; enforced via application-layer
invariants in store.manual_match / manual_unmatch / reconcile.run and
the dedup.preflight_* helpers.

ORM updates (db.py):
- Claim / Remittance: composite PK via explicit PrimaryKeyConstraint in
  __table_args__ (column order matches SQL: batch_id, id).
- New Claim.matched_remittance_batch_id column.
- Match / CasAdjustment / ServiceLinePayment / LineReconciliation:
  added the batch side of their composite FK to the parent table.
- SQLAlchemy before_insert events auto-populate the batch side of the
  composite FK from session.new, then identity_map, then SQL fallback.

Production code updates:
- store.manual_match / manual_unmatch: also write
  matched_remittance_batch_id on the claim (was missing).
- api.py manual-match endpoint: same fix.
- reconcile.run: same fix for auto-matched pairs.

Test updates: replaced s.get(Claim, X) with the composite key
(batch_id, id) where batch_id is known, or s.query().filter().first()
where the test only knows the id. Tests that previously inserted a Match
row pointing at a non-existent Remittance now seed the parent Remittance
so the new NOT NULL composite FK is satisfied.
2026-06-21 18:56:18 -06:00
Tyler 890207f40d feat(store): add find_existing_batch_for_claim and find_existing_batch_for_remit 2026-06-21 17:40:10 -06:00
Tyler b6efd0eaee feat(db): drop inline UNIQUE(batch_id, patient_control_number) via migration 0013 2026-06-21 17:39:28 -06:00
24 changed files with 4442 additions and 106 deletions
+1
View File
@@ -7,3 +7,4 @@ __pycache__/
venv/ venv/
build/ build/
dist/ dist/
backend/.venv
+21 -7
View File
@@ -1093,8 +1093,12 @@ def inbox_match_candidate(remit_id: str, body: dict):
if not claim_id: if not claim_id:
raise HTTPException(400, "claim_id required") raise HTTPException(400, "claim_id required")
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
claim = s.get(Claim, claim_id) # Migration 0014: composite PKs. The URL takes only id (not
remit = s.get(Remittance, remit_id) # batch_id); look up by id alone, first match wins. Cross-batch
# duplicates are handled at ingest time by parse-decide-workflow;
# this manual-match endpoint inherits the legacy "id alone" API.
claim = s.query(Claim).filter(Claim.id == claim_id).first()
remit = s.query(Remittance).filter(Remittance.id == remit_id).first()
if claim is None or remit is None: if claim is None or remit is None:
raise HTTPException(404, "claim or remit not found") raise HTTPException(404, "claim or remit not found")
if claim.matched_remittance_id and claim.matched_remittance_id != remit_id: if claim.matched_remittance_id and claim.matched_remittance_id != remit_id:
@@ -1110,6 +1114,11 @@ def inbox_match_candidate(remit_id: str, body: dict):
}, },
) )
claim.matched_remittance_id = remit_id claim.matched_remittance_id = remit_id
# Migration 0014: composite back-reference to remittances.
# matched_remittance_batch_id is the batch side; the SQL-level FK
# to remittances(batch_id, id) can't be declared in SQLite (no
# ALTER CONSTRAINT) so the application keeps both columns in sync.
claim.matched_remittance_batch_id = remit.batch_id
remit.claim_id = claim_id remit.claim_id = claim_id
s.commit() s.commit()
return {"ok": True, "claim_id": claim_id, "remit_id": remit_id} return {"ok": True, "claim_id": claim_id, "remit_id": remit_id}
@@ -1159,7 +1168,8 @@ def inbox_acknowledge_payer_rejected(body: dict):
not_found = 0 not_found = 0
not_rejected = 0 not_rejected = 0
for cid in claim_ids: for cid in claim_ids:
claim = session.get(Claim, cid) # Migration 0014: composite PK; URL takes only claim_id.
claim = session.query(Claim).filter(Claim.id == cid).first()
if claim is None: if claim is None:
not_found += 1 not_found += 1
continue continue
@@ -1229,7 +1239,8 @@ def inbox_resubmit_rejected(
accepted_with_rows: list[tuple[str, "Claim"]] = [] accepted_with_rows: list[tuple[str, "Claim"]] = []
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
for cid in ids: for cid in ids:
c = s.get(Claim, cid) # Migration 0014: composite PK; URL takes only claim_id.
c = s.query(Claim).filter(Claim.id == cid).first()
if c is None: if c is None:
continue continue
if c.state != ClaimState.REJECTED: if c.state != ClaimState.REJECTED:
@@ -1521,7 +1532,8 @@ def serialize_claim_as_837(claim_id: str):
issue, not a transient failure). issue, not a transient failure).
""" """
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
row = s.get(Claim, claim_id) # Migration 0014: composite PK; URL takes only claim_id.
row = s.query(Claim).filter(Claim.id == claim_id).first()
if row is None: if row is None:
return JSONResponse( return JSONResponse(
{"error": "Not found", "detail": f"Claim {claim_id} not found"}, {"error": "Not found", "detail": f"Claim {claim_id} not found"},
@@ -1584,7 +1596,8 @@ def get_claim_line_reconciliation(claim_id: str) -> dict:
from decimal import Decimal from decimal import Decimal
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
claim = s.get(db.Claim, claim_id) # Migration 0014: composite PK; URL takes only claim_id.
claim = s.query(db.Claim).filter(db.Claim.id == claim_id).first()
if claim is None: if claim is None:
raise HTTPException( raise HTTPException(
status_code=404, status_code=404,
@@ -2363,7 +2376,8 @@ def _serialize_claim_for_submit(claim_id: str) -> str:
from cyclone.parsers.serialize_837 import serialize_837 from cyclone.parsers.serialize_837 import serialize_837
from cyclone import db from cyclone import db
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
row = s.get(db.Claim, claim_id) # Migration 0014: composite PK; URL takes only claim_id.
row = s.query(db.Claim).filter(db.Claim.id == claim_id).first()
if row is None: if row is None:
raise ValueError(f"claim {claim_id!r} not found") raise ValueError(f"claim {claim_id!r} not found")
# Re-parse the stored raw_json to get a ClaimOutput # Re-parse the stored raw_json to get a ClaimOutput
+215 -22
View File
@@ -34,6 +34,7 @@ from sqlalchemy import (
) )
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship, sessionmaker from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship, sessionmaker
from sqlalchemy.types import TypeDecorator from sqlalchemy.types import TypeDecorator
from sqlalchemy.schema import PrimaryKeyConstraint
DEFAULT_DB_PATH = Path.home() / ".local" / "share" / "cyclone" / "cyclone.db" DEFAULT_DB_PATH = Path.home() / ".local" / "share" / "cyclone" / "cyclone.db"
@@ -239,9 +240,15 @@ class Batch(Base):
class Claim(Base): class Claim(Base):
__tablename__ = "claims" __tablename__ = "claims"
id: Mapped[str] = mapped_column(String(64), primary_key=True) # Migration 0014: composite PRIMARY KEY (batch_id, id). Enables
# resubmits (same CLM01 in different batches) and makes the
# pre-flight dedup workflow exercisable. The composite PK is declared
# explicitly in __table_args__ below so the column order matches the
# migration (`PRIMARY KEY (batch_id, id)`).
id: Mapped[str] = mapped_column(String(64))
batch_id: Mapped[str] = mapped_column( batch_id: Mapped[str] = mapped_column(
String(32), ForeignKey("batches.id", ondelete="CASCADE"), nullable=False String(32),
ForeignKey("batches.id", ondelete="CASCADE"),
) )
patient_control_number: Mapped[str] = mapped_column(String(64), nullable=False) patient_control_number: Mapped[str] = mapped_column(String(64), nullable=False)
service_date_from: Mapped[Optional[date]] = mapped_column(Date, nullable=True) service_date_from: Mapped[Optional[date]] = mapped_column(Date, nullable=True)
@@ -290,29 +297,38 @@ class Claim(Base):
resubmit_count: Mapped[int] = mapped_column( resubmit_count: Mapped[int] = mapped_column(
Integer, nullable=False, default=0, server_default=text("0") Integer, nullable=False, default=0, server_default=text("0")
) )
# Back-reference to remittances. The composite pointer is split into
# matched_remittance_id + matched_remittance_batch_id. Migration 0014
# drops the SQL-level FK (composite FK to remittances(batch_id, id)
# cannot be declared as inline REFERENCES in SQLite; no ALTER CONSTRAINT).
# App-layer invariants in store.manual_match / manual_unmatch / dedup
# keep these consistent. matched_remittance_id points at
# remittances.id (a non-PK column under composite PK) — SQLAlchemy
# accepts this because FKs reference any column, not just PKs.
matched_remittance_id: Mapped[Optional[str]] = mapped_column( matched_remittance_id: Mapped[Optional[str]] = mapped_column(
# ORM policy: SET NULL on remittance delete. The claim may outlive its
# remittance during reversal/reimport flows; without this, deleting a
# remittance with matched claims raises IntegrityError.
# Note: 0001_initial.sql predates this policy (no ON DELETE clause).
# SQLite ignores FK direction by default unless PRAGMA foreign_keys=ON,
# so the inconsistency is benign there; the ORM is the source of truth
# for PostgreSQL/test environments with FK enforcement. Amendment to the
# migration is deferred to post-SQLite rollout.
String(64), String(64),
ForeignKey("remittances.id", ondelete="SET NULL"), ForeignKey("remittances.id", ondelete="SET NULL"),
nullable=True, nullable=True,
) )
matched_remittance_batch_id: Mapped[Optional[str]] = mapped_column(
String(32),
nullable=True,
)
raw_json: Mapped[Optional[dict]] = mapped_column(JSONText, nullable=True) raw_json: Mapped[Optional[dict]] = mapped_column(JSONText, nullable=True)
batch: Mapped["Batch"] = relationship(back_populates="claims") batch: Mapped["Batch"] = relationship(back_populates="claims")
__table_args__ = ( __table_args__ = (
# Migration 0014: explicit composite PK column order (batch_id, id).
# SQLAlchemy's default composite-PK assembly is by declaration order,
# but `id` is declared first for readability; we override here so
# ``s.get(Claim, (batch_id, id))`` looks up by the right key order.
PrimaryKeyConstraint("batch_id", "id", name="pk_claims"),
# NOTE: no (batch_id, patient_control_number) unique constraint. # NOTE: no (batch_id, patient_control_number) unique constraint.
# X12 837P allows any number of CLM segments inside one 2000B # X12 837P allows any number of CLM segments inside one 2000B
# subscriber loop, so a single member routinely has multiple # subscriber loop, so a single member routinely has multiple
# claims in a single submission batch. Claim identity is provided # claims in a single submission batch. Claim identity is provided
# by ``id`` (= CLM01 claim_id), which is the primary key. # by the composite primary key (batch_id, id).
Index("ix_claims_state", "state"), Index("ix_claims_state", "state"),
Index("ix_claims_patient_control_number", "patient_control_number"), Index("ix_claims_patient_control_number", "patient_control_number"),
Index("ix_claims_service_date_from", "service_date_from"), Index("ix_claims_service_date_from", "service_date_from"),
@@ -322,17 +338,19 @@ class Claim(Base):
class Remittance(Base): class Remittance(Base):
__tablename__ = "remittances" __tablename__ = "remittances"
id: Mapped[str] = mapped_column(String(64), primary_key=True) # Migration 0014: composite PRIMARY KEY (batch_id, id). Same rationale
# as Claim: enables resubmits (same CLP01 in different batches). The
# composite PK is declared explicitly in __table_args__ below so the
# column order matches the migration.
id: Mapped[str] = mapped_column(String(64))
batch_id: Mapped[str] = mapped_column( batch_id: Mapped[str] = mapped_column(
String(32), ForeignKey("batches.id", ondelete="CASCADE"), nullable=False String(32),
ForeignKey("batches.id", ondelete="CASCADE"),
) )
payer_claim_control_number: Mapped[str] = mapped_column(String(64), nullable=False) payer_claim_control_number: Mapped[str] = mapped_column(String(64), nullable=False)
# ORM policy: no ON DELETE (forward FK from Remittance → Claim). A claim # Forward FK from Remittance → Claim. As with Claim.matched_remittance_id,
# is unlikely to be deleted in normal flow (audit trail); if it ever is, # migration 0014 drops the SQL-level FK (cannot declare composite FK
# the application layer (T7 reconciliation) must clear this FK. The # inline in SQLite). App-layer invariants keep these consistent.
# migration predates this rationale; amendment deferred to post-SQLite
# rollout. SQLite without `PRAGMA foreign_keys=ON` does not enforce,
# so the divergence is benign in this engine.
claim_id: Mapped[Optional[str]] = mapped_column( claim_id: Mapped[Optional[str]] = mapped_column(
String(64), ForeignKey("claims.id"), nullable=True String(64), ForeignKey("claims.id"), nullable=True
) )
@@ -352,14 +370,24 @@ class Remittance(Base):
batch: Mapped["Batch"] = relationship(back_populates="remittances") batch: Mapped["Batch"] = relationship(back_populates="remittances")
cas_adjustments: Mapped[list["CasAdjustment"]] = relationship( cas_adjustments: Mapped[list["CasAdjustment"]] = relationship(
back_populates="remittance", cascade="all, delete-orphan" back_populates="remittance",
cascade="all, delete-orphan",
# Migration 0014: CasAdjustment has TWO FKs to remittances
# (remittance_id and remittance_batch_id). Tell SQLAlchemy to use
# remittance_id for the relationship (the batch_id side is just a
# denormalized copy used for the composite FK constraint).
foreign_keys="CasAdjustment.remittance_id",
) )
__table_args__ = ( __table_args__ = (
# Migration 0014: explicit composite PK column order (batch_id, id).
# See Claim.__table_args__ for the rationale — same as for Claim.
PrimaryKeyConstraint("batch_id", "id", name="pk_remittances"),
# NOTE: no (batch_id, payer_claim_control_number) unique constraint. # NOTE: no (batch_id, payer_claim_control_number) unique constraint.
# An 835 ERA can contain multiple CLP segments that share a # An 835 ERA can contain multiple CLP segments that share a
# payer_claim_control_number (e.g. reversals re-referencing the # payer_claim_control_number (e.g. reversals re-referencing the
# original PCN). Remittance identity is provided by ``id``. # original PCN). Remittance identity is provided by the composite
# PK (batch_id, id).
Index("ix_remittances_claim_id", "claim_id"), Index("ix_remittances_claim_id", "claim_id"),
Index("ix_remittances_payer_claim_control_number", "payer_claim_control_number"), Index("ix_remittances_payer_claim_control_number", "payer_claim_control_number"),
Index("ix_remittances_status_code", "status_code"), Index("ix_remittances_status_code", "status_code"),
@@ -373,6 +401,15 @@ class CasAdjustment(Base):
remittance_id: Mapped[str] = mapped_column( remittance_id: Mapped[str] = mapped_column(
String(64), ForeignKey("remittances.id", ondelete="CASCADE"), nullable=False String(64), ForeignKey("remittances.id", ondelete="CASCADE"), nullable=False
) )
# Migration 0014: remittances.id is no longer a single-column PK; we add
# the batch side of the composite FK to make the constraint declarative
# again. The SQL migration declares it as a real composite FK; here we
# use a plain column reference because the application always queries
# via remittance_id alone (the batch_id is denormalized for FK
# enforcement only — same value for every row with a given remittance_id).
remittance_batch_id: Mapped[str] = mapped_column(
String(32), ForeignKey("remittances.batch_id"), nullable=False
)
group_code: Mapped[str] = mapped_column(String(4), nullable=False) group_code: Mapped[str] = mapped_column(String(4), nullable=False)
reason_code: Mapped[str] = mapped_column(String(8), nullable=False) reason_code: Mapped[str] = mapped_column(String(8), nullable=False)
amount: Mapped[Decimal] = mapped_column(Numeric(12, 2), nullable=False) amount: Mapped[Decimal] = mapped_column(Numeric(12, 2), nullable=False)
@@ -383,7 +420,12 @@ class CasAdjustment(Base):
nullable=True, nullable=True,
) )
remittance: Mapped["Remittance"] = relationship(back_populates="cas_adjustments") remittance: Mapped["Remittance"] = relationship(
back_populates="cas_adjustments",
# See Remittance.cas_adjustments for the rationale: disambiguate
# between the two FKs from CasAdjustment to remittances.
foreign_keys="CasAdjustment.remittance_id",
)
__table_args__ = ( __table_args__ = (
Index("ix_cas_adjustments_remittance_id", "remittance_id"), Index("ix_cas_adjustments_remittance_id", "remittance_id"),
@@ -405,6 +447,10 @@ class ServiceLinePayment(Base):
remittance_id: Mapped[str] = mapped_column( remittance_id: Mapped[str] = mapped_column(
String(64), ForeignKey("remittances.id", ondelete="CASCADE"), nullable=False String(64), ForeignKey("remittances.id", ondelete="CASCADE"), nullable=False
) )
# Migration 0014: see CasAdjustment.remittance_batch_id for the same rationale.
remittance_batch_id: Mapped[str] = mapped_column(
String(32), ForeignKey("remittances.batch_id"), nullable=False
)
line_number: Mapped[int] = mapped_column(Integer, nullable=False) line_number: Mapped[int] = mapped_column(Integer, nullable=False)
procedure_qualifier: Mapped[str] = mapped_column(String(4), nullable=False) procedure_qualifier: Mapped[str] = mapped_column(String(4), nullable=False)
procedure_code: Mapped[str] = mapped_column(String(16), nullable=False) procedure_code: Mapped[str] = mapped_column(String(16), nullable=False)
@@ -453,6 +499,10 @@ class LineReconciliation(Base):
claim_id: Mapped[str] = mapped_column( claim_id: Mapped[str] = mapped_column(
String(64), ForeignKey("claims.id", ondelete="CASCADE"), nullable=False String(64), ForeignKey("claims.id", ondelete="CASCADE"), nullable=False
) )
# Migration 0014: see CasAdjustment.remittance_batch_id for the same rationale.
batch_id: Mapped[str] = mapped_column(
String(32), ForeignKey("claims.batch_id"), nullable=False
)
claim_service_line_number: Mapped[Optional[int]] = mapped_column( claim_service_line_number: Mapped[Optional[int]] = mapped_column(
Integer, nullable=True Integer, nullable=True
) )
@@ -491,9 +541,17 @@ class Match(Base):
String(64), ForeignKey("claims.id", ondelete="CASCADE"), String(64), ForeignKey("claims.id", ondelete="CASCADE"),
nullable=False, nullable=False,
) )
# Migration 0014: batch side of the composite FK to claims.
batch_id: Mapped[str] = mapped_column(
String(32), ForeignKey("claims.batch_id"), nullable=False
)
remittance_id: Mapped[str] = mapped_column( remittance_id: Mapped[str] = mapped_column(
String(64), ForeignKey("remittances.id", ondelete="CASCADE"), nullable=False String(64), ForeignKey("remittances.id", ondelete="CASCADE"), nullable=False
) )
# Migration 0014: batch side of the composite FK to remittances.
remittance_batch_id: Mapped[str] = mapped_column(
String(32), ForeignKey("remittances.batch_id"), nullable=False
)
strategy: Mapped[str] = mapped_column(String(16), nullable=False) strategy: Mapped[str] = mapped_column(String(16), nullable=False)
matched_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False) matched_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
prior_claim_state: Mapped[Optional[ClaimState]] = mapped_column( prior_claim_state: Mapped[Optional[ClaimState]] = mapped_column(
@@ -836,3 +894,138 @@ class ClearhouseORM(Base):
filename_block_json: Mapped[dict] = mapped_column(JSONText, nullable=False) filename_block_json: Mapped[dict] = mapped_column(JSONText, nullable=False)
sftp_block_json: Mapped[dict] = mapped_column(JSONText, nullable=False) sftp_block_json: Mapped[dict] = mapped_column(JSONText, nullable=False)
updated_at: Mapped[str] = mapped_column(String(32), nullable=False) updated_at: Mapped[str] = mapped_column(String(32), nullable=False)
# =============================================================================
# Migration 0014: ORM `before_insert` events auto-populate the batch side
# of composite FKs.
# =============================================================================
#
# **What.** The four listeners below (``_match_before_insert``,
# ``_cas_before_insert``, ``_slp_before_insert``, ``_lr_before_insert``)
# run on every INSERT of a ``Match`` / ``CasAdjustment`` /
# ``ServiceLinePayment`` / ``LineReconciliation`` row. They look up the
# parent ``Claim`` / ``Remittance`` and copy its ``batch_id`` (and
# ``remittance_batch_id``) into the new row's composite-FK column.
#
# **Why.** After migration 0014, every child table has a composite FK to
# its parent: ``(batch_id, claim_id)`` for Match / LineReconciliation and
# ``(remittance_batch_id, remittance_id)`` for CasAdjustment /
# ServiceLinePayment. Both sides are ``NOT NULL`` at the SQL level, so
# every insert must supply ``batch_id``. Application code that creates
# these rows usually has the parent ``claim_id`` / ``remittance_id`` in
# scope (often as a string) but does not always have ``batch_id``
# readily available — the batch is created elsewhere in the same ingest
# transaction, or the parent is fetched by id alone. Threading
# ``batch_id`` through every call site would be error-prone. The events
# solve this transparently: caller passes the parent id as before, and
# the event fills in the batch side.
#
# **Lookup order** (preferred-to-fallback, to minimize round-trips):
# 1. ``session.new`` — pending parents added in this transaction
# (covers the common ingest/match flow where parent + child are
# added in the same session).
# 2. ``session.identity_map`` — already-persisted parents loaded
# earlier in this session.
# 3. ``session.execute(select(...))`` — DB query as a last resort.
# ``autoflush`` is OFF on our ``SessionLocal``, so this won't
# trigger an unintended flush of ``session.new``.
#
# **When it fails.** If the parent is in none of the above (e.g. the
# caller passed a parent id that was never added to the session and is
# not in the DB), the column stays NULL and the INSERT fails with
# ``IntegrityError: NOT NULL constraint failed: <table>.batch_id`` (or
# ``remittance_batch_id``). The error message is misleading — it looks
# like the column was forgotten, not that the parent id was wrong.
# Workarounds:
# * If the parent exists in another session, ``session.add(parent)``
# first (or use ``session.merge(parent)``) so the lookup finds it
# in ``session.new`` / ``session.identity_map``.
# * If you have the parent object in scope and know its ``batch_id``,
# set ``child.batch_id = ...`` explicitly — the event only fills
# in when the column is None, so explicit values win.
# * If the parent is on a different connection entirely, populate the
# column by hand before ``session.add(child)``.
#
# **Idempotency.** The events only act when ``target.batch_id is None``
# (or ``remittance_batch_id``). Application code may explicitly set
# either column — e.g., a bulk-loader that already has batch_id in hand
# skips the lookup.
#
# **Scope.** ``propagate=True`` so subclasses of these mappers (in
# tests, mainly) inherit the listeners. Production code does not use
# mapper inheritance for these tables.
from sqlalchemy import event as _sa_event # noqa: E402
from sqlalchemy.orm import Session as _Session # noqa: E402
def _resolve_claim_batch_id(session: "_Session", claim_id: "str | None") -> "str | None":
"""Look up a Claim's batch_id, preferring session-cached state to SQL."""
if claim_id is None:
return None
for obj in session.new:
if isinstance(obj, Claim) and obj.id == claim_id:
return obj.batch_id
for obj in session.identity_map.values():
if isinstance(obj, Claim) and obj.id == claim_id:
return obj.batch_id
from sqlalchemy import select as _select
return session.execute(
_select(Claim.batch_id).where(Claim.id == claim_id)
).scalar_one_or_none()
def _resolve_remit_batch_id(session: "_Session", remittance_id: "str | None") -> "str | None":
"""Look up a Remittance's batch_id, preferring session-cached state to SQL."""
if remittance_id is None:
return None
for obj in session.new:
if isinstance(obj, Remittance) and obj.id == remittance_id:
return obj.batch_id
for obj in session.identity_map.values():
if isinstance(obj, Remittance) and obj.id == remittance_id:
return obj.batch_id
from sqlalchemy import select as _select
return session.execute(
_select(Remittance.batch_id).where(Remittance.id == remittance_id)
).scalar_one_or_none()
@_sa_event.listens_for(Match, "before_insert", propagate=True)
def _match_before_insert(mapper, connection, target):
if target.batch_id is None or target.remittance_batch_id is None:
session = _Session.object_session(target)
if session is None:
return
if target.batch_id is None:
target.batch_id = _resolve_claim_batch_id(session, target.claim_id)
if target.remittance_batch_id is None:
target.remittance_batch_id = _resolve_remit_batch_id(session, target.remittance_id)
@_sa_event.listens_for(CasAdjustment, "before_insert", propagate=True)
def _cas_before_insert(mapper, connection, target):
if target.remittance_batch_id is None:
session = _Session.object_session(target)
if session is None:
return
target.remittance_batch_id = _resolve_remit_batch_id(session, target.remittance_id)
@_sa_event.listens_for(ServiceLinePayment, "before_insert", propagate=True)
def _slp_before_insert(mapper, connection, target):
if target.remittance_batch_id is None:
session = _Session.object_session(target)
if session is None:
return
target.remittance_batch_id = _resolve_remit_batch_id(session, target.remittance_id)
@_sa_event.listens_for(LineReconciliation, "before_insert", propagate=True)
def _lr_before_insert(mapper, connection, target):
if target.batch_id is None:
session = _Session.object_session(target)
if session is None:
return
target.batch_id = _resolve_claim_batch_id(session, target.claim_id)
+39 -12
View File
@@ -47,31 +47,58 @@ def _strip_comments(sql: str) -> str:
return "\n".join(lines) return "\n".join(lines)
def run(engine: sa.Engine) -> None: def _apply_migrations_until(engine: sa.Engine, target_version: int | None) -> None:
"""Apply any pending migrations. Idempotent.""" """Apply migrations up to ``target_version`` (inclusive).
migrations_dir = MIGRATIONS_DIR
if not migrations_dir.exists():
return # no migrations directory yet — fresh checkout, no-op
migration_files = sorted(migrations_dir.glob("*.sql"))
if not migration_files:
return
If ``target_version`` is ``None``, apply every pending migration. If a
migration's version is greater than ``target_version``, stop iterating.
Idempotent: migrations whose version is <= the current ``PRAGMA user_version``
are skipped.
"""
with engine.begin() as conn: with engine.begin() as conn:
current = conn.exec_driver_sql("PRAGMA user_version").scalar() or 0 current = conn.exec_driver_sql("PRAGMA user_version").scalar() or 0
for path in migration_files: for path in sorted(MIGRATIONS_DIR.glob("*.sql")):
sql = path.read_text() sql = path.read_text()
version = _migration_version(sql, path.name) version = _migration_version(sql, path.name)
if version <= current: if version <= current:
continue continue
if target_version is not None and version > target_version:
break
statements_sql = _strip_comments(sql) statements_sql = _strip_comments(sql)
statements = [s.strip() for s in statements_sql.split(";") if s.strip()] statements = [s.strip() for s in statements_sql.split(";") if s.strip()]
with engine.begin() as conn: with engine.begin() as conn:
for stmt in statements: for stmt in statements:
conn.exec_driver_sql(stmt) conn.exec_driver_sql(stmt)
conn.exec_driver_sql(f"PRAGMA user_version = {version}") conn.exec_driver_sql(f"PRAGMA user_version = {version}")
current = version current = version
def run(engine: sa.Engine) -> None:
"""Apply any pending migrations. Idempotent."""
if not MIGRATIONS_DIR.exists():
return # no migrations directory yet — fresh checkout, no-op
if not list(MIGRATIONS_DIR.glob("*.sql")):
return
_apply_migrations_until(engine, None)
def migrate_to_version(engine: sa.Engine, target_version: int) -> None:
"""Apply migrations up to and including ``target_version``. Idempotent.
Reads the current ``PRAGMA user_version`` and applies any migrations
whose version is in the range ``(current, target_version]``. Stops
before applying migrations with version > ``target_version`` so the
caller can seed data between N and N+1.
Useful for tests that need to seed data after migrations ``0001..N``
have applied but before ``N+1`` runs exercise a migration against
real, pre-existing rows rather than only verifying that post-migration
inserts round-trip.
Calling with ``target_version`` <= the current version is a no-op.
"""
if not MIGRATIONS_DIR.exists():
return
_apply_migrations_until(engine, target_version)
@@ -0,0 +1,59 @@
-- version: 13
-- Drop the inline UNIQUE(batch_id, patient_control_number) on claims.
-- Migration 0003 attempted DROP INDEX IF EXISTS uq_claims_batch_pcn but
-- the constraint is inline in CREATE TABLE, so the drop was a no-op.
-- The only way to remove an inline UNIQUE in SQLite is table recreation.
--
-- X12 837P allows any number of CLM segments per 2000B subscriber loop;
-- claim identity is provided by the primary key (claims.id = CLM01).
-- The remittances table had a parallel constraint already removed in 0003
-- (because that one WAS a named index), so this migration only touches
-- claims.
--
-- The migration runner (db_migrate.py) wraps each .sql in an implicit
-- transaction via engine.begin(), so we MUST NOT use BEGIN/COMMIT.
-- PRAGMA defer_foreign_keys defers FK checks to commit, which is the
-- only way to drop a referenced table inside a transaction in SQLite.
PRAGMA defer_foreign_keys = ON;
CREATE TABLE claims_new (
id TEXT PRIMARY KEY,
batch_id TEXT NOT NULL REFERENCES batches(id) ON DELETE CASCADE,
patient_control_number TEXT NOT NULL,
service_date_from DATE,
service_date_to DATE,
charge_amount NUMERIC(12, 2) NOT NULL DEFAULT 0,
provider_npi TEXT,
payer_id TEXT,
state TEXT NOT NULL DEFAULT 'submitted',
state_before_reversal TEXT,
matched_remittance_id TEXT REFERENCES remittances(id),
raw_json TEXT,
rejection_reason TEXT,
rejected_at TIMESTAMP,
resubmit_count INTEGER NOT NULL DEFAULT 0,
state_changed_at TIMESTAMP,
payer_rejected_at TEXT,
payer_rejected_reason TEXT,
payer_rejected_status_code TEXT,
payer_rejected_by_277ca_id TEXT,
payer_rejected_acknowledged_at TEXT,
payer_rejected_acknowledged_actor TEXT
-- NO UNIQUE (batch_id, patient_control_number) — removed.
);
INSERT INTO claims_new SELECT * FROM claims;
DROP TABLE claims;
ALTER TABLE claims_new RENAME TO claims;
-- Recreate secondary indexes (same names, same columns as initial schema
-- plus later migrations).
CREATE INDEX ix_claims_state ON claims(state);
CREATE INDEX ix_claims_patient_control_number ON claims(patient_control_number);
CREATE INDEX ix_claims_service_date_from ON claims(service_date_from);
CREATE INDEX ix_claims_state_changed_at ON claims(state, state_changed_at);
CREATE INDEX idx_claims_payer_rejected_at ON claims(payer_rejected_at);
CREATE INDEX idx_claims_payer_rejected_unack
ON claims(payer_rejected_at)
WHERE payer_rejected_acknowledged_at IS NULL;
@@ -0,0 +1,253 @@
-- version: 14
-- Relax PRIMARY KEYs on `claims` and `remittances` from single-column (id)
-- to composite (batch_id, id). Enables resubmits (same CLM01 / CLP01 in
-- different batches) and makes the pre-flight dedup workflow exercisable.
--
-- Strategy (mirrors 0013): table recreation with PRAGMA
-- defer_foreign_keys. We must recreate every table that has an FK pointing
-- at `claims(id)` or `remittances(id)` so that FK constraints can be
-- updated to point at the new composite PK.
--
-- FKs that need updating (verified via pragma_foreign_key_list on the
-- pre-migration schema):
-- - remittances.claim_id -> claims(id) becomes -> (batch_id, id)
-- - claims.matched_remittance_id -> remittances(id) becomes -> (batch_id, id)
-- - matches.claim_id -> claims(id) ON DELETE CASCADE becomes -> (batch_id, id)
-- - matches.remittance_id -> remittances(id) ON DELETE CASCADE becomes -> (batch_id, id)
-- - cas_adjustments.remittance_id -> remittances(id) ON DELETE CASCADE becomes -> (batch_id, id)
-- - service_line_payments.remittance_id -> remittances(id) ON DELETE CASCADE becomes -> (batch_id, id)
-- - line_reconciliations.claim_id -> claims(id) ON DELETE CASCADE becomes -> (batch_id, id)
--
-- The cross-table FKs (remittances.claim_id, claims.matched_remittance_id)
-- can NOT be SQL-enforced with composite PKs because SQLite has no
-- ALTER TABLE ADD CONSTRAINT and DROP/ADD CONSTRAINT for an existing FK.
-- We keep them as plain TEXT columns (no REFERENCES clause) and rely on
-- application-layer invariants (store.manual_match / manual_unmatch,
-- dedup.preflight_*) to keep them consistent. The application code already
-- owns those paths; this is a deliberate trade-off documented in 0001's
-- comments ("amendment deferred to post-SQLite rollout").
--
-- The single-column FKs from claims/remittances DOWN to other tables
-- (matches, cas_adjustments, service_line_payments, line_reconciliations)
-- MUST be updated to composite FKs because those tables are recreated with
-- composite FK columns. We add a `batch_id` (or `remittance_batch_id`)
-- column to each and JOIN against the parent table to populate it during
-- the migration.
--
-- Why PRAGMA defer_foreign_keys (and not foreign_keys=OFF): the migration
-- runner wraps each .sql file in `engine.begin()` (a transaction). Inside a
-- transaction, `PRAGMA defer_foreign_keys = ON` defers FK checks to COMMIT,
-- which is the only safe way to drop a referenced table. `PRAGMA
-- foreign_keys` cannot be changed inside a transaction in SQLite, so the
-- defer_foreign_keys approach is what works inside the runner's transaction.
PRAGMA defer_foreign_keys = ON;
-- ============================================================================
-- Step 1: recreate `remittances` with composite PK (batch_id, id).
-- ============================================================================
-- Column shape: every column from 0001_initial.sql + claim_level_adjustment_amount
-- (added by 0006_line_reconciliation.sql). The `claim_id` column is kept
-- without an SQL-level FK — composite-FK to claims is enforced at the
-- application layer (see plan).
CREATE TABLE remittances_new (
id TEXT NOT NULL,
batch_id TEXT NOT NULL REFERENCES batches(id) ON DELETE CASCADE,
payer_claim_control_number TEXT NOT NULL,
claim_id TEXT,
status_code TEXT NOT NULL,
status_label TEXT,
total_charge NUMERIC(12, 2) NOT NULL DEFAULT 0,
total_paid NUMERIC(12, 2) NOT NULL DEFAULT 0,
patient_responsibility NUMERIC(12, 2),
adjustment_amount NUMERIC(12, 2) NOT NULL DEFAULT 0,
claim_level_adjustment_amount NUMERIC(12, 2) NOT NULL DEFAULT 0,
received_at DATETIME NOT NULL,
service_date DATE,
is_reversal INTEGER NOT NULL DEFAULT 0,
raw_json TEXT,
PRIMARY KEY (batch_id, id)
);
INSERT INTO remittances_new
SELECT id, batch_id, payer_claim_control_number, claim_id, status_code,
status_label, total_charge, total_paid, patient_responsibility,
adjustment_amount, claim_level_adjustment_amount, received_at,
service_date, is_reversal, raw_json
FROM remittances;
DROP TABLE remittances;
ALTER TABLE remittances_new RENAME TO remittances;
CREATE INDEX ix_remittances_claim_id ON remittances(claim_id);
CREATE INDEX ix_remittances_payer_claim_control_number ON remittances(payer_claim_control_number);
CREATE INDEX ix_remittances_status_code ON remittances(status_code);
-- ============================================================================
-- Step 2: recreate `claims` with composite PK (batch_id, id).
-- ============================================================================
-- Column shape: every column from 0013_drop_claims_unique_constraint.sql
-- (which itself consolidated 0001 + 0004 + 0008 + 0010), plus a new
-- `matched_remittance_batch_id` column. The back-reference to remittances
-- becomes a 2-column pointer (matched_remittance_id + matched_remittance_batch_id)
-- but the SQL-level FK is dropped (no ALTER CONSTRAINT in SQLite); see plan.
--
-- Column order MUST match the 0013 column order exactly so that
-- `INSERT INTO claims_new SELECT ... FROM claims` populates the right
-- columns. The trailing NULL fills the new matched_remittance_batch_id.
CREATE TABLE claims_new (
id TEXT NOT NULL,
batch_id TEXT NOT NULL REFERENCES batches(id) ON DELETE CASCADE,
patient_control_number TEXT NOT NULL,
service_date_from DATE,
service_date_to DATE,
charge_amount NUMERIC(12, 2) NOT NULL DEFAULT 0,
provider_npi TEXT,
payer_id TEXT,
state TEXT NOT NULL DEFAULT 'submitted',
state_before_reversal TEXT,
-- matched_remittance_id: kept without an SQL-level FK (composite
-- back-reference requires the application layer; see plan header).
matched_remittance_id TEXT,
raw_json TEXT,
rejection_reason TEXT,
rejected_at TIMESTAMP,
resubmit_count INTEGER NOT NULL DEFAULT 0,
state_changed_at TIMESTAMP,
payer_rejected_at TEXT,
payer_rejected_reason TEXT,
payer_rejected_status_code TEXT,
payer_rejected_by_277ca_id TEXT,
payer_rejected_acknowledged_at TEXT,
payer_rejected_acknowledged_actor TEXT,
-- matched_remittance_batch_id: NEW. Always NULL on migration because
-- the pre-0014 schema only stored matched_remittance_id (single column).
-- App code populates it on manual_match / match_auto going forward.
matched_remittance_batch_id TEXT,
PRIMARY KEY (batch_id, id)
);
INSERT INTO claims_new
SELECT id, batch_id, patient_control_number, service_date_from,
service_date_to, charge_amount, provider_npi, payer_id, state,
state_before_reversal, matched_remittance_id, raw_json,
rejection_reason, rejected_at, resubmit_count, state_changed_at,
payer_rejected_at, payer_rejected_reason,
payer_rejected_status_code, payer_rejected_by_277ca_id,
payer_rejected_acknowledged_at, payer_rejected_acknowledged_actor,
NULL
FROM claims;
DROP TABLE claims;
ALTER TABLE claims_new RENAME TO claims;
CREATE INDEX ix_claims_state ON claims(state);
CREATE INDEX ix_claims_patient_control_number ON claims(patient_control_number);
CREATE INDEX ix_claims_service_date_from ON claims(service_date_from);
CREATE INDEX ix_claims_state_changed_at ON claims(state, state_changed_at);
CREATE INDEX idx_claims_payer_rejected_at ON claims(payer_rejected_at);
CREATE INDEX idx_claims_payer_rejected_unack
ON claims(payer_rejected_at)
WHERE payer_rejected_acknowledged_at IS NULL;
-- ============================================================================
-- Step 3: recreate `matches`, `cas_adjustments`, `service_line_payments`,
-- `line_reconciliations` so their FKs point at the new composite PKs.
--
-- At this point the OLD `claims` and `remittances` tables are gone (dropped
-- in Steps 1 and 2) — only the recreated (composite-PK) versions exist.
-- The JOIN below reads the NEW parents, which have identical data
-- (same row count, same id values, same batch_id values); we only need
-- `batch_id` from each parent to populate the composite-FK column.
-- ============================================================================
CREATE TABLE matches_new (
id INTEGER PRIMARY KEY AUTOINCREMENT,
claim_id TEXT NOT NULL,
batch_id TEXT NOT NULL,
remittance_id TEXT NOT NULL,
remittance_batch_id TEXT NOT NULL,
strategy TEXT NOT NULL,
matched_at DATETIME NOT NULL,
prior_claim_state TEXT,
is_reversal INTEGER NOT NULL DEFAULT 0,
FOREIGN KEY (batch_id, claim_id) REFERENCES claims(batch_id, id) ON DELETE CASCADE,
FOREIGN KEY (remittance_batch_id, remittance_id) REFERENCES remittances(batch_id, id) ON DELETE CASCADE
);
INSERT INTO matches_new
SELECT m.id, m.claim_id, c.batch_id, m.remittance_id, r.batch_id,
m.strategy, m.matched_at, m.prior_claim_state, m.is_reversal
FROM matches m
JOIN claims c ON c.id = m.claim_id
JOIN remittances r ON r.id = m.remittance_id;
DROP TABLE matches;
ALTER TABLE matches_new RENAME TO matches;
CREATE INDEX ix_matches_claim_id ON matches(claim_id);
CREATE INDEX ix_matches_remittance_id ON matches(remittance_id);
CREATE INDEX ix_matches_matched_at ON matches(matched_at);
CREATE TABLE cas_adjustments_new (
id INTEGER PRIMARY KEY AUTOINCREMENT,
remittance_id TEXT NOT NULL,
remittance_batch_id TEXT NOT NULL,
group_code TEXT NOT NULL,
reason_code TEXT NOT NULL,
amount NUMERIC(12, 2) NOT NULL,
quantity NUMERIC(10, 2),
service_line_payment_id INTEGER REFERENCES service_line_payments(id) ON DELETE SET NULL,
FOREIGN KEY (remittance_batch_id, remittance_id) REFERENCES remittances(batch_id, id) ON DELETE CASCADE
);
INSERT INTO cas_adjustments_new
SELECT ca.id, ca.remittance_id, r.batch_id, ca.group_code, ca.reason_code,
ca.amount, ca.quantity, ca.service_line_payment_id
FROM cas_adjustments ca
JOIN remittances r ON r.id = ca.remittance_id;
DROP TABLE cas_adjustments;
ALTER TABLE cas_adjustments_new RENAME TO cas_adjustments;
CREATE INDEX ix_cas_adjustments_remittance_id ON cas_adjustments(remittance_id);
CREATE INDEX ix_cas_adjustments_service_line_payment_id ON cas_adjustments(service_line_payment_id);
CREATE TABLE service_line_payments_new (
id INTEGER PRIMARY KEY AUTOINCREMENT,
remittance_id TEXT NOT NULL,
remittance_batch_id TEXT NOT NULL,
line_number INTEGER NOT NULL,
procedure_qualifier VARCHAR(4) NOT NULL,
procedure_code VARCHAR(16) NOT NULL,
modifiers_json TEXT NOT NULL DEFAULT '[]',
charge NUMERIC(12, 2) NOT NULL,
payment NUMERIC(12, 2) NOT NULL,
units NUMERIC(10, 2),
unit_type VARCHAR(8),
service_date DATE,
ref_benefit_plan VARCHAR(64),
superseded_by_id INTEGER REFERENCES service_line_payments(id) ON DELETE SET NULL,
FOREIGN KEY (remittance_batch_id, remittance_id) REFERENCES remittances(batch_id, id) ON DELETE CASCADE
);
INSERT INTO service_line_payments_new
SELECT slp.id, slp.remittance_id, r.batch_id, slp.line_number,
slp.procedure_qualifier, slp.procedure_code, slp.modifiers_json,
slp.charge, slp.payment, slp.units, slp.unit_type, slp.service_date,
slp.ref_benefit_plan, slp.superseded_by_id
FROM service_line_payments slp
JOIN remittances r ON r.id = slp.remittance_id;
DROP TABLE service_line_payments;
ALTER TABLE service_line_payments_new RENAME TO service_line_payments;
CREATE INDEX ix_service_line_payments_remittance_id ON service_line_payments(remittance_id);
CREATE INDEX ix_service_line_payments_procedure_code_date ON service_line_payments(procedure_code, service_date);
CREATE TABLE line_reconciliations_new (
id INTEGER PRIMARY KEY AUTOINCREMENT,
claim_id VARCHAR(64) NOT NULL,
batch_id VARCHAR(64) NOT NULL,
claim_service_line_number INTEGER,
service_line_payment_id INTEGER REFERENCES service_line_payments(id) ON DELETE SET NULL,
status VARCHAR(16) NOT NULL,
match_score INTEGER,
reconciled_at TIMESTAMP NOT NULL,
FOREIGN KEY (batch_id, claim_id) REFERENCES claims(batch_id, id) ON DELETE CASCADE
);
INSERT INTO line_reconciliations_new
SELECT lr.id, lr.claim_id, c.batch_id, lr.claim_service_line_number,
lr.service_line_payment_id, lr.status, lr.match_score, lr.reconciled_at
FROM line_reconciliations lr
JOIN claims c ON c.id = lr.claim_id;
DROP TABLE line_reconciliations;
ALTER TABLE line_reconciliations_new RENAME TO line_reconciliations;
CREATE INDEX ix_line_reconciliations_claim_id ON line_reconciliations(claim_id);
CREATE INDEX ix_line_reconciliations_claim_service_line_number ON line_reconciliations(claim_service_line_number);
CREATE INDEX ix_line_reconciliations_service_line_payment_id ON line_reconciliations(service_line_payment_id);
+3
View File
@@ -287,6 +287,9 @@ def run(session, batch_id: str) -> ReconcileResult:
else: else:
m.claim.state = ClaimState.REVERSED m.claim.state = ClaimState.REVERSED
m.claim.matched_remittance_id = m.remittance.id m.claim.matched_remittance_id = m.remittance.id
# Migration 0014: composite back-reference to remittances — both
# sides needed for get_claim_detail to JOIN back to remittances.
m.claim.matched_remittance_batch_id = m.remittance.batch_id
# Symmetric FK: also set Remittance.claim_id so list_unmatched (which # Symmetric FK: also set Remittance.claim_id so list_unmatched (which
# filters on Remittance.claim_id IS NULL) correctly drops auto-matched # filters on Remittance.claim_id IS NULL) correctly drops auto-matched
# remits from the orphan bucket. Manual matches do this too. # remits from the orphan bucket. Manual matches do this too.
+87 -11
View File
@@ -64,6 +64,40 @@ class AlreadyMatchedError(Exception):
""" """
def find_existing_batch_for_claim(claim_id: str) -> str | None:
"""Return the batch_id of the first batch containing this claim id, or None.
Pure read; opens a short-lived session. Used by the 837 409 handler to
surface which prior batch already holds the same CLM01.
"""
from sqlalchemy import select
from cyclone.db import Claim
with db.SessionLocal()() as s:
row = s.execute(
select(Claim.batch_id).where(Claim.id == claim_id).limit(1)
).first()
return row[0] if row else None
def find_existing_batch_for_remit(remit_id: str) -> str | None:
"""Return the batch_id of the first batch containing this remit id, or None.
Pure read; opens a short-lived session. Used by the 835 409 handler.
`remit_id` is the PK on `remittances.id` (= payer_claim_control_number = CLP01).
"""
from sqlalchemy import select
from cyclone.db import Remittance
with db.SessionLocal()() as s:
row = s.execute(
select(Remittance.batch_id).where(Remittance.id == remit_id).limit(1)
).first()
return row[0] if row else None
class NotMatchedError(Exception): class NotMatchedError(Exception):
"""Raised by ``CycloneStore.manual_unmatch`` when the claim has no match. """Raised by ``CycloneStore.manual_unmatch`` when the claim has no match.
@@ -952,7 +986,10 @@ class CycloneStore:
if isinstance(record, BatchRecord837): if isinstance(record, BatchRecord837):
result: ParseResult = record.result result: ParseResult = record.result
for claim in result.claims: for claim in result.claims:
if s.get(Claim, claim.claim_id) is not None: # Migration 0014: claims PK is composite (batch_id, id).
# Duplicate-check is per-batch now — same CLM01 in a
# DIFFERENT batch is allowed (resubmits / retry).
if s.get(Claim, (record.id, claim.claim_id)) is not None:
log.warning( log.warning(
"add: claim %s already exists; skipping (batch=%s)", "add: claim %s already exists; skipping (batch=%s)",
claim.claim_id, record.id, claim.claim_id, record.id,
@@ -978,7 +1015,9 @@ class CycloneStore:
result835: ParseResult835 = record.result result835: ParseResult835 = record.result
payer_name = result835.payer.name payer_name = result835.payer.name
for cp in result835.claims: for cp in result835.claims:
if s.get(Remittance, cp.payer_claim_control_number) is not None: # Migration 0014: remittances PK is composite. Per-batch
# duplicate-check; cross-batch CLP01 is allowed.
if s.get(Remittance, (record.id, cp.payer_claim_control_number)) is not None:
log.warning( log.warning(
"add: remittance %s already exists; skipping (batch=%s)", "add: remittance %s already exists; skipping (batch=%s)",
cp.payer_claim_control_number, record.id, cp.payer_claim_control_number, record.id,
@@ -1062,7 +1101,11 @@ class CycloneStore:
try: try:
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
for cid in claim_ids: for cid in claim_ids:
row = s.get(Claim, cid) # Migration 0014: composite PK. The event log
# (ActivityEvent.claim_id) only stores the id, not
# batch_id, so look up by id alone within this batch
# context (record.id).
row = s.get(Claim, (record.id, cid))
if row is None: if row is None:
continue continue
ui = to_ui_claim_from_orm( ui = to_ui_claim_from_orm(
@@ -1071,7 +1114,7 @@ class CycloneStore:
) )
self._sync_publish(event_bus, "claim_written", ui) self._sync_publish(event_bus, "claim_written", ui)
for rid in remit_ids: for rid in remit_ids:
row = s.get(Remittance, rid) row = s.get(Remittance, (record.id, rid))
if row is None: if row is None:
continue continue
ui = to_ui_remittance_from_orm( ui = to_ui_remittance_from_orm(
@@ -1218,7 +1261,12 @@ class CycloneStore:
per-line payments + adjustments without a second fetch. per-line payments + adjustments without a second fetch.
""" """
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
row = s.get(Remittance, remittance_id) # Migration 0014: composite PK. The public store.get_remittance
# API takes only remittance_id (callers may not know batch_id),
# so we look up by id alone. If duplicates exist across
# batches (resubmits), we return the first match — the same
# semantics the old single-PK s.get() provided.
row = s.query(Remittance).filter(Remittance.id == remittance_id).first()
if row is None: if row is None:
return None return None
cas_rows = ( cas_rows = (
@@ -1290,7 +1338,9 @@ class CycloneStore:
from cyclone import db as _db from cyclone import db as _db
with _db.SessionLocal()() as s: with _db.SessionLocal()() as s:
row = s.get(Claim, claim_id) # Migration 0014: see get_remittance above — public API takes
# only claim_id; look up by id alone, return first match.
row = s.query(Claim).filter(Claim.id == claim_id).first()
if row is None: if row is None:
return None return None
@@ -1330,7 +1380,12 @@ class CycloneStore:
] ]
if row.matched_remittance_id is not None: if row.matched_remittance_id is not None:
remit = s.get(Remittance, row.matched_remittance_id) # Migration 0014: composite PK; matched_remittance_batch_id
# is the batch side of the back-reference.
remit = s.get(
Remittance,
(row.matched_remittance_batch_id, row.matched_remittance_id),
)
if remit is not None: if remit is not None:
status = ( status = (
"reconciled" "reconciled"
@@ -1983,7 +2038,12 @@ class CycloneStore:
from cyclone import reconcile as _reconcile from cyclone import reconcile as _reconcile
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
claim = s.get(Claim, claim_id) # Migration 0014: public API takes only claim_id / remit_id.
# Look up by id alone — if duplicates exist across batches,
# the first match wins (the manual_match workflow is per-batch
# so the user knows which batch they mean; cross-batch
# collisions are handled at ingest time by parse-decide-workflow).
claim = s.query(Claim).filter(Claim.id == claim_id).first()
if claim is None: if claim is None:
raise LookupError(f"claim {claim_id} not found") raise LookupError(f"claim {claim_id} not found")
if claim.matched_remittance_id is not None: if claim.matched_remittance_id is not None:
@@ -1992,7 +2052,7 @@ class CycloneStore:
f"{claim.matched_remittance_id}" f"{claim.matched_remittance_id}"
) )
remit = s.get(Remittance, remit_id) remit = s.query(Remittance).filter(Remittance.id == remit_id).first()
if remit is None: if remit is None:
raise LookupError(f"remittance {remit_id} not found") raise LookupError(f"remittance {remit_id} not found")
@@ -2031,6 +2091,12 @@ class CycloneStore:
)) ))
claim.state = new_state claim.state = new_state
claim.matched_remittance_id = remit_id claim.matched_remittance_id = remit_id
# Migration 0014: composite back-reference to remittances. The
# claim side needs both id and batch_id (we drop the SQL-level
# cross-table FK because SQLite has no ALTER CONSTRAINT;
# matched_remittance_id alone is not enough to look up the
# remittance under composite PK).
claim.matched_remittance_batch_id = remit.batch_id
# Symmetric FK update — see list_unmatched docstring for why # Symmetric FK update — see list_unmatched docstring for why
# we don't rely on T10 to do this. # we don't rely on T10 to do this.
remit.claim_id = claim_id remit.claim_id = claim_id
@@ -2100,7 +2166,9 @@ class CycloneStore:
6. Return ``{"claim": <ui>, "deletedMatches": <count>}``. 6. Return ``{"claim": <ui>, "deletedMatches": <count>}``.
""" """
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
claim = s.get(Claim, claim_id) # Migration 0014: see manual_match — public API takes only
# claim_id; look up by id alone, first match wins.
claim = s.query(Claim).filter(Claim.id == claim_id).first()
if claim is None: if claim is None:
raise LookupError(f"claim {claim_id} not found") raise LookupError(f"claim {claim_id} not found")
if claim.matched_remittance_id is None: if claim.matched_remittance_id is None:
@@ -2134,12 +2202,20 @@ class CycloneStore:
claim.state = restored_state claim.state = restored_state
claim.matched_remittance_id = None claim.matched_remittance_id = None
# Migration 0014: clear the batch side of the composite
# back-reference too. Both columns must move together.
claim.matched_remittance_batch_id = None
# Clear the symmetric FK on the remittance so list_unmatched # Clear the symmetric FK on the remittance so list_unmatched
# surfaces the pair again. The remittance may have been # surfaces the pair again. The remittance may have been
# deleted between the match and this call — guard with a # deleted between the match and this call — guard with a
# get() so we don't blow up on a stale FK. # get() so we don't blow up on a stale FK.
if latest is not None: if latest is not None:
paired_remit = s.get(Remittance, latest.remittance_id) # Migration 0014: composite PK — latest is a Match row
# which has remittance_batch_id (the batch side of the FK).
paired_remit = s.get(
Remittance,
(latest.remittance_batch_id, latest.remittance_id),
)
if paired_remit is not None: if paired_remit is not None:
paired_remit.claim_id = None paired_remit.claim_id = None
+2 -1
View File
@@ -68,7 +68,8 @@ def test_999_set_rejection_moves_claim_to_rejected_state(client: TestClient):
assert r.status_code == 200, r.text assert r.status_code == 200, r.text
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
c = s.get(Claim, "CLP-1") # Migration 0014: composite PK (batch_id, id).
c = s.get(Claim, ("B-1", "CLP-1"))
assert c.state == ClaimState.REJECTED assert c.state == ClaimState.REJECTED
assert c.rejected_at is not None assert c.rejected_at is not None
assert "999" in (c.rejection_reason or "") assert "999" in (c.rejection_reason or "")
+6 -4
View File
@@ -51,19 +51,21 @@ def test_migration_0002_creates_acks_table():
def test_migration_latest_idempotent_on_fresh_db(): def test_migration_latest_idempotent_on_fresh_db():
"""Re-running the migration on the same DB must be a no-op (PRAGMA """Re-running the migration on the same DB must be a no-op (PRAGMA
user_version already at the latest version currently 12 after user_version already at the latest version currently 14 after
0004-0006 line_reconciliation, 0005 ta1_acks, SP9's 0007 0004-0006 line_reconciliation, 0005 ta1_acks, SP9's 0007
providers/payers/clearhouse, SP10's 0008 payer_rejected, providers/payers/clearhouse, SP10's 0008 payer_rejected,
SP11's 0009 audit_log, SP14's 0010 payer_rejected_acknowledged, SP11's 0009 audit_log, SP14's 0010 payer_rejected_acknowledged,
SP16's 0011 processed_inbound_files, SP17's 0012 db_backups).""" SP16's 0011 processed_inbound_files, SP17's 0012 db_backups,
SP19's 0013 drop_claims_unique_constraint, SP20's 0014
relax_claims_remits_pk)."""
with db.engine().begin() as c: with db.engine().begin() as c:
v1 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0 v1 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
assert v1 == 12 assert v1 == 14
# A second run should not raise and should not bump the version. # A second run should not raise and should not bump the version.
db_migrate.run(db.engine()) db_migrate.run(db.engine())
with db.engine().begin() as c: with db.engine().begin() as c:
v2 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0 v2 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
assert v2 == 12 assert v2 == 14
def test_add_ack_persists_row(): def test_add_ack_persists_row():
+4 -2
View File
@@ -98,8 +98,10 @@ class TestParse277CAEndpointHappyPath:
files={"file": ("minimal_277ca.txt", text, "text/plain")}, files={"file": ("minimal_277ca.txt", text, "text/plain")},
) )
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
c1 = s.get(Claim, "c1") # Migration 0014: composite PK (batch_id, id). The _seed_claim
c2 = s.get(Claim, "c2") # helper uses batch_id="BATCH-1".
c1 = s.get(Claim, ("BATCH-1", "c1"))
c2 = s.get(Claim, ("BATCH-1", "c2"))
assert c1.payer_rejected_at is None assert c1.payer_rejected_at is None
assert c2.payer_rejected_at is not None assert c2.payer_rejected_at is not None
assert c2.payer_rejected_status_code == "A6" assert c2.payer_rejected_status_code == "A6"
+57
View File
@@ -357,3 +357,60 @@ def test_prodfile_cross_pipeline_reconciles(client: TestClient):
assert claim_rows >= len(unique_837_pcns), ( assert claim_rows >= len(unique_837_pcns), (
f"claim_rows {claim_rows} < unique_837_pcns {len(unique_837_pcns)}" f"claim_rows {claim_rows} < unique_837_pcns {len(unique_837_pcns)}"
) )
def test_409_response_includes_existing_batch_id_for_837(client: TestClient) -> None:
"""When a CLM01 already exists in a prior batch, the 409 body has existing_batch_id."""
from datetime import datetime, timezone
from cyclone import db as _db
from cyclone.db import Batch, Claim
# Seed a prior batch with claim CLM-X.
with _db.SessionLocal()() as s:
s.add(Batch(
id="PRIOR", kind="837p", input_filename="prior.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Claim(id="CLM-X", batch_id="PRIOR", patient_control_number="M"))
s.commit()
# Build a file with two CLM* segments both using CLM-X (forces PK collision).
text = (
"ISA*00* *00* *ZZ*SUBMITTERID *ZZ*RECEIVERID "
"*240101*1200*^*00501*000000001*0*P*:~\n"
"GS*HC*SUBMITTERID*RECEIVERID*20240101*1200*1*X*005010X222A1~\n"
"ST*837*0001*005010X222A1~\n"
"BHT*0019*00*1*20240101*1200*CH~\n"
"NM1*41*2*SUBMITTER*****46*SUBMITTERID~\n"
"PER*IC*CONTACT*TE*5555555555~\n"
"NM1*40*2*RECEIVER*****46*RECEIVERID~\n"
"HL*1**20*1~\n"
"NM1*85*2*BILLING*****XX*1881068062~\n"
"N3*123 MAIN*\nN4*DENVER*CO*80202~\n"
"REF*EI*123456789~\n"
"HL*2*1*22*0~\n"
"SBR*P*18*******CI~\n"
"NM1*IL*1*DOE*JOHN****MI*M~\n"
"N3*456 ELM*\nN4*DENVER*CO*80202~\n"
"DMG*D8*19700101*M~\n"
"NM1*PR*2*MEDICAID*****PI*MCD~\n"
"CLM*CLM-X*100***11:B:1*Y*A*Y*Y~\n"
"LX*1~\nSV1*HC:99213*100*UN*1***1~\n"
"DTP*472*D8*20240101~\n"
"CLM*CLM-X*100***11:B:1*Y*A*Y*Y~\n"
"LX*2~\nSV1*HC:99213*100*UN*1***1~\n"
"DTP*472*D8*20240101~\n"
"SE*30*0001~\n"
"GE*1*1~\n"
"IEA*1*000000001~\n"
)
resp = client.post(
"/api/parse-837",
files={"file": ("dup.txt", text, "text/plain")},
headers={"Accept": "application/json"},
)
assert resp.status_code == 409, resp.text
body = resp.json()
assert body.get("existing_batch_id") == "PRIOR"
+16 -8
View File
@@ -70,7 +70,9 @@ class TestApply277CARejectionsHappyPath:
assert outcome.matched == ["c1"] assert outcome.matched == ["c1"]
assert outcome.orphans == [] assert outcome.orphans == []
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
c = s.get(Claim, "c1") # Migration 0014: composite PK; only one claim with id "c1"
# exists in this test, so filter-by-id returns it.
c = s.query(Claim).filter(Claim.id == "c1").first()
assert c.payer_rejected_at is not None assert c.payer_rejected_at is not None
assert c.payer_rejected_status_code == "A6" assert c.payer_rejected_status_code == "A6"
assert "A6" in (c.payer_rejected_reason or "") assert "A6" in (c.payer_rejected_reason or "")
@@ -116,14 +118,16 @@ class TestApply277CARejectionsIdempotent:
return s.query(Claim).filter_by(patient_control_number="CLAIM001").first() return s.query(Claim).filter_by(patient_control_number="CLAIM001").first()
outcome1 = apply_277ca_rejections(s, result, claim_lookup=_lookup, two77ca_id="ACK-1") outcome1 = apply_277ca_rejections(s, result, claim_lookup=_lookup, two77ca_id="ACK-1")
assert outcome1.matched == ["c1"] assert outcome1.matched == ["c1"]
original_reason = s.get(Claim, "c1").payer_rejected_reason # Migration 0014: composite PK — see test_rejected_status_stamps_matching_claim.
original_at = s.get(Claim, "c1").payer_rejected_at original_reason = s.query(Claim).filter(Claim.id == "c1").first().payer_rejected_reason
original_at = s.query(Claim).filter(Claim.id == "c1").first().payer_rejected_at
# Run again with same code. # Run again with same code.
outcome2 = apply_277ca_rejections(s, result, claim_lookup=_lookup, two77ca_id="ACK-1") outcome2 = apply_277ca_rejections(s, result, claim_lookup=_lookup, two77ca_id="ACK-1")
assert outcome2.matched == [] assert outcome2.matched == []
assert outcome2.already_rejected == ["c1"] assert outcome2.already_rejected == ["c1"]
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
c = s.get(Claim, "c1") # Migration 0014: see above — composite PK.
c = s.query(Claim).filter(Claim.id == "c1").first()
# Reason and timestamp unchanged. # Reason and timestamp unchanged.
assert c.payer_rejected_reason == original_reason assert c.payer_rejected_reason == original_reason
assert c.payer_rejected_at == original_at assert c.payer_rejected_at == original_at
@@ -143,7 +147,8 @@ class TestApply277CAOnlyRejectsRejected:
outcome = apply_277ca_rejections(s, result, claim_lookup=_lookup, two77ca_id="ACK-1") outcome = apply_277ca_rejections(s, result, claim_lookup=_lookup, two77ca_id="ACK-1")
assert outcome.matched == [] assert outcome.matched == []
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
c = s.get(Claim, "c1") # Migration 0014: composite PK.
c = s.query(Claim).filter(Claim.id == "c1").first()
assert c.payer_rejected_at is None assert c.payer_rejected_at is None
@@ -183,9 +188,12 @@ class TestApply277CAMultipleStatuses:
outcome = apply_277ca_rejections(s, result, claim_lookup=_lookup, two77ca_id="ACK-1") outcome = apply_277ca_rejections(s, result, claim_lookup=_lookup, two77ca_id="ACK-1")
assert outcome.matched == ["c2"] assert outcome.matched == ["c2"]
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
assert s.get(Claim, "c1").payer_rejected_at is None # Migration 0014: composite PK — filter-by-id returns the
assert s.get(Claim, "c2").payer_rejected_at is not None # single claim with that id (no cross-batch duplicates in
assert s.get(Claim, "c3").payer_rejected_at is None # this test fixture).
assert s.query(Claim).filter(Claim.id == "c1").first().payer_rejected_at is None
assert s.query(Claim).filter(Claim.id == "c2").first().payer_rejected_at is not None
assert s.query(Claim).filter(Claim.id == "c3").first().payer_rejected_at is None
# --------------------------------------------------------------------------- # # --------------------------------------------------------------------------- #
+541
View File
@@ -2,6 +2,7 @@
from __future__ import annotations from __future__ import annotations
from datetime import datetime, timezone
from pathlib import Path from pathlib import Path
import pytest import pytest
@@ -113,3 +114,543 @@ def test_run_ignores_non_sql_files(
).all() ).all()
assert len(rows) == 0 assert len(rows) == 0
assert _user_version(engine) == 1 assert _user_version(engine) == 1
def test_drop_claims_unique_constraint_migration(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Migration 0013 must drop the inline UNIQUE(batch_id, patient_control_number)
on claims by recreating the table. Idempotent and preserves data.
We copy the real 0013 file (so this test exercises the real migration, not
a synthetic one) and stub 0001-0012 with a synthetic 0001 that has the
historical inline UNIQUE. If 0013 doesn't exist as a real file, the runner
will leave user_version at 1 and the test will fail.
"""
monkeypatch.setattr(db_migrate, "MIGRATIONS_DIR", tmp_path)
# Synthetic 0001: claims with INLINE UNIQUE(batch_id, patient_control_number).
# The schema mirrors the real 0001 column shape so 0013's
# ``INSERT INTO claims_new SELECT * FROM claims`` works against this fixture.
(tmp_path / "0001_initial.sql").write_text(
"-- version: 1\n"
"CREATE TABLE batches (id TEXT PRIMARY KEY, kind TEXT NOT NULL, "
"input_filename TEXT NOT NULL, parsed_at DATETIME NOT NULL);\n"
"CREATE TABLE claims ("
"id TEXT PRIMARY KEY, batch_id TEXT NOT NULL REFERENCES batches(id) ON DELETE CASCADE,"
"patient_control_number TEXT NOT NULL, service_date_from DATE, service_date_to DATE,"
"charge_amount NUMERIC(12, 2) NOT NULL DEFAULT 0, provider_npi TEXT, payer_id TEXT,"
"state TEXT NOT NULL DEFAULT 'submitted', state_before_reversal TEXT,"
"matched_remittance_id TEXT REFERENCES remittances(id), raw_json TEXT,"
"rejection_reason TEXT, rejected_at TIMESTAMP, resubmit_count INTEGER NOT NULL DEFAULT 0,"
"state_changed_at TIMESTAMP, payer_rejected_at TEXT, payer_rejected_reason TEXT,"
"payer_rejected_status_code TEXT, payer_rejected_by_277ca_id TEXT,"
"payer_rejected_acknowledged_at TEXT, payer_rejected_acknowledged_actor TEXT,"
"UNIQUE (batch_id, patient_control_number));\n"
)
# Copy the REAL 0013 from the source tree. If it's missing, the test fails
# because user_version will stay at 1 and we'll never get to assert it == 13.
real_migrations = Path(db_migrate.__file__).parent / "migrations"
real_0013 = real_migrations / "0013_drop_claims_unique_constraint.sql"
assert real_0013.exists(), (
f"Real migration 0013 missing at {real_0013} — this test is meant to "
f"exercise the real migration, not a synthetic one."
)
(tmp_path / "0013_drop_claims_unique_constraint.sql").write_text(real_0013.read_text())
engine = _fresh_engine(tmp_path)
db_migrate.run(engine)
assert _user_version(engine) == 13, (
"0013 did not apply; user_version did not reach 13. Either the migration "
"file is missing, or it has a syntax error."
)
# Before-0013 assertion: with the inline UNIQUE, two claims sharing
# (batch_id, patient_control_number) on the same batch must fail. The
# recreation must remove that constraint, so this insert must now succeed.
with engine.begin() as c:
c.exec_driver_sql(
"INSERT INTO batches(id, kind, input_filename, parsed_at) "
"VALUES ('b1', '837p', 'x.txt', '2026-01-01')"
)
c.exec_driver_sql(
"INSERT INTO claims(id, batch_id, patient_control_number) VALUES ('c1', 'b1', 'M')"
)
c.exec_driver_sql(
"INSERT INTO claims(id, batch_id, patient_control_number) VALUES ('c2', 'b1', 'M')"
)
ids = [r[0] for r in c.exec_driver_sql("SELECT id FROM claims ORDER BY id").all()]
assert ids == ["c1", "c2"], (
"Both claims with same (batch_id, patient_control_number) must persist "
"after 0013 — the UNIQUE(batch_id, patient_control_number) is gone."
)
def test_migration_0013_is_idempotent(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Re-running db_migrate.run on a v13 DB is a no-op."""
monkeypatch.setattr(db_migrate, "MIGRATIONS_DIR", tmp_path)
# Use synthetic 0001 + copy the real 0013 so we exercise the real file.
(tmp_path / "0001_initial.sql").write_text(
"-- version: 1\n"
"CREATE TABLE batches (id TEXT PRIMARY KEY, kind TEXT NOT NULL, "
"input_filename TEXT NOT NULL, parsed_at DATETIME NOT NULL);\n"
"CREATE TABLE claims ("
"id TEXT PRIMARY KEY, batch_id TEXT NOT NULL REFERENCES batches(id) ON DELETE CASCADE,"
"patient_control_number TEXT NOT NULL, service_date_from DATE, service_date_to DATE,"
"charge_amount NUMERIC(12, 2) NOT NULL DEFAULT 0, provider_npi TEXT, payer_id TEXT,"
"state TEXT NOT NULL DEFAULT 'submitted', state_before_reversal TEXT,"
"matched_remittance_id TEXT REFERENCES remittances(id), raw_json TEXT,"
"rejection_reason TEXT, rejected_at TIMESTAMP, resubmit_count INTEGER NOT NULL DEFAULT 0,"
"state_changed_at TIMESTAMP, payer_rejected_at TEXT, payer_rejected_reason TEXT,"
"payer_rejected_status_code TEXT, payer_rejected_by_277ca_id TEXT,"
"payer_rejected_acknowledged_at TEXT, payer_rejected_acknowledged_actor TEXT,"
"UNIQUE (batch_id, patient_control_number));\n"
)
real_migrations = Path(db_migrate.__file__).parent / "migrations"
real_0013 = real_migrations / "0013_drop_claims_unique_constraint.sql"
assert real_0013.exists(), "Real 0013 missing — test cannot exercise it."
(tmp_path / "0013_drop_claims_unique_constraint.sql").write_text(real_0013.read_text())
engine = _fresh_engine(tmp_path)
db_migrate.run(engine) # applies both
version_after_first = _user_version(engine)
assert version_after_first == 13
db_migrate.run(engine) # second call: no-op
assert _user_version(engine) == version_after_first
# =============================================================================
# Migration 0014: relax claims/remittances PK to composite (batch_id, id)
# =============================================================================
# Synthetic 0001 with the FULL post-0013 column shape, so 0014's
# INSERT...SELECT statements find every column they expect on the source
# tables. Mirrors the shape after 0001 + 0004 + 0006 + 0008 + 0010 + 0013.
# The tables and columns here are exactly what 0014 reads from; we do NOT
# add migrations 0002..0012 because they're orthogonal to the FK chain
# 0014 walks (no claims/remittances FKs to acks, audit_log, etc).
_0014_SYNTHETIC_0001 = (
"-- version: 1\n"
"CREATE TABLE batches ("
"id TEXT PRIMARY KEY, kind TEXT NOT NULL, input_filename TEXT NOT NULL,"
"parsed_at DATETIME NOT NULL, totals_json TEXT, validation_json TEXT,"
"raw_result_json TEXT);\n"
"CREATE TABLE claims ("
"id TEXT PRIMARY KEY, batch_id TEXT NOT NULL REFERENCES batches(id) ON DELETE CASCADE,"
"patient_control_number TEXT NOT NULL, service_date_from DATE, service_date_to DATE,"
"charge_amount NUMERIC(12, 2) NOT NULL DEFAULT 0, provider_npi TEXT, payer_id TEXT,"
"state TEXT NOT NULL DEFAULT 'submitted', state_before_reversal TEXT,"
"matched_remittance_id TEXT REFERENCES remittances(id), raw_json TEXT,"
"rejection_reason TEXT, rejected_at TIMESTAMP,"
"resubmit_count INTEGER NOT NULL DEFAULT 0, state_changed_at TIMESTAMP,"
"payer_rejected_at TEXT, payer_rejected_reason TEXT,"
"payer_rejected_status_code TEXT, payer_rejected_by_277ca_id TEXT,"
"payer_rejected_acknowledged_at TEXT, payer_rejected_acknowledged_actor TEXT);\n"
"CREATE TABLE remittances ("
"id TEXT PRIMARY KEY, batch_id TEXT NOT NULL REFERENCES batches(id) ON DELETE CASCADE,"
"payer_claim_control_number TEXT NOT NULL, claim_id TEXT REFERENCES claims(id),"
"status_code TEXT NOT NULL, status_label TEXT,"
"total_charge NUMERIC(12, 2) NOT NULL DEFAULT 0,"
"total_paid NUMERIC(12, 2) NOT NULL DEFAULT 0,"
"patient_responsibility NUMERIC(12, 2),"
"adjustment_amount NUMERIC(12, 2) NOT NULL DEFAULT 0,"
"claim_level_adjustment_amount NUMERIC(12, 2) NOT NULL DEFAULT 0,"
"received_at DATETIME NOT NULL, service_date DATE,"
"is_reversal INTEGER NOT NULL DEFAULT 0, raw_json TEXT);\n"
"CREATE TABLE matches ("
"id INTEGER PRIMARY KEY AUTOINCREMENT,"
"claim_id TEXT NOT NULL REFERENCES claims(id) ON DELETE CASCADE,"
"remittance_id TEXT NOT NULL REFERENCES remittances(id) ON DELETE CASCADE,"
"strategy TEXT NOT NULL, matched_at DATETIME NOT NULL,"
"prior_claim_state TEXT, is_reversal INTEGER NOT NULL DEFAULT 0);\n"
"CREATE TABLE cas_adjustments ("
"id INTEGER PRIMARY KEY AUTOINCREMENT,"
"remittance_id TEXT NOT NULL REFERENCES remittances(id) ON DELETE CASCADE,"
"group_code TEXT NOT NULL, reason_code TEXT NOT NULL,"
"amount NUMERIC(12, 2) NOT NULL, quantity NUMERIC(10, 2),"
"service_line_payment_id INTEGER REFERENCES service_line_payments(id) ON DELETE SET NULL);\n"
"CREATE TABLE service_line_payments ("
"id INTEGER PRIMARY KEY AUTOINCREMENT,"
"remittance_id VARCHAR(64) NOT NULL REFERENCES remittances(id) ON DELETE CASCADE,"
"line_number INTEGER NOT NULL, procedure_qualifier VARCHAR(4) NOT NULL,"
"procedure_code VARCHAR(16) NOT NULL,"
"modifiers_json TEXT NOT NULL DEFAULT '[]',"
"charge NUMERIC(12, 2) NOT NULL, payment NUMERIC(12, 2) NOT NULL,"
"units NUMERIC(10, 2), unit_type VARCHAR(8), service_date DATE,"
"ref_benefit_plan VARCHAR(64),"
"superseded_by_id INTEGER REFERENCES service_line_payments(id) ON DELETE SET NULL);\n"
"CREATE TABLE line_reconciliations ("
"id INTEGER PRIMARY KEY AUTOINCREMENT,"
"claim_id VARCHAR(64) NOT NULL REFERENCES claims(id) ON DELETE CASCADE,"
"claim_service_line_number INTEGER,"
"service_line_payment_id INTEGER REFERENCES service_line_payments(id) ON DELETE SET NULL,"
"status VARCHAR(16) NOT NULL, match_score INTEGER, reconciled_at TIMESTAMP NOT NULL);\n"
)
def _apply_0014_only(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> sa.Engine:
"""Apply just 0014 (with a synthetic 0001 that has the post-0013 column
shape) on a fresh engine. Returns the engine with user_version=14.
"""
monkeypatch.setattr(db_migrate, "MIGRATIONS_DIR", tmp_path)
(tmp_path / "0001_initial.sql").write_text(_0014_SYNTHETIC_0001)
real_migrations = Path(db_migrate.__file__).parent / "migrations"
real_0014 = real_migrations / "0014_relax_claims_remits_pk.sql"
assert real_0014.exists(), (
f"Real migration 0014 missing at {real_0014} — these tests are "
f"meant to exercise the real migration, not a synthetic one."
)
(tmp_path / "0014_relax_claims_remits_pk.sql").write_text(real_0014.read_text())
engine = _fresh_engine(tmp_path)
db_migrate.run(engine)
assert _user_version(engine) == 14, (
"0014 did not apply; user_version did not reach 14. Either the "
"migration file is missing, or it has a syntax error."
)
return engine
def test_migration_0014_relaxes_claims_pk_to_composite(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Migration 0014 changes claims PK from single-column id to (batch_id, id).
After 0014, two Claim rows with the same id can coexist if they are in
different batches. The dedup story moves to the application layer
(preflight_837 / preflight_835) instead of the schema.
"""
engine = _apply_0014_only(tmp_path, monkeypatch)
# After 0014: same id in two different batches must coexist.
with engine.begin() as c:
c.exec_driver_sql(
"INSERT INTO batches(id, kind, input_filename, parsed_at) "
"VALUES ('B1', '837p', 'b1.txt', '2026-01-01')"
)
c.exec_driver_sql(
"INSERT INTO batches(id, kind, input_filename, parsed_at) "
"VALUES ('B2', '837p', 'b2.txt', '2026-01-02')"
)
c.exec_driver_sql(
"INSERT INTO claims(id, batch_id, patient_control_number, "
"state) VALUES ('CLM-A', 'B1', 'M1', 'submitted')"
)
c.exec_driver_sql(
"INSERT INTO claims(id, batch_id, patient_control_number, "
"state) VALUES ('CLM-A', 'B2', 'M1', 'submitted')"
)
rows = c.exec_driver_sql(
"SELECT id, batch_id FROM claims WHERE id='CLM-A' ORDER BY batch_id"
).all()
assert rows == [("CLM-A", "B1"), ("CLM-A", "B2")], (
"After 0014, the same claims.id in two different batches must "
"coexist. Composite PK is (batch_id, id)."
)
def test_migration_0014_relaxes_remittances_pk_to_composite(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Same shape for remittances: same CLP01 can exist in two batches."""
engine = _apply_0014_only(tmp_path, monkeypatch)
with engine.begin() as c:
c.exec_driver_sql(
"INSERT INTO batches(id, kind, input_filename, parsed_at) "
"VALUES ('B1', '835', 'b1.txt', '2026-01-01')"
)
c.exec_driver_sql(
"INSERT INTO batches(id, kind, input_filename, parsed_at) "
"VALUES ('B2', '835', 'b2.txt', '2026-01-02')"
)
c.exec_driver_sql(
"INSERT INTO remittances(id, batch_id, payer_claim_control_number, "
"status_code, received_at) "
"VALUES ('CLP-A', 'B1', 'CLP-A', '1', '2026-01-01')"
)
c.exec_driver_sql(
"INSERT INTO remittances(id, batch_id, payer_claim_control_number, "
"status_code, received_at) "
"VALUES ('CLP-A', 'B2', 'CLP-A', '1', '2026-01-02')"
)
rows = c.exec_driver_sql(
"SELECT id, batch_id FROM remittances WHERE id='CLP-A' ORDER BY batch_id"
).all()
assert rows == [("CLP-A", "B1"), ("CLP-A", "B2")], (
"After 0014, the same remittances.id in two different batches must "
"coexist. Composite PK is (batch_id, id)."
)
def test_migration_0014_preserves_existing_data(tmp_path, monkeypatch):
"""Data inserted at v=13 must survive the v=14 table recreation.
0014 uses ``INSERT INTO claims_new SELECT * FROM claims`` (and the
remittance-side equivalent) to populate the recreated tables. Any rows
present before 0014 must still be there after 0014 finishes that is
the contract that lets the schema change ship without data loss.
This test exercises the real 0014 against real, pre-existing rows:
1. Apply 0001-0013 via ``migrate_to_version``.
2. Seed a claim and a remittance in v=13 (raw SQL the ORM model
declares ``matched_remittance_batch_id`` which doesn't exist at v=13).
3. Apply 0014 via ``migrate_to_version``.
4. Verify the seeded rows survive, with the same id and batch_id.
5. Verify the SAME id can be inserted in a DIFFERENT batch after 0014
(proving the composite PK took effect a single-column PK would
have rejected the second insert).
Uses a fresh ``sa.Engine`` (not the module-global ``db.engine()``)
because the conftest autouse fixture has already initialized the
module engine at v=14 we need a clean engine that starts at v=0
so ``migrate_to_version(13)`` actually applies 0001..0013.
"""
from sqlalchemy.orm import sessionmaker
engine = _fresh_engine(tmp_path)
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False, expire_on_commit=False)
from cyclone.db_migrate import migrate_to_version
from cyclone.db import Batch, Claim
# Apply 0001-0013 only.
migrate_to_version(engine, 13)
assert _user_version(engine) == 13
# Seed a claim and a remittance in v=13. We use raw SQL here because
# the ORM Claim/Remittance models declare post-0014 columns
# (``matched_remittance_batch_id``) that don't exist at v=13. The
# migration's data-preservation contract is independent of how the
# data got there — raw SQL inserts count as "data that needs to
# survive the migration" just like ORM inserts do.
with engine.begin() as c:
c.exec_driver_sql(
"INSERT INTO batches(id, kind, input_filename, parsed_at) "
"VALUES ('B-V13', '837p', 'x.txt', '2026-01-01')"
)
c.exec_driver_sql(
"INSERT INTO claims(id, batch_id, patient_control_number, "
"state, state_changed_at) VALUES "
"('CLM-PRESERVED', 'B-V13', 'M', 'submitted', '2026-01-01')"
)
c.exec_driver_sql(
"INSERT INTO batches(id, kind, input_filename, parsed_at) "
"VALUES ('B-V13R', '835', 'x.txt', '2026-01-02')"
)
c.exec_driver_sql(
"INSERT INTO remittances(id, batch_id, payer_claim_control_number, "
"status_code, received_at) VALUES "
"('CLP-PRESERVED', 'B-V13R', 'CLP-PRESERVED', '1', '2026-01-02')"
)
# Apply 0014. Data must survive.
migrate_to_version(engine, 14)
assert _user_version(engine) == 14
with engine.connect() as c:
claim = c.exec_driver_sql(
"SELECT id, batch_id, patient_control_number "
"FROM claims WHERE id='CLM-PRESERVED'"
).first()
remit = c.exec_driver_sql(
"SELECT id, batch_id, payer_claim_control_number "
"FROM remittances WHERE id='CLP-PRESERVED'"
).first()
assert claim == ("CLM-PRESERVED", "B-V13", "M")
assert remit == ("CLP-PRESERVED", "B-V13R", "CLP-PRESERVED")
# Confirm that AFTER v=14, the same id can be inserted in a different
# batch via the ORM (now that the composite PK exists, this is the
# canonical insert path).
with SessionLocal() as s:
s.add(Batch(id="B-V14", kind="837p", input_filename="y.txt",
parsed_at=datetime(2026, 1, 3, tzinfo=timezone.utc),
raw_result_json={}))
s.add(Claim(id="CLM-PRESERVED", batch_id="B-V14",
patient_control_number="M2",
state_changed_at=datetime(2026, 1, 3, tzinfo=timezone.utc)))
s.commit()
with engine.connect() as c:
rows = c.exec_driver_sql(
"SELECT batch_id FROM claims WHERE id='CLM-PRESERVED' ORDER BY batch_id"
).all()
assert [r[0] for r in rows] == ["B-V13", "B-V14"]
def test_migration_0014_preserves_joint_fk_rows(tmp_path, monkeypatch):
"""Migration 0014 also recreates the JOIN-FK tables (matches,
cas_adjustments, service_line_payments, line_reconciliations). Each of
those tables has a composite FK to claims/remittances whose ``batch_id``
side gets populated by JOIN in the migration. Insert representative
rows at v=13 and verify they survive 0014 with the new batch_id column
populated.
This is the complement to ``test_migration_0014_preserves_existing_data``
(which covers the claims/remittances tables) and exercises the
``INSERT INTO ..._new SELECT ... JOIN parent`` statements that the
simpler test cannot.
Like the simpler test, we seed at v=13 via raw SQL the ORM model
declares ``batch_id`` on the JOIN-FK tables but that column is only
added by 0014 itself, so an ORM insert at v=13 would fail.
"""
from sqlalchemy.orm import sessionmaker
engine = _fresh_engine(tmp_path)
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False, expire_on_commit=False)
from cyclone.db_migrate import migrate_to_version
migrate_to_version(engine, 13)
# Seed parent rows + one row per JOIN-FK child table (raw SQL at v=13).
with engine.begin() as c:
c.exec_driver_sql(
"INSERT INTO batches(id, kind, input_filename, parsed_at) "
"VALUES ('B-JOIN', '837p', 'j.txt', '2026-01-01')"
)
c.exec_driver_sql(
"INSERT INTO claims(id, batch_id, patient_control_number, "
"state, state_changed_at) VALUES "
"('CLM-J', 'B-JOIN', 'M', 'submitted', '2026-01-01')"
)
c.exec_driver_sql(
"INSERT INTO batches(id, kind, input_filename, parsed_at) "
"VALUES ('B-JOIN-R', '835', 'jr.txt', '2026-01-02')"
)
c.exec_driver_sql(
"INSERT INTO remittances(id, batch_id, payer_claim_control_number, "
"status_code, received_at) VALUES "
"('CLP-J', 'B-JOIN-R', 'CLP-J', '1', '2026-01-02')"
)
# Seed the four JOIN-FK child rows. At v=13 these tables only have
# the parent-id columns (no batch_id side yet) — 0014 adds the
# batch_id columns and JOINs against the parents to populate them.
c.exec_driver_sql(
"INSERT INTO service_line_payments("
"remittance_id, line_number, procedure_qualifier, procedure_code, "
"modifiers_json, charge, payment) VALUES "
"('CLP-J', 1, 'HC', '99213', '[]', 100.00, 80.00)"
)
c.exec_driver_sql(
"INSERT INTO cas_adjustments("
"remittance_id, group_code, reason_code, amount) VALUES "
"('CLP-J', 'CO', '45', 20.00)"
)
c.exec_driver_sql(
"INSERT INTO line_reconciliations("
"claim_id, status, match_score, reconciled_at) VALUES "
"('CLM-J', 'matched', 95, '2026-01-02')"
)
c.exec_driver_sql(
"INSERT INTO matches("
"claim_id, remittance_id, strategy, matched_at) VALUES "
"('CLM-J', 'CLP-J', 'exact', '2026-01-02')"
)
# Apply 0014 — the JOIN-FK tables get the batch_id column populated by
# the migration's INSERT...SELECT...JOIN statements.
migrate_to_version(engine, 14)
with engine.connect() as c:
rows = c.exec_driver_sql(
"SELECT remittance_id, remittance_batch_id "
"FROM service_line_payments WHERE remittance_id='CLP-J'"
).all()
assert rows == [("CLP-J", "B-JOIN-R")], (
"service_line_payments.remittance_batch_id must be populated "
"by 0014's JOIN against remittances; got " + repr(rows)
)
rows = c.exec_driver_sql(
"SELECT remittance_id, remittance_batch_id "
"FROM cas_adjustments WHERE remittance_id='CLP-J'"
).all()
assert rows == [("CLP-J", "B-JOIN-R")], (
"cas_adjustments.remittance_batch_id must be populated by 0014; got "
+ repr(rows)
)
rows = c.exec_driver_sql(
"SELECT claim_id, batch_id FROM line_reconciliations WHERE claim_id='CLM-J'"
).all()
assert rows == [("CLM-J", "B-JOIN")], (
"line_reconciliations.batch_id must be populated by 0014; got "
+ repr(rows)
)
rows = c.exec_driver_sql(
"SELECT claim_id, batch_id, remittance_id, remittance_batch_id "
"FROM matches WHERE claim_id='CLM-J'"
).all()
assert rows == [("CLM-J", "B-JOIN", "CLP-J", "B-JOIN-R")], (
"matches batch_id + remittance_batch_id must be populated by 0014; "
"got " + repr(rows)
)
# =============================================================================
# migrate_to_version: edge-case behavior
# =============================================================================
# These tests use a fresh ``sa.Engine`` (not ``db.engine()``) because the
# autouse conftest fixture has already initialized the module engine at the
# latest version. We need a clean engine that starts at ``user_version=0``
# so that ``migrate_to_version(N)`` actually exercises the apply / no-op /
# beyond-max branches.
def test_migrate_to_version_idempotent(tmp_path, monkeypatch):
"""Calling migrate_to_version(engine, N) twice is equivalent to calling once."""
from cyclone.db_migrate import migrate_to_version
engine = _fresh_engine(tmp_path)
migrate_to_version(engine, 5)
with engine.begin() as conn:
v1 = conn.exec_driver_sql("PRAGMA user_version").scalar()
migrate_to_version(engine, 5)
with engine.begin() as conn:
v2 = conn.exec_driver_sql("PRAGMA user_version").scalar()
assert v1 == v2 == 5
def test_migrate_to_version_lower_than_current_is_noop(tmp_path, monkeypatch):
"""migrate_to_version(engine, 3) after migrate_to_version(engine, 13) is a no-op."""
from cyclone.db_migrate import migrate_to_version
engine = _fresh_engine(tmp_path)
migrate_to_version(engine, 13)
with engine.begin() as conn:
v_after_13 = conn.exec_driver_sql("PRAGMA user_version").scalar()
migrate_to_version(engine, 3) # lower, no-op
with engine.begin() as conn:
v_after_3 = conn.exec_driver_sql("PRAGMA user_version").scalar()
assert v_after_13 == v_after_3 == 13
def test_migrate_to_version_zero_on_fresh_db(tmp_path, monkeypatch):
"""migrate_to_version(engine, 0) on a fresh DB is a no-op."""
from cyclone.db_migrate import migrate_to_version
engine = _fresh_engine(tmp_path)
migrate_to_version(engine, 0)
with engine.begin() as conn:
v = conn.exec_driver_sql("PRAGMA user_version").scalar()
assert v == 0
def test_migrate_to_version_beyond_max_applies_all(tmp_path, monkeypatch):
"""migrate_to_version(engine, 999) applies every migration (equivalent to run)."""
import re
from cyclone.db_migrate import MIGRATIONS_DIR, migrate_to_version
engine = _fresh_engine(tmp_path)
migrate_to_version(engine, 999)
with engine.begin() as conn:
v = conn.exec_driver_sql("PRAGMA user_version").scalar()
# Should equal the highest version present in the migrations directory.
max_version = max(
int(re.match(r"^(\d+)", p.name).group(1))
for p in MIGRATIONS_DIR.glob("*.sql")
)
assert v == max_version
+31 -17
View File
@@ -72,8 +72,9 @@ def test_create_and_query_claim_with_default_state():
s.add(c) s.add(c)
s.commit() s.commit()
# Migration 0014: composite PK (batch_id, id). PK lookups use tuples.
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
loaded = s.get(Claim, "CLM-1") loaded = s.get(Claim, ("b1", "CLM-1"))
assert loaded is not None assert loaded is not None
assert loaded.state == ClaimState.SUBMITTED assert loaded.state == ClaimState.SUBMITTED
assert loaded.charge_amount == Decimal("124.00") assert loaded.charge_amount == Decimal("124.00")
@@ -99,7 +100,7 @@ def test_claim_uses_db_default_when_state_omitted():
s.commit() s.commit()
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
loaded = s.get(Claim, "CLM-1") loaded = s.get(Claim, ("b1", "CLM-1"))
assert loaded is not None assert loaded is not None
assert loaded.state == ClaimState.SUBMITTED assert loaded.state == ClaimState.SUBMITTED
@@ -123,7 +124,7 @@ def test_state_before_reversal_round_trips():
s.commit() s.commit()
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
loaded = s.get(Claim, "CLM-1") loaded = s.get(Claim, ("b1", "CLM-1"))
assert loaded is not None assert loaded is not None
assert loaded.state_before_reversal == ClaimState.PAID assert loaded.state_before_reversal == ClaimState.PAID
@@ -154,7 +155,8 @@ def test_batch_cascade_delete_drops_claims():
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
assert s.get(Batch, "b1") is None assert s.get(Batch, "b1") is None
assert s.get(Claim, "CLM-1") is None # Composite PK lookup — both batch_id AND id are required.
assert s.get(Claim, ("b1", "CLM-1")) is None
def test_create_and_query_remittance(): def test_create_and_query_remittance():
@@ -178,7 +180,7 @@ def test_create_and_query_remittance():
s.commit() s.commit()
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
loaded = s.get(Remittance, "CLP-1") loaded = s.get(Remittance, ("b1", "CLP-1"))
assert loaded is not None assert loaded is not None
assert loaded.status_code == "1" assert loaded.status_code == "1"
assert loaded.total_paid == Decimal("124.00") assert loaded.total_paid == Decimal("124.00")
@@ -201,10 +203,14 @@ def test_cas_adjustment_aggregation():
) )
s.add(r) s.add(r)
s.flush() s.flush()
# Migration 0014: remittance_batch_id is required (NOT NULL) on
# CasAdjustment; it mirrors remittance_id's batch.
s.add_all([ s.add_all([
CasAdjustment(remittance_id="CLP-1", group_code="CO", reason_code="45", CasAdjustment(remittance_id="CLP-1", remittance_batch_id="b1",
group_code="CO", reason_code="45",
amount=Decimal("30.00")), amount=Decimal("30.00")),
CasAdjustment(remittance_id="CLP-1", group_code="PR", reason_code="1", CasAdjustment(remittance_id="CLP-1", remittance_batch_id="b1",
group_code="PR", reason_code="1",
amount=Decimal("32.00")), amount=Decimal("32.00")),
]) ])
s.commit() s.commit()
@@ -235,15 +241,16 @@ def test_remittance_raw_json_round_trips_dict():
s.commit() s.commit()
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
loaded = s.get(Remittance, "CLP-1") loaded = s.get(Remittance, ("b1", "CLP-1"))
assert loaded.raw_json == {"x12_segments": ["CLP", "NM1", "SVC"], "version": "005010X221A1"} assert loaded.raw_json == {"x12_segments": ["CLP", "NM1", "SVC"], "version": "005010X221A1"}
def test_remittance_allows_duplicate_pcn_in_same_batch(): def test_remittance_allows_duplicate_pcn_in_same_batch():
"""An 835 ERA can carry multiple CLP segments with the same """An 835 ERA can carry multiple CLP segments with the same
payer_claim_control_number (e.g. reversals re-referencing the payer_claim_control_number (e.g. reversals re-referencing the
original PCN). Remittance identity is by ``id`` (CLP01), not by original PCN). Remittance identity is by the composite PK
(batch_id, PCN). This test documents the absence of the constraint (batch_id, id) id (CLP01) distinguishes rows in the same batch.
This test documents the absence of the (batch_id, PCN) constraint
removed in migration 0003. removed in migration 0003.
""" """
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
@@ -286,12 +293,13 @@ def test_remittance_cascade_delete_drops_cas_adjustments():
) )
s.add(r) s.add(r)
s.flush() s.flush()
s.add(CasAdjustment(remittance_id="CLP-1", group_code="CO", s.add(CasAdjustment(remittance_id="CLP-1", remittance_batch_id="b1",
reason_code="45", amount=Decimal("30.00"))) group_code="CO", reason_code="45",
amount=Decimal("30.00")))
s.commit() s.commit()
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
r = s.get(Remittance, "CLP-1") r = s.get(Remittance, ("b1", "CLP-1"))
s.delete(r) s.delete(r)
s.commit() s.commit()
@@ -318,8 +326,10 @@ def test_create_match_and_activity_event():
status_code="1", total_charge=Decimal("124.00"), total_paid=Decimal("124.00"), status_code="1", total_charge=Decimal("124.00"), total_paid=Decimal("124.00"),
received_at=datetime(2026, 6, 19, tzinfo=timezone.utc), received_at=datetime(2026, 6, 19, tzinfo=timezone.utc),
)) ))
# Migration 0014: Match needs batch_id + remittance_batch_id (NOT NULL).
m = Match( m = Match(
claim_id="CLM-1", remittance_id="CLP-1", claim_id="CLM-1", batch_id="b1",
remittance_id="CLP-1", remittance_batch_id="b1",
strategy="auto", is_reversal=False, strategy="auto", is_reversal=False,
matched_at=datetime(2026, 6, 19, 13, 0, tzinfo=timezone.utc), matched_at=datetime(2026, 6, 19, 13, 0, tzinfo=timezone.utc),
) )
@@ -361,9 +371,13 @@ def test_match_multiple_rows_per_claim():
status_code="1", received_at=datetime.now(timezone.utc))) status_code="1", received_at=datetime.now(timezone.utc)))
s.add(Remittance(id="CLP-2", batch_id="b1", payer_claim_control_number="y", s.add(Remittance(id="CLP-2", batch_id="b1", payer_claim_control_number="y",
status_code="1", received_at=datetime.now(timezone.utc))) status_code="1", received_at=datetime.now(timezone.utc)))
s.add(Match(claim_id="CLM-1", remittance_id="CLP-1", strategy="auto", # Migration 0014: batch_id + remittance_batch_id required on Match.
matched_at=datetime.now(timezone.utc))) s.add(Match(claim_id="CLM-1", batch_id="b1",
s.add(Match(claim_id="CLM-1", remittance_id="CLP-2", strategy="manual", remittance_id="CLP-1", remittance_batch_id="b1",
strategy="auto", matched_at=datetime.now(timezone.utc)))
s.add(Match(claim_id="CLM-1", batch_id="b1",
remittance_id="CLP-2", remittance_batch_id="b1",
strategy="manual",
matched_at=datetime.now(timezone.utc), matched_at=datetime.now(timezone.utc),
is_reversal=True, prior_claim_state=ClaimState.PAID)) is_reversal=True, prior_claim_state=ClaimState.PAID))
# No IntegrityError — two Match rows per claim are now allowed. # No IntegrityError — two Match rows per claim are now allowed.
+10 -4
View File
@@ -75,7 +75,9 @@ def test_match_endpoint_links_remit_to_claim(client: TestClient):
assert r.status_code == 200, r.text assert r.status_code == 200, r.text
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
c = s.get(Claim, "C1") # Migration 0014: composite PK — filter-by-id returns the single
# claim with this id (no cross-batch duplicates in this fixture).
c = s.query(Claim).filter(Claim.id == "C1").first()
assert c.matched_remittance_id == "R1" assert c.matched_remittance_id == "R1"
@@ -119,7 +121,8 @@ def test_resubmit_moves_rejected_to_submitted_and_increments_count(client: TestC
assert r.status_code == 200, r.text assert r.status_code == 200, r.text
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
c = s.get(Claim, "C1") # Migration 0014: composite PK (batch_id, id).
c = s.get(Claim, ("B-1", "C1"))
assert c.state == ClaimState.SUBMITTED assert c.state == ClaimState.SUBMITTED
assert c.rejection_reason is None assert c.rejection_reason is None
assert c.resubmit_count == 3 assert c.resubmit_count == 3
@@ -175,7 +178,9 @@ def _seed_rejected_claim_with_raw_837(client: TestClient) -> str:
real_id = r.json()["claims"][0]["claim_id"] real_id = r.json()["claims"][0]["claim_id"]
_seed_batch() _seed_batch()
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
c = s.get(Claim, real_id) # Migration 0014: filter-by-id since the parse-837 batch id is
# a fresh UUID we don't know here.
c = s.query(Claim).filter(Claim.id == real_id).first()
assert c is not None assert c is not None
c.state = ClaimState.REJECTED c.state = ClaimState.REJECTED
c.rejection_reason = "test fixture" c.rejection_reason = "test fixture"
@@ -216,7 +221,8 @@ def test_resubmit_with_download_returns_zip_with_one_x12_per_accepted_claim(clie
# Side effect: claim actually moved to SUBMITTED. # Side effect: claim actually moved to SUBMITTED.
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
c = s.get(Claim, real_id) # Migration 0014: filter-by-id (batch_id is a fresh UUID).
c = s.query(Claim).filter(Claim.id == real_id).first()
assert c.state == ClaimState.SUBMITTED assert c.state == ClaimState.SUBMITTED
+6 -3
View File
@@ -116,17 +116,20 @@ def _seed_pair_with_two_lines():
reconcile_run(s, bid_835) reconcile_run(s, bid_835)
s.commit() s.commit()
return claim_id return claim_id, bid_837
def test_done_today_lane_includes_matched_remittance_line_counts(client): def test_done_today_lane_includes_matched_remittance_line_counts(client):
claim_id = _seed_pair_with_two_lines() claim_id, bid_837 = _seed_pair_with_two_lines()
# Backfill state_changed_at so the claim lands in the done_today lane # Backfill state_changed_at so the claim lands in the done_today lane
# (reconcile.run() doesn't write it; the API surface is exercised by # (reconcile.run() doesn't write it; the API surface is exercised by
# the existing manual-match flow which sets it). # the existing manual-match flow which sets it).
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
cl = s.get(db.Claim, claim_id) # Migration 0014: composite PK (batch_id, id). Use filter-by-id
# since we don't have a single canonical batch_id (the test seeds
# both 837 and 835 batches under fresh UUIDs).
cl = s.query(db.Claim).filter(db.Claim.id == claim_id).first()
cl.state_changed_at = datetime.now(timezone.utc) cl.state_changed_at = datetime.now(timezone.utc)
s.commit() s.commit()
+4 -2
View File
@@ -131,9 +131,11 @@ def test_done_today_includes_recent_terminal_states():
# Force C1's state_changed_at to be recent, C2 to be old. # Force C1's state_changed_at to be recent, C2 to be old.
now = datetime.now(timezone.utc) now = datetime.now(timezone.utc)
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
c1 = s.get(Claim, "C1") # Migration 0014: composite PK — filter-by-id returns the single
# claim with that id (no cross-batch duplicates in this test).
c1 = s.query(Claim).filter(Claim.id == "C1").first()
c1.state_changed_at = now - timedelta(hours=2) c1.state_changed_at = now - timedelta(hours=2)
c2 = s.get(Claim, "C2") c2 = s.query(Claim).filter(Claim.id == "C2").first()
c2.state_changed_at = now - timedelta(hours=30) c2.state_changed_at = now - timedelta(hours=30)
s.commit() s.commit()
+7 -3
View File
@@ -111,7 +111,9 @@ def test_rejection_moves_submitted_claim_to_rejected_state():
assert result.matched == ["CLP-1"] assert result.matched == ["CLP-1"]
assert result.orphans == [] assert result.orphans == []
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
c = s.get(Claim, "CLP-1") # Migration 0014: composite PK — claim is in batch "B-1"; use
# filter-by-id since the test has only one claim with this id.
c = s.query(Claim).filter(Claim.id == "CLP-1").first()
assert c.state == ClaimState.REJECTED assert c.state == ClaimState.REJECTED
assert c.rejected_at is not None assert c.rejected_at is not None
assert "999" in (c.rejection_reason or "") assert "999" in (c.rejection_reason or "")
@@ -130,7 +132,8 @@ def test_accepted_set_leaves_claim_alone():
assert result.matched == [] assert result.matched == []
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
c = s.get(Claim, "CLP-1") # Migration 0014: composite PK — see test_rejection_moves...
c = s.query(Claim).filter(Claim.id == "CLP-1").first()
assert c.state == ClaimState.SUBMITTED assert c.state == ClaimState.SUBMITTED
assert c.rejected_at is None assert c.rejected_at is None
assert c.rejection_reason is None assert c.rejection_reason is None
@@ -162,5 +165,6 @@ def test_already_rejected_claim_is_idempotent():
assert result.matched == [] # no-op assert result.matched == [] # no-op
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
c = s.get(Claim, "CLP-1") # Migration 0014: composite PK — see test_rejection_moves...
c = s.query(Claim).filter(Claim.id == "CLP-1").first()
assert c.state == ClaimState.REJECTED assert c.state == ClaimState.REJECTED
@@ -67,7 +67,9 @@ def test_acknowledge_marks_unacknowledged_claim():
assert body["not_rejected"] == 0 assert body["not_rejected"] == 0
with db.SessionLocal()() as session: with db.SessionLocal()() as session:
c = session.get(Claim, "C1") # Migration 0014: composite PK — filter-by-id returns the single
# claim with this id (no cross-batch duplicates in this fixture).
c = session.query(Claim).filter(Claim.id == "C1").first()
assert c.payer_rejected_acknowledged_at is not None assert c.payer_rejected_acknowledged_at is not None
assert c.payer_rejected_acknowledged_actor == "operator" assert c.payer_rejected_acknowledged_actor == "operator"
# Original fields stay intact for audit. # Original fields stay intact for audit.
+27 -4
View File
@@ -271,7 +271,9 @@ def test_run_matches_and_updates_state(fixture_835):
assert result.unmatched_claims == 0 assert result.unmatched_claims == 0
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
claim = s.get(Claim, "CLM-1") # Migration 0014: claims PK is composite (batch_id, id). The test
# only has one claim with id "CLM-1" so filter-by-id returns it.
claim = s.query(Claim).filter(Claim.id == "CLM-1").first()
assert claim.state == ClaimState.PAID assert claim.state == ClaimState.PAID
assert claim.matched_remittance_id == "CLP-1:b1xxxxxx" assert claim.matched_remittance_id == "CLP-1:b1xxxxxx"
@@ -303,13 +305,32 @@ def test_run_reversal_flips_paid_to_reversed(fixture_835):
# (9 days, outside window) which produced no match. # (9 days, outside window) which produced no match.
_make_claim(s, "CLM-1", "PCN-A", "100.00", date(2026, 6, 5), _make_claim(s, "CLM-1", "PCN-A", "100.00", date(2026, 6, 5),
state=ClaimState.PAID) state=ClaimState.PAID)
# Match row already exists from prior reconcile. # Prior Match exists from an earlier reconcile — represented by a
# Match row pointing at a Remittance that already happened. Migration
# 0014: composite FK requires the parent Remittance row to exist, so
# we add a prior batch to host it (separate batch so reconcile.run
# below only sees CLP-REV in batch "b1"). The prior Match row is just
# a hint that the claim was paid earlier — we don't care about the
# prior Match's batch membership here.
_make_batch(s, batch_id="b-PRIOR", kind="835")
s.add(Remittance(
id="CLP-OLD:bPRIORxxxx", batch_id="b-PRIOR",
payer_claim_control_number="PCN-A",
status_code="1", total_charge=Decimal("100.00"),
total_paid=Decimal("100.00"),
received_at=datetime(2026, 6, 1, tzinfo=timezone.utc),
service_date=date(2026, 6, 1),
))
s.flush()
s.add(Match( s.add(Match(
claim_id="CLM-1", remittance_id="CLP-OLD:b1xxxxxx", claim_id="CLM-1", batch_id="b1",
remittance_id="CLP-OLD:bPRIORxxxx",
remittance_batch_id="b-PRIOR",
strategy="auto", matched_at=datetime.now(timezone.utc), strategy="auto", matched_at=datetime.now(timezone.utc),
is_reversal=False, is_reversal=False,
)) ))
s.flush() s.flush()
# The reversal hits batch "b1" — the one reconcile.run() targets.
_make_remit(s, "CLP-REV", "PCN-A", "22", "100.00", "-100.00", _make_remit(s, "CLP-REV", "PCN-A", "22", "100.00", "-100.00",
date(2026, 6, 10), is_reversal=True) date(2026, 6, 10), is_reversal=True)
s.commit() s.commit()
@@ -321,7 +342,9 @@ def test_run_reversal_flips_paid_to_reversed(fixture_835):
assert result.matched == 1 assert result.matched == 1
with db.SessionLocal()() as s: with db.SessionLocal()() as s:
claim = s.get(Claim, "CLM-1") # Migration 0014: see test_run_matches_and_updates_state — use
# filter-by-id since the test has one claim with id "CLM-1".
claim = s.query(Claim).filter(Claim.id == "CLM-1").first()
assert claim.state == ClaimState.REVERSED assert claim.state == ClaimState.REVERSED
# Match rows: original + reversal. # Match rows: original + reversal.
matches = s.query(Match).filter(Match.claim_id == "CLM-1").all() matches = s.query(Match).filter(Match.claim_id == "CLM-1").all()
+55
View File
@@ -201,3 +201,58 @@ def test_persistence_across_session():
loaded = s2.get_batch("b-x") loaded = s2.get_batch("b-x")
assert loaded is not None assert loaded is not None
assert loaded["kind"] == "837p" assert loaded["kind"] == "837p"
def test_find_existing_batch_for_claim_returns_none_for_unknown():
"""Unknown claim_id -> None."""
from cyclone import store as store_mod # noqa: F401
assert store_mod.find_existing_batch_for_claim("nope") is None
def test_find_existing_batch_for_claim_returns_batch_id():
"""Known claim_id -> batch_id of the holding batch."""
from cyclone import db as _db
from cyclone.db import Batch, Claim
from datetime import datetime, timezone
with _db.SessionLocal()() as s:
s.add(Batch(
id="B1", kind="837p", input_filename="x.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Claim(id="CLM-A", batch_id="B1", patient_control_number="M1"))
s.commit()
from cyclone import store as store_mod # noqa: F401
assert store_mod.find_existing_batch_for_claim("CLM-A") == "B1"
def test_find_existing_batch_for_remit_returns_none_for_unknown():
"""Unknown remit id -> None."""
from cyclone import store as store_mod # noqa: F401
assert store_mod.find_existing_batch_for_remit("nope") is None
def test_find_existing_batch_for_remit_returns_batch_id():
"""Known remit id -> batch_id of the holding batch."""
from cyclone import db as _db
from cyclone.db import Batch, Remittance
from datetime import datetime, timezone
with _db.SessionLocal()() as s:
s.add(Batch(
id="B2", kind="835", input_filename="y.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Remittance(
id="CLP-A", batch_id="B2",
payer_claim_control_number="CLP-A",
status_code="1",
received_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
))
s.commit()
from cyclone import store as store_mod # noqa: F401
assert store_mod.find_existing_batch_for_remit("CLP-A") == "B2"
+6 -3
View File
@@ -221,7 +221,8 @@ def test_manual_match_pairs_claim_with_orphan_remit():
# Match row was persisted with strategy="manual". # Match row was persisted with strategy="manual".
with db.SessionLocal()() as session: with db.SessionLocal()() as session:
from cyclone.db import Claim, Match from cyclone.db import Claim, Match
claim_row = session.get(Claim, claim_id) # Migration 0014: composite PK — claim ingested under batch "b-837".
claim_row = session.get(Claim, ("b-837", claim_id))
assert claim_row is not None assert claim_row is not None
assert claim_row.matched_remittance_id == remit_id assert claim_row.matched_remittance_id == remit_id
match_rows = ( match_rows = (
@@ -417,7 +418,8 @@ def test_manual_match_populates_line_reconciliation_rows():
assert unmatched.service_line_payment_id is None assert unmatched.service_line_payment_id is None
# Remittance aggregates were recomputed (0 CAS in this test). # Remittance aggregates were recomputed (0 CAS in this test).
r = session.get(Remittance, remit_id) # Migration 0014: composite PK — remit ingested under batch "b-835-sp7".
r = session.get(Remittance, ("b-835-sp7", remit_id))
assert r is not None assert r is not None
assert r.adjustment_amount == Decimal("0") assert r.adjustment_amount == Decimal("0")
assert r.claim_level_adjustment_amount == Decimal("0") assert r.claim_level_adjustment_amount == Decimal("0")
@@ -583,7 +585,8 @@ def test_manual_match_idempotent_line_reconciliation():
# CLP-level CAS aggregate now reflects the inserted row. # CLP-level CAS aggregate now reflects the inserted row.
from cyclone.db import Remittance from cyclone.db import Remittance
r = session.get(Remittance, remit_id) # Migration 0014: composite PK — remit ingested under batch "b-835-idemp".
r = session.get(Remittance, ("b-835-idemp", remit_id))
assert r is not None assert r is not None
assert r.adjustment_amount == Decimal("20.00") assert r.adjustment_amount == Decimal("20.00")
assert r.claim_level_adjustment_amount == Decimal("20.00") assert r.claim_level_adjustment_amount == Decimal("20.00")
File diff suppressed because it is too large Load Diff