Tyler
7290cac643
The implementer caught that claims.id is a single-column PK, which prevents the same CLM01 from existing in multiple batches. That makes the spec's pre-flight 409 workflow unreachable and resubmits impossible. Migration 0014 relaxes the PKs to composite (batch_id, id) on both claims and remittances, and updates all FKs that referenced the old single-column PK. Task 1.3 in the plan is now the migration; the previous Task 1.3 (helper tightening) is renumbered to Task 1.4 and will run after 0014 lands.
106 KiB
106 KiB
Parse → Detect → Decide Workflow Implementation Plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: Replace the 409-on-persist workflow with parse-first, pre-flight dedup, structured 409 with full parse result + collision summary, ?force=true to skip-and-continue, and DELETE /api/batches/{id} for cascade cleanup. Frontend gets an error panel with four actions (force-insert, open prior, delete prior, pick different file).
Architecture: Pre-flight dedup runs between validation and persist in both parse_837 and parse_835 endpoints. New dedup.py module owns CollisionReport + preflight_837/preflight_835. New ?force=true query param bypasses pre-flight; the existing per-row s.get(Claim, claim_id) dedup in store.add silently skips colliding rows, and the 200 response includes skipped_claim_ids. New DELETE /api/batches/{id} endpoint hard-deletes with cascade; refuses if any claim is past submitted state. Frontend ApiError carries collisions + parseResult; Upload.tsx panel renders the full parse result and the four actions.
Tech Stack: Python 3.11+, SQLAlchemy 2.x, FastAPI, SQLite (sqlcipher3 optional), React 18 + TanStack Query + Radix UI + sonner, Vitest + React Testing Library, pytest.
Spec: docs/superpowers/specs/2026-06-21-cyclone-parse-decide-workflow-design.md — read fully before starting.
Worktree setup (one-time):
cd /Users/openclaw/dev/cyclone
# claims-unique-fix worktree already exists from prior SP work; Tasks 1.1 and 1.2 are done on it.
cd .worktrees/claims-unique-fix
# venv is symlinked to main repo's .venv
# Use PYTHONPATH override to make Python import the worktree's source, not the main repo's:
export PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src:$PYTHONPATH
All commits happen in this worktree. Merge to main via fast-forward when each phase ends.
Phase 1 — Backend pre-flight dedup module + schema foundation (mostly done; Task 1.3 is new foundation work added after spec/plan review revealed that claims.id PK prevents the workflow from firing)
Task 1.1: Migration 0013 drops inline UNIQUE — DONE
Files:
backend/src/cyclone/migrations/0013_drop_claims_unique_constraint.sql(created)backend/tests/test_db_migrate.py(modified)
Commit b6efd0e on claims-unique-fix. No further action.
Task 1.2: Store helpers find_existing_batch_for_claim / find_existing_batch_for_remit — DONE
Files:
backend/src/cyclone/store.py(modified)backend/tests/test_store.py(modified)
Commit 890207f on claims-unique-fix. No further action.
Task 1.3: Migration 0014 — relax claims and remittances PKs to composite (batch_id, id)
The current schema has claims.id and remittances.id as single-column
PRIMARY KEYs. This makes the spec'd "cross-batch CLM01 collision" workflow
unreachable: the same CLM01 cannot exist in two batches, so the pre-flight
409 path can never fire, force-insert can never skip anything, and resubmits
are impossible. Migration 0014 relaxes the PKs to composite (batch_id, id)
and updates every FK that referenced the old single-column PK.
After 0014, the pre-flight dedup is genuinely exercisable end-to-end and the entire workflow in the spec actually fires on real data.
Files:
-
Create:
backend/src/cyclone/migrations/0014_relax_claims_remits_pk.sql -
Modify:
backend/src/cyclone/db.py(drop the redundant single-columnunique=Trueonid, if any; add composite PK markers) -
Modify:
backend/tests/test_db_migrate.py(0014 tests) -
Step 1: Inspect the live schema and identify every FK
Before writing the migration, enumerate every FK that points at claims(id) or remittances(id). Run:
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -c "
from cyclone import db
from sqlalchemy import inspect
ins = inspect(db.engine())
for tbl in ins.get_table_names():
for fk in ins.get_foreign_keys(tbl):
if 'claims' in (fk.get('referred_table') or '') or 'remittances' in (fk.get('referred_table') or ''):
print(tbl, fk['constrained_columns'], '->', fk['referred_table'], fk['referred_columns'])
"
You should see (at minimum) FKs from remittances, matches, cas_adjustments, line_reconciliations, activity_events (if FK-declared there), and claims.matched_remittance_id. Capture this list — the migration must update each one to include batch_id.
- Step 2: Write the failing migration tests
Add to backend/tests/test_db_migrate.py:
def test_migration_0014_relaxes_claims_pk_to_composite(tmp_path, monkeypatch):
"""Migration 0014 changes claims PK from single-column id to (batch_id, id).
After 0014, two Claim rows with the same id can coexist if they are
in different batches. Same for Remittance. The dedup story moves to
the application layer (preflight_837 / preflight_835) instead of the
schema.
"""
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
from cyclone import db, migrations
# Synthesize 0001 + 0013 + 0014 by calling the migration runner with
# only those versions. Use the existing migrate runner pattern from
# test_db_migrate.py.
db._reset_for_tests()
db.init_db() # runs all migrations up to current
# After full migration: try to insert two claims with the same id
# in different batches.
from datetime import datetime, timezone
from cyclone.db import Batch, Claim
with db.SessionLocal()() as s:
s.add(Batch(id="B1", kind="837p", input_filename="b1.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={}))
s.add(Batch(id="B2", kind="837p", input_filename="b2.txt",
parsed_at=datetime(2026, 1, 2, tzinfo=timezone.utc),
raw_result_json={}))
s.add(Claim(id="CLM-A", batch_id="B1", patient_control_number="M1",
state_changed_at=datetime(2026, 1, 1, tzinfo=timezone.utc)))
s.add(Claim(id="CLM-A", batch_id="B2", patient_control_number="M1",
state_changed_at=datetime(2026, 1, 2, tzinfo=timezone.utc)))
s.commit() # must NOT raise
with db.SessionLocal()() as s:
rows = s.execute(db.text("SELECT id, batch_id FROM claims WHERE id='CLM-A' ORDER BY batch_id")).all()
assert rows == [("CLM-A", "B1"), ("CLM-A", "B2")]
def test_migration_0014_relaxes_remittances_pk_to_composite(tmp_path, monkeypatch):
"""Same shape for remittances: same CLP01 can exist in two batches."""
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
from cyclone import db
from datetime import datetime, timezone
from cyclone.db import Batch, Remittance
db._reset_for_tests()
db.init_db()
with db.SessionLocal()() as s:
s.add(Batch(id="B1", kind="835", input_filename="b1.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={}))
s.add(Batch(id="B2", kind="835", input_filename="b2.txt",
parsed_at=datetime(2026, 1, 2, tzinfo=timezone.utc),
raw_result_json={}))
s.add(Remittance(id="CLP-A", batch_id="B1",
payer_claim_control_number="CLP-A",
status_code="1",
received_at=datetime(2026, 1, 1, tzinfo=timezone.utc)))
s.add(Remittance(id="CLP-A", batch_id="B2",
payer_claim_control_number="CLP-A",
status_code="1",
received_at=datetime(2026, 1, 2, tzinfo=timezone.utc)))
s.commit()
with db.SessionLocal()() as s:
rows = s.execute(db.text("SELECT id, batch_id FROM remittances WHERE id='CLP-A' ORDER BY batch_id")).all()
assert rows == [("CLP-A", "B1"), ("CLP-A", "B2")]
def test_migration_0014_preserves_existing_data(tmp_path, monkeypatch):
"""If the DB already has claim/remittance rows when 0014 runs, the rows
survive the table recreation (INSERT INTO new SELECT * FROM old).
"""
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
# Apply migrations 0001 through 0013, insert a row, then apply 0014.
# (See test_db_migrate.py for the migrate-to-version helper.)
from cyclone import db
from datetime import datetime, timezone
from cyclone.db import Batch, Claim
db._reset_for_tests()
db.init_db() # up to 0014
with db.SessionLocal()() as s:
s.add(Batch(id="B-OLD", kind="837p", input_filename="x.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={}))
s.add(Claim(id="CLM-OLD", batch_id="B-OLD", patient_control_number="M",
state_changed_at=datetime(2026, 1, 1, tzinfo=timezone.utc)))
s.commit()
# Verify row still exists.
with db.SessionLocal()() as s:
row = s.execute(db.text("SELECT id, batch_id FROM claims WHERE id='CLM-OLD'")).first()
assert row == ("CLM-OLD", "B-OLD")
- Step 3: Run the new tests, expect FAIL
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest tests/test_db_migrate.py -k "0014" --no-header -q
Expected: FAIL with IntegrityError: UNIQUE constraint failed: claims.id (the same row insertion that succeeds after 0014 fails before it).
- Step 4: Write migration 0014
Create backend/src/cyclone/migrations/0014_relax_claims_remits_pk.sql:
-- version: 14
-- Relax PRIMARY KEYs on `claims` and `remittances` from single-column (id)
-- to composite (batch_id, id). Enables resubmits (same CLM01 / CLP01 in
-- different batches) and makes the pre-flight dedup workflow exercisable.
--
-- Strategy (mirrors 0013): table recreation with PRAGMA
-- defer_foreign_keys. We must recreate every table that has an FK pointing
-- at `claims(id)` or `remittances(id)` so that FK constraints can be
-- updated to point at the new composite PK.
--
-- FKs that need updating (verified via inspect(db.engine()) on the live
-- schema before this migration was written):
-- - claims.matched_remittance_id -> remittances(id) becomes -> (batch_id, id)
-- - remittances.claim_id -> claims(id) becomes -> (batch_id, id)
-- - matches.claim_id -> claims(id) ON DELETE CASCADE becomes -> (batch_id, id)
-- - matches.remittance_id -> remittances(id) ON DELETE CASCADE becomes -> (batch_id, id)
-- - cas_adjustments.remittance_id -> remittances(id) ON DELETE CASCADE becomes -> (batch_id, id)
-- - line_reconciliations.remittance_id -> remittances(id) ON DELETE CASCADE becomes -> (batch_id, id)
-- - line_reconciliations.claim_id -> claims(id) ON DELETE CASCADE becomes -> (batch_id, id)
--
-- activity_events has no FKs to claims/remittances in 0001 (verified), so
-- we don't recreate it.
PRAGMA defer_foreign_keys = ON;
PRAGMA foreign_keys = OFF; -- required for table recreation in SQLite
-- Step 1: recreate `remittances` with composite PK
CREATE TABLE remittances_new (
id TEXT NOT NULL,
batch_id TEXT NOT NULL REFERENCES batches(id) ON DELETE CASCADE,
payer_claim_control_number TEXT NOT NULL,
claim_id TEXT, -- FK to claims(batch_id, id) added after claims is recreated
status_code TEXT NOT NULL,
status_label TEXT,
total_charge NUMERIC(12, 2) NOT NULL DEFAULT 0,
total_paid NUMERIC(12, 2) NOT NULL DEFAULT 0,
patient_responsibility NUMERIC(12, 2),
adjustment_amount NUMERIC(12, 2) NOT NULL DEFAULT 0,
received_at DATETIME NOT NULL,
service_date DATE,
is_reversal INTEGER NOT NULL DEFAULT 0,
raw_json TEXT,
PRIMARY KEY (batch_id, id)
);
INSERT INTO remittances_new
SELECT id, batch_id, payer_claim_control_number, claim_id, status_code,
status_label, total_charge, total_paid, patient_responsibility,
adjustment_amount, received_at, service_date, is_reversal, raw_json
FROM remittances;
DROP TABLE remittances;
ALTER TABLE remittances_new RENAME TO remittances;
CREATE INDEX ix_remittances_claim_id ON remittances(claim_id);
CREATE INDEX ix_remittances_payer_claim_control_number ON remittances(payer_claim_control_number);
CREATE INDEX ix_remittances_status_code ON remittances(status_code);
-- Step 2: recreate `claims` with composite PK
CREATE TABLE claims_new (
id TEXT NOT NULL,
batch_id TEXT NOT NULL REFERENCES batches(id) ON DELETE CASCADE,
patient_control_number TEXT NOT NULL,
service_date_from DATE,
service_date_to DATE,
charge_amount NUMERIC(12, 2) NOT NULL DEFAULT 0,
provider_npi TEXT,
payer_id TEXT,
state TEXT NOT NULL DEFAULT 'submitted',
state_before_reversal TEXT,
matched_remittance_id TEXT, -- FK added after remittances is recreated
raw_json TEXT,
PRIMARY KEY (batch_id, id)
);
INSERT INTO claims_new
SELECT id, batch_id, patient_control_number, service_date_from,
service_date_to, charge_amount, provider_npi, payer_id, state,
state_before_reversal, matched_remittance_id, raw_json
FROM claims;
DROP TABLE claims;
ALTER TABLE claims_new RENAME TO claims;
CREATE INDEX ix_claims_state ON claims(state);
CREATE INDEX ix_claims_patient_control_number ON claims(patient_control_number);
CREATE INDEX ix_claims_service_date_from ON claims(service_date_from);
-- Step 3: recreate `matches`, `cas_adjustments`, `line_reconciliations` so
-- their FKs point at the new composite PKs.
CREATE TABLE matches_new (
id INTEGER PRIMARY KEY AUTOINCREMENT,
claim_id TEXT NOT NULL,
batch_id TEXT NOT NULL, -- NEW: paired with claim_id for the composite FK
remittance_id TEXT NOT NULL,
remittance_batch_id TEXT NOT NULL, -- NEW: paired with remittance_id
strategy TEXT NOT NULL,
matched_at DATETIME NOT NULL,
prior_claim_state TEXT,
is_reversal INTEGER NOT NULL DEFAULT 0,
FOREIGN KEY (batch_id, claim_id) REFERENCES claims(batch_id, id) ON DELETE CASCADE,
FOREIGN KEY (remittance_batch_id, remittance_id) REFERENCES remittances(batch_id, id) ON DELETE CASCADE
);
INSERT INTO matches_new
SELECT m.id, m.claim_id, c.batch_id, m.remittance_id, r.batch_id,
m.strategy, m.matched_at, m.prior_claim_state, m.is_reversal
FROM matches m
JOIN claims c ON c.id = m.claim_id
JOIN remittances r ON r.id = m.remittance_id;
DROP TABLE matches;
ALTER TABLE matches_new RENAME TO matches;
CREATE INDEX ix_matches_claim_id ON matches(claim_id);
CREATE INDEX ix_matches_remittance_id ON matches(remittance_id);
CREATE INDEX ix_matches_matched_at ON matches(matched_at);
CREATE TABLE cas_adjustments_new (
id INTEGER PRIMARY KEY AUTOINCREMENT,
remittance_id TEXT NOT NULL,
remittance_batch_id TEXT NOT NULL,
group_code TEXT NOT NULL,
reason_code TEXT NOT NULL,
amount NUMERIC(12, 2) NOT NULL,
quantity NUMERIC(10, 2),
FOREIGN KEY (remittance_batch_id, remittance_id) REFERENCES remittances(batch_id, id) ON DELETE CASCADE
);
INSERT INTO cas_adjustments_new
SELECT ca.id, ca.remittance_id, r.batch_id, ca.group_code, ca.reason_code,
ca.amount, ca.quantity
FROM cas_adjustments ca
JOIN remittances r ON r.id = ca.remittance_id;
DROP TABLE cas_adjustments;
ALTER TABLE cas_adjustments_new RENAME TO cas_adjustments;
CREATE INDEX ix_cas_adjustments_remittance_id ON cas_adjustments(remittance_id);
-- line_reconciliations: see 0006. Its columns are remittance_id, claim_id,
-- service_line_payment_id, superseded_by_id, service_line_payment_id. We
-- need to add remittance_batch_id and batch_id columns for the new FKs.
-- (The full column list must match 0006.)
CREATE TABLE line_reconciliations_new (
id INTEGER PRIMARY KEY AUTOINCREMENT,
remittance_id VARCHAR(64) NOT NULL,
remittance_batch_id VARCHAR(64) NOT NULL,
claim_id VARCHAR(64) NOT NULL,
batch_id VARCHAR(64) NOT NULL,
service_line_payment_id INTEGER REFERENCES service_line_payments(id) ON DELETE SET NULL,
superseded_by_id INTEGER REFERENCES service_line_payments(id) ON DELETE SET NULL,
service_line_payment_id_new INTEGER REFERENCES service_line_payments(id) ON DELETE SET NULL,
-- ... (carry over every column from 0006 + service_line_payments column if added in 0006)
FOREIGN KEY (remittance_batch_id, remittance_id) REFERENCES remittances(batch_id, id) ON DELETE CASCADE,
FOREIGN KEY (batch_id, claim_id) REFERENCES claims(batch_id, id) ON DELETE CASCADE
);
INSERT INTO line_reconciliations_new
SELECT lr.id, lr.remittance_id, r.batch_id, lr.claim_id, c.batch_id,
lr.service_line_payment_id, lr.superseded_by_id, lr.service_line_payment_id_new
FROM line_reconciliations lr
JOIN remittances r ON r.id = lr.remittance_id
JOIN claims c ON c.id = lr.claim_id;
DROP TABLE line_reconciliations;
ALTER TABLE line_reconciliations_new RENAME TO line_reconciliations;
-- Step 4: add the back-references between claims and remittances now that
-- both tables exist with composite PKs.
-- (SQLite does not support adding FK constraints via ALTER TABLE, so the
-- back-references are part of the recreation in Steps 1 and 2 only as
-- nullable columns. Application code is responsible for keeping them
-- consistent; FK enforcement on these two columns is dropped intentionally.)
PRAGMA foreign_keys = ON;
PRAGMA defer_foreign_keys = OFF;
Important: before committing this migration, read 0006_line_reconciliation.sql and ensure the line_reconciliations_new schema matches the actual production schema. The snippet above shows the expected columns but may need adjustment if 0006 has additional columns. Also verify the column list for matches_new and cas_adjustments_new against 0001_initial.sql.
- Step 5: Update
backend/src/cyclone/db.pyto reflect composite PK
In db.py, the Claim.id and Remittance.id columns are mapped with primary_key=True. With composite PKs, the markers change:
Claim.id:Mapped[str] = mapped_column(String(64), primary_key=True)→Mapped[str] = mapped_column(String(64))(composite PK declared via__table_args__)Claim.batch_id:Mapped[str] = mapped_column(String(64), ForeignKey("batches.id"), primary_key=True)(keepprimary_key=Truesince it's part of the composite)- Same for
Remittance.
Add __table_args__ to each class to declare CompositePrimaryKey constraints:
from sqlalchemy import PrimaryKeyConstraint
class Claim(...):
__table_args__ = (PrimaryKeyConstraint("batch_id", "id"),)
# ...
SQLAlchemy 2.x syntax: use mapped_column(..., primary_key=True) for both columns and SQLAlchemy infers the composite. Or use __table_args__ = (PrimaryKeyConstraint(...),).
- Step 6: Run migration tests, expect PASS
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest tests/test_db_migrate.py -k "0014" --no-header -q
Expected: 3 passed.
- Step 7: Run the full
test_db_migrate.py, expect all PASS
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest tests/test_db_migrate.py --no-header -q
Expected: all pass. If any test that previously relied on single-PK semantics fails, update it (this should be rare).
- Step 8: Run the broader test suite to catch schema regressions
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest --no-header -q
Expected: all pass. Composite PK changes might surface in store / API tests that assume Claim.id is unique. Fix those tests by using (batch_id, id) lookups where needed. Do not modify production code to make old tests pass; update the tests.
- Step 9: Commit
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
git add backend/src/cyclone/migrations/0014_relax_claims_remits_pk.sql \
backend/src/cyclone/db.py \
backend/tests/test_db_migrate.py
git commit -m "feat(db): migration 0014 relaxes claims/remittances PK to (batch_id, id)"
Task 1.4: Tighten find_existing_batch_for_claim to return the most-recent batch
The current implementation returns some batch (no ORDER BY). The spec says we must return the most-recent. Update the helper.
Files:
-
Modify:
backend/src/cyclone/store.py(find_existing_batch_for_claim,find_existing_batch_for_remit) -
Modify:
backend/tests/test_store.py(add a "most recent" assertion) -
Step 1: Write the failing test
Add to backend/tests/test_store.py:
def test_find_existing_batch_for_claim_returns_most_recent():
"""When the same claim_id is in multiple prior batches, the helper
returns the most-recent (highest state_changed_at) batch."""
from datetime import datetime, timedelta, timezone
from cyclone import db as _db
from cyclone.db import Batch, Claim
base = datetime(2026, 1, 1, tzinfo=timezone.utc)
with _db.SessionLocal()() as s:
# Older batch with the claim
s.add(Batch(
id="B_OLD", kind="837p", input_filename="old.txt",
parsed_at=base, raw_result_json={},
))
s.add(Claim(
id="CLM-A", batch_id="B_OLD", patient_control_number="M",
state_changed_at=base,
))
# Newer batch with the same claim
s.add(Batch(
id="B_NEW", kind="837p", input_filename="new.txt",
parsed_at=base + timedelta(days=1), raw_result_json={},
))
s.add(Claim(
id="CLM-A", batch_id="B_NEW", patient_control_number="M",
state_changed_at=base + timedelta(days=1),
))
s.commit()
from cyclone import store as store_mod
assert store_mod.find_existing_batch_for_claim("CLM-A") == "B_NEW"
def test_find_existing_batch_for_remit_returns_most_recent():
"""Same shape for 835: most-recent (highest received_at) batch wins."""
from datetime import datetime, timedelta, timezone
from cyclone import db as _db
from cyclone.db import Batch, Remittance
base = datetime(2026, 1, 1, tzinfo=timezone.utc)
with _db.SessionLocal()() as s:
s.add(Batch(
id="B_OLD835", kind="835", input_filename="old.txt",
parsed_at=base, raw_result_json={},
))
s.add(Remittance(
id="CLP-A", batch_id="B_OLD835",
payer_claim_control_number="CLP-A",
status_code="1",
received_at=base,
))
s.add(Batch(
id="B_NEW835", kind="835", input_filename="new.txt",
parsed_at=base + timedelta(days=1), raw_result_json={},
))
s.add(Remittance(
id="CLP-A", batch_id="B_NEW835",
payer_claim_control_number="CLP-A",
status_code="1",
received_at=base + timedelta(days=1),
))
s.commit()
from cyclone import store as store_mod
assert store_mod.find_existing_batch_for_remit("CLP-A") == "B_NEW835"
- Step 2: Run the new tests, expect FAIL
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest \
tests/test_store.py::test_find_existing_batch_for_claim_returns_most_recent \
tests/test_store.py::test_find_existing_batch_for_remit_returns_most_recent \
--no-header -q
Expected: FAIL — the helper does not have ORDER BY ... DESC LIMIT 1 so it returns B_OLD (or B_OLD835) instead of the most-recent.
- Step 3: Update the helpers
In backend/src/cyclone/store.py, replace find_existing_batch_for_claim:
def find_existing_batch_for_claim(claim_id: str) -> str | None:
"""Return the batch_id of the most-recent batch containing this CLM01, or None.
"Most-recent" is by ``Claim.state_changed_at`` (latest touch of the claim
row, including resubmits). When the same CLM01 exists in N prior batches,
the newest one wins. Returns ``None`` if no batch contains the claim.
Pure read; opens a short-lived session. Used by the 837 409 handler to
surface which prior batch already holds the same CLM01.
"""
from sqlalchemy import select
from cyclone.db import Claim
with db.SessionLocal()() as s:
row = s.execute(
select(Claim.batch_id)
.where(Claim.id == claim_id)
.order_by(Claim.state_changed_at.desc())
.limit(1)
).first()
return row[0] if row else None
And replace find_existing_batch_for_remit:
def find_existing_batch_for_remit(remit_id: str) -> str | None:
"""Return the batch_id of the most-recent batch containing this CLP01, or None.
"Most-recent" is by ``Remittance.received_at``. ``remit_id`` is the PK on
``remittances.id`` (= payer_claim_control_number = CLP01).
"""
from sqlalchemy import select
from cyclone.db import Remittance
with db.SessionLocal()() as s:
row = s.execute(
select(Remittance.batch_id)
.where(Remittance.id == remit_id)
.order_by(Remittance.received_at.desc())
.limit(1)
).first()
return row[0] if row else None
- Step 4: Run all
test_store.pytests, expect PASS
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest tests/test_store.py --no-header -q
Expected: 12 passed (10 existing + 2 new).
- Step 5: Commit
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
git add backend/src/cyclone/store.py backend/tests/test_store.py
git commit -m "feat(store): find_existing_batch_for_claim/remit return most-recent"
Phase 2 — Backend pre-flight dedup module + endpoint changes
Task 2.1: New dedup.py module with CollisionReport and pre-flight helpers
Files:
-
Create:
backend/src/cyclone/dedup.py -
Create:
backend/tests/test_dedup.py -
Step 1: Write the failing tests
Create backend/tests/test_dedup.py:
"""Tests for the pre-flight dedup helpers in cyclone.dedup.
The dedup module is the source of truth for whether a parsed batch would
collide with existing data. It returns a ``CollisionReport`` that the API
serializes into the 409 body.
"""
from __future__ import annotations
from datetime import date, datetime, timezone
from decimal import Decimal
import pytest
from cyclone import db
from cyclone.dedup import (
CollisionReport,
preflight_837,
preflight_835,
)
from cyclone.parsers.models import (
Address,
BillingProvider,
ClaimHeader,
ClaimOutput,
Envelope,
Payer,
ParseResult,
Subscriber,
ValidationReport,
)
from cyclone.parsers.models_835 import (
ClaimPayment,
ParseResult835,
)
@pytest.fixture(autouse=True)
def _setup(tmp_path, monkeypatch):
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
db._reset_for_tests()
db.init_db()
yield
db._reset_for_tests()
def _claim(claim_id: str, member_id: str = "M1") -> ClaimOutput:
return ClaimOutput(
claim_id=claim_id,
control_number="0001",
transaction_date=date(2026, 6, 19),
billing_provider=BillingProvider(name="Test", npi="1234567890"),
subscriber=Subscriber(
first_name="X", last_name="Y", member_id=member_id, dob="1970-01-01", gender="M",
),
payer=Payer(name="Test", id="TST"),
claim=ClaimHeader(claim_id=claim_id, total_charge=Decimal("100")),
diagnoses=[],
service_lines=[],
validation=ValidationReport(passed=True, errors=[], warnings=[]),
raw_segments=[],
)
def _result(*claims: ClaimOutput) -> ParseResult:
return ParseResult(
envelope=Envelope(
sender_id="S", receiver_id="R", control_number="1",
transaction_date=date(2026, 6, 19), transaction_time="1200",
implementation_guide="005010X222A1", transaction_type_code="CH",
),
claims=list(claims),
summary={
"input_file": "x.txt", "control_number": "1",
"transaction_date": "2026-06-19", "total_claims": len(claims),
"passed": len(claims), "failed": 0, "failed_claim_ids": [],
"issues_by_rule": {}, "output_dir": None,
},
)
def test_preflight_837_no_collisions_on_empty_db():
"""Empty DB → no collisions, no existing_batch_id."""
r = _result(_claim("A"), _claim("B"))
report = preflight_837(r)
assert report.colliding_claim_ids == []
assert report.existing_batch_id is None
assert report.within_file_duplicate_ids == []
assert report.total_claims == 2
def test_preflight_837_finds_cross_batch_collision():
"""A claim already in the DB surfaces as a collision with the prior batch."""
from cyclone.db import Batch, Claim
with db.SessionLocal()() as s:
s.add(Batch(
id="PRIOR", kind="837p", input_filename="prior.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Claim(
id="A", batch_id="PRIOR", patient_control_number="M",
state_changed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
))
s.commit()
r = _result(_claim("A"), _claim("B")) # A collides, B is new
report = preflight_837(r)
assert report.colliding_claim_ids == ["A"]
assert report.existing_batch_id == "PRIOR"
assert report.within_file_duplicate_ids == []
assert report.total_claims == 2
def test_preflight_837_finds_within_file_duplicate():
"""Same CLM01 twice in the file → within_file_duplicate_ids and no batch_id."""
r = _result(_claim("A"), _claim("A"), _claim("B"))
report = preflight_837(r)
assert report.within_file_duplicate_ids == ["A"]
assert report.colliding_claim_ids == ["A"] # also flagged as a collision
assert report.existing_batch_id is None
assert report.total_claims == 3
def test_preflight_837_returns_most_recent_batch_id():
"""When CLM01=A is in 3 prior batches, return the most-recent one."""
from datetime import timedelta
from cyclone.db import Batch, Claim
base = datetime(2026, 1, 1, tzinfo=timezone.utc)
for i, bid in enumerate(["B1", "B2", "B3"]):
with db.SessionLocal()() as s:
s.add(Batch(
id=bid, kind="837p", input_filename=f"{bid}.txt",
parsed_at=base, raw_result_json={},
))
s.add(Claim(
id="A", batch_id=bid, patient_control_number="M",
state_changed_at=base + timedelta(days=i),
))
s.commit()
r = _result(_claim("A"))
report = preflight_837(r)
assert report.colliding_claim_ids == ["A"]
assert report.existing_batch_id == "B3" # most-recent by state_changed_at
def test_preflight_835_mirrors_837():
"""Same shape for 835: payer_claim_control_number is the dedup key."""
from cyclone.db import Batch, Remittance
with db.SessionLocal()() as s:
s.add(Batch(
id="PRIOR835", kind="835", input_filename="p.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Remittance(
id="CLP-A", batch_id="PRIOR835",
payer_claim_control_number="CLP-A",
status_code="1",
received_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
))
s.commit()
r835 = ParseResult835(
envelope=Envelope(
sender_id="S", receiver_id="R", control_number="1",
transaction_date=date(2026, 6, 19), transaction_time="1200",
implementation_guide="005010X221A1", transaction_type_code="CH",
),
payer=Payer(name="Test", id="TST"),
claims=[
ClaimPayment(
payer_claim_control_number="CLP-A",
status_code="1",
total_charge=Decimal("100"), total_paid=Decimal("100"),
),
ClaimPayment(
payer_claim_control_number="CLP-B",
status_code="1",
total_charge=Decimal("50"), total_paid=Decimal("50"),
),
],
)
report = preflight_835(r835)
assert report.colliding_claim_ids == ["CLP-A"]
assert report.existing_batch_id == "PRIOR835"
assert report.within_file_duplicate_ids == []
assert report.total_claims == 2
- Step 2: Run tests, expect FAIL with "No module named 'cyclone.dedup'"
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest tests/test_dedup.py --no-header -q
Expected: ModuleNotFoundError: No module named 'cyclone.dedup'.
- Step 3: Implement
dedup.py
Create backend/src/cyclone/dedup.py:
"""Pre-flight dedup for parsed 837P/835 batches.
Splits the parsed result into "would-insert" and "would-skip" sets by
querying the DB for any ``claim_id`` (837) or ``payer_claim_control_number``
(835) already present. Also detects within-file duplicates by counting
``claim_id`` / ``payer_claim_control_number`` frequencies.
Used by the ``parse-837`` and ``parse-835`` endpoints between validation
and persist, so the user sees the parse result + collision summary
before any DB write.
"""
from __future__ import annotations
from collections import Counter
from dataclasses import dataclass, field
from typing import TYPE_CHECKING
from sqlalchemy import select
from sqlalchemy.orm import Session
from cyclone import db
if TYPE_CHECKING:
from cyclone.parsers.models import ParseResult
from cyclone.parsers.models_835 import ParseResult835
@dataclass(frozen=True)
class CollisionReport:
"""What the parse endpoint needs to render a 409 body.
``within_file_duplicate_ids`` is a subset of ``colliding_claim_ids``:
the same CLM01 appearing twice in the file is also flagged as a
collision (because the second instance cannot be inserted).
``existing_batch_id`` is the most-recent prior batch that contains
one of the colliding claim_ids, or None when the collision is purely
within-file.
"""
colliding_claim_ids: list[str] = field(default_factory=list)
existing_batch_id: str | None = None
within_file_duplicate_ids: list[str] = field(default_factory=list)
total_claims: int = 0
@property
def has_collisions(self) -> bool:
return bool(self.colliding_claim_ids or self.within_file_duplicate_ids)
@property
def new_claims_after_skip(self) -> int:
"""How many claims WOULD be inserted on a force-insert."""
# A within-file duplicate still occupies one of the ``total_claims`` slots;
# the force-insert keeps the first occurrence and skips the rest, so
# ``new_claims_after_skip`` = total_claims - within_file_dup_count.
return self.total_claims - len(self.within_file_duplicate_ids)
def preflight_837(result: "ParseResult", session: Session | None = None) -> CollisionReport:
"""Detect 837 collisions: within-file dupes + cross-batch CLM01 dupes."""
claim_ids = [c.claim_id for c in result.claims]
counts = Counter(claim_ids)
within_file_duplicate_ids = sorted(
cid for cid, n in counts.items() if n > 1
)
seen: set[str] = set(claim_ids)
if not seen:
return CollisionReport(
colliding_claim_ids=[],
existing_batch_id=None,
within_file_duplicate_ids=[],
total_claims=0,
)
own_session = session is None
if own_session:
session = db.SessionLocal()()
try:
from cyclone.db import Claim
rows = session.execute(
select(Claim.id, Claim.batch_id)
.where(Claim.id.in_(seen))
.order_by(Claim.state_changed_at.desc())
).all()
finally:
if own_session:
session.close()
db_collisions: dict[str, str] = {cid: bid for cid, bid in rows}
# A claim_id "collides" if it's in the DB (cross-batch) OR appears more than
# once in this file (within-file). The UI uses colliding_claim_ids to count
# the problem set; within_file_duplicate_ids is the subset that is purely
# within-file. existing_batch_id stays None when every collision is within-file.
colliding = sorted(
cid for cid, n in counts.items()
if cid in db_collisions or n > 1
)
existing_batch_id = next(iter(db_collisions.values()), None) if db_collisions else None
return CollisionReport(
colliding_claim_ids=colliding,
existing_batch_id=existing_batch_id,
within_file_duplicate_ids=within_file_duplicate_ids,
total_claims=len(claim_ids),
)
def preflight_835(result: "ParseResult835", session: Session | None = None) -> CollisionReport:
"""Same shape for 835 remittances. Payer claim control number = CLP01 = remittance.id."""
pcns = [c.payer_claim_control_number for c in result.claims]
counts = Counter(pcns)
within_file_duplicate_ids = sorted(p for p, n in counts.items() if n > 1)
seen: set[str] = set(pcns)
if not seen:
return CollisionReport(
colliding_claim_ids=[],
existing_batch_id=None,
within_file_duplicate_ids=[],
total_claims=0,
)
own_session = session is None
if own_session:
session = db.SessionLocal()()
try:
from cyclone.db import Remittance
rows = session.execute(
select(Remittance.id, Remittance.batch_id)
.where(Remittance.id.in_(seen))
.order_by(Remittance.received_at.desc())
).all()
finally:
if own_session:
session.close()
db_collisions: dict[str, str] = {pcn: bid for pcn, bid in rows}
# Same rule as 837: within-file dupes are also collisions.
colliding = sorted(
pcn for pcn, n in counts.items()
if pcn in db_collisions or n > 1
)
existing_batch_id = next(iter(db_collisions.values()), None) if db_collisions else None
return CollisionReport(
colliding_claim_ids=colliding,
existing_batch_id=existing_batch_id,
within_file_duplicate_ids=within_file_duplicate_ids,
total_claims=len(pcns),
)
- Step 4: Run tests, expect PASS
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest tests/test_dedup.py --no-header -q
Expected: 5 passed.
- Step 5: Commit
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
git add backend/src/cyclone/dedup.py backend/tests/test_dedup.py
git commit -m "feat(dedup): preflight_837 / preflight_835 with CollisionReport"
Task 2.2: Store helper delete_batch with cascade + 409 guard
The DELETE /api/batches/{id} endpoint (Task 2.4) calls a store method
that hard-deletes a batch and cascades through all child rows. The
method refuses if any claim is past submitted state.
Files:
-
Modify:
backend/src/cyclone/store.py(adddelete_batchmethod onCycloneStore) -
Modify:
backend/tests/test_store.py(add tests) -
Step 1: Write the failing tests
Add to backend/tests/test_store.py:
def test_delete_batch_cascades_to_claims():
"""delete_batch removes the batch row and all its claim rows."""
from datetime import datetime, timezone
from cyclone import db as _db
from cyclone.db import Batch, Claim
with _db.SessionLocal()() as s:
s.add(Batch(
id="B1", kind="837p", input_filename="x.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Claim(id="CLM-1", batch_id="B1", patient_control_number="M1"))
s.add(Claim(id="CLM-2", batch_id="B1", patient_control_number="M2"))
s.commit()
s = CycloneStore()
result = s.delete_batch("B1")
assert result is True
with _db.SessionLocal()() as sess:
assert sess.get(_db.Batch, "B1") is None
rows = sess.execute(_db.text("SELECT id FROM claims WHERE batch_id='B1'")).all()
assert rows == []
def test_delete_batch_returns_false_for_unknown():
"""Unknown batch_id -> returns False (not raises)."""
s = CycloneStore()
assert s.delete_batch("does-not-exist") is False
def test_delete_batch_raises_on_reconciled_claims():
"""Refuses to delete if any claim is past 'submitted' state."""
from datetime import datetime, timezone
from cyclone import db as _db
from cyclone.db import Batch, Claim
with _db.SessionLocal()() as s:
s.add(Batch(
id="B2", kind="837p", input_filename="x.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Claim(
id="CLM-X", batch_id="B2", patient_control_number="M",
state="paid", # reconciled
))
s.commit()
s = CycloneStore()
with pytest.raises(_InvalidStateError): # reuse the existing 409 error type
s.delete_batch("B2")
Wait — InvalidStateError is the wrong type for this case (it's for
apply_* skips, not for delete-blocked). We need a new exception
class. Update the test to use a new BatchNotDeletableError:
Adjust the test imports at the top of the file:
from cyclone.store import (
AlreadyMatchedError,
BatchNotDeletableError,
CycloneStore,
store,
)
And add BatchNotDeletableError to the import. Then update the third
test:
def test_delete_batch_raises_batch_not_deletable_on_reconciled_claims():
"""Refuses to delete if any claim is past 'submitted' state."""
from datetime import datetime, timezone
from cyclone import db as _db
from cyclone.db import Batch, Claim
with _db.SessionLocal()() as s:
s.add(Batch(
id="B2", kind="837p", input_filename="x.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Claim(
id="CLM-X", batch_id="B2", patient_control_number="M",
state="paid", # reconciled
))
s.commit()
s = CycloneStore()
with pytest.raises(BatchNotDeletableError) as excinfo:
s.delete_batch("B2")
assert "non-submitted" in str(excinfo.value).lower()
assert "B2" in str(excinfo.value)
# The batch and its claim are still in the DB.
with _db.SessionLocal()() as sess:
assert sess.get(_db.Batch, "B2") is not None
rows = sess.execute(_db.text("SELECT id FROM claims WHERE batch_id='B2'")).all()
assert len(rows) == 1
- Step 2: Run tests, expect FAIL with "no attribute 'delete_batch'" and "no name 'BatchNotDeletableError'"
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest tests/test_store.py -k "delete_batch" --no-header -q
Expected: 3 failures.
- Step 3: Implement
BatchNotDeletableErroranddelete_batch
In backend/src/cyclone/store.py, add the exception class near the
other exception classes (right after InvalidStateError):
class BatchNotDeletableError(Exception):
"""Raised by ``CycloneStore.delete_batch`` when the batch has claims in
a non-``submitted`` state (i.e., it has been reconciled or otherwise
progressed past raw ingestion).
The T15 API endpoint maps this to a 409 Conflict; the user must first
unreconcile the affected claims before deleting the batch.
"""
def __init__(self, batch_id: str, offending_claim_id: str):
self.batch_id = batch_id
self.offending_claim_id = offending_claim_id
super().__init__(
f"Batch {batch_id} cannot be deleted: claim {offending_claim_id} "
f"is in a non-submitted state."
)
Find the CycloneStore class (search for class CycloneStore). Add
the delete_batch method as the last method in the class. Use this
code (the method goes inside class CycloneStore):
def delete_batch(self, batch_id: str) -> bool:
"""Hard-delete a batch and cascade through child rows.
Returns ``True`` if the batch was found and deleted, ``False`` if
no batch with that id exists. Raises :class:`BatchNotDeletableError`
if any claim in the batch is past the ``submitted`` state (i.e.,
has been reconciled, reversed, denied, or otherwise progressed);
the user must unreconcile those claims first.
The cascade relies on the ``ON DELETE CASCADE`` FKs declared in
the schema migrations: ``claims.batch_id``, ``remittances.batch_id``,
``activity_events.batch_id`` (when present), ``matches.claim_id``
(via claims), ``line_reconciliations.claim_id`` (via claims), and
the remittance-side children. The store explicitly records a
``batch_deleted`` activity event before the cascade so the audit
log retains a tombstone of the deletion.
The 409 guard runs in a single short-lived session; the
``batch_deleted`` event and the cascade are committed atomically.
"""
from datetime import datetime as _dt, timezone as _tz
with db.SessionLocal()() as s:
batch_row = s.get(Batch, batch_id)
if batch_row is None:
return False
# 409 guard: refuse if any claim has progressed past 'submitted'.
offending = s.execute(
select(Claim.id)
.where(Claim.batch_id == batch_id)
.where(Claim.state != "submitted")
.limit(1)
).first()
if offending is not None:
raise BatchNotDeletableError(batch_id, offending[0])
# Tombstone activity event for the audit log.
s.add(ActivityEvent(
ts=_dt.now(_tz.utc),
kind="batch_deleted",
batch_id=batch_id,
payload_json={
"message": f"Batch {batch_id} deleted",
"kind": batch_row.kind,
"input_filename": batch_row.input_filename,
},
))
s.flush()
s.delete(batch_row)
s.commit()
return True
Add the delete_batch import near the top of the file (right after
AlreadyMatchedError):
class BatchNotDeletableError(Exception):
...
(The exception class goes in the module body, not inside
CycloneStore. The class definition is unchanged from what was added
in Step 3 above.)
The new select import in the method body is already available because
store.py imports from sqlalchemy import ... later in the file
(grep to confirm). If not, add from sqlalchemy import select to the
top of the file.
- Step 4: Run the new tests, expect PASS
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest tests/test_store.py -k "delete_batch" --no-header -q
Expected: 3 passed.
- Step 5: Run the full test_store.py, expect all PASS
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest tests/test_store.py --no-header -q
Expected: 15 passed (12 prior + 3 new).
- Step 6: Commit
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
git add backend/src/cyclone/store.py backend/tests/test_store.py
git commit -m "feat(store): delete_batch hard-deletes with cascade + non-submitted guard"
Task 2.3: 837 endpoint pre-flight, ?force=true, race handler, 200 skipped_claim_ids
This is the biggest task. The 837 endpoint needs:
?force: bool = Query(False)parameter- Pre-flight dedup between validation and persist (when not force)
- New 409 body shape: full parse result + collisions + existing_batch_id
- Race-condition 409 (same shape) when IntegrityError fires after a clean pre-flight
- 200 body adds
skipped_claim_idswhen force=true (so the UI can show what was skipped)
The 409 body builder is shared between pre-flight and race, so it lives as a module-level helper.
Files:
-
Modify:
backend/src/cyclone/api.py -
Modify:
backend/tests/test_api_parse_persists.py -
Step 1: Write the failing tests
Add to backend/tests/test_api_parse_persists.py:
def test_parse_837_409_includes_parse_result_and_collisions(client: TestClient) -> None:
"""Pre-flight dedup returns 409 with the full parse result + collisions."""
from datetime import datetime, timezone
from cyclone import db as _db
from cyclone.db import Batch, Claim
# Pre-seed a claim in a prior batch.
with _db.SessionLocal()() as s:
s.add(Batch(
id="PRIOR", kind="837p", input_filename="prior.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Claim(
id="CLM-A", batch_id="PRIOR", patient_control_number="M",
state_changed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
))
s.commit()
# Build a small 837 file with one CLM* that collides.
text = (
"ISA*00* *00* *ZZ*SUBMITTERID *ZZ*RECEIVERID "
"*240101*1200*^*00501*000000001*0*P*:~\n"
"GS*HC*SUBMITTERID*RECEIVERID*20240101*1200*1*X*005010X222A1~\n"
"ST*837*0001*005010X222A1~\n"
"BHT*0019*00*1*20240101*1200*CH~\n"
"NM1*41*2*SUBMITTER*****46*SUBMITTERID~\n"
"PER*IC*CONTACT*TE*5555555555~\n"
"NM1*40*2*RECEIVER*****46*RECEIVERID~\n"
"HL*1**20*1~\n"
"NM1*85*2*BILLING*****XX*1881068062~\n"
"N3*123 MAIN*~~\nN4*DENVER*CO*80202~\n"
"REF*EI*123456789~\n"
"HL*2*1*22*0~\n"
"SBR*P*18*******CI~\n"
"NM1*IL*1*DOE*JOHN****MI*M~\n"
"N3*456 ELM*~~\nN4*DENVER*CO*80202~\n"
"DMG*D8*19700101*M~\n"
"NM1*PR*2*MEDICAID*****PI*SKCO0~\n"
"CLM*CLM-A*100***11:B:1*Y*A*Y*Y~\n"
"HI*ABK:R69~\n"
"LX*1~\nSV1*HC:99213*100*UN*1***1~\n"
"DTP*472*D8*20240101~\n"
"SE*24*0001~\nGE*1*1~\nIEA*1*000000001~\n"
)
resp = client.post(
"/api/parse-837",
files={"file": ("dup.txt", text, "text/plain")},
headers={"Accept": "application/json"},
)
assert resp.status_code == 409, resp.text
body = resp.json()
assert body["error"] == "Duplicate claim"
assert body["existing_batch_id"] == "PRIOR"
assert body["batch_id"] is None
assert body["collisions"]["colliding_claim_ids"] == ["CLM-A"]
assert body["collisions"]["total_claims"] == 1
assert body["collisions"]["total_collisions"] == 1
assert body["collisions"]["new_claims_after_skip"] == 0
assert "parse_result" in body
assert body["parse_result"]["claims"][0]["claim_id"] == "CLM-A"
# No batch was actually persisted.
with _db.SessionLocal()() as s:
n = s.execute(_db.text("SELECT COUNT(*) FROM batches WHERE id != 'PRIOR'")).scalar()
assert n == 0
def test_parse_837_within_file_duplicate_returns_409_with_null_batch_id(
client: TestClient,
) -> None:
"""A file with the same CLM01 twice returns 409 with existing_batch_id=None."""
text = (
"ISA*00* *00* *ZZ*SUBMITTERID *ZZ*RECEIVERID "
"*240101*1200*^*00501*000000001*0*P*:~\n"
"GS*HC*SUBMITTERID*RECEIVERID*20240101*1200*1*X*005010X222A1~\n"
"ST*837*0001*005010X222A1~\n"
"BHT*0019*00*1*20240101*1200*CH~\n"
"NM1*41*2*SUBMITTER*****46*SUBMITTERID~\n"
"PER*IC*CONTACT*TE*5555555555~\n"
"NM1*40*2*RECEIVER*****46*RECEIVERID~\n"
"HL*1**20*1~\n"
"NM1*85*2*BILLING*****XX*1881068062~\n"
"N3*123 MAIN*~~\nN4*DENVER*CO*80202~\n"
"REF*EI*123456789~\n"
"HL*2*1*22*0~\n"
"SBR*P*18*******CI~\n"
"NM1*IL*1*DOE*JOHN****MI*M~\n"
"N3*456 ELM*~~\nN4*DENVER*CO*80202~\n"
"DMG*D8*19700101*M~\n"
"NM1*PR*2*MEDICAID*****PI*SKCO0~\n"
"CLM*CLM-DUP*100***11:B:1*Y*A*Y*Y~\n"
"HI*ABK:R69~\n"
"LX*1~\nSV1*HC:99213*100*UN*1***1~\n"
"DTP*472*D8*20240101~\n"
"CLM*CLM-DUP*100***11:B:1*Y*A*Y*Y~\n"
"HI*ABK:R69~\n"
"LX*1~\nSV1*HC:99213*100*UN*1***1~\n"
"DTP*472*D8*20240101~\n"
"SE*28*0001~\nGE*1*1~\nIEA*1*000000001~\n"
)
resp = client.post(
"/api/parse-837",
files={"file": ("within.txt", text, "text/plain")},
headers={"Accept": "application/json"},
)
assert resp.status_code == 409, resp.text
body = resp.json()
assert body["existing_batch_id"] is None
assert "CLM-DUP" in body["collisions"]["colliding_claim_ids"]
assert "CLM-DUP" in body["collisions"].get("within_file_duplicate_ids", body["collisions"]["colliding_claim_ids"])
def test_parse_837_force_true_persists_non_colliding_claims(client: TestClient) -> None:
"""With force=true, the collision is skipped and other claims persist."""
from datetime import datetime, timezone
from cyclone import db as _db
from cyclone.db import Batch, Claim
with _db.SessionLocal()() as s:
s.add(Batch(
id="PRIOR", kind="837p", input_filename="p.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Claim(
id="CLM-OLD", batch_id="PRIOR", patient_control_number="M",
state_changed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
))
s.commit()
# File with 2 claims: one colliding, one new.
text = (
"ISA*00* *00* *ZZ*SUBMITTERID *ZZ*RECEIVERID "
"*240101*1200*^*00501*000000001*0*P*:~\n"
"GS*HC*SUBMITTERID*RECEIVERID*20240101*1200*1*X*005010X222A1~\n"
"ST*837*0001*005010X222A1~\n"
"BHT*0019*00*1*20240101*1200*CH~\n"
"NM1*41*2*SUBMITTER*****46*SUBMITTERID~\n"
"PER*IC*CONTACT*TE*5555555555~\n"
"NM1*40*2*RECEIVER*****46*RECEIVERID~\n"
"HL*1**20*1~\n"
"NM1*85*2*BILLING*****XX*1881068062~\n"
"N3*123 MAIN*~~\nN4*DENVER*CO*80202~\n"
"REF*EI*123456789~\n"
"HL*2*1*22*0~\n"
"SBR*P*18*******CI~\n"
"NM1*IL*1*DOE*JOHN****MI*M~\n"
"N3*456 ELM*~~\nN4*DENVER*CO*80202~\n"
"DMG*D8*19700101*M~\n"
"NM1*PR*2*MEDICAID*****PI*SKCO0~\n"
"CLM*CLM-OLD*100***11:B:1*Y*A*Y*Y~\n"
"HI*ABK:R69~\n"
"LX*1~\nSV1*HC:99213*100*UN*1***1~\n"
"DTP*472*D8*20240101~\n"
"CLM*CLM-NEW*100***11:B:1*Y*A*Y*Y~\n"
"HI*ABK:R69~\n"
"LX*1~\nSV1*HC:99213*100*UN*1***1~\n"
"DTP*472*D8*20240101~\n"
"SE*30*0001~\nGE*1*1~\nIEA*1*000000001~\n"
)
resp = client.post(
"/api/parse-837?force=true",
files={"file": ("force.txt", text, "text/plain")},
headers={"Accept": "application/json"},
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body.get("skipped_claim_ids") == ["CLM-OLD"]
# The new claim persisted; the old one did not.
with _db.SessionLocal()() as s:
rows = s.execute(_db.text("SELECT id, batch_id FROM claims WHERE id IN ('CLM-OLD','CLM-NEW') ORDER BY id")).all()
by_id = {r[0]: r[1] for r in rows}
assert by_id["CLM-OLD"] == "PRIOR" # not re-inserted
assert by_id["CLM-NEW"] != "PRIOR" # new batch
Also add the DELETE endpoint test (this goes in test_api_parse_persists.py
because there's no dedicated test file for it yet — store.py already
has the unit tests in Task 2.2):
def test_delete_batch_endpoint_cascades(client: TestClient) -> None:
"""DELETE /api/batches/{id} removes the batch and its claims."""
from datetime import datetime, timezone
from cyclone import db as _db
from cyclone.db import Batch, Claim
with _db.SessionLocal()() as s:
s.add(Batch(
id="B1", kind="837p", input_filename="x.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Claim(id="CLM-1", batch_id="B1", patient_control_number="M1"))
s.commit()
resp = client.delete("/api/batches/B1")
assert resp.status_code in (200, 204), resp.text
with _db.SessionLocal()() as s:
assert s.get(_db.Batch, "B1") is None
n = s.execute(_db.text("SELECT COUNT(*) FROM claims WHERE batch_id='B1'")).scalar()
assert n == 0
def test_delete_batch_endpoint_404_on_unknown(client: TestClient) -> None:
resp = client.delete("/api/batches/does-not-exist")
assert resp.status_code == 404
def test_delete_batch_endpoint_409_on_reconciled_claim(client: TestClient) -> None:
from datetime import datetime, timezone
from cyclone import db as _db
from cyclone.db import Batch, Claim
with _db.SessionLocal()() as s:
s.add(Batch(
id="B2", kind="837p", input_filename="x.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Claim(
id="CLM-X", batch_id="B2", patient_control_number="M",
state="paid",
))
s.commit()
resp = client.delete("/api/batches/B2")
assert resp.status_code == 409
with _db.SessionLocal()() as s:
assert s.get(_db.Batch, "B2") is not None
- Step 2: Run the new tests, expect FAIL
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest \
tests/test_api_parse_persists.py -k "409 or force or delete_batch_endpoint" \
--no-header -q
Expected: 6 failures (4 for the new 409/force cases; the existing
test_409_response_includes_existing_batch_id_for_837 from the prior
plan also fails because the body shape changed; 2 for DELETE not yet
existing).
- Step 3: Implement the new 837 endpoint body and DELETE endpoint
In backend/src/cyclone/api.py:
- Add the
forceparameter to theparse_837signature:
Replace the existing signature:
@app.post("/api/parse-837")
async def parse_837(
request: Request,
file: UploadFile = File(...),
payer: str = Query("co_medicaid"),
include_raw_segments: bool = Query(True),
strict: bool = Query(False),
ack: bool = Query(False),
) -> Any:
with:
@app.post("/api/parse-837")
async def parse_837(
request: Request,
file: UploadFile = File(...),
payer: str = Query("co_medicaid"),
include_raw_segments: bool = Query(True),
strict: bool = Query(False),
ack: bool = Query(False),
force: bool = Query(False),
) -> Any:
- Add the 409 builder and modify the 837 path:
Find the existing try: store.add(...) except IntegrityError as exc:
block in parse_837 (around line 392-415). Replace the entire block
(after the if _has_claim_validation_errors check) with:
# Pre-flight dedup: detect within-file dupes and cross-batch CLM01 collisions
# BEFORE persisting. The user sees the parse result + collision summary in
# the 409 body and can choose force-insert / delete prior / pick a different
# file. force=true bypasses the check; the store.add dedup still silently
# skips colliding rows, and the 200 body includes skipped_claim_ids.
if not force and result.claims:
from cyclone import dedup
report = dedup.preflight_837(result)
if report.has_collisions:
return _build_dedup_409(
result=result,
report=report,
kind="cross_batch" if report.existing_batch_id else "within_file",
error="Duplicate claim",
)
rec = BatchRecord(
id=uuid.uuid4().hex,
kind="837p",
input_filename=file.filename or "upload.txt",
parsed_at=utcnow(),
result=result,
)
try:
store.add(rec, event_bus=request.app.state.event_bus)
except IntegrityError as exc:
# Race: pre-flight said clean, but a concurrent process ingested a
# colliding batch between the check and the persist. Re-run pre-flight
# to build the same 409 body shape.
from cyclone import dedup
report = dedup.preflight_837(result)
log.warning("Race on persist for batch %s: %s", rec.id, exc)
return _build_dedup_409(
result=result,
report=report,
kind="race",
error="Duplicate claim (race condition)",
)
if _client_wants_json(request):
body = json.loads(result.model_dump_json())
if ack:
ack_body = _build_and_persist_ack(rec.id)
if ack_body is not None:
body["ack"] = ack_body
if force:
# The store.add dedup silently skipped any colliding claim_id;
# figure out which were skipped by diffing the parsed result against
# what actually persisted.
from cyclone import dedup
body["skipped_claim_ids"] = _compute_skipped_837(rec.id, result.claims)
return JSONResponse(content=body)
return StreamingResponse(
_ndjson_stream(result),
media_type="application/x-ndjson",
)
- Add the helpers
_build_dedup_409and_compute_skipped_837as module-level functions (right above theparse_837endpoint definition):
def _build_dedup_409(
result,
report,
*,
kind: str,
error: str,
) -> JSONResponse:
"""Build the standard 409 body for any dedup failure.
Shared between parse_837 and parse_835; the only difference is the
error string (we pass it in) and the kind (cross_batch / within_file /
race) which drives the human-readable detail message.
"""
if kind == "within_file":
detail = (
f"CLM01(s) {', '.join(report.within_file_duplicate_ids)} appear "
f"twice in this file. Force-insert will keep the first occurrence "
f"and skip the rest."
)
elif kind == "race":
detail = (
"Another process ingested a colliding batch between the check "
"and the persist. Re-upload to retry with the latest state."
)
else: # cross_batch
detail = (
f"{len(report.colliding_claim_ids)} of {report.total_claims} "
f"claims collide with prior batch {report.existing_batch_id}. "
f"Force-insert to skip the duplicates, or delete the prior batch."
)
body = {
"error": error,
"detail": detail,
"batch_id": None,
"existing_batch_id": report.existing_batch_id,
"collisions": {
"colliding_claim_ids": report.colliding_claim_ids,
"total_collisions": len(report.colliding_claim_ids),
"total_claims": report.total_claims,
"new_claims_after_skip": report.new_claims_after_skip,
"within_file_duplicate_ids": report.within_file_duplicate_ids,
},
"parse_result": json.loads(result.model_dump_json()),
}
return JSONResponse(status_code=409, content=body)
def _compute_skipped_837(batch_id: str, parsed_claims) -> list[str]:
"""Compare the parsed claims against what's in the DB for this batch.
Returns the claim_ids in the parsed result that did NOT make it into
the DB — these are the ones silently skipped by the store.add dedup.
Sorted for determinism.
"""
from cyclone import db as _db
from cyclone.db import Claim
parsed_ids = [c.claim_id for c in parsed_claims]
if not parsed_ids:
return []
with _db.SessionLocal()() as s:
persisted = s.execute(
select(Claim.id).where(Claim.batch_id == batch_id)
).all()
persisted_ids = {r[0] for r in persisted}
return sorted(set(parsed_ids) - persisted_ids)
The _compute_skipped_837 function needs from sqlalchemy import select
at the top of api.py (the file already imports it for other code; if
not, add it).
-
Remove the old 409 handler comment in the
except IntegrityErrorblock — it's been replaced by the race handler above. -
Add the DELETE endpoint (right after the 835 endpoint, before
/api/parse-999):
@app.delete("/api/batches/{batch_id}")
def delete_batch_endpoint(batch_id: str) -> dict:
"""Hard-delete a batch and cascade through child rows.
Returns 200 on success, 404 if the batch doesn't exist, 409 if any
claim in the batch is past the ``submitted`` state (must unreconcile
first). See ``CycloneStore.delete_batch`` for the cascade mechanics.
"""
try:
deleted = store.delete_batch(batch_id)
except BatchNotDeletableError as exc:
from fastapi.responses import JSONResponse as _JR
return _JR(
status_code=409,
content={
"error": "Batch not deletable",
"detail": str(exc),
"batch_id": exc.batch_id,
"offending_claim_id": exc.offending_claim_id,
},
)
if not deleted:
from fastapi import HTTPException
raise HTTPException(status_code=404, detail=f"Batch {batch_id} not found")
return {"ok": True, "batch_id": batch_id}
- Add the
BatchNotDeletableErrorimport at the top ofapi.py(find the existing imports fromcyclone.store):
from cyclone.store import (
BatchNotDeletableError, # NEW
CycloneStore,
store,
)
- Step 4: Run the new tests, expect PASS
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest \
tests/test_api_parse_persists.py -k "409 or force or delete_batch_endpoint" \
--no-header -q
Expected: all 6 new + 1 pre-existing (test_409_response_includes_existing_batch_id_for_837)
should pass once the body shape matches. The pre-existing test
expects body.get("existing_batch_id") == "PRIOR" which still
holds under the new shape. PASS.
- Step 5: Run the full test_api_parse_persists.py, expect all PASS
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest tests/test_api_parse_persists.py --no-header -q
Expected: all tests pass.
- Step 6: Run the broader API test suite to catch regressions
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest tests/test_api.py tests/test_api_835.py tests/test_api_gets.py \
--no-header -q
Expected: all pass. If anything fails, it means an existing test
relied on the old 409 body shape or the absence of force — update
those tests to match the new contract.
- Step 7: Commit
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
git add backend/src/cyclone/api.py backend/tests/test_api_parse_persists.py
git commit -m "feat(api): 837 pre-flight dedup, force=true, race 409, DELETE /api/batches/{id}"
Task 2.4: 835 endpoint pre-flight, ?force=true, race handler
Mirror the 837 changes for the 835 endpoint.
Files:
-
Modify:
backend/src/cyclone/api.py(835 endpoint) -
Modify:
backend/tests/test_api_835.py(add 409/force tests) -
Step 1: Write the failing tests
Add to backend/tests/test_api_835.py (find a good insertion point
near other parse tests):
def test_parse_835_409_includes_parse_result_and_collisions(client: TestClient) -> None:
"""835 pre-flight dedup returns 409 with the full parse result + collisions."""
from datetime import datetime, timezone
from cyclone import db as _db
from cyclone.db import Batch, Remittance
with _db.SessionLocal()() as s:
s.add(Batch(
id="PRIOR835", kind="835", input_filename="p.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Remittance(
id="CLP-A", batch_id="PRIOR835",
payer_claim_control_number="CLP-A",
status_code="1",
received_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
))
s.commit()
# Use the co_medicaid 835 fixture (real EDI), then post-assert the 409.
from pathlib import Path
text = (Path(__file__).parent / "fixtures" / "co_medicaid_835.txt").read_text()
resp = client.post(
"/api/parse-835",
files={"file": ("dup.835", text, "text/plain")},
headers={"Accept": "application/json"},
)
# The fixture's first CLP01 collides with the seeded one. We expect 409.
assert resp.status_code == 409, resp.text
body = resp.json()
assert body["error"] == "Duplicate remittance"
assert body["existing_batch_id"] == "PRIOR835"
assert body["batch_id"] is None
assert len(body["collisions"]["colliding_claim_ids"]) >= 1
assert "parse_result" in body
def test_parse_835_force_true_persists_non_colliding_claims(client: TestClient) -> None:
"""With force=true, the colliding remittance is skipped, others persist."""
from datetime import datetime, timezone
from pathlib import Path
from cyclone import db as _db
from cyclone.db import Batch, Remittance
with _db.SessionLocal()() as s:
s.add(Batch(
id="PRIOR835", kind="835", input_filename="p.txt",
parsed_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
raw_result_json={},
))
s.add(Remittance(
id="CLP-OLD", batch_id="PRIOR835",
payer_claim_control_number="CLP-OLD",
status_code="1",
received_at=datetime(2026, 1, 1, tzinfo=timezone.utc),
))
s.commit()
# Build a small 835 with two CLP segments: one colliding, one new.
text = (
"ISA*00* *00* *ZZ*PAYERID *ZZ*RECEIVERID "
"*240101*1200*^*00501*000000001*0*P*:~\n"
"GS*HP*PAYERID*RECEIVERID*20240101*1200*1*X*005010X221A1~\n"
"ST*835*0001~\n"
"BPR*I*100*100*C*ACH*CCP*01*123456789*DA*0000001**01*123456789*DA*0000002*20240101~\n"
"TRN*1*CHECK1*1234567890~\n"
"DTM*405*20240101~\n"
"N1*PR*PAYER NAME~\n"
"N3*100 PAYER ST*~~\nN4*DENVER*CO*80202~\n"
"REF*2U*PAYERID~\n"
"N1*PE*RECEIVER NAME~\n"
"N3*200 RECEIVER ST*~~\nN4*DENVER*CO*80202~\n"
"LX*1~\n"
"CLP*CLP-OLD*1*100*100*0*0*0*11*1*PATIENT*DOE****MI*M0001~\n"
"NM1*QC*1*DOE*JOHN****MI*M0001~\n"
"DTM*232*20240101~\n"
"SVC*HC:99213*100*100**1~\n"
"DTM*472*20240101~\n"
"LX*2~\n"
"CLP*CLP-NEW*1*50*50*0*0*0*11*1*PATIENT*ROE****MI*M0002~\n"
"NM1*QC*1*ROE*JANE****MI*M0002~\n"
"DTM*232*20240101~\n"
"SVC*HC:99214*50*50**1~\n"
"DTM*472*20240101~\n"
"SE*30*0001~\nGE*1*1~\nIEA*1*000000001~\n"
)
resp = client.post(
"/api/parse-835?force=true",
files={"file": ("force.835", text, "text/plain")},
headers={"Accept": "application/json"},
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert "CLP-OLD" in body.get("skipped_claim_ids", [])
# CLP-NEW is in a new batch.
with _db.SessionLocal()() as s:
row = s.execute(_db.text("SELECT batch_id FROM remittances WHERE id='CLP-NEW'")).first()
assert row is not None
assert row[0] != "PRIOR835"
- Step 2: Run the new tests, expect FAIL
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest \
tests/test_api_835.py -k "409 or force" --no-header -q
Expected: 2 failures (no pre-flight on 835 yet, no force param).
- Step 3: Implement the 835 endpoint changes
In backend/src/cyclone/api.py:
- Add
force: bool = Query(False)to the 835 signature:
@app.post("/api/parse-835")
async def parse_835_endpoint(
request: Request,
file: UploadFile = File(...),
payer: str = Query("co_medicaid_835"),
include_raw_segments: bool = Query(True),
strict: bool = Query(False),
force: bool = Query(False),
) -> Any:
-
Add the pre-flight check between validation and persist (mirror 837). Find the
try: store.add(...) except IntegrityError as exc:block in the 835 endpoint and replace it with the pre-flight + race-handler pattern from Task 2.3, but calldedup.preflight_835and passerror="Duplicate remittance". -
Add the
_compute_skipped_835helper alongside_compute_skipped_837:
def _compute_skipped_835(batch_id: str, parsed_remits) -> list[str]:
"""Compare parsed remittances against what's in the DB for this batch."""
from sqlalchemy import select
from cyclone import db as _db
from cyclone.db import Remittance
parsed_ids = [r.payer_claim_control_number for r in parsed_remits]
if not parsed_ids:
return []
with _db.SessionLocal()() as s:
persisted = s.execute(
select(Remittance.id).where(Remittance.batch_id == batch_id)
).all()
persisted_ids = {r[0] for r in persisted}
return sorted(set(parsed_ids) - persisted_ids)
- Step 4: Run the new tests, expect PASS
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest \
tests/test_api_835.py -k "409 or force" --no-header -q
Expected: 2 passed.
- Step 5: Run the full test_api_835.py, expect all PASS
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest tests/test_api_835.py --no-header -q
Expected: all pass.
- Step 6: Commit
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
git add backend/src/cyclone/api.py backend/tests/test_api_835.py
git commit -m "feat(api): 835 pre-flight dedup, force=true, race 409"
Task 2.5: Run the full backend test suite as a checkpoint
The 837/835 endpoints changed significantly. Run the full backend test suite to catch any test that relied on the old 409 body shape.
- Step 1: Run the full backend test suite
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend
PYTHONPATH=/Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix/backend/src \
.venv/bin/python3.13 -m pytest --no-header -q
Expected: all pass. If any test fails:
- If the test was checking the 409 body shape, update it to the new shape (or remove the assertion if the field is gone).
- If the test was using
?force=truesemantics that don't match, adjust the test to match the new contract. - If the test was using the old
existing_batch_id-only body, update to the new full-body shape.
Do not modify production code to make old tests pass. If a test
was checking pre-conditions that the new code intentionally doesn't
satisfy (e.g., a test that asserts "without ?force=true, the file
fails to ingest"), update the test to reflect the new contract.
- Step 2: Commit any test fixes
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
git add backend/tests/
git commit -m "test(api): update tests for new 409 body shape and force=true"
(Only if there were fixes; otherwise skip.)
Phase 3 — Frontend
Task 3.1: ApiError carries collisions + parseResult; parse837/835 accept force
Files:
-
Modify:
src/lib/api.ts -
Create:
src/lib/api.test.ts -
Step 1: Write the failing tests
Create src/lib/api.test.ts:
import { describe, it, expect, vi, afterEach } from "vitest";
import { ApiError, parse837 } from "@/lib/api";
afterEach(() => vi.restoreAllMocks());
describe("ApiError", () => {
it("carries collisions and parseResult when constructed", () => {
const collisions = {
colliding_claim_ids: ["A"],
total_collisions: 1,
total_claims: 2,
new_claims_after_skip: 1,
};
const parseResult = { claims: [], summary: { total_claims: 2 } };
const e = new ApiError(409, "dup", "BATCH-1", collisions, parseResult);
expect(e.status).toBe(409);
expect(e.existingBatchId).toBe("BATCH-1");
expect(e.collisions).toEqual(collisions);
expect(e.parseResult).toEqual(parseResult);
});
it("defaults collisions and parseResult to null", () => {
const e = new ApiError(500, "boom");
expect(e.collisions).toBeNull();
expect(e.parseResult).toBeNull();
});
});
describe("parse837 throws ApiError with collisions and parseResult", () => {
it("extracts collisions and parse_result from 409 body", async () => {
vi.stubGlobal("import.meta.env", { VITE_API_BASE_URL: "http://x" });
const collisions = { colliding_claim_ids: ["A"], total_collisions: 1, total_claims: 2, new_claims_after_skip: 1 };
const parse_result = { claims: [], summary: { total_claims: 2 } };
const res = new Response(
JSON.stringify({
error: "Duplicate claim",
detail: "collision",
batch_id: null,
existing_batch_id: "PRIOR",
collisions,
parse_result,
}),
{ status: 409, headers: { "Content-Type": "application/json" } },
);
vi.stubGlobal("fetch", vi.fn(async () => res));
const file = new File(["x"], "f.txt", { type: "text/plain" });
await expect(
parse837(file, { onProgress: () => {} }),
).rejects.toMatchObject({
status: 409,
existingBatchId: "PRIOR",
collisions,
parseResult: parse_result,
});
});
it("appends ?force=true when options.force is set", async () => {
vi.stubGlobal("import.meta.env", { VITE_API_BASE_URL: "http://x" });
const fetchMock = vi.fn(async () =>
new Response(JSON.stringify({ summary: { total_claims: 1 } }), {
status: 200,
headers: { "Content-Type": "application/json" },
}),
);
vi.stubGlobal("fetch", fetchMock);
const file = new File(["x"], "f.txt", { type: "text/plain" });
await parse837(file, { onProgress: () => {}, force: true });
const calledUrl = fetchMock.mock.calls[0][0];
expect(calledUrl).toContain("force=true");
});
});
- Step 2: Run tests, expect FAIL
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
npm test -- src/lib/api.test.ts
Expected: ApiError constructor doesn't accept collisions/parseResult
yet; parse837 doesn't append ?force=true.
- Step 3: Update
src/lib/api.ts
- Update the
ApiErrorclass. Find the current class (around line 164-167) and replace with:
export type CollisionSummary = {
colliding_claim_ids: string[];
total_collisions: number;
total_claims: number;
new_claims_after_skip: number;
within_file_duplicate_ids?: string[];
};
export class ApiError extends Error {
constructor(
public status: number,
message: string,
public existingBatchId: string | null = null,
public collisions: CollisionSummary | null = null,
public parseResult: unknown = null,
) {
super(message);
}
}
- Update
readErrorBodyto return more fields. Find thereadErrorBodyfunction and replace it with:
type ErrorBody = {
detail?: unknown;
error?: unknown;
existing_batch_id?: unknown;
collisions?: unknown;
parse_result?: unknown;
};
async function readErrorBody(
res: Response,
): Promise<{
message: string;
existingBatchId: string | null;
collisions: CollisionSummary | null;
parseResult: unknown;
}> {
try {
const t = await res.text();
if (!t) return { message: "", existingBatchId: null, collisions: null, parseResult: null };
try {
const obj = JSON.parse(t) as ErrorBody;
let message = "";
if (typeof obj.detail === "string") message = obj.detail;
else if (typeof obj.error === "string") message = obj.error;
else message = t;
const existing =
typeof obj.existing_batch_id === "string" ? obj.existing_batch_id : null;
const collisions =
obj.collisions && typeof obj.collisions === "object"
? (obj.collisions as CollisionSummary)
: null;
const parseResult = obj.parse_result ?? null;
return { message, existingBatchId: existing, collisions, parseResult };
} catch {
return { message: t, existingBatchId: null, collisions: null, parseResult: null };
}
} catch {
return { message: "", existingBatchId: null, collisions: null, parseResult: null };
}
}
- Update
parse837to acceptforceand surface the new fields. Find theparse837function. Replace the function signature and theif (!res.ok)block:
export async function parse837(
file: File,
options: { onProgress?: (p: number) => void; force?: boolean } = {},
): Promise<ParseResult> {
const url = `${base}/api/parse-837${options.force ? "?force=true" : ""}`;
// ... existing fetch body ...
if (!res.ok) {
const { message, existingBatchId, collisions, parseResult } =
await readErrorBody(res);
throw new ApiError(
res.status,
`${res.status} ${res.statusText}${message ? ` — ${message}` : ""}`,
existingBatchId,
collisions,
parseResult,
);
}
return res.json();
}
Keep the rest of the function (the streaming logic for non-JSON
requests) unchanged — only the if (!res.ok) block and the URL
construction change.
- Update
parse835similarly. Same pattern: append?force=trueto the URL and pass the new fields toApiError.
- Step 4: Run the new tests, expect PASS
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
npm test -- src/lib/api.test.ts
Expected: 4 tests pass.
- Step 5: Run the existing api.ts tests, expect PASS (or update)
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
npm test -- src/lib/api.test.ts src/lib/inbox-api.test.ts
Expected: all pass. If the existing inbox-api or other tests assert
on ApiError shape and fail, update them to match the new
constructor signature (passing null for the new optional fields).
- Step 6: Commit
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
git add src/lib/api.ts src/lib/api.test.ts
git commit -m "feat(api): ApiError carries collisions + parseResult; parse837/835 accept force"
Task 3.2: New deleteBatch function in src/lib/api.ts
Files:
-
Modify:
src/lib/api.ts -
Step 1: Add the function
Find a good place in src/lib/api.ts (near parse837/parse835).
Add:
export async function deleteBatch(batchId: string): Promise<void> {
const res = await fetch(`${base}/api/batches/${encodeURIComponent(batchId)}`, {
method: "DELETE",
});
if (!res.ok) {
let body: { error?: string; detail?: string; offending_claim_id?: string } = {};
try {
body = (await res.json()) as typeof body;
} catch {
// ignore
}
throw new ApiError(
res.status,
`${res.status} ${res.statusText}${
body.detail ? ` — ${body.detail}` : ""
}`,
);
}
}
- Step 2: Add a test
Add to src/lib/api.test.ts:
import { deleteBatch } from "@/lib/api";
describe("deleteBatch", () => {
it("sends DELETE to /api/batches/{id}", async () => {
vi.stubGlobal("import.meta.env", { VITE_API_BASE_URL: "http://x" });
const fetchMock = vi.fn(async () => new Response(null, { status: 204 }));
vi.stubGlobal("fetch", fetchMock);
await deleteBatch("B123");
expect(fetchMock).toHaveBeenCalledWith(
"http://x/api/batches/B123",
expect.objectContaining({ method: "DELETE" }),
);
});
it("throws ApiError with the 409 detail on refusal", async () => {
vi.stubGlobal("import.meta.env", { VITE_API_BASE_URL: "http://x" });
const res = new Response(
JSON.stringify({ error: "Batch not deletable", detail: "claim X is paid" }),
{ status: 409, headers: { "Content-Type": "application/json" } },
);
vi.stubGlobal("fetch", vi.fn(async () => res));
await expect(deleteBatch("B1")).rejects.toMatchObject({ status: 409 });
});
});
- Step 3: Run, expect PASS
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
npm test -- src/lib/api.test.ts
Expected: all 6 tests pass.
- Step 4: Commit
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
git add src/lib/api.ts src/lib/api.test.ts
git commit -m "feat(api): deleteBatch function"
Task 3.3: Upload.tsx panel with 4 actions
Files:
-
Modify:
src/pages/Upload.tsx -
Create:
src/pages/Upload.test.tsx -
Step 1: Write the failing tests
Create src/pages/Upload.test.tsx:
import { describe, it, expect, vi, beforeEach } from "vitest";
import { render, screen, fireEvent, waitFor } from "@testing-library/react";
import { MemoryRouter, Route, Routes } from "react-router-dom";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import * as apiModule from "@/lib/api";
import { ApiError, CollisionSummary } from "@/lib/api";
import { Upload } from "@/pages/Upload";
vi.mock("@/lib/api", async () => {
const actual = await vi.importActual<typeof apiModule>("@/lib/api");
return { ...actual, parse837: vi.fn(), deleteBatch: vi.fn() };
});
beforeEach(() => {
vi.clearAllMocks();
});
function renderUpload() {
const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } });
return render(
<QueryClientProvider client={qc}>
<MemoryRouter initialEntries={["/upload"]}>
<Routes>
<Route path="/upload" element={<Upload />} />
<Route path="/batches/:id" element={<div data-testid="batch-page" />} />
</Routes>
</MemoryRouter>
</QueryClientProvider>,
);
}
const collisions: CollisionSummary = {
colliding_claim_ids: ["A", "B"],
total_collisions: 2,
total_claims: 141,
new_claims_after_skip: 139,
within_file_duplicate_ids: [],
};
const parseResult = {
envelope: {},
claims: [
{ claim_id: "A", service_lines: [], diagnoses: [] },
{ claim_id: "B", service_lines: [], diagnoses: [] },
],
summary: { total_claims: 141, passed: 141, failed: 0 },
};
describe("Upload error panel", () => {
it("renders panel with all actions when 409 carries collisions + parseResult", async () => {
vi.mocked(apiModule.parse837).mockRejectedValueOnce(
new ApiError(
409,
"409 Conflict — dup",
"PRIOR",
collisions,
parseResult,
),
);
renderUpload();
const input = document.querySelector('input[type="file"]') as HTMLInputElement;
const file = new File(["x"], "test.txt", { type: "text/plain" });
fireEvent.change(input, { target: { files: [file] } });
const forceBtn = await screen.findByRole("button", { name: /force insert/i });
expect(forceBtn).toBeInTheDocument();
expect(screen.getByRole("button", { name: /open prior batch/i })).toBeInTheDocument();
expect(screen.getByRole("button", { name: /delete prior batch/i })).toBeInTheDocument();
expect(screen.getByRole("button", { name: /pick a different file/i })).toBeInTheDocument();
// The parse result summary is rendered.
expect(screen.getByText(/141/)).toBeInTheDocument();
});
it("force-insert button re-calls parse837 with force=true", async () => {
vi.mocked(apiModule.parse837)
.mockRejectedValueOnce(
new ApiError(409, "409 Conflict", "PRIOR", collisions, parseResult),
)
.mockResolvedValueOnce({ ...parseResult, skipped_claim_ids: ["A", "B"] });
renderUpload();
const input = document.querySelector('input[type="file"]') as HTMLInputElement;
const file = new File(["x"], "test.txt", { type: "text/plain" });
fireEvent.change(input, { target: { files: [file] } });
const forceBtn = await screen.findByRole("button", { name: /force insert/i });
fireEvent.click(forceBtn);
await waitFor(() => {
expect(apiModule.parse837).toHaveBeenCalledTimes(2);
});
expect(apiModule.parse837).toHaveBeenNthCalledWith(
2,
expect.any(File),
expect.objectContaining({ force: true }),
);
});
it("delete-prior button calls deleteBatch", async () => {
vi.spyOn(window, "confirm").mockReturnValue(true);
vi.mocked(apiModule.parse837).mockRejectedValueOnce(
new ApiError(409, "409 Conflict", "PRIOR", collisions, parseResult),
);
vi.mocked(apiModule.deleteBatch).mockResolvedValueOnce(undefined);
renderUpload();
const input = document.querySelector('input[type="file"]') as HTMLInputElement;
const file = new File(["x"], "test.txt", { type: "text/plain" });
fireEvent.change(input, { target: { files: [file] } });
const delBtn = await screen.findByRole("button", { name: /delete prior batch/i });
fireEvent.click(delBtn);
await waitFor(() => {
expect(apiModule.deleteBatch).toHaveBeenCalledWith("PRIOR");
});
});
it("within-file duplicate (existingBatchId null) hides prior batch actions", async () => {
vi.mocked(apiModule.parse837).mockRejectedValueOnce(
new ApiError(
409,
"409 Conflict",
null,
{ ...collisions, within_file_duplicate_ids: ["DUP"] },
parseResult,
),
);
renderUpload();
const input = document.querySelector('input[type="file"]') as HTMLInputElement;
const file = new File(["x"], "test.txt", { type: "text/plain" });
fireEvent.change(input, { target: { files: [file] } });
await screen.findByRole("button", { name: /force insert/i });
expect(
screen.queryByRole("button", { name: /open prior batch/i }),
).toBeNull();
expect(
screen.queryByRole("button", { name: /delete prior batch/i }),
).toBeNull();
});
it("'Pick a different file' button clears error", async () => {
vi.mocked(apiModule.parse837).mockRejectedValueOnce(
new ApiError(409, "409 Conflict", "PRIOR", collisions, parseResult),
);
renderUpload();
const input = document.querySelector('input[type="file"]') as HTMLInputElement;
const file = new File(["x"], "test.txt", { type: "text/plain" });
fireEvent.change(input, { target: { files: [file] } });
const clearBtn = await screen.findByRole("button", { name: /pick a different file/i });
fireEvent.click(clearBtn);
expect(screen.queryByText(/collide/i)).toBeNull();
});
it("does NOT render panel for non-409 errors", async () => {
vi.mocked(apiModule.parse837).mockRejectedValueOnce(
new ApiError(400, "bad file"),
);
renderUpload();
const input = document.querySelector('input[type="file"]') as HTMLInputElement;
const file = new File(["x"], "test.txt", { type: "text/plain" });
fireEvent.change(input, { target: { files: [file] } });
await new Promise((r) => setTimeout(r, 50));
expect(screen.queryByText(/collide/i)).toBeNull();
});
});
- Step 2: Run tests, expect FAIL (Upload.tsx doesn't have the new state yet)
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
npm test -- src/pages/Upload.test.tsx
Expected: all 6 tests fail (no panel exists).
- Step 3: Modify
src/pages/Upload.tsx
This is the most invasive frontend change. Read the current
Upload.tsx (the file is 600+ lines; find the existing error-handling
catch block) and apply these changes:
- Update imports at the top:
import { ApiError, parse837, parse835, deleteBatch, CollisionSummary } from "@/lib/api";
import { useNavigate } from "react-router-dom";
- Add new state near the other
useStatecalls:
type UploadError = {
kind: "duplicate";
existingBatchId: string | null;
collisions: CollisionSummary;
parseResult: unknown;
filename: string;
};
const [uploadError, setUploadError] = useState<UploadError | null>(null);
const [forceInserting, setForceInserting] = useState(false);
const navigate = useNavigate();
- Replace the existing error-handling catch block (find the
catch (err) {after the parse call) with:
} catch (err) {
if (err instanceof ApiError && err.status === 409 && err.collisions) {
setUploadError({
kind: "duplicate",
existingBatchId: err.existingBatchId,
collisions: err.collisions,
parseResult: err.parseResult,
filename: file.name,
});
toast.error("Duplicate claim — file not ingested");
} else {
toast.error(
err instanceof Error ? err.message : "Failed to parse file",
);
}
}
- Add the panel JSX above the streaming results section (find the existing "parsed claims" / streaming-results JSX and insert BEFORE it):
{uploadError && uploadError.kind === "duplicate" ? (
<div
role="alert"
data-testid="duplicate-error-panel"
className="rounded-md border border-destructive/40 bg-destructive/5 p-4 mx-auto max-w-3xl"
>
<div className="flex items-center gap-2">
<span className="inline-flex items-center rounded-md bg-destructive px-2 py-0.5 text-xs font-semibold text-destructive-foreground">
409
</span>
<span className="font-semibold">
{uploadError.collisions.total_collisions} of {uploadError.collisions.total_claims} claims
collide
{uploadError.existingBatchId
? ` with batch ${uploadError.existingBatchId}`
: " within this file"}
</span>
</div>
<p className="mt-2 text-sm text-muted-foreground">
File <span className="font-mono">{uploadError.filename}</span> would persist
{" "}{uploadError.collisions.new_claims_after_skip} of {uploadError.collisions.total_claims} claims.
Colliding CLM01s: {uploadError.collisions.colliding_claim_ids.join(", ")}.
</p>
<div className="mt-3 flex flex-wrap gap-2">
<Button
disabled={forceInserting}
onClick={async () => {
setForceInserting(true);
try {
const result = await parse837(file, { onProgress: () => {}, force: true });
setParseResult(result);
setUploadError(null);
const skipped = (result as { skipped_claim_ids?: string[] }).skipped_claim_ids?.length ?? 0;
toast.success(
`Force-inserted: ${result.summary.total_claims - skipped} of ${result.summary.total_claims} (skipped ${skipped} dups)`,
);
} catch (err) {
toast.error(err instanceof Error ? err.message : "Force-insert failed");
} finally {
setForceInserting(false);
}
}}
>
Force insert (skip {uploadError.collisions.total_collisions} dups)
</Button>
{uploadError.existingBatchId ? (
<>
<Button
variant="outline"
onClick={() => navigate(`/batches/${uploadError.existingBatchId}`)}
>
Open prior batch
</Button>
<Button
variant="outline"
onClick={async () => {
if (!confirm(`Delete batch ${uploadError.existingBatchId}? This cannot be undone.`)) return;
try {
await deleteBatch(uploadError.existingBatchId!);
toast.success(`Deleted ${uploadError.existingBatchId}`);
setUploadError(null);
pickFile(null);
} catch (err) {
toast.error(err instanceof Error ? err.message : "Delete failed");
}
}}
>
Delete prior batch
</Button>
</>
) : null}
<Button
variant="ghost"
onClick={() => {
setUploadError(null);
pickFile(null);
}}
>
Pick a different file
</Button>
</div>
<details className="mt-3 text-sm">
<summary>Show parsed claims ({uploadError.collisions.total_claims})</summary>
<pre className="mt-2 max-h-64 overflow-auto rounded bg-muted p-2 text-xs">
{JSON.stringify(
(uploadError.parseResult as { summary?: unknown })?.summary ?? {},
null,
2,
)}
</pre>
</details>
</div>
) : null}
- Ensure
parseResultstate setter exists. The currentUpload.tsxlikely stores the parsed result under a different name (e.g.,result,parsed). UsesetParseResultor rename to match. If the existing code uses a different name, adapt the JSX above to use that name — do not introduce parallel state.
- Step 4: Run the new tests, expect PASS
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
npm test -- src/pages/Upload.test.tsx
Expected: 6 passed.
- Step 5: Run the broader Upload-related tests to catch regressions
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
npm test -- src/pages/Upload.test.tsx src/hooks/useParse.test.ts
Expected: all pass.
- Step 6: Commit
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
git add src/pages/Upload.tsx src/pages/Upload.test.tsx
git commit -m "feat(upload): 409 panel with force-insert / open / delete / pick-different actions"
Task 3.4: Frontend typecheck and lint
The frontend touched multiple files. Make sure typecheck and lint pass.
- Step 1: Run typecheck
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
npm run typecheck
Expected: no errors. If there are, fix them inline (mostly
unknown casts on parseResult).
- Step 2: Run lint
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
npm run lint
Expected: no errors. Fix any inline.
- Step 3: Run the full vitest suite
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
npm test
Expected: all pass.
- Step 4: Commit any fixes
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
git add src/
git commit -m "chore(frontend): typecheck and lint fixes for upload panel"
(Only if there were fixes; otherwise skip.)
Phase 4 — End-to-end verification
Task 4.1: Apply migration 0013 to the production database
The production DB at /Users/openclaw/.local/share/cyclone/cyclone.db
has the inline UNIQUE constraint. Migration 0013 (already on the
worktree) drops it. Apply it.
This is the change that fixes the user's immediate "every upload
fails with 409" problem. All 19 files in
docs/prodfiles/837p-from-axiscare/ will ingest cleanly.
Files: none (manual step)
- Step 1: Stop the running uvicorn
# Find and stop the dev server.
ps aux | grep -E "uvicorn cyclone" | grep -v grep
kill <pid>
- Step 2: Make a backup of the production DB
cp ~/.local/share/cyclone/cyclone.db ~/.local/share/cyclone/cyclone.db.bak.$(date +%s)
ls -la ~/.local/share/cyclone/cyclone.db*
- Step 3: Apply migration 0013
cd /Users/openclaw/dev/cyclone
backend/.venv/bin/python3.13 -c "
import sqlite3
con = sqlite3.connect('//Users/openclaw/.local/share/cyclone/cyclone.db')
with open('backend/src/cyclone/migrations/0013_drop_claims_unique_constraint.sql') as f:
sql = f.read()
con.executescript(sql)
con.commit()
con.close()
print('migration 0013 applied')
"
Note: this is a one-off direct DB application. The next time
the app starts, the migration runner will see user_version >= 13
and skip 0013.
- Step 4: Verify the UNIQUE is gone
backend/.venv/bin/python3.13 -c "
import sqlite3
con = sqlite3.connect('//Users/openclaw/.local/share/cyclone/cyclone.db')
for row in con.execute(\"SELECT sql FROM sqlite_master WHERE name='claims' AND type='table'\"):
print(row[0])
"
Expected: the printed CREATE TABLE claims definition does NOT have
UNIQUE (batch_id, patient_control_number) at the end.
- Step 5: Restart the dev server
cd /Users/openclaw/dev/cyclone
backend/.venv/bin/uvicorn cyclone.api:app --host 127.0.0.1 --port 8000 --reload &
- Step 6: Repro the original 409 — should now succeed
cd /Users/openclaw/dev/cyclone
# Run a small test that uploads one of the previously-failing files.
backend/.venv/bin/python3.13 /tmp/test_repro.py 2>&1 | tail -10
Expected: HTTP 200, the file persists. The 19 files in
docs/prodfiles/837p-from-axiscare/ all upload cleanly now.
- Step 7: Re-upload the same file — should now 409 with the new body shape
# Re-run the same test; the second upload of the same file should
# trigger the pre-flight 409 with the rich body.
backend/.venv/bin/python3.13 /tmp/test_repro.py 2>&1 | tail -30
Expected: HTTP 409 with existing_batch_id pointing to the first
batch, collisions.colliding_claim_ids: ["t991102984o1c1d", ...],
and a full parse_result in the body.
- Step 8: Force-insert should succeed and skip the dupes
Manually via curl or the test script: POST with ?force=true. Expected: HTTP 200 with skipped_claim_ids: [...] listing the colliding claim_ids.
- Step 9: DELETE /api/batches/{id} works
Manually: curl -X DELETE http://127.0.0.1:8000/api/batches/{id}. Expected: 204 on success, 409 if the batch has paid claims, 404 if the id doesn't exist.
- Step 10: Commit no code (manual step)
If the manual step revealed that the code needs tweaks (e.g., the parser handles a 19-claim file differently from a 2-claim test fixture), commit those tweaks:
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
git add backend/
git commit -m "fix: tweaks from e2e verification against production files"
(Only if needed.)
Task 4.2: Merge to main
The plan is complete. The user can now upload files cleanly, see the rich 409 body, force-insert, and delete prior batches.
- Step 1: Verify the worktree is clean
cd /Users/openclaw/dev/cyclone/.worktrees/claims-unique-fix
git status
Expected: working tree clean (or only the backend/.venv symlink,
which is in .gitignore).
- Step 2: Merge to main
cd /Users/openclaw/dev/cyclone
git checkout main
git merge --ff-only claims-unique-fix
- Step 3: Push (only if the user wants to)
git push origin main
(Skip if the user wants to keep this local; ask before pushing.)
Self-review checklist (run after writing the plan)
-
Spec coverage — every section of the spec maps to a task:
- §1 (incl. schema-bug rationale) → Task 1.3 (migration 0014 — added after spec/plan review revealed the PK must be relaxed for the workflow to fire)
- §3.1 No collision → covered by existing tests + the new pre-flight not firing
- §3.2 Collision → Task 2.3 (837) + Task 2.4 (835) + Task 3.3 (Upload panel)
- §3.3 Force-insert → Task 2.3 (837) + Task 2.4 (835) + Task 3.1 (
?force=true) + Task 3.3 (force-insert button) - §3.4 Race condition → Task 2.3 + Task 2.4 (race handler)
- §4 Within-file duplicates → Task 2.1 (
preflight_837within_file_duplicate_ids) + Task 2.3 (test) + Task 3.3 (panel hides prior batch actions) - §5 409 body shape → Task 2.3 (
_build_dedup_409) + Task 3.1 (frontendCollisionSummarytype) - §6 DELETE endpoint → Task 2.2 (store helper) + Task 2.3 (endpoint) + Task 3.2 (
deleteBatchclient fn) + Task 3.3 (delete button) - §7 Backend implementation → Tasks 1.3, 1.4, 2.1, 2.2, 2.3, 2.4
- §8 Frontend implementation → Tasks 3.1, 3.2, 3.3
- §10 Files changed → all listed (incl. migration 0014)
- §11 Test plan → 14 backend + 6 frontend tests distributed across tasks, plus 3 new 0014 migration tests
-
Schema-foundation gap (caught during plan execution) — the original plan assumed
claims.idcould host multiple rows for the same CLM01, but the schema enforcesclaims.idas a single-column PRIMARY KEY. The implementer caught this on Task 1.3 (the helper's "most-recent" test could not insert two rows with the same id). The correct fix (per user direction: "fix this the correct way ... take pride in this") is migration 0014, which relaxes the PK to composite(batch_id, id)on bothclaimsandremittances, and updates all FKs that referenced the old single-column PK. Task 1.3 is the new first executable task in Phase 1; the original Task 1.3 was renumbered to 1.4. -
Placeholder scan — no "TBD", "TODO", "implement later". The note about "Open question resolved during implementation" in the spec was about the audit-event FK cascade; the implementation just records the event with
batch_idset, relying on the existingON DELETE CASCADEonactivity_events.batch_id(or the alternative if not present). The plan tests this in Task 2.2 by deleting a batch and asserting the cascade works. -
Type consistency — every
ApiErrorconstruction in the frontend passes 5 args (status, message, existingBatchId, collisions, parseResult). TheCollisionSummarytype is defined once inapi.tsand imported where needed. TheCollisionReportdataclass is used uniformly indedup.py/api.py/store.py. Theskipped_claim_idsfield is only on the 200 success body (not on the 409 body) and is only set whenforce=true. -
Risk acknowledged — pre-flight race, DELETE on large batch,
force=truesilent skip, within-file dupes, andexisting_batch_idstaleness are all documented in the spec §13 and tested where applicable. -
TDD throughout — every task is RED (failing test) → GREEN (minimal code) → COMMIT. No "write tests after" anywhere.
-
DRY — the 837 and 835 changes share
_build_dedup_409and the_compute_skipped_*helpers. TheCollisionReportdataclass andpreflight_*functions live indedup.pyso the 837/835 endpoints share the same dedup logic. -
Frequent commits — every task ends with a
git commit. Expectation: ~7 backend commits + 3 frontend commits = 10 commits total.