cyclone/docs/superpowers/specs/2026-07-07-cyclone-submit-batch-canonical-flow-design.md

Sub-project 37 — Canonical submit-batch flow: Design Spec

Date: 2026-07-07 Status: Draft, awaiting user sign-off Branch: sp37-submit-batch-canonical-flow (off main, post-SP36 merge) Aesthetic direction: No UI changes. Pure backend structural + data-model addition — the public HTTP surface gains one new endpoint; existing CLIs and endpoints are preserved unchanged.

---

1. Scope

Today, cyclone's claims table is essentially empty in production (clm-1 is the only row as of 2026-07-07). The reason is structural: dzinesco generates the 837P files (in ingest/corrected/batch-*-claims/*.x12), cyclone's resubmit-rejected-claims CLI only validates and re-uploads them to Gainwell's MFT — it never writes the claims to the DB. When 999 acknowledgments come back referencing those batches, every AK2 set-response becomes an orphan because there is no Batch row to join against.

The 999→claim join (the D10 two-pass join in claim_acks.lookup_claims_for_ack_set_response) has two paths:

• Pass 1 (primary): Batch.envelope.control_number == 999's set_control_number. This currently fails because the 999's set_control_number (AK201) echoes the source 837's ST02 (transaction set control number), not the ISA13 (interchange control number). The parsed Envelope model stores ISA13 in control_number, so the keys never collide even when the DB row exists.
• Pass 2 (fallback): Claim.patient_control_number == 999's set_control_number. This also fails because the _claim_837_row ORM builder sets patient_control_number = claim.claim_id (CLM01), and CLM01 ≠ ST02 in production batches.

The fix is twofold: (a) make cyclone write a Batch + Claim rows at submission time so there is something to join against, and (b) make the join key actually match — by storing the ST02 explicitly on Batch and using it in Pass 1.

In scope:

• A new top-level CLI cyclone submit-batch that does parse → DB write → SFTP upload per file, in that order.
• A new HTTP endpoint POST /api/submit-batch with the same semantics.
• A shared helper module cyclone.submission.submit_file so the CLI and HTTP path share one source of truth.
• A new nullable column Batch.transaction_set_control_number, populated from the parsed 837's ST02 on every add_record write.
• An Alembic migration 0013_add_batch_txn_set_control_number.py (additive, nullable, backfills from raw_result_json.envelope.transaction_set_control_number where present).
• An update to claim_acks.batch_envelope_index so Pass 1 also matches on transaction_set_control_number (Pass 2 unchanged).
• Per-claim claim_submitted activity events and per-file clearhouse.submitted audit events (same shape resubmit-rejected-claims already emits).

Out of scope (explicit):

• Deprecating parse-837 CLI or /api/parse-837 endpoint. They remain for one-off testing and dry-run validation.
• Deprecating resubmit-rejected-claims. It remains the "validate-then-upload without DB write" path for the dzinesco-generated files that aren't ready for canonical tracking.
• UI changes. The Claims page already renders the fields a new tracked claim needs; the existing claim_written live-tail event fires automatically from add_record.
• Bulk backfill of the existing ingest/corrected/batch-*-claims/*.x12 backlog. A follow-up SP will walk that directory once submit-batch is proven.
• Any change to the claims row schema beyond the new Batch column.
• SFTP auth work. This box stays in manual mode (post-2026-07-07 incident); submit-batch works in stub mode for local testing and against a whitelisted IP for production.

2. Decisions (locked during brainstorming)

These were each a multiple-choice question with operator sign-off on 2026-07-07.

1. Where does the DB write happen? New canonical submit flow. submit-batch parses → writes to DB → uploads, replacing neither parse-837 nor resubmit-rejected-claims but superseding them as the preferred path.
2. DB vs upload ordering? DB-first, upload-second. If DB write fails, the file never goes on the wire. If DB write succeeds but upload fails, re-running is safe (idempotent DB write, idempotent SFTP put).
3. Join key for 999 acks? Add Batch.transaction_set_control_number column. Update Pass 1 to match on it. Don't repurpose envelope.control_number (preserves backward compat with existing joins) and don't force CLM01 == ST02 (Gainwell's existing 999s use different values).
4. Deprecation posture? Add submit-batch; preserve parse-837 and resubmit-rejected-claims. Document submit-batch as preferred in CLAUDE.md + RUNBOOK. Future SP can deprecate if usage shifts.

3. Architecture

The new flow is a single, small public surface — one CLI command, one HTTP endpoint, one shared helper. The helper owns the parse → write → upload sequence and is the only place that knows about the order.

Module layout

A new package cyclone.submission (under backend/src/cyclone/submission/) holds the shared helper:

• __init__.py — re-exports submit_file, SubmitResult, SubmitOutcome
• core.py — submit_file(path: Path, *, sftp_block, actor: str, event_bus=None, validate: bool = True) -> SubmitResult. Owns its own DB session for the parse-write step, returns a result dataclass the caller inspects.
• cli.py — Click handler for cyclone submit-batch. Walks --ingest-dir, calls submit_file per file, aggregates counts, exits 0/1/2 per cyclone-cli convention.
• api.py — FastAPI handler for POST /api/submit-batch. Mounted in api_routers/submission.py. Calls submit_file per file. Auth-gated by the existing matrix_gate.

submit_file algorithm

The walker globs <ingest_dir>/batch-*-claims/*.x12 (mirrors resubmit-rejected-claims exactly — same pattern, same _-prefix skip for AppleDouble residue). For each matched file:

1. Read bytes; if validate=True, run parse_837_text(bytes, payer_cfg). On parse failure, record {path, "parse_failed", exc} and return without writing or uploading.
2. Run a payer-id check (must be CO_TXIX); on mismatch, record and return.
3. Call store.add_record(record, event_bus=...). On DB failure, record and return (no SFTP call).
4. Build the remote path {outbound_root}/{filename}. Open an SftpClient (the seeded singleton). On SFTP connect failure, record and return — the DB row is already written; the re-run will skip it via idempotency.
5. sftp_client.stat(remote_path) — if st_size == local_size, mark as skipped (already uploaded; do not re-emit audit). Otherwise sftp_client.write_file(remote_path, bytes).
6. On successful upload, emit one clearhouse.submitted audit event with source_file, batch_id, actor.

Idempotency

Two layers, both per-file:

• DB layer: store.add_record already does a s.get(Claim, claim.claim_id) check before insert. A re-parse of the same file logs "claim already exists; skipping" and continues. Re-running submit-batch on an already-tracked file is a no-op on the DB.
• SFTP layer: the sftp.stat().st_size == local_size check before write_file short-circuits already-uploaded files.

Failure modes summary

• Parse fails → no DB write, no SFTP put, file counted as failed.
• Payer-id mismatch → same as parse fail.
• DB write fails → no SFTP put, file counted as failed. The DB session rolls back; no orphan rows.
• SFTP connect fails → DB row already committed; file counted as failed. Re-run is safe.
• SFTP stat succeeds + size match → counted as skipped; no audit event.
• SFTP write fails → file counted as failed. DB row already committed. Re-run is safe.
• Crash between DB write and SFTP put → DB row exists, file never landed. Re-run will see the size mismatch (remote doesn't have it) and upload.

4. Data flow & error handling

POST /api/submit-batch

Request body (JSON):

• ingest_dir: string — root directory holding batch-*-claims/*.x12 subfolders (mirrors the CLI flag).
• validate: bool = true — pass-through to submit_file.
• actor: string = "api-submit-batch" — audit actor tag.
• limit: int | null = null — stop after N files (smoke test).

Response body (200):

• submitted: int — files uploaded this run.
• skipped: int — files already on remote (size match).
• failed: int — files that hit parse / payer / DB / SFTP errors.
• results: list<{file: string, outcome: "submitted"|"skipped"|"parse_failed"|"payer_mismatch"|"db_failed"|"sftp_failed", batch_id: string|null, error: string|null}> — per-file detail.

Status codes:

• 200 — run completed (may have failures; details in the response body).
• 401 — no auth cookie (matrix_gate).
• 422 — body validation failure.
• 500 — unexpected exception (caught by the existing @app.exception_handler(Exception)).

CLI exit codes (per cyclone-cli convention)

• 0 — run completed, even with some per-file failures (the per-file detail is in stdout).
• 1 — unexpected exception (uncaught error in the walker or the helper itself).
• 2 — config-level failure (no clearhouse seeded, SFTP block in stub mode for non-stubbed invocation, ingest-dir doesn't exist).

Audit + live-tail events

• claim_submitted (one per Claim row) — emitted by add_record already; no new code path.
• clearhouse.submitted (one per successful upload) — emitted by submit_file after the SFTP put succeeds. Same actor tag and payload shape as the current resubmit-rejected-claims path uses.

Migration 0013_add_batch_txn_set_control_number.py

• op.add_column('batches', sa.Column('transaction_set_control_number', sa.String(32), nullable=True)).
• Backfill: UPDATE batches SET transaction_set_control_number = json_extract(raw_result_json, '$.envelope.transaction_set_control_number') WHERE raw_result_json IS NOT NULL AND json_extract(raw_result_json, '$.envelope.transaction_set_control_number') IS NOT NULL.
• Reversible: drop column.
• The down migration is op.drop_column('batches', 'transaction_set_control_number').

Join-key update in claim_acks.batch_envelope_index

The existing function reads every Batch row whose raw_result_json has an envelope and returns dict[str, str] keyed by envelope.control_number. The updated version returns the same shape — a single dict[str, str] — but populated from two columns:

• Each row contributes one entry keyed by envelope.control_number (preserved).
• Each row also contributes one entry keyed by transaction_set_control_number (new), when that column is non-NULL.

If both keys resolve to the same value (Gainwell batches where the ST02 happens to equal the ISA13 — unusual but legal), the second insert is a no-op. If they differ, both keys point to the same batch.id — a single idx.get(set_control_number) lookup will match either one. The dict may grow by up to len(batches) new entries; callers that previously assumed len(index) == len(batches) are unaffected because that invariant was already loose (multiple ST02s can share one ISA13).

lookup_claims_for_ack_set_response Pass 1 is unchanged at the call site — one idx.get(set_control_number) lookup that succeeds whether the source 837's ST02 matches a transaction_set_control_number row or an envelope.control_number row. Pass 2 (PCN) fires as today if both miss.

5. Testing approach

Per cyclone-tests skill: pytest under backend/tests/, two flavors — test_api_* (FastAPI integration) and test_* (pure unit). Frontend tests out of scope (no UI changes).

Unit tests (backend/tests/test_submission.py)

• submit_file happy path — mock SFTP, real DB; assert one Batch + N Claim rows + one audit event.
• submit_file idempotency on DB — call submit_file twice with the same file; assert second call returns skipped (DB no-op).
• submit_file idempotency on SFTP — mock SFTP stat to return matching size; assert skipped, no second write_file call, no second audit event.
• submit_file parse fail — pass a bad 837; assert no DB write, no SFTP call, parse_failed outcome.
• submit_file payer mismatch — pass a valid 837 with wrong payer_id; assert no DB write, no SFTP call, payer_mismatch outcome.
• submit_file DB fail — monkey-patch add_record to raise; assert no SFTP call, db_failed outcome.
• submit_file SFTP fail — mock SFTP write_file to raise; assert sftp_failed outcome, DB row still present.
• submit_file re-run after partial failure — submit a file that fails SFTP, then submit again with SFTP working; assert second run uploads and emits audit.

Integration tests (backend/tests/test_api_submit_batch.py)

• POST /api/submit-batch happy path — 3 valid 837 fixtures in ingest_dir; assert 200 + body shape.
• POST /api/submit-batch auth gate — no cookie → 401 (matrix_gate).
• POST /api/submit-batch body validation — missing ingest_dir → 422.

Migration tests (backend/tests/test_migration_0013.py)

• Up + down on a fresh DB; column exists + is nullable; backfill populates the column from raw JSON.

Join-key update tests (backend/tests/test_claim_acks_index.py)

• batch_envelope_index returns both maps; Pass 1 finds a claim by ST02 even when ISA13 differs; Pass 1 finds by ISA13 (backward compat); Pass 2 fallback still works.

Live test (after merge)

Walk an actual ingest/batch-*-claims/ directory via the CLI; verify:

• claims table has new rows with the expected patient_control_number (CLM01).
• batches table has new rows with transaction_set_control_number matching the parsed ST02.
• Live-tail claim_written events fire on the API.

Then drop a synthetic 999 (with the matching AK2 set_control_number) into ingest/, run pull-inbound, verify the ClaimAck link row is created (not an orphan).

6. Threat model (post-SP24 alignment)

No new attack surface. The new endpoint and CLI:

• Are gated by the existing matrix_gate (HTTP) and CYCLONE_AUTH_DISABLED env var.
• Use the same SFTP client and credential paths as resubmit-rejected-claims today.
• Write to the same store facade; no new SQL surface.
• Emit the same audit events with the same payload shape.

The transaction_set_control_number column is operator-visible (it's in the raw batches row), but no auth boundary depends on it being absent. PII exposure is unchanged (the column carries the same value the 999 ack would have echoed back — already public-to-Gainwell).

7. Risks & mitigations

• Risk: The new submission package is the third "command runner" in the codebase (alongside cli.py's standalone commands and scheduler.py). Mitigation: Keep submit_file as a pure helper with no FastAPI/Click dependencies; the API and CLI handlers are thin wrappers. Document the layering in submission/__init__.py's module docstring.
• Risk: resubmit-rejected-claims and submit-batch will diverge over time (one writes to DB, one doesn't). Mitigation: Both call the same submit_file helper if submit-batch's --write-to-db flag is set to true (default). Document the divergence point in RUNBOOK.
• Risk: Migration backfill could be slow on a large DB. Mitigation: UPDATE ... WHERE json_extract(...) IS NOT NULL is one pass; on a million-row DB this is sub-second. Index on transaction_set_control_number not added (low cardinality per batch; existing ix_claims_patient_control_number covers the hot path).
• Risk: Live-tail event volume doubles when submit-batch runs (one claim_submitted per claim in addition to the historical claim_written). Mitigation: This is desired behavior — the operator wants the activity feed to show submissions. Document in the RUNBOOK that submitting a 1000-claim batch will publish ~1000 events.

8. Rollout

• Branch: sp37-submit-batch-canonical-flow, off main as of 710104f.
• Implementation order:
  1. Migration 0013 + backfill (additive, no behavior change yet).
  2. Update claim_acks.batch_envelope_index + Pass 1 lookup (no behavior change yet — the new map is built but unused until submit-batch writes rows).
  3. cyclone/submission/core.py helper + unit tests.
  4. CLI handler cyclone submit-batch + smoke test.
  5. HTTP handler POST /api/submit-batch + integration tests.
  6. Live test with one of the existing ingest/batch-*-claims/*.x12 directories (will be in stub mode on this host).
• Live test cadence: per the user's standing directive — live-test after each significant step, run autoreview, commit.
• Merge shape: single atomic merge commit per SP-N convention. No squash. No rebase.
• Post-merge: RUNBOOK update for the canonical flow; CLAUDE.md update to mark submit-batch as preferred.

9. Open questions

None blocking the spec. Tracked for follow-up SPs:

• Backfill command — once submit-batch is proven, a cyclone backfill-claims CLI that walks the existing ingest/corrected/batch-*-claims/*.x12 backlog (without uploading — they're already on Gainwell's side) and writes the DB rows. This is what would un-orphan the 1491+ orphan 999 acks currently in the DB.
• Deprecation timeline for resubmit-rejected-claims — once submit-batch is the default, consider folding resubmit-rejected-claims into a --no-write-to-db flag on submit-batch to consolidate the surface.
• Cross-link to 277CA acks — same transaction_set_control_number join key will resolve 277CA orphans if/when those arrive. The migration and index update cover both 999 and 277CA since both use Pass 1.