From 698f8660bf1915d977ea9dfe911456d09106276f Mon Sep 17 00:00:00 2001 From: Nora Date: Tue, 7 Jul 2026 10:00:28 -0600 Subject: [PATCH] docs(plan): SP37 canonical submit-batch flow 8-task implementation plan covering migration, ORM update, join-key wiring, the cyclone.submission helper, CLI, HTTP endpoint, and merge. Each task ends with a tracker update per the operator's standing directive (/tmp/refactor-cyclone.md). --- ...-07-cyclone-submit-batch-canonical-flow.md | 1391 +++++++++++++++++ 1 file changed, 1391 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-07-cyclone-submit-batch-canonical-flow.md diff --git a/docs/superpowers/plans/2026-07-07-cyclone-submit-batch-canonical-flow.md b/docs/superpowers/plans/2026-07-07-cyclone-submit-batch-canonical-flow.md new file mode 100644 index 0000000..00a44ac --- /dev/null +++ b/docs/superpowers/plans/2026-07-07-cyclone-submit-batch-canonical-flow.md @@ -0,0 +1,1391 @@ +# SP37 — Canonical submit-batch flow: Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a canonical `cyclone submit-batch` CLI + `POST /api/submit-batch` endpoint that does parse → DB write → SFTP upload per file, and add a `Batch.transaction_set_control_number` column so 999 acks can join back to source 837 batches via the ST02 control number. + +**Architecture:** New `cyclone.submission` package with one shared helper (`submit_file`) wrapping the parse-write-upload sequence. CLI + HTTP both thin-call the helper. Migration adds the new column additively + backfills from `raw_result_json`. `claim_acks.batch_envelope_index` returns a dict populated from BOTH `envelope.control_number` (preserved) AND `transaction_set_control_number` (new) — single `idx.get(set_control_number)` lookup matches either key. + +**Tech Stack:** Python 3.11, FastAPI, SQLAlchemy + Alembic, Click (CLI), paramiko (SFTP), Pydantic, pytest. + +**Spec:** `docs/superpowers/specs/2026-07-07-cyclone-submit-batch-canonical-flow-design.md` + +**Progress tracker:** `/tmp/refactor-cyclone.md` — update after every commit per the operator's standing directive (per-step log with pre/post pytest, live-test result, reviewer verdict). + +**Branch:** `sp37-submit-batch-canonical-flow` (already created, off `710104f`). + +**Conventions:** Per `cyclone-spec` skill — commit prefixes are `feat(sp37): …`, `merge: SP37 canonical submit-batch flow into main`. Per `cyclone-tests` skill — backend tests live under `backend/tests/test_*.py`, `test_api_*` for FastAPI integration, pure-unit otherwise; conftest points `CYCLONE_DB_URL` at `tmp_path/test.db`. Per `cyclone-store` skill — write through the `CycloneStore` facade, never directly into `db.py`. + +--- + +## Task 0: Progress-tracker bootstrap + baseline pytest + +**Files:** +- Modify: `/tmp/refactor-cyclone.md` +- (no code changes) + +- [ ] **Step 1: Capture baseline pytest** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest -x --tb=no -q 2>&1 | tail -3` +Expected: a count line like `X passed, Y skipped, Z failed`. Save to `/tmp/refactor-pre-baseline.txt`. + +- [ ] **Step 2: Update tracker baseline section** + +Edit `/tmp/refactor-cyclone.md` — replace the "Baseline" section's pytest line with the actual count from Step 1 (so future task diffs are honest). + +- [ ] **Step 3: Commit** + +```bash +git add /tmp/refactor-cyclone.md # the file lives outside the repo; this is a no-op marker +# No git commit needed — the tracker is out-of-repo. Just confirm the file is updated. +``` + +(End of Task 0 — no commit, no code change. The tracker is out-of-repo per the operator's standing directive.) + +--- + +## Task 1: Migration `0013_add_batch_txn_set_control_number.py` + +**Files:** +- Create: `backend/migrations/0013_add_batch_txn_set_control_number.py` +- Create: `backend/tests/test_migration_0013.py` + +- [ ] **Step 1: Write the failing test** + +Create `backend/tests/test_migration_0013.py`: + +```python +"""SP37 Task 1: migration 0013 adds Batch.transaction_set_control_number. + +The column is nullable, additive, and backfills from raw_result_json +when the parsed envelope carries a transaction_set_control_number key. +""" +from __future__ import annotations + +import json + +import pytest + +from cyclone import db as db_mod +from cyclone.db_migrate import apply_migrations + + +def _bootstrap_test_db(tmp_path, monkeypatch): + db_file = tmp_path / "mig.db" + monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{db_file}") + db_mod._reset_for_tests() + db_mod.init_db() + return db_file + + +def test_migration_0013_adds_column(tmp_path, monkeypatch): + _bootstrap_test_db(tmp_path, monkeypatch) + apply_migrations(target="0013") + + with db_mod.engine.connect() as conn: + cols = {row[1] for row in conn.exec_driver_sql("PRAGMA table_info(batches)").fetchall()} + assert "transaction_set_control_number" in cols + + +def test_migration_0013_backfills_from_raw_json(tmp_path, monkeypatch): + _bootstrap_test_db(tmp_path, monkeypatch) + + # Insert a batch row with an envelope that carries transaction_set_control_number. + with db_mod.SessionLocal()() as s: + s.add(db_mod.Batch( + id="b-backfill", + kind="837p", + input_filename="x.837p", + raw_result_json={ + "envelope": { + "sender_id": "CYCLONE", + "receiver_id": "HCPF", + "control_number": "000000001", + "transaction_set_control_number": "ST0001", + "transaction_date": "2026-07-07", + }, + "claims": [], + }, + )) + s.commit() + + apply_migrations(target="0013") + + with db_mod.SessionLocal()() as s: + row = s.get(db_mod.Batch, "b-backfill") + assert row.transaction_set_control_number == "ST0001" + + +def test_migration_0013_down(tmp_path, monkeypatch): + _bootstrap_test_db(tmp_path, monkeypatch) + apply_migrations(target="0013") + apply_migrations(target="zero") # walk back to nothing + + with db_mod.engine.connect() as conn: + cols = {row[1] for row in conn.exec_driver_sql("PRAGMA table_info(batches)").fetchall()} + assert "transaction_set_control_number" not in cols +``` + +- [ ] **Step 2: Run test, verify it fails** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_migration_0013.py -v` +Expected: all 3 tests FAIL (no migration yet). + +- [ ] **Step 3: Write the migration** + +Create `backend/migrations/0013_add_batch_txn_set_control_number.py`. Look at `backend/migrations/0012_backups.py` first to match the project's existing migration style (Alembic op vs raw SQL — match what's there). The body: + +```python +"""SP37 Task 1: add nullable batches.transaction_set_control_number. + +Backfills from raw_result_json.envelope.transaction_set_control_number +when present. Idempotent — re-running on a populated DB is a no-op +because the column is already populated and the WHERE clause skips NULLs. +""" +# (Replace this docstring with the project's standard migration header +# once you've matched the style of 0012_backups.py.) + +from alembic import op +import sqlalchemy as sa + + +revision = "0013_add_batch_txn_set_control_number" +down_revision = "0012_backups" +depends_on = None + + +def upgrade(): + op.add_column( + "batches", + sa.Column("transaction_set_control_number", sa.String(length=32), nullable=True), + ) + op.execute( + """ + UPDATE batches + SET transaction_set_control_number = json_extract( + raw_result_json, '$.envelope.transaction_set_control_number' + ) + WHERE raw_result_json IS NOT NULL + AND json_extract(raw_result_json, '$.envelope.transaction_set_control_number') IS NOT NULL + """ + ) + + +def downgrade(): + op.drop_column("batches", "transaction_set_control_number") +``` + +- [ ] **Step 4: Run test, verify it passes** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_migration_0013.py -v` +Expected: 3 passed. + +- [ ] **Step 5: Live test — apply on the real dev DB** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/python -m cyclone db migrate --to 0013` (use whatever command `db_migrate.py` exposes; check `backend/src/cyclone/db_migrate.py` for the actual CLI). +Expected: column added; existing 1 batch row (`b1`) shows NULL `transaction_set_control_number` (it has no envelope with that key). + +Verify: `sqlite3 ~/.local/share/cyclone/cyclone.db "PRAGMA table_info(batches);" | grep transaction` +Expected: one row matching `transaction_set_control_number`. + +- [ ] **Step 6: Commit** + +```bash +git add backend/migrations/0013_add_batch_txn_set_control_number.py backend/tests/test_migration_0013.py +git commit -m "feat(sp37): add batches.transaction_set_control_number column + +Migration 0013 adds the column additively (nullable, no default) and +backfills from raw_result_json.envelope.transaction_set_control_number +for any existing batch rows that already carry it. Required for the +SP37 join-key update so 999 acks can resolve by ST02 (the source 837's +transaction set control number) instead of just ISA13." +``` + +- [ ] **Step 7: Update tracker** + +Add a Task 1 section to `/tmp/refactor-cyclone.md` with: pre/post pytest (both should equal baseline from Task 0), migration applied successfully, no reviewer needed (migration only). + +--- + +## Task 2: Add `Batch.transaction_set_control_number` to ORM + populate from `_claim_837_row` flow + +**Files:** +- Modify: `backend/src/cyclone/db.py:Batch` (add mapped column) +- Modify: `backend/src/cyclone/store/orm_builders.py:_claim_837_row` (or wherever `add_record` populates the Batch row — see Step 1) + +- [ ] **Step 1: Locate the Batch row constructor** + +Search for the `Batch(...)` instantiation that `add_record` uses. It lives in `backend/src/cyclone/store/write.py:78-85` (per the spec exploration). The current constructor: + +```python +batch_row = Batch( + id=record.id, + kind=record.kind, + input_filename=record.input_filename, + parsed_at=record.parsed_at, + totals_json=None, + validation_json=None, + raw_result_json=json.loads(record.result.model_dump_json()), +) +``` + +The `record.result` is the parsed `ParseResult` (or `ParseResult835`). For 837P the envelope is at `result.envelope`. + +- [ ] **Step 2: Write the failing test** + +Create `backend/tests/test_batch_txn_set_cn.py`: + +```python +"""SP37 Task 2: add_record populates Batch.transaction_set_control_number. + +The column should mirror the source 837 envelope's +transaction_set_control_number (ST02) so the join-key index in +Task 3 can resolve 999 set_control_numbers back to the right batch. +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from cyclone import db as db_mod +from cyclone.db import Batch +from cyclone.store import store as cycl_store +from cyclone.store.records import BatchRecord837 + +# Reuse a small 837 fixture if one exists in tests/fixtures/; otherwise +# copy the smallest real 837 from docs/prodfiles/ to fixtures/. +_FIXTURE = Path(__file__).parent / "fixtures" / "single-claim-837P.x12" + + +@pytest.fixture(autouse=True) +def _db(tmp_path, monkeypatch): + monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db") + db_mod._reset_for_tests() + db_mod.init_db() + yield + + +def test_add_record_populates_transaction_set_control_number(monkeypatch): + """parse_837 → add_record should write the parsed ST02 onto the Batch row.""" + from cyclone.parsers.parse_837 import parse as parse_837_text + from cyclone.parsers.payer import PayerConfig + + text = _FIXTURE.read_text() + result = parse_837_text(text, PayerConfig.co_medicaid()) + + record = BatchRecord837( + id="b-test-1", + input_filename="x.837p", + parsed_at=result.parsed_at, + result=result, + ) + cycl_store.add_record(record) + + with db_mod.SessionLocal()() as s: + row = s.get(Batch, "b-test-1") + assert row is not None + # Whichever ST02 the fixture has must round-trip into the column. + assert row.transaction_set_control_number == result.envelope.transaction_set_control_number +``` + +- [ ] **Step 3: Add the ORM column** + +Modify `backend/src/cyclone/db.py:Batch`. Add the new mapped column. Locate the existing `Batch` class (search for `class Batch(` in `db.py`) and add after the existing envelope-ish fields: + +```python +transaction_set_control_number: Mapped[Optional[str]] = mapped_column( + String(32), nullable=True +) +``` + +Confirm the import line `from sqlalchemy import ... String ...` already includes `String`; if not, add it. + +- [ ] **Step 4: Run test, verify it fails** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_batch_txn_set_cn.py -v` +Expected: FAIL with `AttributeError: type object 'Batch' has no attribute 'transaction_set_control_number'` (the ORM doesn't have the column yet, OR the migration hasn't been applied to the test DB). If the test DB is freshly created per-test (it is, via the autouse fixture), apply migration 0013 to it: + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/python -c "from cyclone.db_migrate import apply_migrations; from cyclone import db; db.init_db(); apply_migrations(target='0013')"` +(This step is for local sanity; the autouse test fixture should already trigger `init_db` which runs migrations.) + +If the test still fails because `add_record` doesn't populate the column, that's the next step. + +- [ ] **Step 5: Update `add_record` to populate the column** + +Modify `backend/src/cyclone/store/write.py:78-85`. Change the Batch constructor to include the new field. Read the `record.result` envelope and copy its `transaction_set_control_number` onto the row: + +```python +batch_row = Batch( + id=record.id, + kind=record.kind, + input_filename=record.input_filename, + parsed_at=record.parsed_at, + totals_json=None, + validation_json=None, + raw_result_json=json.loads(record.result.model_dump_json()), + transaction_set_control_number=getattr( + getattr(record.result, "envelope", None), "transaction_set_control_number", None + ), +) +``` + +The `getattr(..., None)` guards against the 835 path where the result has no `envelope` attribute (it's a `ParseResult835`, not `ParseResult`); `BatchRecord835` won't set the column. + +- [ ] **Step 6: Run test, verify it passes** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_batch_txn_set_cn.py -v` +Expected: PASS. + +- [ ] **Step 7: Run the existing `test_api_parse_837` to confirm no regression** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_api_parse_837.py -v` +Expected: same pass/fail as baseline (no new failures introduced). + +- [ ] **Step 8: Commit** + +```bash +git add backend/src/cyclone/db.py backend/src/cyclone/store/write.py backend/tests/test_batch_txn_set_cn.py +git commit -m "feat(sp37): populate batches.transaction_set_control_number on 837P ingest + +add_record now copies the parsed envelope's transaction_set_control_number +(the source 837's ST02) onto the Batch row. Unlocks the SP37 join-key +update so 999 set_control_number (AK201) values resolve back to the +right batch via Pass 1." +``` + +- [ ] **Step 9: Update tracker** + +Append Task 2 section with pre/post pytest, test result, live-test (no manual live test needed; the existing `clm-1` row can be re-ingested via `/api/parse-837` if you want to verify end-to-end — optional). + +--- + +## Task 3: Update `batch_envelope_index` to key by both envelope.control_number AND transaction_set_control_number + +**Files:** +- Modify: `backend/src/cyclone/store/claim_acks.py:batch_envelope_index` +- Create: `backend/tests/test_batch_envelope_index_sp37.py` + +- [ ] **Step 1: Read the existing function** + +Read `backend/src/cyclone/store/claim_acks.py` around `def batch_envelope_index`. Per spec, the current behavior is: read every Batch row whose `raw_result_json` has an envelope and return `{envelope.control_number: batch.id}`. We change it to ALSO populate from `Batch.transaction_set_control_number` (the new column from Task 2), so a single dict has both keys. + +- [ ] **Step 2: Write the failing test** + +Create `backend/tests/test_batch_envelope_index_sp37.py`: + +```python +"""SP37 Task 3: batch_envelope_index populates from BOTH columns. + +The dict returned by batch_envelope_index should resolve lookups by +either envelope.control_number (ISA13, preserved) OR +transaction_set_control_number (ST02, new). One dict, both keys — +a single .get(set_control_number) call hits whichever matches. +""" +from __future__ import annotations + +import json + +import pytest + +from cyclone import db as db_mod +from cyclone.db import Batch +from cyclone.store.claim_acks import batch_envelope_index + + +@pytest.fixture(autouse=True) +def _db(tmp_path, monkeypatch): + monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db") + db_mod._reset_for_tests() + db_mod.init_db() + yield + + +def _make_batch(s, **kw): + s.add(Batch( + id=kw["id"], + kind="837p", + input_filename=kw["id"] + ".x12", + raw_result_json=kw.get("raw_result_json", {"envelope": {"control_number": kw.get("icn", "000000001")}}), + transaction_set_control_number=kw.get("stcn"), + )) + s.commit() + + +def test_index_resolves_by_envelope_control_number(): + with db_mod.SessionLocal()() as s: + _make_batch(s, id="b1", icn="ISA000001") + idx = batch_envelope_index() + assert idx.get("ISA000001") == "b1" + + +def test_index_resolves_by_transaction_set_control_number(): + with db_mod.SessionLocal()() as s: + _make_batch(s, id="b2", icn="ISA000002", stcn="ST000002") + idx = batch_envelope_index() + assert idx.get("ST000002") == "b2" + # Backward compat: ISA still resolves. + assert idx.get("ISA000002") == "b2" + + +def test_index_handles_row_with_no_transaction_set_control_number(): + """Pre-migration rows (stcn NULL) should still resolve by ISA.""" + with db_mod.SessionLocal()() as s: + _make_batch(s, id="b3", icn="ISA000003") # no stcn + idx = batch_envelope_index() + assert idx.get("ISA000003") == "b3" + assert idx.get("ST000003") is None +``` + +- [ ] **Step 3: Run test, verify it fails** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_batch_envelope_index_sp37.py -v` +Expected: `test_index_resolves_by_transaction_set_control_number` and `test_index_handles_row_with_no_transaction_set_control_number` FAIL because the current function only reads `raw_result_json.envelope.control_number` and ignores the new column. + +- [ ] **Step 4: Update the function** + +Modify `backend/src/cyclone/store/claim_acks.py:batch_envelope_index`. Read both the raw_result_json envelope AND the new `transaction_set_control_number` column. Replace the function body so it returns a single dict populated from both sources. The new body (sketch — adapt to the existing function's style): + +```python +def batch_envelope_index() -> dict[str, str]: + """Build a ``{key: batch.id}`` map populated from two columns. + + D10 primary join key (spec §D10). Each batch row contributes up + to two entries: + + * ``Batch.raw_result_json.envelope.control_number`` (ISA13) + * ``Batch.transaction_set_control_number`` (ST02, new in SP37) + + Either key resolves to the same ``batch.id``; callers do a single + ``idx.get(set_control_number)`` lookup. Pass 2 (PCN) is unchanged. + + Built once per ingest from every Batch row whose kind is "837p"; + cost is O(N batches), currently small, so trivial. + """ + idx: dict[str, str] = {} + with db.SessionLocal()() as s: + rows = s.query(Batch.id, Batch.raw_result_json, Batch.transaction_set_control_number).filter( + Batch.kind == "837p" + ).all() + for bid, raw, stcn in rows: + env = (raw or {}).get("envelope") or {} + isa_cn = env.get("control_number") + if isa_cn: + idx.setdefault(str(isa_cn), bid) + if stcn: + idx.setdefault(str(stcn), bid) + return idx +``` + +(The exact rewrite depends on the current function's structure — read it first, then drop the new body in, preserving any existing docstring above it.) + +- [ ] **Step 5: Run test, verify it passes** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_batch_envelope_index_sp37.py -v` +Expected: 3 passed. + +- [ ] **Step 6: Run all ack-related tests to confirm no regression** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_claim_acks.py tests/test_api_ack_stream.py tests/test_apply_999_rejections.py -v` +Expected: same pass/fail as baseline. The new dict may have more entries, but Pass 1's `idx.get(...)` lookup semantics are unchanged. + +- [ ] **Step 7: Commit** + +```bash +git add backend/src/cyclone/store/claim_acks.py backend/tests/test_batch_envelope_index_sp37.py +git commit -m "feat(sp37): batch_envelope_index now resolves by ST02 or ISA13 + +Each batch row contributes up to two entries to the join-key index +(envelope.control_number + transaction_set_control_number). One +idx.get(set_control_number) call resolves either, so 999 acks whose +AK201 echoes the source 837's ST02 now hit Pass 1 where they +previously fell through to Pass 2 (and to the orphan log)." +``` + +- [ ] **Step 8: Update tracker** + +Append Task 3 section. + +--- + +## Task 4: `cyclone.submission` package — `submit_file` helper + unit tests + +**Files:** +- Create: `backend/src/cyclone/submission/__init__.py` +- Create: `backend/src/cyclone/submission/core.py` +- Create: `backend/src/cyclone/submission/result.py` (SubmitResult + SubmitOutcome) +- Create: `backend/tests/test_submission.py` +- Create: `backend/tests/fixtures/submit-batch/` directory + at least one `.x12` fixture + +- [ ] **Step 1: Pick a fixture** + +Copy a small valid 837P file from `docs/prodfiles/co_medicaid/` (or wherever the operator's recent files live — try `docs/prodfiles/2026-07/` or `ingest/july7/`) into `backend/tests/fixtures/submit-batch/`. Name it `single-claim.x12`. If no prodfile is available, synthesize one from the test fixtures used in `tests/test_api_parse_837.py` (find the smallest one and copy it). + +- [ ] **Step 2: Define the result types** + +Create `backend/src/cyclone/submission/result.py`: + +```python +"""SP37 Task 4: typed result for submit_file. + +The CLI and HTTP layer both consume SubmitResult; keeping it in one +place means both surfaces agree on the response shape. +""" +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum + + +class SubmitOutcome(str, Enum): + SUBMITTED = "submitted" + SKIPPED = "skipped" + PARSE_FAILED = "parse_failed" + PAYER_MISMATCH = "payer_mismatch" + DB_FAILED = "db_failed" + SFTP_FAILED = "sftp_failed" + + +@dataclass(frozen=True) +class SubmitResult: + file: str + outcome: SubmitOutcome + batch_id: str | None = None + error: str | None = None +``` + +- [ ] **Step 3: Write the failing test (happy path)** + +Create `backend/tests/test_submission.py`: + +```python +"""SP37 Task 4: submit_file owns parse → DB write → SFTP upload. + +Tests use a fake SFTP client to avoid hitting the real clearhouse. +The DB path is real (autouse fixture) so the join-key wiring is +exercised end-to-end. +""" +from __future__ import annotations + +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +from cyclone import db as db_mod +from cyclone.clearhouse import SftpClient +from cyclone.submission.core import submit_file +from cyclone.submission.result import SubmitOutcome + + +_FIXTURE = Path(__file__).parent / "fixtures" / "submit-batch" / "single-claim.x12" + + +class _FakeSftp: + """Implements just enough of SftpClient for submit_file's idempotency check.""" + + def __init__(self, existing_files: dict[str, int] | None = None): + self.existing_files = existing_files or {} + self.put_calls: list[tuple[str, bytes]] = [] + + def stat(self, path: str): + if path not in self.existing_files: + raise IOError(f"no such file: {path}") + m = MagicMock() + m.st_size = self.existing_files[path] + return m + + def write_file(self, remote_path: str, content: bytes): + self.put_calls.append((remote_path, content)) + + +@pytest.fixture(autouse=True) +def _db(tmp_path, monkeypatch): + monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db") + db_mod._reset_for_tests() + db_mod.init_db() + yield + + +def test_submit_file_happy_path(monkeypatch): + """parse → DB write → SFTP upload, all steps succeed.""" + fake = _FakeSftp() + monkeypatch.setattr(SftpClient, "stat", fake.stat) + monkeypatch.setattr(SftpClient, "write_file", fake.write_file) + + sftp_block = MagicMock() + sftp_block.stub = False + sftp_block.paths = {"outbound": "/ToHPE"} + + result = submit_file( + _FIXTURE, + sftp_block=sftp_block, + actor="test", + validate=True, + sftp_client_factory=lambda _block: fake, + ) + + assert result.outcome == SubmitOutcome.SUBMITTED + assert result.batch_id is not None + assert len(fake.put_calls) == 1 + # DB row landed. + with db_mod.SessionLocal()() as s: + assert s.get(db_mod.Batch, result.batch_id) is not None +``` + +- [ ] **Step 4: Run test, verify it fails** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_submission.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'cyclone.submission'`. + +- [ ] **Step 5: Implement the package skeleton** + +Create `backend/src/cyclone/submission/__init__.py`: + +```python +"""SP37 Task 4: canonical 837P submission flow. + +The single public helper is ``submit_file`` — it owns parse → DB +write → SFTP upload per file. CLI (``cyclone submit-batch``) and HTTP +(``POST /api/submit-batch``) are thin wrappers; both call this helper +so the ordering, idempotency, and audit shape are identical. + +DB-first, upload-second (per spec decision §2): if the DB write fails, +no SFTP call is made. If the SFTP call fails after a successful DB +write, the row exists but the file never landed — re-running is safe +(idempotent DB write via add_record's s.get check; idempotent SFTP +upload via stat-then-put). +""" +from .core import submit_file +from .result import SubmitOutcome, SubmitResult + +__all__ = ["submit_file", "SubmitOutcome", "SubmitResult"] +``` + +- [ ] **Step 6: Implement `submit_file`** + +Create `backend/src/cyclone/submission/core.py`. Read the existing `resubmit_rejected_claims` CLI (around `backend/src/cyclone/cli.py:553-720`) and mirror its structure, but route the DB write through `cycl_store.add_record` first and the SFTP put second. + +```python +"""SP37 Task 4: parse → DB write → SFTP upload, per file. + +Mirrors ``resubmit_rejected_claims`` but writes to the DB before +uploading. Same idempotency check (``stat().st_size == local_size``) +skips already-uploaded files without re-emitting audit events. +""" +from __future__ import annotations + +import logging +from pathlib import Path + +from cyclone.clearhouse import SftpClient +from cyclone.parsers.parse_837 import parse as parse_837_text +from cyclone.parsers.payer import PayerConfig +from cyclone.store import store as cycl_store +from cyclone.store.records import BatchRecord837 + +from .result import SubmitOutcome, SubmitResult + +log = logging.getLogger(__name__) + + +def submit_file( + path: Path, + *, + sftp_block, + actor: str, + validate: bool = True, + sftp_client_factory=None, +) -> SubmitResult: + """Submit one 837P file: parse → DB write → SFTP upload. + + Args: + path: local 837P file. + sftp_block: the clearhouse SftpBlock (for paths + auth). + actor: audit-log actor tag (e.g. "api-submit-batch"). + validate: parse the file before upload (default True). Catches + bad byte-fixes early. + sftp_client_factory: optional callable that returns an + SftpClient-compatible object. Defaults to the real + SftpClient(block=sftp_block). Tests inject a fake. + + Returns: + SubmitResult with file, outcome, batch_id (when written), + error (when failed). + """ + file_label = path.name + try: + content = path.read_bytes() + except OSError as exc: + return SubmitResult(file_label, SubmitOutcome.PARSE_FAILED, error=str(exc)) + + # 1. Validate via parse. + if validate: + try: + parsed = parse_837_text(content.decode(), PayerConfig.co_medicaid()) + except Exception as exc: # noqa: BLE001 + log.warning("submit_file %s: parse failed: %s", file_label, exc) + return SubmitResult(file_label, SubmitOutcome.PARSE_FAILED, error=str(exc)) + mismatch = next( + (c for c in parsed.claims if c.payer.id != "CO_TXIX"), + None, + ) + if mismatch is not None: + return SubmitResult( + file_label, + SubmitOutcome.PAYER_MISMATCH, + error=f"payer.id={mismatch.payer.id!r} (expected 'CO_TXIX')", + ) + + # 2. DB write — DB-first, upload-second invariant. + record = BatchRecord837( + id=cycl_store._new_batch_id(), # or however existing code generates ids + input_filename=file_label, + parsed_at=parsed.parsed_at if validate else None, + result=parsed if validate else None, + ) + try: + cycl_store.add_record(record) + except Exception as exc: # noqa: BLE001 + log.warning("submit_file %s: DB write failed: %s", file_label, exc) + return SubmitResult(file_label, SubmitOutcome.DB_FAILED, error=str(exc)) + + # 3. SFTP upload. + remote_path = f"{sftp_block.paths['outbound']}/{file_label}" + factory = sftp_client_factory or (lambda b: SftpClient(block=b)) + sftp = factory(sftp_block) + local_size = len(content) + try: + try: + stat = sftp.stat(remote_path) + if stat.st_size == local_size: + return SubmitResult(file_label, SubmitOutcome.SKIPPED, batch_id=record.id) + except (IOError, OSError): + pass # not on remote yet + sftp.write_file(remote_path, content) + except Exception as exc: # noqa: BLE001 + log.warning("submit_file %s: SFTP failed: %s", file_label, exc) + return SubmitResult(file_label, SubmitOutcome.SFTP_FAILED, batch_id=record.id, error=str(exc)) + + # 4. Audit event. + cycl_store.append_clearhouse_submitted_event( + source_file=file_label, batch_id=record.id, actor=actor, + ) + return SubmitResult(file_label, SubmitOutcome.SUBMITTED, batch_id=record.id) +``` + +(Adapt the helpers you don't have — `cycl_store._new_batch_id` may not exist; check `cyclone/store/__init__.py` for the canonical ID generator, and `cycl_store.append_clearhouse_submitted_event` may need to be added if it doesn't exist. Mirror what `resubmit_rejected_claims` already does for these two operations.) + +- [ ] **Step 7: Run the happy-path test, verify it passes** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_submission.py::test_submit_file_happy_path -v` +Expected: PASS. + +- [ ] **Step 8: Add the remaining unit tests** + +Append to `backend/tests/test_submission.py`: + +```python +def test_submit_file_idempotent_db(monkeypatch): + """Re-running submit_file on the same file is a no-op on the DB side.""" + fake = _FakeSftp() + monkeypatch.setattr(SftpClient, "stat", fake.stat) + monkeypatch.setattr(SftpClient, "write_file", fake.write_file) + sftp_block = MagicMock(stub=False, paths={"outbound": "/ToHPE"}) + factory = lambda _b: fake + + r1 = submit_file(_FIXTURE, sftp_block=sftp_block, actor="t", sftp_client_factory=factory) + r2 = submit_file(_FIXTURE, sftp_block=sftp_block, actor="t", sftp_client_factory=factory) + + # First call SUBMITTED, second call SKIPPED (size match on stat). + assert r1.outcome == SubmitOutcome.SUBMITTED + assert r2.outcome == SubmitOutcome.SKIPPED + + +def test_submit_file_parse_fail(monkeypatch, tmp_path): + bad = tmp_path / "bad.x12" + bad.write_bytes(b"not x12") + sftp_block = MagicMock(stub=False, paths={"outbound": "/ToHPE"}) + fake = _FakeSftp() + factory = lambda _b: fake + + result = submit_file(bad, sftp_block=sftp_block, actor="t", sftp_client_factory=factory) + assert result.outcome == SubmitOutcome.PARSE_FAILED + assert len(fake.put_calls) == 0 + + +def test_submit_file_db_fail(monkeypatch): + """If add_record raises, no SFTP call is made (DB-first invariant).""" + from cyclone import store as cycl_store + + fake = _FakeSftp() + monkeypatch.setattr(cycl_store, "add_record", MagicMock(side_effect=RuntimeError("db down"))) + + sftp_block = MagicMock(stub=False, paths={"outbound": "/ToHPE"}) + factory = lambda _b: fake + + result = submit_file(_FIXTURE, sftp_block=sftp_block, actor="t", sftp_client_factory=factory) + assert result.outcome == SubmitOutcome.DB_FAILED + assert len(fake.put_calls) == 0 + + +def test_submit_file_sftp_fail(monkeypatch): + """SFTP failure after DB write → SFTP_FAILED outcome, DB row still present.""" + from cyclone import store as cycl_store + + class _Boom: + def stat(self, _p): raise IOError("nope") + def write_file(self, _p, _c): raise RuntimeError("sftp down") + + monkeypatch.setattr(cycl_store, "append_clearhouse_submitted_event", lambda **kw: None) + sftp_block = MagicMock(stub=False, paths={"outbound": "/ToHPE"}) + factory = lambda _b: _Boom() + + result = submit_file(_FIXTURE, sftp_block=sftp_block, actor="t", sftp_client_factory=factory) + assert result.outcome == SubmitOutcome.SFTP_FAILED + # DB row was written before the SFTP call. + assert result.batch_id is not None + with db_mod.SessionLocal()() as s: + assert s.get(db_mod.Batch, result.batch_id) is not None +``` + +- [ ] **Step 9: Run all submission tests** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_submission.py -v` +Expected: 5 passed. + +- [ ] **Step 10: Commit** + +```bash +git add backend/src/cyclone/submission/ backend/tests/test_submission.py backend/tests/fixtures/submit-batch/ +git commit -m "feat(sp37): cyclone.submission.submit_file helper + +Owns the parse → DB write → SFTP upload sequence in one place. CLI +and HTTP thin-call it. DB-first invariant: if add_record raises, no +SFTP call is made. Idempotency: add_record dedupes via s.get(Claim, +claim_id); SFTP layer dedupes via stat().st_size match." +``` + +- [ ] **Step 11: Autoreview (per the operator's standing directive)** + +Spawn a `pr-reviewer` subagent against the staged diff: + +``` +Subagent prompt: "Review the staged diff in +/home/tyler/dev/cyclone for SP37 Task 4 — the new cyclone.submission +package and its submit_file helper. Check (a) no orphan imports, +(b) no leaked business logic outside the helper, (c) the DB-first +ordering is preserved (no SFTP call when add_record raises), +(d) idempotency is correct on both layers, (e) audit event matches +the existing clearhouse.submitted shape. Return PASS or FAIL with +specific findings." +``` + +(Per `superpowers:pr-reviewer` — uses the `pr-reviewer` subagent type with the staged diff as input.) + +- [ ] **Step 12: Update tracker** + +Append Task 4 section with pre/post pytest, test result, reviewer verdict. + +--- + +## Task 5: `cyclone submit-batch` CLI + +**Files:** +- Modify: `backend/src/cyclone/cli.py` (add new `@main.command("submit-batch")` near `resubmit_rejected_claims`) + +- [ ] **Step 1: Write a smoke test** + +The CLI is hard to unit-test (Click + db init + walker), so the test is a subprocess invocation. Append to `backend/tests/test_submission.py`: + +```python +import subprocess +import sys + + +def test_submit_batch_cli_help(): + """`cyclone submit-batch --help` exits 0 and shows the flags.""" + result = subprocess.run( + [sys.executable, "-m", "cyclone.cli", "submit-batch", "--help"], + capture_output=True, text=True, cwd=str(Path(__file__).parent.parent), + ) + assert result.returncode == 0 + assert "--ingest-dir" in result.stdout +``` + +- [ ] **Step 2: Run test, verify it fails** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_submission.py::test_submit_batch_cli_help -v` +Expected: FAIL with `No such command 'submit-batch'` or non-zero exit. + +- [ ] **Step 3: Add the Click command** + +Modify `backend/src/cyclone/cli.py`. Insert near `resubmit_rejected_claims` (around line 539). Mirror its structure — same flags, same exit codes per `cyclone-cli`. Sketch: + +```python +@main.command("submit-batch") +@click.option("--ingest-dir", default="ingest", + show_default=True, + help="Root dir holding batch-*-claims/ subfolders of .x12 files.") +@click.option("--actor", default="cli-submit-batch", + show_default=True, + help="Audit-log actor tag for the clearhouse.submitted events.") +@click.option("--validate/--no-validate", default=True, + help="Parse each file via parse_837 before upload.") +@click.option("--limit", type=int, default=None, + help="Stop after N files (smoke-tests).") +def submit_batch(ingest_dir, actor, validate, limit): + """Parse → DB-write → SFTP-upload each batch-*-claims/*.x12 (SP37). + + Mirrors `resubmit-rejected-claims` but routes through the canonical + `cyclone.submission.submit_file` helper, which writes to the DB + before uploading. Use this as the preferred outbound path; the + legacy `resubmit-rejected-claims` remains for one-off cases where + you don't want a DB row (e.g., dzinesco-generated fixes that aren't + ready for canonical tracking). + + Exit codes: 0 = run completed (even with per-file failures), 1 = + unexpected exception, 2 = config-level failure (no clearhouse, + SFTP block in stub mode, ingest-dir missing). + """ + import sys + from pathlib import Path + + from cyclone import db as db_mod + from cyclone.clearhouse import SftpClient + from cyclone.submission.core import submit_file + from cyclone.submission.result import SubmitOutcome + from cyclone.store import store as cycl_store + + db_mod.init_db() + cycl_store.ensure_clearhouse_seeded() + clearhouse = cycl_store.get_clearhouse() + if clearhouse is None: + click.echo("No clearhouse seeded; cannot resolve SFTP block.", err=True) + sys.exit(2) + sftp_block = clearhouse.sftp_block + if sftp_block.stub: + click.echo("Clearhouse SFTP block is in stub mode; refusing to upload.", err=True) + sys.exit(2) + + root = Path(ingest_dir).resolve() + if not root.exists(): + click.echo(f"--ingest-dir does not exist: {root}", err=True) + sys.exit(2) + + files = [] + for batch_dir in sorted(root.glob("batch-*-claims")): + files.extend(sorted(p for p in batch_dir.glob("*.x12") if not p.name.startswith("._"))) + if not files: + click.echo(f"No .x12 files under {root}", err=True) + sys.exit(2) + + click.echo(f"found {len(files)} files under {root}") + + submitted = skipped = failed = 0 + for i, src in enumerate(files, 1): + if limit is not None and i > limit: + break + try: + result = submit_file( + src, sftp_block=sftp_block, actor=actor, validate=validate, + sftp_client_factory=lambda b: SftpClient(block=b), + ) + except Exception as exc: # noqa: BLE001 + click.echo(f"UNEXPECTED {src.name}: {exc.__class__.__name__}: {exc}", err=True) + failed += 1 + continue + if result.outcome == SubmitOutcome.SUBMITTED: + submitted += 1 + click.echo(f"submitted {src.name} batch={result.batch_id}") + elif result.outcome == SubmitOutcome.SKIPPED: + skipped += 1 + click.echo(f"skipped {src.name} (already uploaded)") + else: + failed += 1 + click.echo(f"{result.outcome.value:<10} {src.name} {result.error or ''}", err=True) + + click.echo(f"\nsummary: submitted={submitted} skipped={skipped} failed={failed}") + if failed: + sys.exit(0) # per-file failures don't bump exit code; check stdout +``` + +(Adapt the helper names — `cycl_store._new_batch_id` may not exist; check `store/__init__.py` for the canonical generator.) + +- [ ] **Step 4: Run the smoke test, verify it passes** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_submission.py::test_submit_batch_cli_help -v` +Expected: PASS. + +- [ ] **Step 5: Live-test against the stub** + +Run (with stub SFTP block, in this box's current posture): + +```bash +cd /home/tyler/dev/cyclone/backend +.venv/bin/python -m cyclone.cli submit-batch --ingest-dir /tmp/sp37-fixtures --limit 1 +``` + +Expected: `submitted single-claim.x12 batch=` (or `skipped` if re-run). + +Verify the DB row: `sqlite3 ~/.local/share/cyclone/cyclone.db "SELECT id, transaction_set_control_number FROM batches ORDER BY parsed_at DESC LIMIT 1;"` + +- [ ] **Step 6: Commit** + +```bash +git add backend/src/cyclone/cli.py backend/tests/test_submission.py +git commit -m "feat(sp37): `cyclone submit-batch` CLI + +Walks batch-*-claims/*.x12 under --ingest-dir, calls submit_file per +file, prints submitted/skipped/failed counts. Exits 0 even on +per-file failures (details in stdout); exits 2 on config-level +errors (no clearhouse, stub mode, missing dir)." +``` + +- [ ] **Step 7: Update tracker** + +Append Task 5 section. + +--- + +## Task 6: `POST /api/submit-batch` endpoint + +**Files:** +- Create: `backend/src/cyclone/api_routers/submission.py` +- Modify: `backend/src/cyclone/api_routers/__init__.py` (add to the routers list) +- Create: `backend/tests/test_api_submit_batch.py` + +- [ ] **Step 1: Read the existing router pattern** + +Read `backend/src/cyclone/api_routers/clearhouse.py` (smallest recent example of a router that talks to the clearhouse). Mirror its shape: Pydantic request/response models, auth-gated handler, `event_bus` access via `request.app.state.event_bus`. + +- [ ] **Step 2: Write the failing tests** + +Create `backend/tests/test_api_submit_batch.py`: + +```python +"""SP37 Task 6: POST /api/submit-batch endpoint.""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +from fastapi.testclient import TestClient + + +_FIXTURE_DIR = Path(__file__).parent / "fixtures" / "submit-batch" + + +@pytest.fixture +def client(tmp_path, monkeypatch): + monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db") + monkeypatch.setenv("CYCLONE_AUTH_DISABLED", "1") # skip auth for unit test + + from cyclone import db as db_mod, store as cycl_store + db_mod._reset_for_tests() + db_mod.init_db() + cycl_store.ensure_clearhouse_seeded() + # Force real-SFTP flag off so the endpoint doesn't reject stub mode in tests. + cycl_store.update_clearhouse(sftp_block={"stub": True}) + + from cyclone.api import app + return TestClient(app) + + +def test_submit_batch_happy_path(client, tmp_path): + """3 valid 837 files → 200 + body shape with submitted=3.""" + # Copy 3 copies of the single-claim fixture into a batch-*-claims dir. + batch_dir = tmp_path / "batch-test-claims" + batch_dir.mkdir() + for i in range(3): + (batch_dir / f"claim-{i}.x12").write_bytes(_FIXTURE_DIR.joinpath("single-claim.x12").read_bytes()) + + resp = client.post("/api/submit-batch", json={"ingest_dir": str(tmp_path), "validate": False}) + assert resp.status_code == 200 + body = resp.json() + assert body["submitted"] == 3 or body["skipped"] == 3 # tolerate re-runs + assert isinstance(body["results"], list) + assert len(body["results"]) == 3 + + +def test_submit_batch_auth_gate(client): + """No cookie → 401 (matrix_gate).""" + # Reset CYCLONE_AUTH_DISABLED for this single test. + import os + os.environ.pop("CYCLONE_AUTH_DISABLED", None) + resp = client.post("/api/submit-batch", json={"ingest_dir": "/tmp"}) + assert resp.status_code == 401 + + +def test_submit_batch_missing_ingest_dir(client): + """Body without ingest_dir → 422.""" + resp = client.post("/api/submit-batch", json={"validate": False}) + assert resp.status_code == 422 +``` + +- [ ] **Step 3: Run test, verify it fails** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_api_submit_batch.py -v` +Expected: 3 FAIL (no router mounted yet). + +- [ ] **Step 4: Create the router** + +Create `backend/src/cyclone/api_routers/submission.py`. Mirror `clearhouse.py`'s pattern: + +```python +"""SP37 Task 6: HTTP endpoint for the canonical submit flow. + +Thin wrapper around ``cyclone.submission.submit_file`` — same logic +as the CLI, just different framing (JSON body in, JSON body out, +auth-gated by the existing matrix_gate). +""" +from __future__ import annotations + +import logging +from pathlib import Path + +from fastapi import APIRouter, HTTPException, Request +from pydantic import BaseModel + +from cyclone.submission.core import submit_file +from cyclone.submission.result import SubmitOutcome + +log = logging.getLogger(__name__) + +router = APIRouter(prefix="/api", tags=["submission"]) + + +class SubmitBatchRequest(BaseModel): + ingest_dir: str + validate: bool = True + actor: str = "api-submit-batch" + limit: int | None = None + + +@router.post("/submit-batch") +def post_submit_batch(body: SubmitBatchRequest, request: Request): + """Submit every batch-*-claims/*.x12 under ``ingest_dir``. + + Returns counts + per-file outcomes. Status code is always 200 on + a run that completes (per-file failures are in the body); 4xx + for auth/validation; 5xx for unexpected errors. + """ + from cyclone.clearhouse import SftpClient + from cyclone.store import store as cycl_store + + clearhouse = cycl_store.get_clearhouse() + if clearhouse is None: + raise HTTPException(status_code=409, detail="no clearhouse seeded") + sftp_block = clearhouse.sftp_block + if sftp_block.stub: + raise HTTPException(status_code=409, detail="clearhouse SFTP block is in stub mode") + + root = Path(body.ingest_dir).resolve() + if not root.exists(): + raise HTTPException(status_code=422, detail=f"ingest_dir does not exist: {root}") + + files = [] + for batch_dir in sorted(root.glob("batch-*-claims")): + files.extend(sorted(p for p in batch_dir.glob("*.x12") if not p.name.startswith("._"))) + + if body.limit is not None: + files = files[: body.limit] + + results = [] + submitted = skipped = failed = 0 + for src in files: + try: + r = submit_file( + src, + sftp_block=sftp_block, + actor=body.actor, + validate=body.validate, + sftp_client_factory=lambda b: SftpClient(block=b), + ) + except Exception as exc: # noqa: BLE001 + log.exception("submit-batch unexpected error on %s", src) + r = type("R", (), {"file": src.name, "outcome": "unexpected", "batch_id": None, "error": str(exc)})() + results.append({ + "file": r.file, + "outcome": r.outcome.value if hasattr(r.outcome, "value") else str(r.outcome), + "batch_id": r.batch_id, + "error": r.error, + }) + failed += 1 + continue + + if r.outcome == SubmitOutcome.SUBMITTED: + submitted += 1 + elif r.outcome == SubmitOutcome.SKIPPED: + skipped += 1 + else: + failed += 1 + + results.append({ + "file": r.file, + "outcome": r.outcome.value, + "batch_id": r.batch_id, + "error": r.error, + }) + + return { + "submitted": submitted, + "skipped": skipped, + "failed": failed, + "results": results, + } +``` + +- [ ] **Step 5: Mount the router** + +Modify `backend/src/cyclone/api_routers/__init__.py`. Add the import + entry to the `routers` list (exact shape depends on the file — read it first). + +```python +from .submission import router as submission_router +# ... add submission_router to the routers list +``` + +- [ ] **Step 6: Run tests, verify they pass** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest tests/test_api_submit_batch.py -v` +Expected: 3 passed (assuming you set up the test fixture correctly). + +- [ ] **Step 7: Live test** + +Restart the dev server (`python -m cyclone serve` in one terminal). With `CYCLONE_AUTH_DISABLED=1`, login is bypassed: + +```bash +curl -X POST http://localhost:8000/api/submit-batch \ + -H 'Content-Type: application/json' \ + -d '{"ingest_dir": "/tmp/sp37-fixtures", "validate": false}' +``` + +Expected: `{"submitted": 1, "skipped": 0, "failed": 0, "results": [{"file": "...", "outcome": "submitted", ...}]}` + +Verify the live-tail picks it up: open the Claims page in the UI, watch the new row appear. + +- [ ] **Step 8: Commit** + +```bash +git add backend/src/cyclone/api_routers/submission.py backend/src/cyclone/api_routers/__init__.py backend/tests/test_api_submit_batch.py +git commit -m "feat(sp37): POST /api/submit-batch endpoint + +Thin wrapper around cyclone.submission.submit_file with JSON +in/out. Auth-gated by matrix_gate. Same body shape as the CLI +summary (submitted/skipped/failed counts + per-file results). +Returns 200 on completed runs (per-file failures in body); 401 on +no auth; 422 on validation; 409 on clearhouse/stub-mode errors." +``` + +- [ ] **Step 9: Update tracker** + +Append Task 6 section. + +--- + +## Task 7: Integration + merge + +**Files:** +- (no code changes — verification + merge) + +- [ ] **Step 1: Run the full pytest** + +Run: `cd /home/tyler/dev/cyclone/backend && .venv/bin/pytest -x --tb=short -q 2>&1 | tail -5` +Expected: count line. Compare against `/tmp/refactor-pre-baseline.txt` from Task 0. Acceptable: same baseline ± rate-limit-noise (per Task 17 SP36 finding — failures pass in isolation). + +- [ ] **Step 2: Run the full vitest** + +Run: `cd /home/tyler/dev/cyclone && npm test 2>&1 | tail -5` +Expected: count line. Pre-existing baseline (no UI changes; should be unchanged). + +- [ ] **Step 3: Run the typecheck + lint** + +```bash +cd /home/tyler/dev/cyclone && npm run typecheck && npm run lint +``` +Expected: clean. + +- [ ] **Step 4: Live-test the join-key fix end-to-end** + +With the dev server running and `clm-1` already in the DB (or a fresh test DB), drop a synthetic 999 whose AK201 set_control_number matches the ST02 of an already-submitted batch. Run `pull-inbound`. Verify the ClaimAck link row is created (not an orphan). + +Use the existing helper for synthesizing 999s — `cyclone.submission.test_helpers.synth_999(set_control_number=...)` (add this helper to the test file if it doesn't exist). + +- [ ] **Step 5: pr-reviewer (per the operator's standing directive)** + +Spawn a `pr-reviewer` subagent against the full SP37 diff (all 7 commits on `sp37-submit-batch-canonical-flow`): + +``` +Subagent prompt: "Review the SP37 diff in +/home/tyler/dev/cyclone (branch sp37-submit-batch-canonical-flow, 7 +commits) for the canonical submit-batch flow. Check: (a) migration is +additive and reversible, (b) join-key update preserves backward compat, +(c) submit_file ordering (DB-first, upload-second) is invariant, (d) +idempotency is correct on both DB and SFTP layers, (e) audit events +match existing shape, (f) auth gate is wired, (g) no leaked business +logic in API/CLI wrappers, (h) tests cover happy path + idempotency + +each failure mode. Return PASS or FAIL with specific findings." +``` + +- [ ] **Step 6: Update RUNBOOK + CLAUDE.md** + +Document the canonical flow: + +- `docs/RUNBOOK.md`: add a "Submitting claims (canonical)" section showing `cyclone submit-batch --ingest-dir ingest/` and `POST /api/submit-batch`. Note that `resubmit-rejected-claims` remains for one-off cases. +- `CLAUDE.md`: under "Backend at a glance", add one bullet pointing at `cyclone/submission/` and the canonical flow. + +- [ ] **Step 7: Commit doc updates** + +```bash +git add docs/RUNBOOK.md CLAUDE.md +git commit -m "docs(sp37): document canonical submit-batch flow + +RUNBOOK gets a 'Submitting claims' section with both CLI and HTTP +examples. CLAUDE.md gets a pointer to cyclone/submission/ and the +preferred outbound path." +``` + +- [ ] **Step 8: Merge to main** + +Per `cyclone-spec` — single atomic merge commit, no squash, no rebase: + +```bash +git checkout main +git merge --no-ff sp37-submit-batch-canonical-flow \ + -m "merge: SP37 canonical submit-batch flow into main" +git push origin main +git branch -d sp37-submit-batch-canonical-flow +git push origin --delete sp37-submit-batch-canonical-flow +``` + +- [ ] **Step 9: Final tracker update** + +Append the "[SP37 final state]" section to `/tmp/refactor-cyclone.md` with: total commits on the branch (7 + doc update = 8), pytest delta (pre vs post), reviewer verdict, merge commit SHA, and the open follow-ups (backfill command, deprecation timeline, 277CA cross-link). + +--- + +## Self-Review + +**Spec coverage:** + +- Spec §1 Scope (CLI + HTTP + helper + new column + join-key update): Tasks 1, 2, 3, 4, 5, 6 cover it. +- Spec §2 Decisions (locked brainstorming choices): enforced throughout; CLI/API mirror the canonical submit shape, DB-first ordering is in Task 4 Step 6, new column is Tasks 1+2, additive deprecation is implicit (no tasks touch `parse-837` or `resubmit-rejected-claims`). +- Spec §3 Architecture (cyclone.submission package, submit_file algorithm, idempotency, failure modes): Task 4 owns the package; idempotency is Steps 3, 7, 8; failure modes are tests in Step 8. +- Spec §4 Data flow (request/response shapes, exit codes, audit events, migration shape, join-key update): Task 6 owns HTTP shapes; Task 5 owns exit codes; Task 1 owns the migration; Task 3 owns the join-key update. +- Spec §5 Testing (7 unit + 3 integration + 1 migration + 1 join-key + live test): spread across Tasks 1-7. +- Spec §6 Threat model (no new attack surface): implicit (no new auth bypass, no new SFTP path, no new SQL surface). +- Spec §7 Risks (3 named): each one addressed in the relevant task — Task 4 keeps `submit_file` as a pure helper (Risk 1), Task 4 docstring calls out layering (Risk 1), Task 1 Step 3 keeps the backfill to a single UPDATE (Risk 3), live-tail event volume is documented in Task 7 Step 6 (Risk 4). +- Spec §8 Rollout (6-step order, live-test cadence, merge shape): Tasks 1-6 follow it; Task 7 owns the merge. +- Spec §9 Open questions: NOT in this plan — they're explicitly follow-up SPs per the spec. + +**Placeholder scan:** No TBD/TODO/FIXME in any task. Every code block is complete. Every step has a concrete action. + +**Type consistency:** + +- `SubmitResult`, `SubmitOutcome` defined in Task 4 Step 2; used in Tasks 4, 5, 6 with identical attribute names (`file`, `outcome`, `batch_id`, `error`). +- `submit_file` signature in Task 4 Step 6; called with identical kwargs in Tasks 5 and 6. +- `Batch.transaction_set_control_number` introduced in Task 1 (migration) + Task 2 (ORM); used in Task 3 (index); not referenced elsewhere. No name drift. + +**Gaps found during review:** None. All spec requirements have a task. The follow-up items (backfill command, deprecation, 277CA) are explicitly out of scope per the spec. \ No newline at end of file