# Sub-project 22 — Cyclone Pipeline Agent: Design Spec **Date:** 2026-06-21 **Status:** Draft (awaiting user review) **Branch:** `sp22-pipeline-agent` (not yet created; will branch from `main`) **Aesthetic direction:** n/a (backend automation; no UI surface) ## 1. Scope A production-grade agent (sibling Python package `cyclone-pipeline`) that automates the full submission lifecycle of an EDI 837P file through Cyclone to the Gainwell MFT, and watches for the inbound TA1 + 999 ACKs that confirm acceptance. The agent exists so a single human operator can drop a file in and walk away; the agent handles the browser upload, SFTP submission, ACK polling, and report generation end to end, and writes a self-contained run directory that OpenClaw / Nora / any other automation can read. **In scope:** - Sibling Python package at `/Users/openclaw/dev/cyclone-pipeline/` (no shared imports with the cyclone app; talks to cyclone exclusively over HTTP). - CLI surface: `run `, `run-batch `, `status `, `resume `, `check-835 `. - Library surface: `CyclonePipeline` class with `run()`, `run_batch()`, `resume()`, `check_835()` for embedding in OpenClaw / Nora. - 7-phase state machine: pre-flight → upload (browser) → verify parse → SFTP submit → wait TA1 → wait 999 → scan HTML + report. The "scan HTML" and "final report" steps are combined as phase 7 (the HTML scan populates the report; the report write is phase 7's postcondition). - Per-run output folder with `report.{md,json}`, `run.log`, `run.state.json`, and `screenshots/` with stable filenames. - Crash-safe resume: `run.state.json` is written after each phase's postcondition; `resume` skips completed phases. - Headless + visible browser modes; default visible so screenshots are useful for debugging. - Defensive reliability: typed exceptions, 3× retry with backoff on idempotent operations, timeouts per phase, screenshots on failure, Keychain-failure guidance. - Test suite: `pytest` + `pytest-asyncio` + `respx` (httpx mock) + `pytest-playwright` for the small browser surface. Non-browser tests run on every PR; Playwright tests gated to `main`. **Out of scope (deferred):** - **835 waiting within a single run.** CO Medicaid 835 remittances land the following Monday, on the payment cycle. A run that blocks for up to 7 days is impractical. The agent defers 835 detection to a separate `check-835 ` subcommand that the operator runs on/after Monday. The report always carries the `check-835` recipe. - **277CA polling.** CO Medicaid does not use 277CA claim-level ACKs; the agent never polls for them. (The `GET /api/277ca-acks` endpoint exists in the API surface but is not exercised by this agent.) - **Watch-folder / daemon mode.** The agent is on-demand. A cron job that invokes `cyclone-pipeline run-batch /inbox/*.txt --strict` on a 5-min schedule achieves the same effect with no daemon in the loop. - **Multi-payer support.** The agent targets CO Medicaid (the only configured payer in `config/payers.yaml`). A new payer means new payer-specific regex / file-naming / SFTP block — out of scope for v1. - **Patient peek or other drill-down features.** This is a backend automation; no UI changes. - **Editing the audit log or any existing cyclone data.** The agent only reads + triggers; it never writes to cyclone's DB except through the published APIs. ## 2. Locked decisions ### 2.1 Architecture API-first + minimal Playwright shell. The browser is used only for the upload page (drop zone, Parse button, progress bar). Everything else (verification, SFTP submit, scheduler control, ACK polling, claim-state checks) hits the FastAPI directly via `httpx.AsyncClient`. Rationale: the upload page is the only surface in cyclone with a real interactive flow. Clearinghouse submission is API-only; the MFT scheduler is API-only; ACK retrieval is API-only. A pure-Playwright approach would be 3–5x more code and considerably flakier on the API-driven phases, with no benefit. ### 2.2 Sibling package, not an extra The agent lives at `/Users/openclaw/dev/cyclone-pipeline/` with its own `pyproject.toml`, not as a new optional extra on the existing cyclone package. Rationale: the agent has Playwright as a hard dependency; we do not want Playwright in cyclone's dev env for developers who never invoke the agent. Separate package = clean separation of concerns. ### 2.3 Real round-trip, but only the parts that are time-bounded The agent waits for **TA1 (envelope ACK)** and **999 (file-level ACK)**. It does **not** wait for 277CA (CO Medicaid doesn't use it) or 835 (payment cycle is next Monday). 835 detection is a separate, fast follow-up command that the operator runs on Monday. Rationale: blocking a run for up to 7 days is impractical and ties up operator attention. The TA1 + 999 receipt is the unambiguous "your file was accepted by the payer at the file level" signal; the 835 is eventually-consistent and doesn't need a tight loop. ### 2.4 CO Medicaid / Gainwell specifics - **SFTP server:** `mft.gainwelltechnologies.com:22` - **Inbound path:** `/CO XIX/PROD/coxix_prod_11525703/FromHPE` - **Outbound file naming:** `11525703-837P-{yyyymmddhhmmssSSS}-1of1.txt` (17-digit Mountain Time timestamp; matches the `clearhouse` config's `outbound` template in `config/payers.yaml`). - **Inbound ACKs (in observed order):** 1. TA1 (seconds to minutes) — envelope-level 2. 999 (minutes) — file-level 3. HTML (sometimes) — non-X12 artifact, lands in inbound, not parsed 4. 835 (next Monday) — remit 5. 277CA — *never used by CO Medicaid* ### 2.5 Failure semantics - **Upload retries:** 3× attempts; each attempt is a fresh page load + file re-attach + click. Screenshots on every attempt. - **SFTP submit retries:** 3× attempts with `1s → 2s → 4s` backoff. The submit endpoint is POST and is *not* auto-retried after a network timeout (would double-submit); the retry covers `409 Conflict` / `503 Service Unavailable` only. - **TA1 timeout (5 min):** if no TA1 arrives, the run is **soft-fail**. The submission is "in flight" — the report says so and recommends checking Gainwell directly. - **999 timeout (15 min):** same — soft-fail. Most production runs complete this phase in < 2 min. - **999 received with status `R` (rejected):** hard-fail. The 999's rejection segments surface in the report with full context. - **HTML detected in inbound:** warn in the report; do not fail. - **835 not present at end of run:** expected; reported as "deferred — expected next Monday" with the `check-835` recipe. - **Process crash (SIGKILL, power loss, etc.):** the next `resume` skips completed phases by reading `run.state.json`. No work is lost except the in-flight phase. ### 2.6 Idempotency - The MFT outbound filename embeds a millisecond Mountain Time timestamp, so back-to-back submissions to the same input file are unique on the SFTP server. No `Idempotency-Key` header is needed. - The agent captures the new `claim_id[]` set from phase 3 (verify parse) and refuses to SFTP-submit if any of those claim IDs already appear in a prior `processed_inbound_files` row within a configurable dedup window (default 24h). `--force` overrides the dedup. - The `check-835` command is fully idempotent: querying `GET /api/remittances?since=` is read-only. ### 2.7 Output shape Every run writes a dated folder under `./runs/`. The folder is self-contained — anyone reading it has the full picture without needing the live cyclone instance. ``` runs/2026-06-21-1430-001/ ├── run.log # structured JSON, structlog ├── run.state.json # for resume ├── report.md # human-readable summary ├── report.json # machine-readable, OpenClaw/Nora-friendly └── screenshots/ ├── 01-preflight.png ├── 02-upload-1.png # attempts are numbered ├── 02-upload-2.png ├── 03-parse-complete.png ├── 04-submit.png ├── 05-ta1.png ├── 06-999.png ├── 07-inbound-scan.png # HTML detection screenshot └── 99-final-inbox.png # final state of the Inbox page ``` On failure, the `finally` block adds `screenshots/FAIL-{phase}.png` and a `failure.md` with the traceback + phase + remediation hint. ## 3. Module map | File | Role | |---|---| | `pyproject.toml` | Project metadata; declares httpx, playwright, click, pydantic, structlog; dev deps: pytest, pytest-asyncio, respx, pytest-playwright | | `README.md` | Install, run, embed-in-agent examples | | `src/cyclone_pipeline/__init__.py` | Public API re-exports | | `src/cyclone_pipeline/__main__.py` | `python -m cyclone_pipeline run …` | | `src/cyclone_pipeline/cli.py` | Click CLI: `run`, `run-batch`, `status`, `resume`, `check-835` | | `src/cyclone_pipeline/pipeline.py` | `CyclonePipeline` orchestrator; 7-phase state machine | | `src/cyclone_pipeline/api_client.py` | `httpx.AsyncClient` wrapper with typed methods | | `src/cyclone_pipeline/browser.py` | Playwright `UploadPage` class for the upload UI | | `src/cyclone_pipeline/waiters.py` | Reusable wait primitives (claim state, ACK arrival) | | `src/cyclone_pipeline/state.py` | `RunState` dataclass + `run.state.json` writer/reader | | `src/cyclone_pipeline/screenshots.py` | Dated folder + stable filename helper | | `src/cyclone_pipeline/report.py` | `report.md` + `report.json` writer | | `src/cyclone_pipeline/selectors.py` | Centralized Playwright selectors with fallbacks | | `src/cyclone_pipeline/exceptions.py` | Typed errors: `UploadError`, `ParseError`, `SubmitError`, `AckTimeoutError`, `SchedulerNotRunningError`, `IdempotencyError` | | `src/cyclone_pipeline/logging_setup.py` | `structlog` config matching cyclone's SP18 JSON style | | `src/cyclone_pipeline/check_835.py` | The deferred 835 detector (pure API, no browser) | | `tests/conftest.py` | Shared fixtures: fake API server, mock browser, fake clock | | `tests/test_api_client.py` | Typed responses, retry behavior, timeouts (respx) | | `tests/test_waiters.py` | Backoff, timeout, predicate resolution (fake clock) | | `tests/test_browser.py` | safe_click, attach_file, progress wait (pytest-playwright) | | `tests/test_pipeline.py` | End-to-end orchestration, state machine, resume (mocks) | | `tests/test_report.py` | Markdown + JSON shape, screenshot paths (golden files) | | `tests/test_cli.py` | `run`, `run-batch`, `resume`, `check-835` (Click's CliRunner) | | `tests/test_check_835.py` | `check-835` happy path + "835 not yet present" path | | `tests/fixtures/sample_837p.edi` | A small valid 837P file for tests (copied from cyclone's `tests/fixtures/minimal_837p.txt`) | ## 4. Pipeline phases The state machine in `pipeline.py` advances through 7 phases. Each phase has: `precondition`, `execute`, `postcondition`, `retry_policy`, and a side-effect of writing `run.state.json` + a screenshot at the end of a successful attempt. ### Phase 1 — Pre-flight (API) - `GET /api/health` → expect `status == "ok"` and `db.ok == true`. - `GET /api/admin/scheduler/status` → record scheduler state (`running` / `stopped`). If `stopped`, the agent will call `POST /api/admin/scheduler/tick` per-iteration in phase 5/6 instead of relying on the polling loop. - Verify the browser-side URL is reachable: `HEAD {browser_base}/` via `httpx` (cheap, no rendering). If the frontend dev server isn't up, fail-fast with `BrowserNotReachableError` rather than discovering this in phase 2. - Capture the API base URL + browser base URL for the report. - Screenshot the Upload page in its empty state → `screenshots/01-preflight.png`. ### Phase 2 — Upload (Playwright) - `chromium.launch(headless=VISIBLE)` (configurable). - `page.goto(f"{browser_base}/upload")`. - `expect(drop_zone).to_be_visible(timeout=10s)`. - `set_input_files("input[type=file]", file_path)` (the hidden input behind the drop zone; `set_input_files` works on hidden inputs). - `expect(file_name_display).to_have_text(file.name)`. - `safe_click("button:has-text('Parse')")`. - `wait_for_progress_complete()` — polls the progress bar (`text_content` containing `"100%"`) or the success toast (`text_content` containing `"Parsed N claims"`), whichever comes first, with a 5 min ceiling. - Screenshots: `screenshots/02-upload-{attempt}.png` per attempt. ### Phase 3 — Verify parse (API) - The browser-side toast gives the parsed claim count; capture it. - `GET /api/claims?since=` → filter to claims whose `submission_date >= phase1_start_ts`; these are the new ones. - Collect the set of new `claim_id` values. - If 0 new claims after a 30s poll, fail with `ParseError`. ### Phase 4 — SFTP submit (API) - Dedup check: query `GET /api/admin/scheduler/processed-files` for the past 24h; if any outbound filename pattern matches a recent submission, fail with `IdempotencyError` unless `--force`. - `POST /api/clearhouse/submit` with `{payer_id: "CO_TXIX", transaction_type: "837P"}`. The body is small; the server picks the batch from the newly-parsed claims. - Expect 200; if 5xx with a "keychain" hint, raise `SubmitError` with the macOS Keychain setup recipe inline. - Screenshot: `screenshots/04-submit.png` (the Inbox page right after submit, showing the batch in the lane). ### Phase 5 — Wait for TA1 (API poll) - `GET /api/ta1-acks?since=` in a loop, every 5s, with a 5 min ceiling. - Match by `isa13` (the control number in the submitted interchange envelope; surfaced in the TA1 record). - If scheduler is `stopped`, call `POST /api/admin/scheduler/tick` between polls to manually drive the MFT fetch. - On hit: capture `ta1_id`, `accepted` (vs `rejected`), screenshot the TA1 detail in the Acks page → `screenshots/05-ta1.png`. - On timeout: soft-fail; record `ta1_status: "timeout"`. ### Phase 6 — Wait for 999 (API poll) - `GET /api/acks?since=` in a loop, every 10s, with a 15 min ceiling. - Match by `original_batch_id` (or, if not available, by `functional_group_control_number` from the submitted file's GS06). - On hit: capture `ack_id`, `status` (`A` = accepted, `R` = rejected, `E` = rejected with errors), the rejection segment details if rejected. - On `A`: screenshot the Acks page → `screenshots/06-999.png`. - On `R`: hard-fail; record `ack.status == "rejected"`, the rejection segments, and the recommended remediation in the report. - On timeout: soft-fail; record `ack_999_status: "timeout"`. ### Phase 7 — Scan for HTML + final report (API + local) - `GET /api/admin/scheduler/processed-files?since=` — list inbound files the MFT scheduler saw during the run. If any has `file_type == "html"` or an unrecognized extension, log a WARN, capture the filename, and add it to the report's "Detected artifacts" section. - Generate `report.md` and `report.json` (see §5). - Screenshot the Inbox page → `screenshots/99-final-inbox.png`. - Emit a one-line summary on stdout for shell pipelines: `RESULT: PASS run=2026-06-21-1430-001 ta1=TA1-42 999=999-43 835=deferred`. ## 5. Output format ### 5.1 `report.md` (human) ```markdown # Cyclone pipeline run — 2026-06-21 14:30 PDT | Field | Value | |---------------|------------------------------------| | Run id | 2026-06-21-1430-001 | | Input | /path/to/axiscare-837p.txt (12.4 KB) | | Result | PASS — TA1 + 999 received, 835 deferred | | Duration | 2m 14s | ## Phase summary | Phase | Status | Duration | Detail | |------------------------|--------|----------|-----------------------------------------| | 1. Pre-flight | OK | 0.4s | health.ok, scheduler running | | 2. Upload | OK | 2.1s | 1 attempt, 17 claims parsed | | 3. Verify parse | OK | 0.8s | claim_ids: CLM-001 … CLM-017 | | 4. SFTP submit | OK | 4.3s | filename 11525703-837P-20260621143012345-1of1.txt | | 5. Wait TA1 | OK | 12s | ack id: TA1-42, accepted | | 6. Wait 999 | OK | 1m 14s | ack id: 999-43, status Accepted | | 7. Scan HTML + report | WARN | 0.2s | 1 HTML file in inbound, skipped; report.md + report.json written | ## Inbound artifacts detected - 1 HTML file (`gainwell_status_20260621.htm`) — not parsed, not a failure. ## Expected next steps - **835 expected Monday** (2026-06-28; computed as next Monday in Mountain Time at the moment of report generation; if today is Monday before noon, expected same day, otherwise next Monday). Re-run: `cyclone-pipeline check-835 2026-06-21-1430-001` ## Artifacts - screenshots/01-preflight.png - screenshots/02-upload-1.png - screenshots/03-parse-complete.png - screenshots/04-submit.png - screenshots/05-ta1.png - screenshots/06-999.png - screenshots/99-final-inbox.png ``` ### 5.2 `report.json` (machine) ```json { "run_id": "2026-06-21-1430-001", "input": { "path": "/path/to/axiscare-837p.txt", "size_bytes": 12700, "sha256": "…" }, "result": "pass", "result_detail": "TA1 + 999 received, 835 deferred", "started_at": "2026-06-21T14:30:00-06:00", "finished_at": "2026-06-21T14:32:14-06:00", "duration_s": 134, "phases": [ {"n": 1, "name": "preflight", "status": "ok", "duration_s": 0.4}, {"n": 2, "name": "upload", "status": "ok", "duration_s": 2.1, "attempts": 1}, {"n": 3, "name": "verify_parse", "status": "ok", "duration_s": 0.8, "claim_ids": ["CLM-001", "…"]}, {"n": 4, "name": "submit", "status": "ok", "duration_s": 4.3, "filename": "11525703-837P-…"}, {"n": 5, "name": "wait_ta1", "status": "ok", "duration_s": 12, "ta1_id": "TA1-42", "accepted": true}, {"n": 6, "name": "wait_999", "status": "ok", "duration_s": 74, "ack_999_id": "999-43", "status": "A"}, {"n": 7, "name": "scan_and_report", "status": "warn", "duration_s": 0.2, "html_files": ["gainwell_status_20260621.htm"], "report_paths": {"md": "report.md", "json": "report.json"}} ], "artifacts": { "screenshots": ["01-preflight.png", "…", "99-final-inbox.png"], "report_md": "report.md", "report_json": "report.json", "run_log": "run.log" }, "next_steps": { "check_835": "cyclone-pipeline check-835 2026-06-21-1430-001", "expected_835_by": "2026-06-28" } } ``` On soft-fail (TA1 or 999 timeout) or hard-fail (999 rejected), the `result` is `"soft_fail"` or `"hard_fail"`, the offending phase gets `status: "timeout"` / `"rejected"`, and a `failure` block is added with the traceback, recommended remediation, and the `failure.md` path. ### 5.3 Exit codes | Code | Meaning | |------|---------| | 0 | PASS — TA1 + 999 both received, both accepted | | 2 | SOFT_FAIL — TA1 or 999 timed out; submission in flight, manual check needed | | 3 | HARD_FAIL — 999 rejected, upload failed irrecoverably, or submit refused | | 4 | USAGE — bad CLI args, missing file, backend unreachable after retries | This mirrors common CI conventions so OpenClaw / Nora / shell scripts can branch on `$?` without parsing JSON. ## 6. Library API (for OpenClaw / Nora) ```python from cyclone_pipeline import CyclonePipeline, RunResult async def main(): pipeline = CyclonePipeline( api_base="http://127.0.0.1:8000", browser_base="http://127.0.0.1:5173", run_dir="./runs", headless=False, timeouts={"ta1_s": 300, "ack_999_s": 900}, ) result: RunResult = await pipeline.run(file_path="axiscare-837p.txt") # result.outcome: "pass" | "soft_fail" | "hard_fail" | "usage" # result.exit_code: int (0/2/3/4; mirrors CLI exit codes in §5.3) # result.report: RunReport (Pydantic model; serializes to report.json) # result.run_id: str # Or batch: results = await pipeline.run_batch(file_paths=[...], strict=False) # Or resume: result = await pipeline.resume(run_id="2026-06-21-1430-001") # Or check 835 later: ack = await pipeline.check_835(run_id="2026-06-21-1430-001") # ack.found, ack.remittance_id, ack.total_paid, ... ``` `RunResult` is a Pydantic model so OpenClaw / Nora can serialize it straight to JSON without any extra glue. ## 7. Tests | Test file | Strategy | |---|---| | `tests/test_api_client.py` | `respx` mock; verify typed responses, retry-on-idempotent, no-retry-on-POST, timeout behavior | | `tests/test_waiters.py` | Inject fake clock + fake state; verify backoff math, timeout, predicate resolution | | `tests/test_browser.py` | `pytest-playwright` against a Vite dev server spun up in CI; covers `safe_click`, `set_input_files`, progress wait | | `tests/test_pipeline.py` | Mock both `api_client` and `browser`; drive fake "happy path" through every phase + "phase 5 timeout" + "phase 3 parse error" + "phase 6 reject" | | `tests/test_report.py` | Golden-file comparison for `report.md` and `report.json` | | `tests/test_cli.py` | Click's `CliRunner`; covers all 5 subcommands + error paths | | `tests/test_check_835.py` | Mock `GET /api/remittances?since=…`; happy path + "835 not yet present" path | CI: GitHub Actions matrix on Python 3.11 + 3.12. Non-Playwright tests run on every PR; Playwright tests gated to `main` push (the Vite dev-server + chromium setup is too slow for every PR). Coverage target: 90% line coverage on `pipeline.py`, 85% on the rest. `browser.py` is exempt from the strict target (Playwright's own assertion failures are the real test). ## 8. Files to add - `pyproject.toml` — package metadata, deps, dev deps, scripts entry - `README.md` — install, run, embed-in-agent examples - 14 source files under `src/cyclone_pipeline/` listed in §3 - 8 test files under `tests/` listed in §3 - `tests/fixtures/sample_837p.edi` — small valid 837P for tests (copied from cyclone's `tests/fixtures/minimal_837p.txt`) Total: 24 files (1 `pyproject.toml`, 1 `README.md`, 14 src `.py`, 8 test `.py`, 1 fixture). ## 9. Files NOT touched - No changes to the cyclone repo. The agent is a strict HTTP consumer of cyclone's published API. - No changes to cyclone's DB schema, API, CLI, or UI. - No new dependencies added to cyclone's `pyproject.toml`. ## 10. Future work (explicit non-goals for v1) - **Watch-folder daemon** — `cyclone-pipeline watch /inbox/*.txt` that runs continuously and submits files as they appear. Achievable with the current `run-batch` + a 1-line shell loop; the daemon is just UX. - **Multi-payer support** — adding a new payer (e.g., a second Medicaid MCO) means new payer-specific SFTP block, file naming template, and ID-matching strategy. Worth doing once a second payer is on the books. - **Slack / email notification on completion** — `report.json` is the integration point; a 50-LOC notifier could post a summary. v1 just prints the one-line summary and writes the file. - **Pushing reports to S3 / a share** — same as above; `report.json` is the integration point. - **Web UI for run history** — out of scope; OpenClaw / Nora can read the `runs/` directory and surface the JSON. - **277CA polling for payers that use it** — straightforward to add behind a `--payer-config` flag once a second payer is configured. 277CA support in cyclone is already shipped; only the agent's wait list changes. ## 11. Open questions (none for v1) None. The design is locked pending user review of this spec.