cyclone/docs/RUNBOOK.md

# Cyclone Operator Runbook

Procedures for running Cyclone in production. The end-user README
covers dev setup; this doc is for an operator bringing up SFTP polling
on a fresh host.

## SP25 — Enable SFTP Polling for Real Gainwell MFT

The inbound MFT polling scheduler ships in SP16 and is wired to real
paramiko SFTP in SP13. To turn it on for `mft.gainwelltechnologies.com`:

### Prerequisites

- Python 3.11+ with the `cyclone` package installed (`pip install -e .[dev,sftp]`).
- The macOS `keyring` library is optional. On a Linux server or Docker
  container, the MFT password is supplied via a plain env var (no
  Keychain dependency).
- Outbound TCP/22 to `mft.gainwelltechnologies.com`.

### Env vars

| Variable | Required | Default | Purpose |
|---|---|---|---|
| `CYCLONE_SFTP_PASSWORD` | Yes (real MFT only) | unset | The MFT password. Stripped of whitespace; empty values are treated as unset. |
| `CYCLONE_SFTP_PASSWORD_FILE` | No | unset | Path to a file containing the MFT password. Highest-priority lookup in `secrets.get_secret()`. Standard Docker-secrets pattern. See "Docker secrets variant" below. |
| `CYCLONE_SCHEDULER_AUTOSTART` | No | unset (falsy) | When `1`/`true`/`yes`, the scheduler starts polling on API launch. |
| `CYCLONE_SCHEDULER_POLL_SECONDS` | No | `60` | Seconds between poll cycles. The Gainwell MFT server doesn't push — we pull. |

### First-time setup

1. **Set the password env var.** On the server that runs Cyclone:

   ```bash
   export CYCLONE_SFTP_PASSWORD='the-actual-password'
   ```

   For systemd / Docker, persist this in the service file or
   `docker-compose.yml` (use a `_FILE` companion or a `secrets:`
   block; see your platform's docs).

2. **Confirm the clearhouse SFTP block is configured.** `GET /api/clearhouse` should return a row. If not, run the lifespan once to seed it (any API launch will do).

3. **Flip stub → false and point at real MFT.** Authenticated as an admin:

   ```bash
   # GET the current row, modify the sftp_block, PATCH it back.
   # The endpoint requires a full Clearhouse body, so always
   # round-trip through GET to get the right updated_at + filename_block.
   curl -s http://127.0.0.1:8000/api/clearhouse -b cookies.txt > /tmp/ch.json
   # Edit /tmp/ch.json — set sftp_block.stub to false and adjust host/port/paths/auth.
   # The minimum change for real-MFT mode is:
   #   jq '.sftp_block.stub = false
   #       | .sftp_block.host = "mft.gainwelltechnologies.com"
   #       | .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
   #     /tmp/ch.json > /tmp/ch-patched.json
   curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
     -H 'Content-Type: application/json' \
     -b cookies.txt \
     --data @/tmp/ch-patched.json
   ```

   The endpoint hot-reloads the scheduler. No API restart required.

4. **Start polling.** Either:

   - Set `CYCLONE_SCHEDULER_AUTOSTART=true` and restart the API.
   - Or trigger manually:

     ```bash
     curl -X POST http://127.0.0.1:8000/api/admin/scheduler/start -b cookies.txt
     ```

### Verification

```bash
# Is the loop running?
curl http://127.0.0.1:8000/api/admin/scheduler/status -b cookies.txt

# Force one poll cycle right now:
curl -X POST http://127.0.0.1:8000/api/admin/scheduler/tick -b cookies.txt

# What files has the scheduler seen?
curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?limit=20' -b cookies.txt

# Only the errors?
curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?status=error' -b cookies.txt
```

A successful first poll shows rows in `processed-files` with
`status=ok` and a non-zero `claim_count` for 835/277CA files.

### Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| `status=error` rows with `AuthenticationException` | Wrong password in `CYCLONE_SFTP_PASSWORD` | Re-export with the correct value, then PATCH /api/clearhouse or restart. |
| `status=error` rows with `IOError: [Errno 111] Connection refused` | Outbound TCP/22 blocked, or wrong host/port | Check the firewall; confirm `host` in `GET /api/clearhouse`. |
| Zero rows in `processed-files` after a tick | Inbound MFT dir is empty (HPE hasn't pushed yet) — not an error | Wait. Trigger `tick` again later. |
| `RuntimeError: SFTP: Keychain entry ... missing or stub` | `get_secret()` fell through to Keychain (Linux + missing env var) | Set `CYCLONE_SFTP_PASSWORD`. |
| `RuntimeError: SftpBlock.auth must contain ...` | PATCH set `stub=false` without an `auth` block | PATCH again with a complete `auth` dict. |
| Scheduler never starts (`running: false`) | `CYCLONE_SCHEDULER_AUTOSTART` not set, no manual `start` call | Either set autostart or `POST /api/admin/scheduler/start`. |

### macOS dev box variant

If you prefer the Keychain over env vars on macOS:

```bash
security add-generic-password -s cyclone -a sftp.gainwell.password -w '<password>'
security find-generic-password -s cyclone -a sftp.gainwell.password -w   # verify
```

The env var is the highest-priority lookup; the Keychain is the
fallback. Setting both means the env var wins. To force the Keychain,
unset the env var for that shell.

### Docker secrets variant (SP26)

For the SP23 Docker stack, mount the MFT password as a file rather
than embedding it in `docker-compose.yml`. The compose file already
declares the `cyclone_sftp_password` secret and wires
`CYCLONE_SFTP_PASSWORD_FILE: "/run/secrets/cyclone_sftp_password"`
on the backend service. Create the file once on the host:

```bash
sudo install -m 0600 -o root -g root /dev/null /etc/cyclone/secrets/sftp_password
echo -n 'the-actual-password' | sudo tee /etc/cyclone/secrets/sftp_password > /dev/null
sudo chmod 0600 /etc/cyclone/secrets/sftp_password
```

Then `docker compose up -d`. The backend's `secrets.get_secret()` will
read the file on the next scheduler tick — no env-var export, no
`docker-compose.yml` edit with the password in it. The file takes
precedence over the plain `CYCLONE_SFTP_PASSWORD` env var; setting
both means the file wins.

If the mounted file is missing or unreadable (typo in path, container
started without the secret mount), the scheduler surfaces a
`RuntimeError` at the next tick that names the env var and the
missing path — this is intentional, so a silent fall-through doesn't
mask a real misconfiguration.

## Manual SFTP mode (this box's current posture)

Use this when the host's IP isn't whitelisted with Gainwell's MFT, when
you don't want a daemon polling every minute, or when you prefer to
drag-drop files with FileZilla / WinSCP. The seeded `dzinesco`
clearhouse ships in this mode (`sftp_block.stub: true`); no daemon
changes required.

**Posture.** `SftpClient` reads/writes to local staging instead of
`mft.gainwelltechnologies.com`:

- Outbound (`write_file`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/`
- Inbound (`list_inbound`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/`

(Paths are relative to the backend cwd; create the directories if
absent.)

### Daily flow (manual mode)

1. **Pull inbound from Gainwell** with your SFTP client:
   `/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
2. **Stage locally** in `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
   (or, equivalently, drop them in `/home/tyler/dev/cyclone/ingest/`
   and copy in).
3. **Process** with `pull-inbound` (writes acks/835s to DB, dedupes
   via `processed_inbound_files`):

   ```bash
   cd /home/tyler/dev/cyclone/backend
   mkdir -p "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE"
   cp /home/tyler/dev/cyclone/ingest/*.x12 \
      "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/"
   .venv/bin/python -m cyclone pull-inbound --date 20260701
   .venv/bin/python -m cyclone pull-inbound --date 20260702
   .venv/bin/python -m cyclone pull-inbound --date 20260703
   ```

4. **Submit outbound** with `POST /api/clearhouse/submit` (writes
   serialized 837P files into the stub staging dir; you drag-drop
   them to your SFTP client's `/CO XIX/PROD/coxix_prod_11525703/ToHPE/`).

### Backfill a backlog

`ingest/` often holds days of unprocessed acks. The copy + per-date
`pull-inbound` flow above clears it. Files already in
`processed_inbound_files` are skipped automatically — to re-process,
delete the row first (`DELETE FROM processed_inbound_files WHERE name = ...`).

### Submitting claims (canonical — SP37)

For 837P files generated upstream (dzinesco) that you want cyclone to
**track in the DB** before uploading, use the canonical submit path.
This is the preferred outbound path going forward — it captures the
batch's `transaction_set_control_number` (the ST02 control number that
999 acks reference in AK201) so future 999 ack links resolve instead
of becoming orphans.

```bash
cd /home/tyler/dev/cyclone/backend

# 1. Lay out files in batch-*-claims subdirs under your ingest dir:
#    ingest/batch-2026-07-08-claims/claim-001.x12
#    ingest/batch-2026-07-08-claims/claim-002.x12
#    ...

# 2. CLI — walks ingest/, parses, writes to DB, then SFTP-uploads.
.venv/bin/python -m cyclone submit-batch \
    --ingest-dir /home/tyler/dev/cyclone/ingest \
    --actor cli-submit-batch

# Or via HTTP (auth-gated by matrix_gate):
curl -s -X POST http://localhost:8000/api/submit-batch \
    -H 'Content-Type: application/json' \
    -d '{"ingest_dir": "/home/tyler/dev/cyclone/ingest", "actor": "api-submit-batch"}'
```

Both surfaces share `cyclone.submission.submit_file` for the
parse → DB-write → SFTP-upload chain (DB-first, upload-second
invariant). The walker pattern is identical: `batch-*-claims/*.x12`,
sorted, with `._*` AppleDouble files skipped.

**When to use `submit-batch` vs `resubmit-rejected-claims`:**
- `submit-batch` — canonical path for fresh 837s from dzinesco (or any
  source) that should be tracked in the DB before upload. Default choice.
- `resubmit-rejected-claims` — one-off path for cases where you do NOT
  want a DB row (e.g., dzinesco-generated fixes not ready for canonical
  tracking). Legacy, retained for backward compat.

**Status codes / exit codes:**
- HTTP 200 on completed runs (per-file failures live in the JSON body);
  401 unauthenticated; 404 no clearhouse; 409 stub mode; 422 validation.
- CLI exit 0 on completed runs (per-file failures counted, not bumped);
  2 on config-level failures (no clearhouse / stub mode / missing dir).

### Reissue claims (SP24)

For rebuilding raw 837P files into one IG-correct single-claim X12
file per claim — e.g. after a 999 rejection that surfaced an
in-scope serializer defect — use `cyclone reissue-claims`. This is
the canonical offline-rebuild workflow; the pre-SP24 script
`scripts/reissue_claims.py` is a deprecation shim that prints a
warning at import time.

```bash
cd /home/tyler/dev/cyclone/backend
.venv/bin/python -m cyclone.cli reissue-claims \
    --input-dir /home/tyler/dev/cyclone/july8billing \
    --date 2026-07-08 \
    --output-root /home/tyler/dev/cyclone/dev/rebills \
    --pipeline initial \
    --zip-output xxxclaims.zip
```

What it does:

1. Parse every `*.x12` / `*.txt` / `*.edi` under `--input-dir`.
2. Validate each parsed claim; skip claims with hard errors.
3. Serialize each surviving claim via the IG-correct serializer
   (no `HL*3` / `PAT` / `NM1*QC` when `SBR02 == "18"`).
4. Write one X12 per claim under
   `--output-root/<date>/<pipeline>/` with HCPF-spec filenames.
5. Optionally zip the output to `--zip-output`.

**Exit codes:** 0 success (per-file warnings are non-fatal); 1
IG-correctness guard tripped (the serializer's `PATIENT_LOOP_DEFAULT_INCLUDED`
constant is `True`) or unexpected error; 2 parse / validation failure
on a file AND zero claims survived.

**IG-correctness guard.** On every invocation the CLI refuses to run
if the serializer's default has been flipped back to `True`. This is
the SP24 regression guard — flipping the constant would re-introduce
the 2026-07-08 Edifabric 999 rejection ("2000C HL must be absent
when 2000B SBR02 = '18'"). The guard's only job is to fail loud; it
runs once per invocation and reads one module-level constant.

**No SFTP.** `reissue-claims` is a pure local workflow — no DB write,
no SFTP upload, no Edifabric gate. To push the rebuilt files to
Gainwell, use `cyclone submit-batch --ingest-dir <the emitted dir>`
(see the "Submitting claims" section above) or move the zip manually
per the "Manual SFTP mode" posture.

### Note on per-file parse CLIs

`parse-837` and `parse-835` exist as CLIs but only emit JSON files to
`--output-dir`; they do NOT write to the DB. There is no
`parse-999` / `parse-ta1` / `parse-277ca` CLI in this version of
cyclone — the canonical 999/TA1/277CA/835 ingestion path is
`pull-inbound` → `Scheduler.process_inbound_files`. For inspection of
a single file without DB writes, use the Python API:

```python
from cyclone.parsers.parse_999 import parse as parse_999
result = parse_999(open("path.999.x12").read())
```

### After SP39 lands — corrected-claim push + tracking

The corrected files now live under
`ingest/corrected-v2/v2-batch-<batch-id>-<N>-claims/` instead of the
SP33-era `ingest/corrected/batch-<id>-<N>-claims/` tree. The
originals are preserved for postmortem; the new tree is the
resubmission payload. Push workflow:

1. `cyclone resubmit-rejected-claims --ingest-dir /home/tyler/dev/cyclone/ingest/corrected-v2`
   (one push per batch dir; the CLI is idempotent on
   `(claim_id, interchange_control_number)` so re-running is safe).
2. After Gainwell returns 999 acks, drop them into `ingest/` for
   the standard `cyclone pull-inbound` flow.
3. Track the post-submission outcomes:
   `cyclone resubmissions status [--batch-id=<id>]` — shows
   `pending_999 / 999_accepted / 999_rejected / 277ca_accepted /
   paid / denied_again` per claim, joined against `claim_acks`
   (via the existing SP28/31 auto-link) and `remittances` (via
   CLP→claim).

What SP39 fixed in the serializer (2010BB loop, payer NM1 segment):

- Empty / `SKCO0` / `CO_BHA` payer.id → normalized to `CO_TXIX`
  (Gainwell's companion guide requires `CO_TXIX` for the 2010BB
  NM109; `SKCO0` and `CO_BHA` were silently accepted by Gainwell's
  pre-processor in late 2025 but their SET-level structural
  validator rejects them as of 2026-07).
- Empty / `COHCPF` / `CO_BHA` payer.name → normalized to `CO_TXIX`.
- Foreign payer IDs (anything else) pass through verbatim with a
  WARNING log per substitution.

### Switching from manual → real (and back)

When this host's IP is whitelisted with Gainwell's MFT admin, the
SP25 procedure above flips the block. The CLI one-liner:

```bash
export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
.venv/bin/python -c "
from cyclone import db, store
from cyclone.providers import Clearhouse, SftpBlock
db.init_db()
ch = store.store.get_clearhouse()
sb = ch.sftp_block.model_dump()
sb['stub'] = False
sb['auth'] = {'password_keychain_account': 'sftp.gainwell.password'}
new = Clearhouse(
    id=1, name=ch.name, tpid=ch.tpid,
    submitter_id_qual=ch.submitter_id_qual,
    submitter_name=ch.submitter_name,
    submitter_contact_name=ch.submitter_contact_name,
    submitter_contact_email=ch.submitter_contact_email,
    filename_block=ch.filename_block,
    sftp_block=SftpBlock.model_validate(sb, strict=True),
    updated_at=ch.updated_at,
)
print('updated:', store.store.update_clearhouse(new).sftp_block.stub)
"
```

Hot-reload via `PATCH /api/clearhouse` (preferred over direct DB
write — it also calls `scheduler.reconfigure_scheduler` so the running
daemon picks up the new block without a restart):

```bash
curl -s -b cookies.txt http://127.0.0.1:8000/api/clearhouse > /tmp/ch.json
jq '.sftp_block.stub = false
    | .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
   /tmp/ch.json > /tmp/ch-patched.json
curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
     -H 'Content-Type: application/json' -b cookies.txt \
     --data @/tmp/ch-patched.json
```

To revert, set `stub: true` and `auth: {"method": "keychain", "secret_ref": "sftp.gainwell.password"}`.

### After SP40 lands — Edifabric /v2/x12/validate integration

SP40 wires Edifabric's reference X12 validator into the push flow
as a fail-closed pre-upload gate, plus a CLI / HTTP endpoint for
ad-hoc validation of single files. The integration catches the
same byte-level defects (`PER-04 required`, `SBR-09 required`,
etc.) Edifabric flagged on the SP39 regen `ingest/corrected-v2/`.

**One-time secret setup** (operator-only, never commit):

```bash
# Option A — macOS Keychain (preferred, paid tier key):
cyclone secrets set edifabric.api_key <paid-tier-key>
#   → stores in Keychain under service='cyclone', account='edifabric.api_key'

# Option B — env var (CI / headless / per-process):
export CYCLONE_EDIFABRIC_API_KEY="<paid-tier-key>"

# Option C — file companion (matches the SP25 SFTP-password pattern):
export CYCLONE_EDIFABRIC_API_KEY_FILE=/path/to/edi-key.txt
#   file should be chmod 600, single-line, no trailing newline needed
```

The dev-only `tests/fixtures/edifabric_api_key.txt` holds the
free `EdiNation Developer API` provisional key from the public docs
(`3ecf6b1c5cf34bd797a5f4c57951a1cf`); CI uses the mocked transport
so the key is never actually transmitted.

**Single-file validation — CLI**

```bash
# Status printout + OperationResult summary; exits 0 / 1 / 2.
.venv/bin/python -m cyclone.cli validate-837 ingest/corrected-v2/v2-batch-<id>-<N>-claims/tp-<id>-1of1.x12
```

Exit codes (matches the rest of the validator family):

- `0` — Edifabric returned `Status="success"` (warnings are non-blocking).
- `2` — `Status="error"`: at least one `Details` item has
  `Status="error"`. The CLI prints the first 5 offending segment
  IDs + messages, then exits.
- `1` — Edifabric client config error (no API key, or transport
  exception). The operator must fix `cyc secrets set` before retrying.

**Single-file validation — HTTP (auth-gated)**

```bash
curl -b admin-cookies.txt -X POST \
  -F "file=@ingest/corrected-v2/v2-batch-.../tp-...-1of1.x12" \
  http://127.0.0.1:8000/api/admin/validate-837
# → 200 with the raw OperationResult JSON (Status + Details + LastIndex)
#   Caller inspects Status to decide whether the file is acceptable.
```

`POST /api/admin/validate-837` is matrix-gated (ADMIN_ONLY); 401
without a session cookie, 403 with a viewer cookie.

**Pre-upload gate on resubmit-rejected-claims**

`cyclone resubmit-rejected-claims` runs Edifabric validation
before every SFTP `put`. Any file that returns `Status="error"`
is skipped (not uploaded, no `Resubmission` row inserted), and the
batch continues with the remaining files. After the run:

```bash
.venv/bin/python -m cyclone.cli resubmit-rejected-claims \
  --ingest-dir /home/tyler/dev/cyclone/ingest/corrected-v2
# ...
# EDIFABRIC VALIDATION FAIL  tp-cyc-deadbeef-1of1.x12: 2 error(s)
#   [error] PER: PER-04 is required
#   [error] SBR: SBR-09 is required
# ...
# DONE uploaded=361 skipped=0 failed=0 validated=363 validate_failed=2 ...
#
# 2 file(s) blocked by Edifabric gate:
#   tp-cyc-aaaaaaaa-1of1.x12
#     [error] PER: PER-04 is required
#   tp-cyc-bbbbbbbb-1of1.x12
#     [error] SBR: SBR-09 is required
```

To bypass the gate (dev only — for hot-fix scenarios where
Edifabric is the bottleneck):

```bash
.venv/bin/python -m cyclone.cli resubmit-rejected-claims \
  --ingest-dir ... \
  --skip-edifabric-validate
```

**Expected OperationResult shape** (verbatim from EdiNation docs):

```json
{
  "Status": "success" | "warning" | "error",
  "Details": [
    {"Index": 5, "SegmentId": "PER", "Value": "...",
     "Message": "PER-04 is required", "Status": "error"}
  ],
  "LastIndex": 46
}
```

Warnings (`Status="warning"`) never block. Only `Details` items
with `Status="error"` fail the gate.

**Troubleshooting**

- `API key not configured` → run
  `cyclone secrets set edifabric.api_key <key>` (or set
  `CYCLONE_EDIFABRIC_API_KEY_FILE`).
- `502 Edifabric upstream failure` → check
  https://status.edifabric.com/ (or the API status page); the
  gate treats any 4xx/5xx from Edifabric as a hard fail for that
  file but continues the batch.
- After fixing the byte defect, re-run
  `cyclone resubmit-rejected-claims` for the blocked file —
  idempotent on `(claim_id, interchange_control_number)`.

## Known historical drift — the 804 orphan 999s

The `acks` table may hold several hundred 999 acks whose source 837
batch is not present in the `batches` table — the Inbox "Ack orphans"
lane surfaces them at `GET /api/inbox/ack-orphans`. These are real
production 999s (e.g. sender_id `COMEDASSISTPROG`) whose source 837s
were submitted to HPE clearinghouse before the current
`~/.local/share/cyclone/cyclone.db` snapshot was created. The source
837s themselves were never re-ingested into the current DB, so the
`claims` table has no rows that could link against them. SP37's
canonical submit-batch flow captures ST02 going forward, so the
orphan count is stable — not a forward-looking bug, just historical
drift.

**You cannot auto-link these orphans.** The source 837s are not in
`ingest/`, `backend/var/sftp/staging/`, or any local path — they
were transmitted to HPE and never came back. Cyclone is downstream
of the clearinghouse and does not retain copies of outbound 837s
after SFTP ACK. Do not attempt to re-ingest from SFTP inbound —
those files are the 999 acks themselves, not the source 837s.

### Triage path

1. **Inspect the Inbox > AckOrphansLane** in the UI. Each row is a
   999 ack with no resolvable claim. Sort by `parsed_at DESC` to
   see the most recent first; older orphans are less likely to be
   actionable.
2. **Decide per row.** If the operator can identify the source 837
   outside Cyclone (e.g. a manual record of what was submitted that
   day), manually create a `claims` row + a `claim_acks` link via
   `POST /api/parse-837` followed by `POST /api/inbox/candidates/{remit_id}/match`
   or `POST /api/acks/.../match-claim`. If not, leave the orphan
   alone — it stays as valid audit history.

### Optional: seed synthetic batch rows (one-shot)

`cyclone ack-orphans reconcile` inserts a synthetic `batches` row
for every orphan ST02 that doesn't already have one. The synthetic
row is marked with `input_filename = '<synthetic:orphan-reconcile>'`
so it's trivially distinguishable in queries. Future 999 acks for
the same ST02s will resolve against the batch envelope index (so
the operator can see "this 999 is for a known orphan source") but
will not link to claims (because no claim rows exist).

```bash
# Preview the reconcile without writing rows.
cyclone ack-orphans reconcile --dry-run

# Insert the synthetic batch rows.
cyclone ack-orphans reconcile
```

Re-running `cyclone ack-orphans reconcile` after a successful pass
is a no-op (idempotent). To inspect the per-ST02 breakdown at any
time:

```bash
cyclone ack-orphans status
```

The status command prints a table with ST02, ACK COUNT, and HAS
BATCH columns, ranked by ack count so the heaviest backlog
surfaces first.

### What this is NOT

- **Not a backfill.** No `claims` rows are synthesized — the source
  data is gone.
- **Not auto-runnable.** The reconcile CLI is operator-invoked only;
  it does not run on boot, in the SFTP polling scheduler, or via
  cron.
- **Not a deletion.** The orphan 999s are valid audit history and
  must remain queryable.