cyclone/docs/RUNBOOK.md

# Cyclone Operator Runbook

Procedures for running Cyclone in production. The end-user README
covers dev setup; this doc is for an operator bringing up SFTP polling
on a fresh host.

## SP25 — Enable SFTP Polling for Real Gainwell MFT

The inbound MFT polling scheduler ships in SP16 and is wired to real
paramiko SFTP in SP13. To turn it on for `mft.gainwelltechnologies.com`:

### Prerequisites

- Python 3.11+ with the `cyclone` package installed (`pip install -e .[dev,sftp]`).
- The macOS `keyring` library is optional. On a Linux server or Docker
  container, the MFT password is supplied via a plain env var (no
  Keychain dependency).
- Outbound TCP/22 to `mft.gainwelltechnologies.com`.

### Env vars

| Variable | Required | Default | Purpose |
|---|---|---|---|
| `CYCLONE_SFTP_PASSWORD` | Yes (real MFT only) | unset | The MFT password. Stripped of whitespace; empty values are treated as unset. |
| `CYCLONE_SFTP_PASSWORD_FILE` | No | unset | Path to a file containing the MFT password. Highest-priority lookup in `secrets.get_secret()`. Standard Docker-secrets pattern. See "Docker secrets variant" below. |
| `CYCLONE_SCHEDULER_AUTOSTART` | No | unset (falsy) | When `1`/`true`/`yes`, the scheduler starts polling on API launch. |
| `CYCLONE_SCHEDULER_POLL_SECONDS` | No | `60` | Seconds between poll cycles. The Gainwell MFT server doesn't push — we pull. |

### First-time setup

1. **Set the password env var.** On the server that runs Cyclone:

   ```bash
   export CYCLONE_SFTP_PASSWORD='the-actual-password'
   ```

   For systemd / Docker, persist this in the service file or
   `docker-compose.yml` (use a `_FILE` companion or a `secrets:`
   block; see your platform's docs).

2. **Confirm the clearhouse SFTP block is configured.** `GET /api/clearhouse` should return a row. If not, run the lifespan once to seed it (any API launch will do).

3. **Flip stub → false and point at real MFT.** Authenticated as an admin:

   ```bash
   # GET the current row, modify the sftp_block, PATCH it back.
   # The endpoint requires a full Clearhouse body, so always
   # round-trip through GET to get the right updated_at + filename_block.
   curl -s http://127.0.0.1:8000/api/clearhouse -b cookies.txt > /tmp/ch.json
   # Edit /tmp/ch.json — set sftp_block.stub to false and adjust host/port/paths/auth.
   # The minimum change for real-MFT mode is:
   #   jq '.sftp_block.stub = false
   #       | .sftp_block.host = "mft.gainwelltechnologies.com"
   #       | .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
   #     /tmp/ch.json > /tmp/ch-patched.json
   curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
     -H 'Content-Type: application/json' \
     -b cookies.txt \
     --data @/tmp/ch-patched.json
   ```

   The endpoint hot-reloads the scheduler. No API restart required.

4. **Start polling.** Either:

   - Set `CYCLONE_SCHEDULER_AUTOSTART=true` and restart the API.
   - Or trigger manually:

     ```bash
     curl -X POST http://127.0.0.1:8000/api/admin/scheduler/start -b cookies.txt
     ```

### Verification

```bash
# Is the loop running?
curl http://127.0.0.1:8000/api/admin/scheduler/status -b cookies.txt

# Force one poll cycle right now:
curl -X POST http://127.0.0.1:8000/api/admin/scheduler/tick -b cookies.txt

# What files has the scheduler seen?
curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?limit=20' -b cookies.txt

# Only the errors?
curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?status=error' -b cookies.txt
```

A successful first poll shows rows in `processed-files` with
`status=ok` and a non-zero `claim_count` for 835/277CA files.

### Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| `status=error` rows with `AuthenticationException` | Wrong password in `CYCLONE_SFTP_PASSWORD` | Re-export with the correct value, then PATCH /api/clearhouse or restart. |
| `status=error` rows with `IOError: [Errno 111] Connection refused` | Outbound TCP/22 blocked, or wrong host/port | Check the firewall; confirm `host` in `GET /api/clearhouse`. |
| Zero rows in `processed-files` after a tick | Inbound MFT dir is empty (HPE hasn't pushed yet) — not an error | Wait. Trigger `tick` again later. |
| `RuntimeError: SFTP: Keychain entry ... missing or stub` | `get_secret()` fell through to Keychain (Linux + missing env var) | Set `CYCLONE_SFTP_PASSWORD`. |
| `RuntimeError: SftpBlock.auth must contain ...` | PATCH set `stub=false` without an `auth` block | PATCH again with a complete `auth` dict. |
| Scheduler never starts (`running: false`) | `CYCLONE_SCHEDULER_AUTOSTART` not set, no manual `start` call | Either set autostart or `POST /api/admin/scheduler/start`. |

### macOS dev box variant

If you prefer the Keychain over env vars on macOS:

```bash
security add-generic-password -s cyclone -a sftp.gainwell.password -w '<password>'
security find-generic-password -s cyclone -a sftp.gainwell.password -w   # verify
```

The env var is the highest-priority lookup; the Keychain is the
fallback. Setting both means the env var wins. To force the Keychain,
unset the env var for that shell.

### Docker secrets variant (SP26)

For the SP23 Docker stack, mount the MFT password as a file rather
than embedding it in `docker-compose.yml`. The compose file already
declares the `cyclone_sftp_password` secret and wires
`CYCLONE_SFTP_PASSWORD_FILE: "/run/secrets/cyclone_sftp_password"`
on the backend service. Create the file once on the host:

```bash
sudo install -m 0600 -o root -g root /dev/null /etc/cyclone/secrets/sftp_password
echo -n 'the-actual-password' | sudo tee /etc/cyclone/secrets/sftp_password > /dev/null
sudo chmod 0600 /etc/cyclone/secrets/sftp_password
```

Then `docker compose up -d`. The backend's `secrets.get_secret()` will
read the file on the next scheduler tick — no env-var export, no
`docker-compose.yml` edit with the password in it. The file takes
precedence over the plain `CYCLONE_SFTP_PASSWORD` env var; setting
both means the file wins.

If the mounted file is missing or unreadable (typo in path, container
started without the secret mount), the scheduler surfaces a
`RuntimeError` at the next tick that names the env var and the
missing path — this is intentional, so a silent fall-through doesn't
mask a real misconfiguration.

## Manual SFTP mode (this box's current posture)

Use this when the host's IP isn't whitelisted with Gainwell's MFT, when
you don't want a daemon polling every minute, or when you prefer to
drag-drop files with FileZilla / WinSCP. The seeded `dzinesco`
clearhouse ships in this mode (`sftp_block.stub: true`); no daemon
changes required.

**Posture.** `SftpClient` reads/writes to local staging instead of
`mft.gainwelltechnologies.com`:

- Outbound (`write_file`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/`
- Inbound (`list_inbound`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/`

(Paths are relative to the backend cwd; create the directories if
absent.)

### Daily flow (manual mode)

1. **Pull inbound from Gainwell** with your SFTP client:
   `/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
2. **Stage locally** in `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
   (or, equivalently, drop them in `/home/tyler/dev/cyclone/ingest/`
   and copy in).
3. **Process** with `pull-inbound` (writes acks/835s to DB, dedupes
   via `processed_inbound_files`):

   ```bash
   cd /home/tyler/dev/cyclone/backend
   mkdir -p "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE"
   cp /home/tyler/dev/cyclone/ingest/*.x12 \
      "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/"
   .venv/bin/python -m cyclone pull-inbound --date 20260701
   .venv/bin/python -m cyclone pull-inbound --date 20260702
   .venv/bin/python -m cyclone pull-inbound --date 20260703
   ```

4. **Submit outbound** with `POST /api/clearhouse/submit` (writes
   serialized 837P files into the stub staging dir; you drag-drop
   them to your SFTP client's `/CO XIX/PROD/coxix_prod_11525703/ToHPE/`).

### Backfill a backlog

`ingest/` often holds days of unprocessed acks. The copy + per-date
`pull-inbound` flow above clears it. Files already in
`processed_inbound_files` are skipped automatically — to re-process,
delete the row first (`DELETE FROM processed_inbound_files WHERE name = ...`).

### Submitting claims (canonical — SP37)

For 837P files generated upstream (dzinesco) that you want cyclone to
**track in the DB** before uploading, use the canonical submit path.
This is the preferred outbound path going forward — it captures the
batch's `transaction_set_control_number` (the ST02 control number that
999 acks reference in AK201) so future 999 ack links resolve instead
of becoming orphans.

```bash
cd /home/tyler/dev/cyclone/backend

# 1. Lay out files in batch-*-claims subdirs under your ingest dir:
#    ingest/batch-2026-07-08-claims/claim-001.x12
#    ingest/batch-2026-07-08-claims/claim-002.x12
#    ...

# 2. CLI — walks ingest/, parses, writes to DB, then SFTP-uploads.
.venv/bin/python -m cyclone submit-batch \
    --ingest-dir /home/tyler/dev/cyclone/ingest \
    --actor cli-submit-batch

# Or via HTTP (auth-gated by matrix_gate):
curl -s -X POST http://localhost:8000/api/submit-batch \
    -H 'Content-Type: application/json' \
    -d '{"ingest_dir": "/home/tyler/dev/cyclone/ingest", "actor": "api-submit-batch"}'
```

Both surfaces share `cyclone.submission.submit_file` for the
parse → DB-write → SFTP-upload chain (DB-first, upload-second
invariant). The walker pattern is identical: `batch-*-claims/*.x12`,
sorted, with `._*` AppleDouble files skipped.

**When to use `submit-batch` vs `resubmit-rejected-claims`:**
- `submit-batch` — canonical path for fresh 837s from dzinesco (or any
  source) that should be tracked in the DB before upload. Default choice.
- `resubmit-rejected-claims` — one-off path for cases where you do NOT
  want a DB row (e.g., dzinesco-generated fixes not ready for canonical
  tracking). Legacy, retained for backward compat.

**Status codes / exit codes:**
- HTTP 200 on completed runs (per-file failures live in the JSON body);
  401 unauthenticated; 404 no clearhouse; 409 stub mode; 422 validation.
- CLI exit 0 on completed runs (per-file failures counted, not bumped);
  2 on config-level failures (no clearhouse / stub mode / missing dir).

### Note on per-file parse CLIs

`parse-837` and `parse-835` exist as CLIs but only emit JSON files to
`--output-dir`; they do NOT write to the DB. There is no
`parse-999` / `parse-ta1` / `parse-277ca` CLI in this version of
cyclone — the canonical 999/TA1/277CA/835 ingestion path is
`pull-inbound` → `Scheduler.process_inbound_files`. For inspection of
a single file without DB writes, use the Python API:

```python
from cyclone.parsers.parse_999 import parse as parse_999
result = parse_999(open("path.999.x12").read())
```

### Switching from manual → real (and back)

When this host's IP is whitelisted with Gainwell's MFT admin, the
SP25 procedure above flips the block. The CLI one-liner:

```bash
export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
.venv/bin/python -c "
from cyclone import db, store
from cyclone.providers import Clearhouse, SftpBlock
db.init_db()
ch = store.store.get_clearhouse()
sb = ch.sftp_block.model_dump()
sb['stub'] = False
sb['auth'] = {'password_keychain_account': 'sftp.gainwell.password'}
new = Clearhouse(
    id=1, name=ch.name, tpid=ch.tpid,
    submitter_id_qual=ch.submitter_id_qual,
    submitter_name=ch.submitter_name,
    submitter_contact_name=ch.submitter_contact_name,
    submitter_contact_email=ch.submitter_contact_email,
    filename_block=ch.filename_block,
    sftp_block=SftpBlock.model_validate(sb, strict=True),
    updated_at=ch.updated_at,
)
print('updated:', store.store.update_clearhouse(new).sftp_block.stub)
"
```

Hot-reload via `PATCH /api/clearhouse` (preferred over direct DB
write — it also calls `scheduler.reconfigure_scheduler` so the running
daemon picks up the new block without a restart):

```bash
curl -s -b cookies.txt http://127.0.0.1:8000/api/clearhouse > /tmp/ch.json
jq '.sftp_block.stub = false
    | .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
   /tmp/ch.json > /tmp/ch-patched.json
curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
     -H 'Content-Type: application/json' -b cookies.txt \
     --data @/tmp/ch-patched.json
```

To revert, set `stub: true` and `auth: {"method": "keychain", "secret_ref": "sftp.gainwell.password"}`.

## Known historical drift — the 804 orphan 999s

The `acks` table may hold several hundred 999 acks whose source 837
batch is not present in the `batches` table — the Inbox "Ack orphans"
lane surfaces them at `GET /api/inbox/ack-orphans`. These are real
production 999s (e.g. sender_id `COMEDASSISTPROG`) whose source 837s
were submitted to HPE clearinghouse before the current
`~/.local/share/cyclone/cyclone.db` snapshot was created. The source
837s themselves were never re-ingested into the current DB, so the
`claims` table has no rows that could link against them. SP37's
canonical submit-batch flow captures ST02 going forward, so the
orphan count is stable — not a forward-looking bug, just historical
drift.

**You cannot auto-link these orphans.** The source 837s are not in
`ingest/`, `backend/var/sftp/staging/`, or any local path — they
were transmitted to HPE and never came back. Cyclone is downstream
of the clearinghouse and does not retain copies of outbound 837s
after SFTP ACK. Do not attempt to re-ingest from SFTP inbound —
those files are the 999 acks themselves, not the source 837s.

### Triage path

1. **Inspect the Inbox > AckOrphansLane** in the UI. Each row is a
   999 ack with no resolvable claim. Sort by `parsed_at DESC` to
   see the most recent first; older orphans are less likely to be
   actionable.
2. **Decide per row.** If the operator can identify the source 837
   outside Cyclone (e.g. a manual record of what was submitted that
   day), manually create a `claims` row + a `claim_acks` link via
   `POST /api/parse-837` followed by `POST /api/inbox/candidates/{remit_id}/match`
   or `POST /api/acks/.../match-claim`. If not, leave the orphan
   alone — it stays as valid audit history.

### Optional: seed synthetic batch rows (one-shot)

`cyclone ack-orphans reconcile` inserts a synthetic `batches` row
for every orphan ST02 that doesn't already have one. The synthetic
row is marked with `input_filename = '<synthetic:orphan-reconcile>'`
so it's trivially distinguishable in queries. Future 999 acks for
the same ST02s will resolve against the batch envelope index (so
the operator can see "this 999 is for a known orphan source") but
will not link to claims (because no claim rows exist).

```bash
# Preview the reconcile without writing rows.
cyclone ack-orphans reconcile --dry-run

# Insert the synthetic batch rows.
cyclone ack-orphans reconcile
```

Re-running `cyclone ack-orphans reconcile` after a successful pass
is a no-op (idempotent). To inspect the per-ST02 breakdown at any
time:

```bash
cyclone ack-orphans status
```

The status command prints a table with ST02, ACK COUNT, and HAS
BATCH columns, ranked by ack count so the heaviest backlog
surfaces first.

### What this is NOT

- **Not a backfill.** No `claims` rows are synthesized — the source
  data is gone.
- **Not auto-runnable.** The reconcile CLI is operator-invoked only;
  it does not run on boot, in the SFTP polling scheduler, or via
  cron.
- **Not a deletion.** The orphan 999s are valid audit history and
  must remain queryable.