cyclone/docs/RUNBOOK.md

# Cyclone Operator Runbook

Procedures for running Cyclone in production. The end-user README
covers dev setup; this doc is for an operator bringing up SFTP polling
on a fresh host.

## SP25 — Enable SFTP Polling for Real Gainwell MFT

The inbound MFT polling scheduler ships in SP16 and is wired to real
paramiko SFTP in SP13. To turn it on for `mft.gainwelltechnologies.com`:

### Prerequisites

- Python 3.11+ with the `cyclone` package installed (`pip install -e .[dev,sftp]`).
- The macOS `keyring` library is optional. On a Linux server or Docker
  container, the MFT password is supplied via a plain env var (no
  Keychain dependency).
- Outbound TCP/22 to `mft.gainwelltechnologies.com`.

### Env vars

| Variable | Required | Default | Purpose |
|---|---|---|---|
| `CYCLONE_SFTP_PASSWORD` | Yes (real MFT only) | unset | The MFT password. Stripped of whitespace; empty values are treated as unset. |
| `CYCLONE_SFTP_PASSWORD_FILE` | No | unset | Path to a file containing the MFT password. Highest-priority lookup in `secrets.get_secret()`. Standard Docker-secrets pattern. See "Docker secrets variant" below. |
| `CYCLONE_SCHEDULER_AUTOSTART` | No | unset (falsy) | When `1`/`true`/`yes`, the scheduler starts polling on API launch. |
| `CYCLONE_SCHEDULER_POLL_SECONDS` | No | `60` | Seconds between poll cycles. The Gainwell MFT server doesn't push — we pull. |

### First-time setup

1. **Set the password env var.** On the server that runs Cyclone:

   ```bash
   export CYCLONE_SFTP_PASSWORD='the-actual-password'
   ```

   For systemd / Docker, persist this in the service file or
   `docker-compose.yml` (use a `_FILE` companion or a `secrets:`
   block; see your platform's docs).

2. **Confirm the clearhouse SFTP block is configured.** `GET /api/clearhouse` should return a row. If not, run the lifespan once to seed it (any API launch will do).

3. **Flip stub → false and point at real MFT.** Authenticated as an admin:

   ```bash
   # GET the current row, modify the sftp_block, PATCH it back.
   # The endpoint requires a full Clearhouse body, so always
   # round-trip through GET to get the right updated_at + filename_block.
   curl -s http://127.0.0.1:8000/api/clearhouse -b cookies.txt > /tmp/ch.json
   # Edit /tmp/ch.json — set sftp_block.stub to false and adjust host/port/paths/auth.
   # The minimum change for real-MFT mode is:
   #   jq '.sftp_block.stub = false
   #       | .sftp_block.host = "mft.gainwelltechnologies.com"
   #       | .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
   #     /tmp/ch.json > /tmp/ch-patched.json
   curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
     -H 'Content-Type: application/json' \
     -b cookies.txt \
     --data @/tmp/ch-patched.json
   ```

   The endpoint hot-reloads the scheduler. No API restart required.

4. **Start polling.** Either:

   - Set `CYCLONE_SCHEDULER_AUTOSTART=true` and restart the API.
   - Or trigger manually:

     ```bash
     curl -X POST http://127.0.0.1:8000/api/admin/scheduler/start -b cookies.txt
     ```

### Verification

```bash
# Is the loop running?
curl http://127.0.0.1:8000/api/admin/scheduler/status -b cookies.txt

# Force one poll cycle right now:
curl -X POST http://127.0.0.1:8000/api/admin/scheduler/tick -b cookies.txt

# What files has the scheduler seen?
curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?limit=20' -b cookies.txt

# Only the errors?
curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?status=error' -b cookies.txt
```

A successful first poll shows rows in `processed-files` with
`status=ok` and a non-zero `claim_count` for 835/277CA files.

### Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| `status=error` rows with `AuthenticationException` | Wrong password in `CYCLONE_SFTP_PASSWORD` | Re-export with the correct value, then PATCH /api/clearhouse or restart. |
| `status=error` rows with `IOError: [Errno 111] Connection refused` | Outbound TCP/22 blocked, or wrong host/port | Check the firewall; confirm `host` in `GET /api/clearhouse`. |
| Zero rows in `processed-files` after a tick | Inbound MFT dir is empty (HPE hasn't pushed yet) — not an error | Wait. Trigger `tick` again later. |
| `RuntimeError: SFTP: Keychain entry ... missing or stub` | `get_secret()` fell through to Keychain (Linux + missing env var) | Set `CYCLONE_SFTP_PASSWORD`. |
| `RuntimeError: SftpBlock.auth must contain ...` | PATCH set `stub=false` without an `auth` block | PATCH again with a complete `auth` dict. |
| Scheduler never starts (`running: false`) | `CYCLONE_SCHEDULER_AUTOSTART` not set, no manual `start` call | Either set autostart or `POST /api/admin/scheduler/start`. |

### macOS dev box variant

If you prefer the Keychain over env vars on macOS:

```bash
security add-generic-password -s cyclone -a sftp.gainwell.password -w '<password>'
security find-generic-password -s cyclone -a sftp.gainwell.password -w   # verify
```

The env var is the highest-priority lookup; the Keychain is the
fallback. Setting both means the env var wins. To force the Keychain,
unset the env var for that shell.

### Docker secrets variant (SP26)

For the SP23 Docker stack, mount the MFT password as a file rather
than embedding it in `docker-compose.yml`. The compose file already
declares the `cyclone_sftp_password` secret and wires
`CYCLONE_SFTP_PASSWORD_FILE: "/run/secrets/cyclone_sftp_password"`
on the backend service. Create the file once on the host:

```bash
sudo install -m 0600 -o root -g root /dev/null /etc/cyclone/secrets/sftp_password
echo -n 'the-actual-password' | sudo tee /etc/cyclone/secrets/sftp_password > /dev/null
sudo chmod 0600 /etc/cyclone/secrets/sftp_password
```

Then `docker compose up -d`. The backend's `secrets.get_secret()` will
read the file on the next scheduler tick — no env-var export, no
`docker-compose.yml` edit with the password in it. The file takes
precedence over the plain `CYCLONE_SFTP_PASSWORD` env var; setting
both means the file wins.

If the mounted file is missing or unreadable (typo in path, container
started without the secret mount), the scheduler surfaces a
`RuntimeError` at the next tick that names the env var and the
missing path — this is intentional, so a silent fall-through doesn't
mask a real misconfiguration.

## Manual SFTP mode (this box's current posture)

Use this when the host's IP isn't whitelisted with Gainwell's MFT, when
you don't want a daemon polling every minute, or when you prefer to
drag-drop files with FileZilla / WinSCP. The seeded `dzinesco`
clearhouse ships in this mode (`sftp_block.stub: true`); no daemon
changes required.

**Posture.** `SftpClient` reads/writes to local staging instead of
`mft.gainwelltechnologies.com`:

- Outbound (`write_file`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/`
- Inbound (`list_inbound`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/`

(Paths are relative to the backend cwd; create the directories if
absent.)

### Daily flow (manual mode)

1. **Pull inbound from Gainwell** with your SFTP client:
   `/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
2. **Stage locally** in `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
   (or, equivalently, drop them in `/home/tyler/dev/cyclone/ingest/`
   and copy in).
3. **Process** with `pull-inbound` (writes acks/835s to DB, dedupes
   via `processed_inbound_files`):

   ```bash
   cd /home/tyler/dev/cyclone/backend
   mkdir -p "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE"
   cp /home/tyler/dev/cyclone/ingest/*.x12 \
      "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/"
   .venv/bin/python -m cyclone pull-inbound --date 20260701
   .venv/bin/python -m cyclone pull-inbound --date 20260702
   .venv/bin/python -m cyclone pull-inbound --date 20260703
   ```

4. **Submit outbound** with `POST /api/clearhouse/submit` (writes
   serialized 837P files into the stub staging dir; you drag-drop
   them to your SFTP client's `/CO XIX/PROD/coxix_prod_11525703/ToHPE/`).

### Backfill a backlog

`ingest/` often holds days of unprocessed acks. The copy + per-date
`pull-inbound` flow above clears it. Files already in
`processed_inbound_files` are skipped automatically — to re-process,
delete the row first (`DELETE FROM processed_inbound_files WHERE name = ...`).

### Note on per-file parse CLIs

`parse-837` and `parse-835` exist as CLIs but only emit JSON files to
`--output-dir`; they do NOT write to the DB. There is no
`parse-999` / `parse-ta1` / `parse-277ca` CLI in this version of
cyclone — the canonical 999/TA1/277CA/835 ingestion path is
`pull-inbound` → `Scheduler.process_inbound_files`. For inspection of
a single file without DB writes, use the Python API:

```python
from cyclone.parsers.parse_999 import parse as parse_999
result = parse_999(open("path.999.x12").read())
```

### Switching from manual → real (and back)

When this host's IP is whitelisted with Gainwell's MFT admin, the
SP25 procedure above flips the block. The CLI one-liner:

```bash
export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
.venv/bin/python -c "
from cyclone import db, store
from cyclone.providers import Clearhouse, SftpBlock
db.init_db()
ch = store.store.get_clearhouse()
sb = ch.sftp_block.model_dump()
sb['stub'] = False
sb['auth'] = {'password_keychain_account': 'sftp.gainwell.password'}
new = Clearhouse(
    id=1, name=ch.name, tpid=ch.tpid,
    submitter_id_qual=ch.submitter_id_qual,
    submitter_name=ch.submitter_name,
    submitter_contact_name=ch.submitter_contact_name,
    submitter_contact_email=ch.submitter_contact_email,
    filename_block=ch.filename_block,
    sftp_block=SftpBlock.model_validate(sb, strict=True),
    updated_at=ch.updated_at,
)
print('updated:', store.store.update_clearhouse(new).sftp_block.stub)
"
```

Hot-reload via `PATCH /api/clearhouse` (preferred over direct DB
write — it also calls `scheduler.reconfigure_scheduler` so the running
daemon picks up the new block without a restart):

```bash
curl -s -b cookies.txt http://127.0.0.1:8000/api/clearhouse > /tmp/ch.json
jq '.sftp_block.stub = false
    | .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
   /tmp/ch.json > /tmp/ch-patched.json
curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
     -H 'Content-Type: application/json' -b cookies.txt \
     --data @/tmp/ch-patched.json
```

To revert, set `stub: true` and `auth: {"method": "keychain", "secret_ref": "sftp.gainwell.password"}`.