diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md new file mode 100644 index 0000000..9c9b28f --- /dev/null +++ b/docs/RUNBOOK.md @@ -0,0 +1,113 @@ +# Cyclone Operator Runbook + +Procedures for running Cyclone in production. The end-user README +covers dev setup; this doc is for an operator bringing up SFTP polling +on a fresh host. + +## SP25 — Enable SFTP Polling for Real Gainwell MFT + +The inbound MFT polling scheduler ships in SP16 and is wired to real +paramiko SFTP in SP13. To turn it on for `mft.gainwelltechnologies.com`: + +### Prerequisites + +- Python 3.11+ with the `cyclone` package installed (`pip install -e .[dev,sftp]`). +- The macOS `keyring` library is optional. On a Linux server or Docker + container, the MFT password is supplied via a plain env var (no + Keychain dependency). +- Outbound TCP/22 to `mft.gainwelltechnologies.com`. + +### Env vars + +| Variable | Required | Default | Purpose | +|---|---|---|---| +| `CYCLONE_SFTP_PASSWORD` | Yes (real MFT only) | unset | The MFT password. Stripped of whitespace; empty values are treated as unset. | +| `CYCLONE_SCHEDULER_AUTOSTART` | No | unset (falsy) | When `1`/`true`/`yes`, the scheduler starts polling on API launch. | +| `CYCLONE_SCHEDULER_POLL_SECONDS` | No | `60` | Seconds between poll cycles. The Gainwell MFT server doesn't push — we pull. | + +### First-time setup + +1. **Set the password env var.** On the server that runs Cyclone: + + ```bash + export CYCLONE_SFTP_PASSWORD='the-actual-password' + ``` + + For systemd / Docker, persist this in the service file or + `docker-compose.yml` (use a `_FILE` companion or a `secrets:` + block; see your platform's docs). + +2. **Confirm the clearhouse SFTP block is configured.** `GET /api/clearhouse` should return a row. If not, run the lifespan once to seed it (any API launch will do). + +3. **Flip stub → false and point at real MFT.** Authenticated as an admin: + + ```bash + # GET the current row, modify the sftp_block, PATCH it back. + # The endpoint requires a full Clearhouse body, so always + # round-trip through GET to get the right updated_at + filename_block. + curl -s http://127.0.0.1:8000/api/clearhouse -b cookies.txt > /tmp/ch.json + # Edit /tmp/ch.json — set sftp_block.stub to false and adjust host/port/paths/auth. + # The minimum change for real-MFT mode is: + # jq '.sftp_block.stub = false + # | .sftp_block.host = "mft.gainwelltechnologies.com" + # | .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \ + # /tmp/ch.json > /tmp/ch-patched.json + curl -X PATCH http://127.0.0.1:8000/api/clearhouse \ + -H 'Content-Type: application/json' \ + -b cookies.txt \ + --data @/tmp/ch-patched.json + ``` + + The endpoint hot-reloads the scheduler. No API restart required. + +4. **Start polling.** Either: + + - Set `CYCLONE_SCHEDULER_AUTOSTART=true` and restart the API. + - Or trigger manually: + + ```bash + curl -X POST http://127.0.0.1:8000/api/admin/scheduler/start -b cookies.txt + ``` + +### Verification + +```bash +# Is the loop running? +curl http://127.0.0.1:8000/api/admin/scheduler/status -b cookies.txt + +# Force one poll cycle right now: +curl -X POST http://127.0.0.1:8000/api/admin/scheduler/tick -b cookies.txt + +# What files has the scheduler seen? +curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?limit=20' -b cookies.txt + +# Only the errors? +curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?status=error' -b cookies.txt +``` + +A successful first poll shows rows in `processed-files` with +`status=ok` and a non-zero `claim_count` for 835/277CA files. + +### Troubleshooting + +| Symptom | Likely cause | Fix | +|---|---|---| +| `status=error` rows with `AuthenticationException` | Wrong password in `CYCLONE_SFTP_PASSWORD` | Re-export with the correct value, then PATCH /api/clearhouse or restart. | +| `status=error` rows with `IOError: [Errno 111] Connection refused` | Outbound TCP/22 blocked, or wrong host/port | Check the firewall; confirm `host` in `GET /api/clearhouse`. | +| Zero rows in `processed-files` after a tick | Inbound MFT dir is empty (HPE hasn't pushed yet) — not an error | Wait. Trigger `tick` again later. | +| `RuntimeError: SFTP: Keychain entry ... missing or stub` | `get_secret()` fell through to Keychain (Linux + missing env var) | Set `CYCLONE_SFTP_PASSWORD`. | +| `RuntimeError: SftpBlock.auth must contain ...` | PATCH set `stub=false` without an `auth` block | PATCH again with a complete `auth` dict. | +| Scheduler never starts (`running: false`) | `CYCLONE_SCHEDULER_AUTOSTART` not set, no manual `start` call | Either set autostart or `POST /api/admin/scheduler/start`. | + +### macOS dev box variant + +If you prefer the Keychain over env vars on macOS: + +```bash +security add-generic-password -s cyclone -a sftp.gainwell.password -w '' +security find-generic-password -s cyclone -a sftp.gainwell.password -w # verify +``` + +The env var is the highest-priority lookup; the Keychain is the +fallback. Setting both means the env var wins. To force the Keychain, +unset the env var for that shell.