cyclone/docs/RUNBOOK.md

Cyclone Operator Runbook

Procedures for running Cyclone in production. The end-user README covers dev setup; this doc is for an operator bringing up SFTP polling on a fresh host.

SP25 — Enable SFTP Polling for Real Gainwell MFT

The inbound MFT polling scheduler ships in SP16 and is wired to real paramiko SFTP in SP13. To turn it on for mft.gainwelltechnologies.com:

Prerequisites

• Python 3.11+ with the cyclone package installed (pip install -e .[dev,sftp]).
• The macOS keyring library is optional. On a Linux server or Docker container, the MFT password is supplied via a plain env var (no Keychain dependency).
• Outbound TCP/22 to mft.gainwelltechnologies.com.

Env vars

Variable	Required	Default	Purpose
CYCLONE_SFTP_PASSWORD	Yes (real MFT only)	unset	The MFT password. Stripped of whitespace; empty values are treated as unset.
CYCLONE_SCHEDULER_AUTOSTART	No	unset (falsy)	When 1/true/yes, the scheduler starts polling on API launch.
CYCLONE_SCHEDULER_POLL_SECONDS	No	60	Seconds between poll cycles. The Gainwell MFT server doesn't push — we pull.

First-time setup

1. Set the password env var. On the server that runs Cyclone:

   export CYCLONE_SFTP_PASSWORD='the-actual-password'
   

   For systemd / Docker, persist this in the service file or docker-compose.yml (use a _FILE companion or a secrets: block; see your platform's docs).

2. Confirm the clearhouse SFTP block is configured. GET /api/clearhouse should return a row. If not, run the lifespan once to seed it (any API launch will do).

3. Flip stub → false and point at real MFT. Authenticated as an admin:

   # GET the current row, modify the sftp_block, PATCH it back.
   # The endpoint requires a full Clearhouse body, so always
   # round-trip through GET to get the right updated_at + filename_block.
   curl -s http://127.0.0.1:8000/api/clearhouse -b cookies.txt > /tmp/ch.json
   # Edit /tmp/ch.json — set sftp_block.stub to false and adjust host/port/paths/auth.
   # The minimum change for real-MFT mode is:
   #   jq '.sftp_block.stub = false
   #       | .sftp_block.host = "mft.gainwelltechnologies.com"
   #       | .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
   #     /tmp/ch.json > /tmp/ch-patched.json
   curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
     -H 'Content-Type: application/json' \
     -b cookies.txt \
     --data @/tmp/ch-patched.json
   

   The endpoint hot-reloads the scheduler. No API restart required.

4. Start polling. Either:

   • Set CYCLONE_SCHEDULER_AUTOSTART=true and restart the API.

   • Or trigger manually:

     curl -X POST http://127.0.0.1:8000/api/admin/scheduler/start -b cookies.txt
     

Verification

# Is the loop running?
curl http://127.0.0.1:8000/api/admin/scheduler/status -b cookies.txt

# Force one poll cycle right now:
curl -X POST http://127.0.0.1:8000/api/admin/scheduler/tick -b cookies.txt

# What files has the scheduler seen?
curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?limit=20' -b cookies.txt

# Only the errors?
curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?status=error' -b cookies.txt

A successful first poll shows rows in processed-files with status=ok and a non-zero claim_count for 835/277CA files.

Troubleshooting

Symptom	Likely cause	Fix
status=error rows with AuthenticationException	Wrong password in CYCLONE_SFTP_PASSWORD	Re-export with the correct value, then PATCH /api/clearhouse or restart.
status=error rows with IOError: [Errno 111] Connection refused	Outbound TCP/22 blocked, or wrong host/port	Check the firewall; confirm host in GET /api/clearhouse.
Zero rows in processed-files after a tick	Inbound MFT dir is empty (HPE hasn't pushed yet) — not an error	Wait. Trigger tick again later.
RuntimeError: SFTP: Keychain entry ... missing or stub	get_secret() fell through to Keychain (Linux + missing env var)	Set CYCLONE_SFTP_PASSWORD.
RuntimeError: SftpBlock.auth must contain ...	PATCH set stub=false without an auth block	PATCH again with a complete auth dict.
Scheduler never starts (running: false)	CYCLONE_SCHEDULER_AUTOSTART not set, no manual start call	Either set autostart or POST /api/admin/scheduler/start.

macOS dev box variant

If you prefer the Keychain over env vars on macOS:

security add-generic-password -s cyclone -a sftp.gainwell.password -w '<password>'
security find-generic-password -s cyclone -a sftp.gainwell.password -w   # verify

The env var is the highest-priority lookup; the Keychain is the fallback. Setting both means the env var wins. To force the Keychain, unset the env var for that shell.