cyclone/docs/RUNBOOK.md

Cyclone Operator Runbook

Procedures for running Cyclone in production. The end-user README covers dev setup; this doc is for an operator bringing up SFTP polling on a fresh host.

SP25 — Enable SFTP Polling for Real Gainwell MFT

The inbound MFT polling scheduler ships in SP16 and is wired to real paramiko SFTP in SP13. To turn it on for mft.gainwelltechnologies.com:

Prerequisites

• Python 3.11+ with the cyclone package installed (pip install -e .[dev,sftp]).
• The macOS keyring library is optional. On a Linux server or Docker container, the MFT password is supplied via a plain env var (no Keychain dependency).
• Outbound TCP/22 to mft.gainwelltechnologies.com.

Env vars

Variable	Required	Default	Purpose
CYCLONE_SFTP_PASSWORD	Yes (real MFT only)	unset	The MFT password. Stripped of whitespace; empty values are treated as unset.
CYCLONE_SFTP_PASSWORD_FILE	No	unset	Path to a file containing the MFT password. Highest-priority lookup in secrets.get_secret(). Standard Docker-secrets pattern. See "Docker secrets variant" below.
CYCLONE_SCHEDULER_AUTOSTART	No	unset (falsy)	When 1/true/yes, the scheduler starts polling on API launch.
CYCLONE_SCHEDULER_POLL_SECONDS	No	60	Seconds between poll cycles. The Gainwell MFT server doesn't push — we pull.

First-time setup

1. Set the password env var. On the server that runs Cyclone:

   export CYCLONE_SFTP_PASSWORD='the-actual-password'
   

   For systemd / Docker, persist this in the service file or docker-compose.yml (use a _FILE companion or a secrets: block; see your platform's docs).

2. Confirm the clearhouse SFTP block is configured. GET /api/clearhouse should return a row. If not, run the lifespan once to seed it (any API launch will do).

3. Flip stub → false and point at real MFT. Authenticated as an admin:

   # GET the current row, modify the sftp_block, PATCH it back.
   # The endpoint requires a full Clearhouse body, so always
   # round-trip through GET to get the right updated_at + filename_block.
   curl -s http://127.0.0.1:8000/api/clearhouse -b cookies.txt > /tmp/ch.json
   # Edit /tmp/ch.json — set sftp_block.stub to false and adjust host/port/paths/auth.
   # The minimum change for real-MFT mode is:
   #   jq '.sftp_block.stub = false
   #       | .sftp_block.host = "mft.gainwelltechnologies.com"
   #       | .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
   #     /tmp/ch.json > /tmp/ch-patched.json
   curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
     -H 'Content-Type: application/json' \
     -b cookies.txt \
     --data @/tmp/ch-patched.json
   

   The endpoint hot-reloads the scheduler. No API restart required.

4. Start polling. Either:

   • Set CYCLONE_SCHEDULER_AUTOSTART=true and restart the API.

   • Or trigger manually:

     curl -X POST http://127.0.0.1:8000/api/admin/scheduler/start -b cookies.txt
     

Verification

# Is the loop running?
curl http://127.0.0.1:8000/api/admin/scheduler/status -b cookies.txt

# Force one poll cycle right now:
curl -X POST http://127.0.0.1:8000/api/admin/scheduler/tick -b cookies.txt

# What files has the scheduler seen?
curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?limit=20' -b cookies.txt

# Only the errors?
curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?status=error' -b cookies.txt

A successful first poll shows rows in processed-files with status=ok and a non-zero claim_count for 835/277CA files.

Troubleshooting

Symptom	Likely cause	Fix
status=error rows with AuthenticationException	Wrong password in CYCLONE_SFTP_PASSWORD	Re-export with the correct value, then PATCH /api/clearhouse or restart.
status=error rows with IOError: [Errno 111] Connection refused	Outbound TCP/22 blocked, or wrong host/port	Check the firewall; confirm host in GET /api/clearhouse.
Zero rows in processed-files after a tick	Inbound MFT dir is empty (HPE hasn't pushed yet) — not an error	Wait. Trigger tick again later.
RuntimeError: SFTP: Keychain entry ... missing or stub	get_secret() fell through to Keychain (Linux + missing env var)	Set CYCLONE_SFTP_PASSWORD.
RuntimeError: SftpBlock.auth must contain ...	PATCH set stub=false without an auth block	PATCH again with a complete auth dict.
Scheduler never starts (running: false)	CYCLONE_SCHEDULER_AUTOSTART not set, no manual start call	Either set autostart or POST /api/admin/scheduler/start.

macOS dev box variant

If you prefer the Keychain over env vars on macOS:

security add-generic-password -s cyclone -a sftp.gainwell.password -w '<password>'
security find-generic-password -s cyclone -a sftp.gainwell.password -w   # verify

The env var is the highest-priority lookup; the Keychain is the fallback. Setting both means the env var wins. To force the Keychain, unset the env var for that shell.

Docker secrets variant (SP26)

For the SP23 Docker stack, mount the MFT password as a file rather than embedding it in docker-compose.yml. The compose file already declares the cyclone_sftp_password secret and wires CYCLONE_SFTP_PASSWORD_FILE: "/run/secrets/cyclone_sftp_password" on the backend service. Create the file once on the host:

sudo install -m 0600 -o root -g root /dev/null /etc/cyclone/secrets/sftp_password
echo -n 'the-actual-password' | sudo tee /etc/cyclone/secrets/sftp_password > /dev/null
sudo chmod 0600 /etc/cyclone/secrets/sftp_password

Then docker compose up -d. The backend's secrets.get_secret() will read the file on the next scheduler tick — no env-var export, no docker-compose.yml edit with the password in it. The file takes precedence over the plain CYCLONE_SFTP_PASSWORD env var; setting both means the file wins.

If the mounted file is missing or unreadable (typo in path, container started without the secret mount), the scheduler surfaces a RuntimeError at the next tick that names the env var and the missing path — this is intentional, so a silent fall-through doesn't mask a real misconfiguration.

Manual SFTP mode (this box's current posture)

Use this when the host's IP isn't whitelisted with Gainwell's MFT, when you don't want a daemon polling every minute, or when you prefer to drag-drop files with FileZilla / WinSCP. The seeded dzinesco clearhouse ships in this mode (sftp_block.stub: true); no daemon changes required.

Posture. SftpClient reads/writes to local staging instead of mft.gainwelltechnologies.com:

• Outbound (write_file): ./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/
• Inbound (list_inbound): ./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/

(Paths are relative to the backend cwd; create the directories if absent.)

Daily flow (manual mode)

1. Pull inbound from Gainwell with your SFTP client: /CO XIX/PROD/coxix_prod_11525703/FromHPE/

2. Stage locally in ./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/ (or, equivalently, drop them in /home/tyler/dev/cyclone/ingest/ and copy in).

3. Process with pull-inbound (writes acks/835s to DB, dedupes via processed_inbound_files):

   cd /home/tyler/dev/cyclone/backend
   mkdir -p "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE"
   cp /home/tyler/dev/cyclone/ingest/*.x12 \
      "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/"
   .venv/bin/python -m cyclone pull-inbound --date 20260701
   .venv/bin/python -m cyclone pull-inbound --date 20260702
   .venv/bin/python -m cyclone pull-inbound --date 20260703
   

4. Submit outbound with POST /api/clearhouse/submit (writes serialized 837P files into the stub staging dir; you drag-drop them to your SFTP client's /CO XIX/PROD/coxix_prod_11525703/ToHPE/).

Backfill a backlog

ingest/ often holds days of unprocessed acks. The copy + per-date pull-inbound flow above clears it. Files already in processed_inbound_files are skipped automatically — to re-process, delete the row first (DELETE FROM processed_inbound_files WHERE name = ...).

Submitting claims (canonical — SP37)

For 837P files generated upstream (dzinesco) that you want cyclone to track in the DB before uploading, use the canonical submit path. This is the preferred outbound path going forward — it captures the batch's transaction_set_control_number (the ST02 control number that 999 acks reference in AK201) so future 999 ack links resolve instead of becoming orphans.

cd /home/tyler/dev/cyclone/backend

# 1. Lay out files in batch-*-claims subdirs under your ingest dir:
#    ingest/batch-2026-07-08-claims/claim-001.x12
#    ingest/batch-2026-07-08-claims/claim-002.x12
#    ...

# 2. CLI — walks ingest/, parses, writes to DB, then SFTP-uploads.
.venv/bin/python -m cyclone submit-batch \
    --ingest-dir /home/tyler/dev/cyclone/ingest \
    --actor cli-submit-batch

# Or via HTTP (auth-gated by matrix_gate):
curl -s -X POST http://localhost:8000/api/submit-batch \
    -H 'Content-Type: application/json' \
    -d '{"ingest_dir": "/home/tyler/dev/cyclone/ingest", "actor": "api-submit-batch"}'

Both surfaces share cyclone.submission.submit_file for the parse → DB-write → SFTP-upload chain (DB-first, upload-second invariant). The walker pattern is identical: batch-*-claims/*.x12, sorted, with ._* AppleDouble files skipped.

When to use submit-batch vs resubmit-rejected-claims:

• submit-batch — canonical path for fresh 837s from dzinesco (or any source) that should be tracked in the DB before upload. Default choice.
• resubmit-rejected-claims — one-off path for cases where you do NOT want a DB row (e.g., dzinesco-generated fixes not ready for canonical tracking). Legacy, retained for backward compat.

Status codes / exit codes:

• HTTP 200 on completed runs (per-file failures live in the JSON body); 401 unauthenticated; 404 no clearhouse; 409 stub mode; 422 validation.
• CLI exit 0 on completed runs (per-file failures counted, not bumped); 2 on config-level failures (no clearhouse / stub mode / missing dir).

Note on per-file parse CLIs

parse-837 and parse-835 exist as CLIs but only emit JSON files to --output-dir; they do NOT write to the DB. There is no parse-999 / parse-ta1 / parse-277ca CLI in this version of cyclone — the canonical 999/TA1/277CA/835 ingestion path is pull-inbound → Scheduler.process_inbound_files. For inspection of a single file without DB writes, use the Python API:

from cyclone.parsers.parse_999 import parse as parse_999
result = parse_999(open("path.999.x12").read())

Switching from manual → real (and back)

When this host's IP is whitelisted with Gainwell's MFT admin, the SP25 procedure above flips the block. The CLI one-liner:

export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
.venv/bin/python -c "
from cyclone import db, store
from cyclone.providers import Clearhouse, SftpBlock
db.init_db()
ch = store.store.get_clearhouse()
sb = ch.sftp_block.model_dump()
sb['stub'] = False
sb['auth'] = {'password_keychain_account': 'sftp.gainwell.password'}
new = Clearhouse(
    id=1, name=ch.name, tpid=ch.tpid,
    submitter_id_qual=ch.submitter_id_qual,
    submitter_name=ch.submitter_name,
    submitter_contact_name=ch.submitter_contact_name,
    submitter_contact_email=ch.submitter_contact_email,
    filename_block=ch.filename_block,
    sftp_block=SftpBlock.model_validate(sb, strict=True),
    updated_at=ch.updated_at,
)
print('updated:', store.store.update_clearhouse(new).sftp_block.stub)
"

Hot-reload via PATCH /api/clearhouse (preferred over direct DB write — it also calls scheduler.reconfigure_scheduler so the running daemon picks up the new block without a restart):

curl -s -b cookies.txt http://127.0.0.1:8000/api/clearhouse > /tmp/ch.json
jq '.sftp_block.stub = false
    | .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
   /tmp/ch.json > /tmp/ch-patched.json
curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
     -H 'Content-Type: application/json' -b cookies.txt \
     --data @/tmp/ch-patched.json

To revert, set stub: true and auth: {"method": "keychain", "secret_ref": "sftp.gainwell.password"}.

Known historical drift — the 804 orphan 999s

The acks table may hold several hundred 999 acks whose source 837 batch is not present in the batches table — the Inbox "Ack orphans" lane surfaces them at GET /api/inbox/ack-orphans. These are real production 999s (e.g. sender_id COMEDASSISTPROG) whose source 837s were submitted to HPE clearinghouse before the current ~/.local/share/cyclone/cyclone.db snapshot was created. The source 837s themselves were never re-ingested into the current DB, so the claims table has no rows that could link against them. SP37's canonical submit-batch flow captures ST02 going forward, so the orphan count is stable — not a forward-looking bug, just historical drift.

You cannot auto-link these orphans. The source 837s are not in ingest/, backend/var/sftp/staging/, or any local path — they were transmitted to HPE and never came back. Cyclone is downstream of the clearinghouse and does not retain copies of outbound 837s after SFTP ACK. Do not attempt to re-ingest from SFTP inbound — those files are the 999 acks themselves, not the source 837s.

Triage path

1. Inspect the Inbox > AckOrphansLane in the UI. Each row is a 999 ack with no resolvable claim. Sort by parsed_at DESC to see the most recent first; older orphans are less likely to be actionable.
2. Decide per row. If the operator can identify the source 837 outside Cyclone (e.g. a manual record of what was submitted that day), manually create a claims row + a claim_acks link via POST /api/parse-837 followed by POST /api/inbox/candidates/{remit_id}/match or POST /api/acks/.../match-claim. If not, leave the orphan alone — it stays as valid audit history.

Optional: seed synthetic batch rows (one-shot)

cyclone ack-orphans reconcile inserts a synthetic batches row for every orphan ST02 that doesn't already have one. The synthetic row is marked with input_filename = '<synthetic:orphan-reconcile>' so it's trivially distinguishable in queries. Future 999 acks for the same ST02s will resolve against the batch envelope index (so the operator can see "this 999 is for a known orphan source") but will not link to claims (because no claim rows exist).

# Preview the reconcile without writing rows.
cyclone ack-orphans reconcile --dry-run

# Insert the synthetic batch rows.
cyclone ack-orphans reconcile

Re-running cyclone ack-orphans reconcile after a successful pass is a no-op (idempotent). To inspect the per-ST02 breakdown at any time:

cyclone ack-orphans status

The status command prints a table with ST02, ACK COUNT, and HAS BATCH columns, ranked by ack count so the heaviest backlog surfaces first.

What this is NOT

• Not a backfill. No claims rows are synthesized — the source data is gone.
• Not auto-runnable. The reconcile CLI is operator-invoked only; it does not run on boot, in the SFTP polling scheduler, or via cron.
• Not a deletion. The orphan 999s are valid audit history and must remain queryable.