cyclone/docs/RUNBOOK.md

Cyclone Operator Runbook

Procedures for running Cyclone in production. The end-user README covers dev setup; this doc is for an operator bringing up SFTP polling on a fresh host.

SP25 — Enable SFTP Polling for Real Gainwell MFT

The inbound MFT polling scheduler ships in SP16 and is wired to real paramiko SFTP in SP13. To turn it on for mft.gainwelltechnologies.com:

Prerequisites

• Python 3.11+ with the cyclone package installed (pip install -e .[dev,sftp]).
• The macOS keyring library is optional. On a Linux server or Docker container, the MFT password is supplied via a plain env var (no Keychain dependency).
• Outbound TCP/22 to mft.gainwelltechnologies.com.

Env vars

Variable	Required	Default	Purpose
CYCLONE_SFTP_PASSWORD	Yes (real MFT only)	unset	The MFT password. Stripped of whitespace; empty values are treated as unset.
CYCLONE_SFTP_PASSWORD_FILE	No	unset	Path to a file containing the MFT password. Highest-priority lookup in secrets.get_secret(). Standard Docker-secrets pattern. See "Docker secrets variant" below.
CYCLONE_SCHEDULER_AUTOSTART	No	unset (falsy)	When 1/true/yes, the scheduler starts polling on API launch.
CYCLONE_SCHEDULER_POLL_SECONDS	No	60	Seconds between poll cycles. The Gainwell MFT server doesn't push — we pull.

First-time setup

1. Set the password env var. On the server that runs Cyclone:

   export CYCLONE_SFTP_PASSWORD='the-actual-password'
   

   For systemd / Docker, persist this in the service file or docker-compose.yml (use a _FILE companion or a secrets: block; see your platform's docs).

2. Confirm the clearhouse SFTP block is configured. GET /api/clearhouse should return a row. If not, run the lifespan once to seed it (any API launch will do).

3. Flip stub → false and point at real MFT. Authenticated as an admin:

   # GET the current row, modify the sftp_block, PATCH it back.
   # The endpoint requires a full Clearhouse body, so always
   # round-trip through GET to get the right updated_at + filename_block.
   curl -s http://127.0.0.1:8000/api/clearhouse -b cookies.txt > /tmp/ch.json
   # Edit /tmp/ch.json — set sftp_block.stub to false and adjust host/port/paths/auth.
   # The minimum change for real-MFT mode is:
   #   jq '.sftp_block.stub = false
   #       | .sftp_block.host = "mft.gainwelltechnologies.com"
   #       | .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
   #     /tmp/ch.json > /tmp/ch-patched.json
   curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
     -H 'Content-Type: application/json' \
     -b cookies.txt \
     --data @/tmp/ch-patched.json
   

   The endpoint hot-reloads the scheduler. No API restart required.

4. Start polling. Either:

   • Set CYCLONE_SCHEDULER_AUTOSTART=true and restart the API.

   • Or trigger manually:

     curl -X POST http://127.0.0.1:8000/api/admin/scheduler/start -b cookies.txt
     

Verification

# Is the loop running?
curl http://127.0.0.1:8000/api/admin/scheduler/status -b cookies.txt

# Force one poll cycle right now:
curl -X POST http://127.0.0.1:8000/api/admin/scheduler/tick -b cookies.txt

# What files has the scheduler seen?
curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?limit=20' -b cookies.txt

# Only the errors?
curl 'http://127.0.0.1:8000/api/admin/scheduler/processed-files?status=error' -b cookies.txt

A successful first poll shows rows in processed-files with status=ok and a non-zero claim_count for 835/277CA files.

Troubleshooting

Symptom	Likely cause	Fix
status=error rows with AuthenticationException	Wrong password in CYCLONE_SFTP_PASSWORD	Re-export with the correct value, then PATCH /api/clearhouse or restart.
status=error rows with IOError: [Errno 111] Connection refused	Outbound TCP/22 blocked, or wrong host/port	Check the firewall; confirm host in GET /api/clearhouse.
Zero rows in processed-files after a tick	Inbound MFT dir is empty (HPE hasn't pushed yet) — not an error	Wait. Trigger tick again later.
RuntimeError: SFTP: Keychain entry ... missing or stub	get_secret() fell through to Keychain (Linux + missing env var)	Set CYCLONE_SFTP_PASSWORD.
RuntimeError: SftpBlock.auth must contain ...	PATCH set stub=false without an auth block	PATCH again with a complete auth dict.
Scheduler never starts (running: false)	CYCLONE_SCHEDULER_AUTOSTART not set, no manual start call	Either set autostart or POST /api/admin/scheduler/start.

macOS dev box variant

If you prefer the Keychain over env vars on macOS:

security add-generic-password -s cyclone -a sftp.gainwell.password -w '<password>'
security find-generic-password -s cyclone -a sftp.gainwell.password -w   # verify

The env var is the highest-priority lookup; the Keychain is the fallback. Setting both means the env var wins. To force the Keychain, unset the env var for that shell.

Docker secrets variant (SP26)

For the SP23 Docker stack, mount the MFT password as a file rather than embedding it in docker-compose.yml. The compose file already declares the cyclone_sftp_password secret and wires CYCLONE_SFTP_PASSWORD_FILE: "/run/secrets/cyclone_sftp_password" on the backend service. Create the file once on the host:

sudo install -m 0600 -o root -g root /dev/null /etc/cyclone/secrets/sftp_password
echo -n 'the-actual-password' | sudo tee /etc/cyclone/secrets/sftp_password > /dev/null
sudo chmod 0600 /etc/cyclone/secrets/sftp_password

Then docker compose up -d. The backend's secrets.get_secret() will read the file on the next scheduler tick — no env-var export, no docker-compose.yml edit with the password in it. The file takes precedence over the plain CYCLONE_SFTP_PASSWORD env var; setting both means the file wins.

If the mounted file is missing or unreadable (typo in path, container started without the secret mount), the scheduler surfaces a RuntimeError at the next tick that names the env var and the missing path — this is intentional, so a silent fall-through doesn't mask a real misconfiguration.

Manual SFTP mode (this box's current posture)

Use this when the host's IP isn't whitelisted with Gainwell's MFT, when you don't want a daemon polling every minute, or when you prefer to drag-drop files with FileZilla / WinSCP. The seeded dzinesco clearhouse ships in this mode (sftp_block.stub: true); no daemon changes required.

Posture. SftpClient reads/writes to local staging instead of mft.gainwelltechnologies.com:

• Outbound (write_file): ./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/
• Inbound (list_inbound): ./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/

(Paths are relative to the backend cwd; create the directories if absent.)

Daily flow (manual mode)

1. Pull inbound from Gainwell with your SFTP client: /CO XIX/PROD/coxix_prod_11525703/FromHPE/

2. Stage locally in ./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/ (or, equivalently, drop them in /home/tyler/dev/cyclone/ingest/ and copy in).

3. Process with pull-inbound (writes acks/835s to DB, dedupes via processed_inbound_files):

   cd /home/tyler/dev/cyclone/backend
   mkdir -p "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE"
   cp /home/tyler/dev/cyclone/ingest/*.x12 \
      "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/"
   .venv/bin/python -m cyclone pull-inbound --date 20260701
   .venv/bin/python -m cyclone pull-inbound --date 20260702
   .venv/bin/python -m cyclone pull-inbound --date 20260703
   

4. Submit outbound with POST /api/clearhouse/submit (writes serialized 837P files into the stub staging dir; you drag-drop them to your SFTP client's /CO XIX/PROD/coxix_prod_11525703/ToHPE/).

Backfill a backlog

ingest/ often holds days of unprocessed acks. The copy + per-date pull-inbound flow above clears it. Files already in processed_inbound_files are skipped automatically — to re-process, delete the row first (DELETE FROM processed_inbound_files WHERE name = ...).

Submitting claims (canonical — SP37)

For 837P files generated upstream (dzinesco) that you want cyclone to track in the DB before uploading, use the canonical submit path. This is the preferred outbound path going forward — it captures the batch's transaction_set_control_number (the ST02 control number that 999 acks reference in AK201) so future 999 ack links resolve instead of becoming orphans.

cd /home/tyler/dev/cyclone/backend

# 1. Lay out files in batch-*-claims subdirs under your ingest dir:
#    ingest/batch-2026-07-08-claims/claim-001.x12
#    ingest/batch-2026-07-08-claims/claim-002.x12
#    ...

# 2. CLI — walks ingest/, parses, writes to DB, then SFTP-uploads.
.venv/bin/python -m cyclone submit-batch \
    --ingest-dir /home/tyler/dev/cyclone/ingest \
    --actor cli-submit-batch

# Or via HTTP (auth-gated by matrix_gate):
curl -s -X POST http://localhost:8000/api/submit-batch \
    -H 'Content-Type: application/json' \
    -d '{"ingest_dir": "/home/tyler/dev/cyclone/ingest", "actor": "api-submit-batch"}'

Both surfaces share cyclone.submission.submit_file for the parse → DB-write → SFTP-upload chain (DB-first, upload-second invariant). The walker pattern is identical: batch-*-claims/*.x12, sorted, with ._* AppleDouble files skipped.

When to use submit-batch vs resubmit-rejected-claims:

• submit-batch — canonical path for fresh 837s from dzinesco (or any source) that should be tracked in the DB before upload. Default choice.
• resubmit-rejected-claims — one-off path for cases where you do NOT want a DB row (e.g., dzinesco-generated fixes not ready for canonical tracking). Legacy, retained for backward compat.

Status codes / exit codes:

• HTTP 200 on completed runs (per-file failures live in the JSON body); 401 unauthenticated; 404 no clearhouse; 409 stub mode; 422 validation.
• CLI exit 0 on completed runs (per-file failures counted, not bumped); 2 on config-level failures (no clearhouse / stub mode / missing dir).

Reissue claims (SP24)

For rebuilding raw 837P files into one IG-correct single-claim X12 file per claim — e.g. after a 999 rejection that surfaced an in-scope serializer defect — use cyclone reissue-claims. This is the canonical offline-rebuild workflow; the pre-SP24 script scripts/reissue_claims.py is a deprecation shim that prints a warning at import time.

cd /home/tyler/dev/cyclone/backend
.venv/bin/python -m cyclone.cli reissue-claims \
    --input-dir /home/tyler/dev/cyclone/july8billing \
    --date 2026-07-08 \
    --output-root /home/tyler/dev/cyclone/dev/rebills \
    --pipeline initial \
    --zip-output xxxclaims.zip

What it does:

1. Parse every *.x12 / *.txt / *.edi under --input-dir.
2. Validate each parsed claim; skip claims with hard errors.
3. Serialize each surviving claim via the IG-correct serializer (no HL*3 / PAT / NM1*QC when SBR02 == "18").
4. Write one X12 per claim under --output-root/<date>/<pipeline>/ with HCPF-spec filenames.
5. Optionally zip the output to --zip-output.

Exit codes: 0 success (per-file warnings are non-fatal); 1 IG-correctness guard tripped (the serializer's PATIENT_LOOP_DEFAULT_INCLUDED constant is True) or unexpected error; 2 parse / validation failure on a file AND zero claims survived.

IG-correctness guard. On every invocation the CLI refuses to run if the serializer's default has been flipped back to True. This is the SP24 regression guard — flipping the constant would re-introduce the 2026-07-08 Edifabric 999 rejection ("2000C HL must be absent when 2000B SBR02 = '18'"). The guard's only job is to fail loud; it runs once per invocation and reads one module-level constant.

No SFTP. reissue-claims is a pure local workflow — no DB write, no SFTP upload, no Edifabric gate. To push the rebuilt files to Gainwell, use cyclone submit-batch --ingest-dir <the emitted dir> (see the "Submitting claims" section above) or move the zip manually per the "Manual SFTP mode" posture.

Note on per-file parse CLIs

parse-837 and parse-835 exist as CLIs but only emit JSON files to --output-dir; they do NOT write to the DB. There is no parse-999 / parse-ta1 / parse-277ca CLI in this version of cyclone — the canonical 999/TA1/277CA/835 ingestion path is pull-inbound → Scheduler.process_inbound_files. For inspection of a single file without DB writes, use the Python API:

from cyclone.parsers.parse_999 import parse as parse_999
result = parse_999(open("path.999.x12").read())

SP39 — corrected-claim push + tracking

The corrected files now live under ingest/corrected-v2/v2-batch-<batch-id>-<N>-claims/ instead of the SP33-era ingest/corrected/batch-<id>-<N>-claims/ tree. The originals are preserved for postmortem; the new tree is the resubmission payload. Push workflow:

1. cyclone resubmit-rejected-claims --ingest-dir /home/tyler/dev/cyclone/ingest/corrected-v2 (one push per batch dir; the CLI is idempotent on (claim_id, interchange_control_number) so re-running is safe).
2. After Gainwell returns 999 acks, drop them into ingest/ for the standard cyclone pull-inbound flow.
3. Track the post-submission outcomes: cyclone resubmissions status [--batch-id=<id>] — shows pending_999 / 999_accepted / 999_rejected / 277ca_accepted / paid / denied_again per claim, joined against claim_acks (via the existing SP28/31 auto-link) and remittances (via CLP→claim).

What SP39 fixed in the serializer (2010BB loop, payer NM1 segment):

• Empty / SKCO0 / CO_BHA payer.id → normalized to CO_TXIX (Gainwell's companion guide requires CO_TXIX for the 2010BB NM109; SKCO0 and CO_BHA were silently accepted by Gainwell's pre-processor in late 2025 but their SET-level structural validator rejects them as of 2026-07).
• Empty / COHCPF / CO_BHA payer.name → normalized to CO_TXIX.
• Foreign payer IDs (anything else) pass through verbatim with a WARNING log per substitution.

Switching from manual → real (and back)

When this host's IP is whitelisted with Gainwell's MFT admin, the SP25 procedure above flips the block. The CLI one-liner:

export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
.venv/bin/python -c "
from cyclone import db, store
from cyclone.providers import Clearhouse, SftpBlock
db.init_db()
ch = store.store.get_clearhouse()
sb = ch.sftp_block.model_dump()
sb['stub'] = False
sb['auth'] = {'password_keychain_account': 'sftp.gainwell.password'}
new = Clearhouse(
    id=1, name=ch.name, tpid=ch.tpid,
    submitter_id_qual=ch.submitter_id_qual,
    submitter_name=ch.submitter_name,
    submitter_contact_name=ch.submitter_contact_name,
    submitter_contact_email=ch.submitter_contact_email,
    filename_block=ch.filename_block,
    sftp_block=SftpBlock.model_validate(sb, strict=True),
    updated_at=ch.updated_at,
)
print('updated:', store.store.update_clearhouse(new).sftp_block.stub)
"

Hot-reload via PATCH /api/clearhouse (preferred over direct DB write — it also calls scheduler.reconfigure_scheduler so the running daemon picks up the new block without a restart):

curl -s -b cookies.txt http://127.0.0.1:8000/api/clearhouse > /tmp/ch.json
jq '.sftp_block.stub = false
    | .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
   /tmp/ch.json > /tmp/ch-patched.json
curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
     -H 'Content-Type: application/json' -b cookies.txt \
     --data @/tmp/ch-patched.json

To revert, set stub: true and auth: {"method": "keychain", "secret_ref": "sftp.gainwell.password"}.

SP40 — Edifabric /v2/x12/validate integration

SP40 wires Edifabric's reference X12 validator into the push flow as a fail-closed pre-upload gate, plus a CLI / HTTP endpoint for ad-hoc validation of single files. The integration catches the same byte-level defects (PER-04 required, SBR-09 required, etc.) Edifabric flagged on the SP39 regen ingest/corrected-v2/.

One-time secret setup (operator-only, never commit):

# Option A — macOS Keychain (preferred, paid tier key):
cyclone secrets set edifabric.api_key <paid-tier-key>
#   → stores in Keychain under service='cyclone', account='edifabric.api_key'

# Option B — env var (CI / headless / per-process):
export CYCLONE_EDIFABRIC_API_KEY="<paid-tier-key>"

# Option C — file companion (matches the SP25 SFTP-password pattern):
export CYCLONE_EDIFABRIC_API_KEY_FILE=/path/to/edi-key.txt
#   file should be chmod 600, single-line, no trailing newline needed

The dev-only tests/fixtures/edifabric_api_key.txt holds the free EdiNation Developer API provisional key from the public docs (3ecf6b1c5cf34bd797a5f4c57951a1cf); CI uses the mocked transport so the key is never actually transmitted.

Single-file validation — CLI

# Status printout + OperationResult summary; exits 0 / 1 / 2.
.venv/bin/python -m cyclone.cli validate-837 ingest/corrected-v2/v2-batch-<id>-<N>-claims/tp-<id>-1of1.x12

Exit codes (matches the rest of the validator family):

• 0 — Edifabric returned Status="success" (warnings are non-blocking).
• 2 — Status="error": at least one Details item has Status="error". The CLI prints the first 5 offending segment IDs + messages, then exits.
• 1 — Edifabric client config error (no API key, or transport exception). The operator must fix cyc secrets set before retrying.

Single-file validation — HTTP (auth-gated)

curl -b admin-cookies.txt -X POST \
  -F "file=@ingest/corrected-v2/v2-batch-.../tp-...-1of1.x12" \
  http://127.0.0.1:8000/api/admin/validate-837
# → 200 with the raw OperationResult JSON (Status + Details + LastIndex)
#   Caller inspects Status to decide whether the file is acceptable.

POST /api/admin/validate-837 is matrix-gated (ADMIN_ONLY); 401 without a session cookie, 403 with a viewer cookie.

Pre-upload gate on resubmit-rejected-claims

cyclone resubmit-rejected-claims runs Edifabric validation before every SFTP put. Any file that returns Status="error" is skipped (not uploaded, no Resubmission row inserted), and the batch continues with the remaining files. After the run:

.venv/bin/python -m cyclone.cli resubmit-rejected-claims \
  --ingest-dir /home/tyler/dev/cyclone/ingest/corrected-v2
# ...
# EDIFABRIC VALIDATION FAIL  tp-cyc-deadbeef-1of1.x12: 2 error(s)
#   [error] PER: PER-04 is required
#   [error] SBR: SBR-09 is required
# ...
# DONE uploaded=361 skipped=0 failed=0 validated=363 validate_failed=2 ...
#
# 2 file(s) blocked by Edifabric gate:
#   tp-cyc-aaaaaaaa-1of1.x12
#     [error] PER: PER-04 is required
#   tp-cyc-bbbbbbbb-1of1.x12
#     [error] SBR: SBR-09 is required

To bypass the gate (dev only — for hot-fix scenarios where Edifabric is the bottleneck):

.venv/bin/python -m cyclone.cli resubmit-rejected-claims \
  --ingest-dir ... \
  --skip-edifabric-validate

Expected OperationResult shape (verbatim from EdiNation docs):

{
  "Status": "success" | "warning" | "error",
  "Details": [
    {"Index": 5, "SegmentId": "PER", "Value": "...",
     "Message": "PER-04 is required", "Status": "error"}
  ],
  "LastIndex": 46
}

Warnings (Status="warning") never block. Only Details items with Status="error" fail the gate.

Troubleshooting

• API key not configured → run cyclone secrets set edifabric.api_key <key> (or set CYCLONE_EDIFABRIC_API_KEY_FILE).
• 502 Edifabric upstream failure → check https://status.edifabric.com/ (or the API status page); the gate treats any 4xx/5xx from Edifabric as a hard fail for that file but continues the batch.
• After fixing the byte defect, re-run cyclone resubmit-rejected-claims for the blocked file — idempotent on (claim_id, interchange_control_number).

Known historical drift — the 804 orphan 999s

The acks table may hold several hundred 999 acks whose source 837 batch is not present in the batches table — the Inbox "Ack orphans" lane surfaces them at GET /api/inbox/ack-orphans. These are real production 999s (e.g. sender_id COMEDASSISTPROG) whose source 837s were submitted to HPE clearinghouse before the current ~/.local/share/cyclone/cyclone.db snapshot was created. The source 837s themselves were never re-ingested into the current DB, so the claims table has no rows that could link against them. SP37's canonical submit-batch flow captures ST02 going forward, so the orphan count is stable — not a forward-looking bug, just historical drift.

You cannot auto-link these orphans. The source 837s are not in ingest/, backend/var/sftp/staging/, or any local path — they were transmitted to HPE and never came back. Cyclone is downstream of the clearinghouse and does not retain copies of outbound 837s after SFTP ACK. Do not attempt to re-ingest from SFTP inbound — those files are the 999 acks themselves, not the source 837s.

Triage path

1. Inspect the Inbox > AckOrphansLane in the UI. Each row is a 999 ack with no resolvable claim. Sort by parsed_at DESC to see the most recent first; older orphans are less likely to be actionable.
2. Decide per row. If the operator can identify the source 837 outside Cyclone (e.g. a manual record of what was submitted that day), manually create a claims row + a claim_acks link via POST /api/parse-837 followed by POST /api/inbox/candidates/{remit_id}/match or POST /api/acks/.../match-claim. If not, leave the orphan alone — it stays as valid audit history.

Optional: seed synthetic batch rows (one-shot)

cyclone ack-orphans reconcile inserts a synthetic batches row for every orphan ST02 that doesn't already have one. The synthetic row is marked with input_filename = '<synthetic:orphan-reconcile>' so it's trivially distinguishable in queries. Future 999 acks for the same ST02s will resolve against the batch envelope index (so the operator can see "this 999 is for a known orphan source") but will not link to claims (because no claim rows exist).

# Preview the reconcile without writing rows.
cyclone ack-orphans reconcile --dry-run

# Insert the synthetic batch rows.
cyclone ack-orphans reconcile

Re-running cyclone ack-orphans reconcile after a successful pass is a no-op (idempotent). To inspect the per-ST02 breakdown at any time:

cyclone ack-orphans status

The status command prints a table with ST02, ACK COUNT, and HAS BATCH columns, ranked by ack count so the heaviest backlog surfaces first.

What this is NOT

• Not a backfill. No claims rows are synthesized — the source data is gone.
• Not auto-runnable. The reconcile CLI is operator-invoked only; it does not run on boot, in the SFTP polling scheduler, or via cron.
• Not a deletion. The orphan 999s are valid audit history and must remain queryable.