docs: document manual SFTP mode for this host (post-SP36)

- .env.example: add GAINWELL_SFTP_* mirror vars with bridge instructions
- .gitignore: exclude ingest/ + macOS AppleDouble residue
- CLAUDE.md: 'Production SFTP posture' section (env bridge, inbound drop
  zone, auth caveat, ingestion paths, daemon hot-reload caveat)
- docs/RUNBOOK.md: 'Manual SFTP mode' section (daily flow, backfill,
  parse-CLI caveat, manual→real switch procedure)

Recalibration after the auth-failed incident from 103.14.26.95 — local
SFTP uploads won't work until Gainwell whitelists this IP, so the
canonical flow is now manual file copy + cyclone pull-inbound.
This commit is contained in:
Nora
2026-07-07 09:48:57 -06:00
parent 4b56416414
commit 710104f491
4 changed files with 166 additions and 1 deletions
+20
View File
@@ -18,3 +18,23 @@ VITE_API_BASE_URL=
# Optional. Set to 1 to disable auth entirely (DEV ONLY). When set, # Optional. Set to 1 to disable auth entirely (DEV ONLY). When set,
# the backend auto-grants admin access without checking credentials. # the backend auto-grants admin access without checking credentials.
# CYCLONE_AUTH_DISABLED=0 # CYCLONE_AUTH_DISABLED=0
# ─── Gainwell / HCPF SFTP credentials (operator convention) ─────────────
# These mirror what /home/tyler/EDI/scripts/upload_claims.sh exports and
# what the operator's SFTP client uses. Cyclone's secrets module looks up
# CYCLONE_SFTP_PASSWORD (or _FILE), not GAINWELL_SFTP_PASS — to bridge:
#
# export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
#
# or for the daemon, set CYCLONE_SFTP_PASSWORD_FILE to a 0600 file.
GAINWELL_SFTP_USER=colorado-fts\coxix_prod_11525703
GAINWELL_SFTP_HOST=mft.gainwelltechnologies.com
GAINWELL_SFTP_PASS=
GAINWELL_REMOTE_DIR=/CO XIX/PROD/coxix_prod_11525703/ToHPE
# CYCLONE_SFTP_PASSWORD= # mirror of GAINWELL_SFTP_PASS
# CYCLONE_SFTP_PASSWORD_FILE= # path to a 0600 file containing it
# CYCLONE_SCHEDULER_AUTOSTART= # 1/true/yes to start the MFT poll loop on API launch
# CYCLONE_SCHEDULER_POLL_SECONDS=60 # poll interval when autostart is on
# CYCLONE_TAIL_HEARTBEAT_S=15 # NDJSON stream heartbeat interval
+10
View File
@@ -20,6 +20,16 @@ build/
*.swp *.swp
*.swo *.swo
# macOS extraction residue (AppleDouble / __MACOSX from unzip on macOS).
# These are paired with every regular file in an extracted zip — never data.
__MACOSX/
._*
# Operator drop zone (untracked working dir; contents are HCPF-delivered
# inbound files, never source). Use `mkdir -p ingest && touch ingest/.gitkeep`
# if you want the directory itself in the repo.
ingest/
# Local config # Local config
.env .env
.env.local .env.local
+28
View File
@@ -137,6 +137,34 @@ Config: `config/payers.yaml` is the on-disk source for providers / payers / clea
Secrets live in the macOS Keychain (via `keyring` + `cyclone.secrets`): SQLCipher key (service `cyclone`, account `cyclone.db.key`), SFTP password, backup passphrase. No secrets on disk in plaintext. Secrets live in the macOS Keychain (via `keyring` + `cyclone.secrets`): SQLCipher key (service `cyclone`, account `cyclone.db.key`), SFTP password, backup passphrase. No secrets on disk in plaintext.
## Production SFTP posture (this box: manual mode)
This host runs in **manual SFTP mode** against Gainwell's MOVEit Transfer MFT at `mft.gainwelltechnologies.com`. The seeded `dzinesco` clearhouse keeps `sftp_block.stub: true`; the operator moves files to/from Gainwell with their SFTP client and drops inbound files into `ingest/` for cyclone to ingest. Do NOT flip `stub` to `false` from this host — see "Auth caveat" below.
### Env var convention
The operator's shell exports the Gainwell creds as `GAINWELL_SFTP_USER`, `GAINWELL_SFTP_PASS`, `GAINWELL_SFTP_HOST`, `GAINWELL_REMOTE_DIR`. Cyclone's `secrets.get_secret()` looks up `CYCLONE_SFTP_PASSWORD` (or `CYCLONE_SFTP_PASSWORD_FILE`) per `secrets._ENV_NAME_FOR`. The two are NOT the same name — bridge them per-call or in your shell rc:
```bash
export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
```
### Inbound drop zone
`/home/tyler/dev/cyclone/ingest/` is the operator-maintained staging dir for inbound MFT files (999 acks, TA1, 835 remittances, 277CA claim acks). Files here are NOT auto-watched; processing happens via the procedure in `docs/RUNBOOK.md` § "Manual SFTP mode". At time of writing this dir holds ~1500 unprocessed 999 acks + 1 TA1 + 1 835 from early July 2026 — the local ingest flow clears them.
### Auth caveat (do not retry)
Paramiko reaches `MOVEit Transfer SFTP` cleanly (SSH kex completes, MOVEit offers password auth) but `AuthenticationException: Authentication failed` is returned despite the operator confirming the credentials are known-good from another host. Most likely cause: MOVEit Transfer's IP-based access control — this host's public IP `103.14.26.95` is not whitelisted. Repeated auth attempts risk account lockout. Do NOT keep guessing. Resolve by either: (a) whitelisting this IP with Gainwell's MFT admin, or (b) staying in manual mode and moving files via the operator's SFTP client.
### Inbound ingestion paths
The canonical way to write 999/TA1/277CA/835 rows into the DB is `Scheduler.process_inbound_files`, exposed as the CLI `cyclone pull-inbound --date YYYYMMDD` and the HTTP `POST /api/admin/scheduler/pull-inbound`. **There is no `parse-999` / `parse-ta1` / `parse-277ca` CLI command** (CLAUDE.md's "CLI" section above is aspirational on those three). The `parse-837` and `parse-835` CLIs exist but only emit JSON files to `--output-dir`; they do NOT write to the DB. For DB writes, use `pull-inbound` (which dedupes via `processed_inbound_files`).
### Daemon hot-reload
`python -m cyclone serve` runs as root (started by `tini`) and keeps the SFTP block in memory. Flipping `stub` directly in the DB (`store.update_clearhouse(...)`) does NOT auto-reload the daemon's view — either restart the daemon or use `PATCH /api/clearhouse` (which calls `scheduler.reconfigure_scheduler` to hot-reload).
## Frontend at a glance ## Frontend at a glance
`src/` is React 18 + TypeScript + Vite. Routes register in `src/App.tsx` (11 pages, all under a `<Layout>` route wrapper). Pages are pure renderers — every page pairs with a `use<X>` data hook in `src/hooks/` and renders a `<PageHeader>` + a table/list/KPI grid. Drawers (`ClaimDrawer/`, `RemitDrawer/`, plus the new `ProviderDrawer/` and `AckDrawer/`) are mounted by the page and their open/close state is mirrored to the URL via `useDrawerUrlState` so deep-links round-trip. Drill-stack navigation is provided by `<DrillStackProvider>` in `src/components/drill/`. `src/` is React 18 + TypeScript + Vite. Routes register in `src/App.tsx` (11 pages, all under a `<Layout>` route wrapper). Pages are pure renderers — every page pairs with a `use<X>` data hook in `src/hooks/` and renders a `<PageHeader>` + a table/list/KPI grid. Drawers (`ClaimDrawer/`, `RemitDrawer/`, plus the new `ProviderDrawer/` and `AckDrawer/`) are mounted by the page and their open/close state is mirrored to the URL via `useDrawerUrlState` so deep-links round-trip. Drill-stack navigation is provided by `<DrillStackProvider>` in `src/components/drill/`.
+107
View File
@@ -138,3 +138,110 @@ started without the secret mount), the scheduler surfaces a
`RuntimeError` at the next tick that names the env var and the `RuntimeError` at the next tick that names the env var and the
missing path — this is intentional, so a silent fall-through doesn't missing path — this is intentional, so a silent fall-through doesn't
mask a real misconfiguration. mask a real misconfiguration.
## Manual SFTP mode (this box's current posture)
Use this when the host's IP isn't whitelisted with Gainwell's MFT, when
you don't want a daemon polling every minute, or when you prefer to
drag-drop files with FileZilla / WinSCP. The seeded `dzinesco`
clearhouse ships in this mode (`sftp_block.stub: true`); no daemon
changes required.
**Posture.** `SftpClient` reads/writes to local staging instead of
`mft.gainwelltechnologies.com`:
- Outbound (`write_file`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/`
- Inbound (`list_inbound`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
(Paths are relative to the backend cwd; create the directories if
absent.)
### Daily flow (manual mode)
1. **Pull inbound from Gainwell** with your SFTP client:
`/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
2. **Stage locally** in `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
(or, equivalently, drop them in `/home/tyler/dev/cyclone/ingest/`
and copy in).
3. **Process** with `pull-inbound` (writes acks/835s to DB, dedupes
via `processed_inbound_files`):
```bash
cd /home/tyler/dev/cyclone/backend
mkdir -p "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE"
cp /home/tyler/dev/cyclone/ingest/*.x12 \
"./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/"
.venv/bin/python -m cyclone pull-inbound --date 20260701
.venv/bin/python -m cyclone pull-inbound --date 20260702
.venv/bin/python -m cyclone pull-inbound --date 20260703
```
4. **Submit outbound** with `POST /api/clearhouse/submit` (writes
serialized 837P files into the stub staging dir; you drag-drop
them to your SFTP client's `/CO XIX/PROD/coxix_prod_11525703/ToHPE/`).
### Backfill a backlog
`ingest/` often holds days of unprocessed acks. The copy + per-date
`pull-inbound` flow above clears it. Files already in
`processed_inbound_files` are skipped automatically — to re-process,
delete the row first (`DELETE FROM processed_inbound_files WHERE name = ...`).
### Note on per-file parse CLIs
`parse-837` and `parse-835` exist as CLIs but only emit JSON files to
`--output-dir`; they do NOT write to the DB. There is no
`parse-999` / `parse-ta1` / `parse-277ca` CLI in this version of
cyclone — the canonical 999/TA1/277CA/835 ingestion path is
`pull-inbound` → `Scheduler.process_inbound_files`. For inspection of
a single file without DB writes, use the Python API:
```python
from cyclone.parsers.parse_999 import parse as parse_999
result = parse_999(open("path.999.x12").read())
```
### Switching from manual → real (and back)
When this host's IP is whitelisted with Gainwell's MFT admin, the
SP25 procedure above flips the block. The CLI one-liner:
```bash
export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
.venv/bin/python -c "
from cyclone import db, store
from cyclone.providers import Clearhouse, SftpBlock
db.init_db()
ch = store.store.get_clearhouse()
sb = ch.sftp_block.model_dump()
sb['stub'] = False
sb['auth'] = {'password_keychain_account': 'sftp.gainwell.password'}
new = Clearhouse(
id=1, name=ch.name, tpid=ch.tpid,
submitter_id_qual=ch.submitter_id_qual,
submitter_name=ch.submitter_name,
submitter_contact_name=ch.submitter_contact_name,
submitter_contact_email=ch.submitter_contact_email,
filename_block=ch.filename_block,
sftp_block=SftpBlock.model_validate(sb, strict=True),
updated_at=ch.updated_at,
)
print('updated:', store.store.update_clearhouse(new).sftp_block.stub)
"
```
Hot-reload via `PATCH /api/clearhouse` (preferred over direct DB
write — it also calls `scheduler.reconfigure_scheduler` so the running
daemon picks up the new block without a restart):
```bash
curl -s -b cookies.txt http://127.0.0.1:8000/api/clearhouse > /tmp/ch.json
jq '.sftp_block.stub = false
| .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
/tmp/ch.json > /tmp/ch-patched.json
curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
-H 'Content-Type: application/json' -b cookies.txt \
--data @/tmp/ch-patched.json
```
To revert, set `stub: true` and `auth: {"method": "keychain", "secret_ref": "sftp.gainwell.password"}`.