docs: document manual SFTP mode for this host (post-SP36)
- .env.example: add GAINWELL_SFTP_* mirror vars with bridge instructions - .gitignore: exclude ingest/ + macOS AppleDouble residue - CLAUDE.md: 'Production SFTP posture' section (env bridge, inbound drop zone, auth caveat, ingestion paths, daemon hot-reload caveat) - docs/RUNBOOK.md: 'Manual SFTP mode' section (daily flow, backfill, parse-CLI caveat, manual→real switch procedure) Recalibration after the auth-failed incident from 103.14.26.95 — local SFTP uploads won't work until Gainwell whitelists this IP, so the canonical flow is now manual file copy + cyclone pull-inbound.
This commit is contained in:
Nora
@@ -18,3 +18,23 @@ VITE_API_BASE_URL=
|
||||
# Optional. Set to 1 to disable auth entirely (DEV ONLY). When set,
|
||||
# the backend auto-grants admin access without checking credentials.
|
||||
# CYCLONE_AUTH_DISABLED=0
|
||||
|
||||
# ─── Gainwell / HCPF SFTP credentials (operator convention) ─────────────
|
||||
# These mirror what /home/tyler/EDI/scripts/upload_claims.sh exports and
|
||||
# what the operator's SFTP client uses. Cyclone's secrets module looks up
|
||||
# CYCLONE_SFTP_PASSWORD (or _FILE), not GAINWELL_SFTP_PASS — to bridge:
|
||||
#
|
||||
# export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
|
||||
#
|
||||
# or for the daemon, set CYCLONE_SFTP_PASSWORD_FILE to a 0600 file.
|
||||
GAINWELL_SFTP_USER=colorado-fts\coxix_prod_11525703
|
||||
GAINWELL_SFTP_HOST=mft.gainwelltechnologies.com
|
||||
GAINWELL_SFTP_PASS=
|
||||
GAINWELL_REMOTE_DIR=/CO XIX/PROD/coxix_prod_11525703/ToHPE
|
||||
|
||||
# CYCLONE_SFTP_PASSWORD= # mirror of GAINWELL_SFTP_PASS
|
||||
# CYCLONE_SFTP_PASSWORD_FILE= # path to a 0600 file containing it
|
||||
|
||||
# CYCLONE_SCHEDULER_AUTOSTART= # 1/true/yes to start the MFT poll loop on API launch
|
||||
# CYCLONE_SCHEDULER_POLL_SECONDS=60 # poll interval when autostart is on
|
||||
# CYCLONE_TAIL_HEARTBEAT_S=15 # NDJSON stream heartbeat interval
|
||||
+10
@@ -20,6 +20,16 @@ build/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# macOS extraction residue (AppleDouble / __MACOSX from unzip on macOS).
|
||||
# These are paired with every regular file in an extracted zip — never data.
|
||||
__MACOSX/
|
||||
._*
|
||||
|
||||
# Operator drop zone (untracked working dir; contents are HCPF-delivered
|
||||
# inbound files, never source). Use `mkdir -p ingest && touch ingest/.gitkeep`
|
||||
# if you want the directory itself in the repo.
|
||||
ingest/
|
||||
|
||||
# Local config
|
||||
.env
|
||||
.env.local
|
||||
|
||||
@@ -137,6 +137,34 @@ Config: `config/payers.yaml` is the on-disk source for providers / payers / clea
|
||||
|
||||
Secrets live in the macOS Keychain (via `keyring` + `cyclone.secrets`): SQLCipher key (service `cyclone`, account `cyclone.db.key`), SFTP password, backup passphrase. No secrets on disk in plaintext.
|
||||
|
||||
## Production SFTP posture (this box: manual mode)
|
||||
|
||||
This host runs in **manual SFTP mode** against Gainwell's MOVEit Transfer MFT at `mft.gainwelltechnologies.com`. The seeded `dzinesco` clearhouse keeps `sftp_block.stub: true`; the operator moves files to/from Gainwell with their SFTP client and drops inbound files into `ingest/` for cyclone to ingest. Do NOT flip `stub` to `false` from this host — see "Auth caveat" below.
|
||||
|
||||
### Env var convention
|
||||
|
||||
The operator's shell exports the Gainwell creds as `GAINWELL_SFTP_USER`, `GAINWELL_SFTP_PASS`, `GAINWELL_SFTP_HOST`, `GAINWELL_REMOTE_DIR`. Cyclone's `secrets.get_secret()` looks up `CYCLONE_SFTP_PASSWORD` (or `CYCLONE_SFTP_PASSWORD_FILE`) per `secrets._ENV_NAME_FOR`. The two are NOT the same name — bridge them per-call or in your shell rc:
|
||||
|
||||
```bash
|
||||
export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
|
||||
```
|
||||
|
||||
### Inbound drop zone
|
||||
|
||||
`/home/tyler/dev/cyclone/ingest/` is the operator-maintained staging dir for inbound MFT files (999 acks, TA1, 835 remittances, 277CA claim acks). Files here are NOT auto-watched; processing happens via the procedure in `docs/RUNBOOK.md` § "Manual SFTP mode". At time of writing this dir holds ~1500 unprocessed 999 acks + 1 TA1 + 1 835 from early July 2026 — the local ingest flow clears them.
|
||||
|
||||
### Auth caveat (do not retry)
|
||||
|
||||
Paramiko reaches `MOVEit Transfer SFTP` cleanly (SSH kex completes, MOVEit offers password auth) but `AuthenticationException: Authentication failed` is returned despite the operator confirming the credentials are known-good from another host. Most likely cause: MOVEit Transfer's IP-based access control — this host's public IP `103.14.26.95` is not whitelisted. Repeated auth attempts risk account lockout. Do NOT keep guessing. Resolve by either: (a) whitelisting this IP with Gainwell's MFT admin, or (b) staying in manual mode and moving files via the operator's SFTP client.
|
||||
|
||||
### Inbound ingestion paths
|
||||
|
||||
The canonical way to write 999/TA1/277CA/835 rows into the DB is `Scheduler.process_inbound_files`, exposed as the CLI `cyclone pull-inbound --date YYYYMMDD` and the HTTP `POST /api/admin/scheduler/pull-inbound`. **There is no `parse-999` / `parse-ta1` / `parse-277ca` CLI command** (CLAUDE.md's "CLI" section above is aspirational on those three). The `parse-837` and `parse-835` CLIs exist but only emit JSON files to `--output-dir`; they do NOT write to the DB. For DB writes, use `pull-inbound` (which dedupes via `processed_inbound_files`).
|
||||
|
||||
### Daemon hot-reload
|
||||
|
||||
`python -m cyclone serve` runs as root (started by `tini`) and keeps the SFTP block in memory. Flipping `stub` directly in the DB (`store.update_clearhouse(...)`) does NOT auto-reload the daemon's view — either restart the daemon or use `PATCH /api/clearhouse` (which calls `scheduler.reconfigure_scheduler` to hot-reload).
|
||||
|
||||
## Frontend at a glance
|
||||
|
||||
`src/` is React 18 + TypeScript + Vite. Routes register in `src/App.tsx` (11 pages, all under a `<Layout>` route wrapper). Pages are pure renderers — every page pairs with a `use<X>` data hook in `src/hooks/` and renders a `<PageHeader>` + a table/list/KPI grid. Drawers (`ClaimDrawer/`, `RemitDrawer/`, plus the new `ProviderDrawer/` and `AckDrawer/`) are mounted by the page and their open/close state is mirrored to the URL via `useDrawerUrlState` so deep-links round-trip. Drill-stack navigation is provided by `<DrillStackProvider>` in `src/components/drill/`.
|
||||
|
||||
+107
@@ -138,3 +138,110 @@ started without the secret mount), the scheduler surfaces a
|
||||
`RuntimeError` at the next tick that names the env var and the
|
||||
missing path — this is intentional, so a silent fall-through doesn't
|
||||
mask a real misconfiguration.
|
||||
|
||||
## Manual SFTP mode (this box's current posture)
|
||||
|
||||
Use this when the host's IP isn't whitelisted with Gainwell's MFT, when
|
||||
you don't want a daemon polling every minute, or when you prefer to
|
||||
drag-drop files with FileZilla / WinSCP. The seeded `dzinesco`
|
||||
clearhouse ships in this mode (`sftp_block.stub: true`); no daemon
|
||||
changes required.
|
||||
|
||||
**Posture.** `SftpClient` reads/writes to local staging instead of
|
||||
`mft.gainwelltechnologies.com`:
|
||||
|
||||
- Outbound (`write_file`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/`
|
||||
- Inbound (`list_inbound`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
|
||||
|
||||
(Paths are relative to the backend cwd; create the directories if
|
||||
absent.)
|
||||
|
||||
### Daily flow (manual mode)
|
||||
|
||||
1. **Pull inbound from Gainwell** with your SFTP client:
|
||||
`/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
|
||||
2. **Stage locally** in `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
|
||||
(or, equivalently, drop them in `/home/tyler/dev/cyclone/ingest/`
|
||||
and copy in).
|
||||
3. **Process** with `pull-inbound` (writes acks/835s to DB, dedupes
|
||||
via `processed_inbound_files`):
|
||||
|
||||
```bash
|
||||
cd /home/tyler/dev/cyclone/backend
|
||||
mkdir -p "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE"
|
||||
cp /home/tyler/dev/cyclone/ingest/*.x12 \
|
||||
"./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/"
|
||||
.venv/bin/python -m cyclone pull-inbound --date 20260701
|
||||
.venv/bin/python -m cyclone pull-inbound --date 20260702
|
||||
.venv/bin/python -m cyclone pull-inbound --date 20260703
|
||||
```
|
||||
|
||||
4. **Submit outbound** with `POST /api/clearhouse/submit` (writes
|
||||
serialized 837P files into the stub staging dir; you drag-drop
|
||||
them to your SFTP client's `/CO XIX/PROD/coxix_prod_11525703/ToHPE/`).
|
||||
|
||||
### Backfill a backlog
|
||||
|
||||
`ingest/` often holds days of unprocessed acks. The copy + per-date
|
||||
`pull-inbound` flow above clears it. Files already in
|
||||
`processed_inbound_files` are skipped automatically — to re-process,
|
||||
delete the row first (`DELETE FROM processed_inbound_files WHERE name = ...`).
|
||||
|
||||
### Note on per-file parse CLIs
|
||||
|
||||
`parse-837` and `parse-835` exist as CLIs but only emit JSON files to
|
||||
`--output-dir`; they do NOT write to the DB. There is no
|
||||
`parse-999` / `parse-ta1` / `parse-277ca` CLI in this version of
|
||||
cyclone — the canonical 999/TA1/277CA/835 ingestion path is
|
||||
`pull-inbound` → `Scheduler.process_inbound_files`. For inspection of
|
||||
a single file without DB writes, use the Python API:
|
||||
|
||||
```python
|
||||
from cyclone.parsers.parse_999 import parse as parse_999
|
||||
result = parse_999(open("path.999.x12").read())
|
||||
```
|
||||
|
||||
### Switching from manual → real (and back)
|
||||
|
||||
When this host's IP is whitelisted with Gainwell's MFT admin, the
|
||||
SP25 procedure above flips the block. The CLI one-liner:
|
||||
|
||||
```bash
|
||||
export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
|
||||
.venv/bin/python -c "
|
||||
from cyclone import db, store
|
||||
from cyclone.providers import Clearhouse, SftpBlock
|
||||
db.init_db()
|
||||
ch = store.store.get_clearhouse()
|
||||
sb = ch.sftp_block.model_dump()
|
||||
sb['stub'] = False
|
||||
sb['auth'] = {'password_keychain_account': 'sftp.gainwell.password'}
|
||||
new = Clearhouse(
|
||||
id=1, name=ch.name, tpid=ch.tpid,
|
||||
submitter_id_qual=ch.submitter_id_qual,
|
||||
submitter_name=ch.submitter_name,
|
||||
submitter_contact_name=ch.submitter_contact_name,
|
||||
submitter_contact_email=ch.submitter_contact_email,
|
||||
filename_block=ch.filename_block,
|
||||
sftp_block=SftpBlock.model_validate(sb, strict=True),
|
||||
updated_at=ch.updated_at,
|
||||
)
|
||||
print('updated:', store.store.update_clearhouse(new).sftp_block.stub)
|
||||
"
|
||||
```
|
||||
|
||||
Hot-reload via `PATCH /api/clearhouse` (preferred over direct DB
|
||||
write — it also calls `scheduler.reconfigure_scheduler` so the running
|
||||
daemon picks up the new block without a restart):
|
||||
|
||||
```bash
|
||||
curl -s -b cookies.txt http://127.0.0.1:8000/api/clearhouse > /tmp/ch.json
|
||||
jq '.sftp_block.stub = false
|
||||
| .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
|
||||
/tmp/ch.json > /tmp/ch-patched.json
|
||||
curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
|
||||
-H 'Content-Type: application/json' -b cookies.txt \
|
||||
--data @/tmp/ch-patched.json
|
||||
```
|
||||
|
||||
To revert, set `stub: true` and `auth: {"method": "keychain", "secret_ref": "sftp.gainwell.password"}`.
|
||||
|
||||
Reference in New Issue
Block a user