diff --git a/.env.example b/.env.example index 2734c8b..190683b 100644 --- a/.env.example +++ b/.env.example @@ -17,4 +17,24 @@ VITE_API_BASE_URL= # Optional. Set to 1 to disable auth entirely (DEV ONLY). When set, # the backend auto-grants admin access without checking credentials. -# CYCLONE_AUTH_DISABLED=0 \ No newline at end of file +# CYCLONE_AUTH_DISABLED=0 + +# ─── Gainwell / HCPF SFTP credentials (operator convention) ───────────── +# These mirror what /home/tyler/EDI/scripts/upload_claims.sh exports and +# what the operator's SFTP client uses. Cyclone's secrets module looks up +# CYCLONE_SFTP_PASSWORD (or _FILE), not GAINWELL_SFTP_PASS — to bridge: +# +# export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS" +# +# or for the daemon, set CYCLONE_SFTP_PASSWORD_FILE to a 0600 file. +GAINWELL_SFTP_USER=colorado-fts\coxix_prod_11525703 +GAINWELL_SFTP_HOST=mft.gainwelltechnologies.com +GAINWELL_SFTP_PASS= +GAINWELL_REMOTE_DIR=/CO XIX/PROD/coxix_prod_11525703/ToHPE + +# CYCLONE_SFTP_PASSWORD= # mirror of GAINWELL_SFTP_PASS +# CYCLONE_SFTP_PASSWORD_FILE= # path to a 0600 file containing it + +# CYCLONE_SCHEDULER_AUTOSTART= # 1/true/yes to start the MFT poll loop on API launch +# CYCLONE_SCHEDULER_POLL_SECONDS=60 # poll interval when autostart is on +# CYCLONE_TAIL_HEARTBEAT_S=15 # NDJSON stream heartbeat interval \ No newline at end of file diff --git a/.gitignore b/.gitignore index 479a213..6eee15b 100644 --- a/.gitignore +++ b/.gitignore @@ -20,6 +20,16 @@ build/ *.swp *.swo +# macOS extraction residue (AppleDouble / __MACOSX from unzip on macOS). +# These are paired with every regular file in an extracted zip — never data. +__MACOSX/ +._* + +# Operator drop zone (untracked working dir; contents are HCPF-delivered +# inbound files, never source). Use `mkdir -p ingest && touch ingest/.gitkeep` +# if you want the directory itself in the repo. +ingest/ + # Local config .env .env.local diff --git a/CLAUDE.md b/CLAUDE.md index 5421874..8dfdba6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -137,6 +137,34 @@ Config: `config/payers.yaml` is the on-disk source for providers / payers / clea Secrets live in the macOS Keychain (via `keyring` + `cyclone.secrets`): SQLCipher key (service `cyclone`, account `cyclone.db.key`), SFTP password, backup passphrase. No secrets on disk in plaintext. +## Production SFTP posture (this box: manual mode) + +This host runs in **manual SFTP mode** against Gainwell's MOVEit Transfer MFT at `mft.gainwelltechnologies.com`. The seeded `dzinesco` clearhouse keeps `sftp_block.stub: true`; the operator moves files to/from Gainwell with their SFTP client and drops inbound files into `ingest/` for cyclone to ingest. Do NOT flip `stub` to `false` from this host — see "Auth caveat" below. + +### Env var convention + +The operator's shell exports the Gainwell creds as `GAINWELL_SFTP_USER`, `GAINWELL_SFTP_PASS`, `GAINWELL_SFTP_HOST`, `GAINWELL_REMOTE_DIR`. Cyclone's `secrets.get_secret()` looks up `CYCLONE_SFTP_PASSWORD` (or `CYCLONE_SFTP_PASSWORD_FILE`) per `secrets._ENV_NAME_FOR`. The two are NOT the same name — bridge them per-call or in your shell rc: + +```bash +export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS" +``` + +### Inbound drop zone + +`/home/tyler/dev/cyclone/ingest/` is the operator-maintained staging dir for inbound MFT files (999 acks, TA1, 835 remittances, 277CA claim acks). Files here are NOT auto-watched; processing happens via the procedure in `docs/RUNBOOK.md` § "Manual SFTP mode". At time of writing this dir holds ~1500 unprocessed 999 acks + 1 TA1 + 1 835 from early July 2026 — the local ingest flow clears them. + +### Auth caveat (do not retry) + +Paramiko reaches `MOVEit Transfer SFTP` cleanly (SSH kex completes, MOVEit offers password auth) but `AuthenticationException: Authentication failed` is returned despite the operator confirming the credentials are known-good from another host. Most likely cause: MOVEit Transfer's IP-based access control — this host's public IP `103.14.26.95` is not whitelisted. Repeated auth attempts risk account lockout. Do NOT keep guessing. Resolve by either: (a) whitelisting this IP with Gainwell's MFT admin, or (b) staying in manual mode and moving files via the operator's SFTP client. + +### Inbound ingestion paths + +The canonical way to write 999/TA1/277CA/835 rows into the DB is `Scheduler.process_inbound_files`, exposed as the CLI `cyclone pull-inbound --date YYYYMMDD` and the HTTP `POST /api/admin/scheduler/pull-inbound`. **There is no `parse-999` / `parse-ta1` / `parse-277ca` CLI command** (CLAUDE.md's "CLI" section above is aspirational on those three). The `parse-837` and `parse-835` CLIs exist but only emit JSON files to `--output-dir`; they do NOT write to the DB. For DB writes, use `pull-inbound` (which dedupes via `processed_inbound_files`). + +### Daemon hot-reload + +`python -m cyclone serve` runs as root (started by `tini`) and keeps the SFTP block in memory. Flipping `stub` directly in the DB (`store.update_clearhouse(...)`) does NOT auto-reload the daemon's view — either restart the daemon or use `PATCH /api/clearhouse` (which calls `scheduler.reconfigure_scheduler` to hot-reload). + ## Frontend at a glance `src/` is React 18 + TypeScript + Vite. Routes register in `src/App.tsx` (11 pages, all under a `` route wrapper). Pages are pure renderers — every page pairs with a `use` data hook in `src/hooks/` and renders a `` + a table/list/KPI grid. Drawers (`ClaimDrawer/`, `RemitDrawer/`, plus the new `ProviderDrawer/` and `AckDrawer/`) are mounted by the page and their open/close state is mirrored to the URL via `useDrawerUrlState` so deep-links round-trip. Drill-stack navigation is provided by `` in `src/components/drill/`. diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index c549ae8..1271f43 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -138,3 +138,110 @@ started without the secret mount), the scheduler surfaces a `RuntimeError` at the next tick that names the env var and the missing path — this is intentional, so a silent fall-through doesn't mask a real misconfiguration. + +## Manual SFTP mode (this box's current posture) + +Use this when the host's IP isn't whitelisted with Gainwell's MFT, when +you don't want a daemon polling every minute, or when you prefer to +drag-drop files with FileZilla / WinSCP. The seeded `dzinesco` +clearhouse ships in this mode (`sftp_block.stub: true`); no daemon +changes required. + +**Posture.** `SftpClient` reads/writes to local staging instead of +`mft.gainwelltechnologies.com`: + +- Outbound (`write_file`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/` +- Inbound (`list_inbound`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/` + +(Paths are relative to the backend cwd; create the directories if +absent.) + +### Daily flow (manual mode) + +1. **Pull inbound from Gainwell** with your SFTP client: + `/CO XIX/PROD/coxix_prod_11525703/FromHPE/` +2. **Stage locally** in `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/` + (or, equivalently, drop them in `/home/tyler/dev/cyclone/ingest/` + and copy in). +3. **Process** with `pull-inbound` (writes acks/835s to DB, dedupes + via `processed_inbound_files`): + + ```bash + cd /home/tyler/dev/cyclone/backend + mkdir -p "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE" + cp /home/tyler/dev/cyclone/ingest/*.x12 \ + "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/" + .venv/bin/python -m cyclone pull-inbound --date 20260701 + .venv/bin/python -m cyclone pull-inbound --date 20260702 + .venv/bin/python -m cyclone pull-inbound --date 20260703 + ``` + +4. **Submit outbound** with `POST /api/clearhouse/submit` (writes + serialized 837P files into the stub staging dir; you drag-drop + them to your SFTP client's `/CO XIX/PROD/coxix_prod_11525703/ToHPE/`). + +### Backfill a backlog + +`ingest/` often holds days of unprocessed acks. The copy + per-date +`pull-inbound` flow above clears it. Files already in +`processed_inbound_files` are skipped automatically — to re-process, +delete the row first (`DELETE FROM processed_inbound_files WHERE name = ...`). + +### Note on per-file parse CLIs + +`parse-837` and `parse-835` exist as CLIs but only emit JSON files to +`--output-dir`; they do NOT write to the DB. There is no +`parse-999` / `parse-ta1` / `parse-277ca` CLI in this version of +cyclone — the canonical 999/TA1/277CA/835 ingestion path is +`pull-inbound` → `Scheduler.process_inbound_files`. For inspection of +a single file without DB writes, use the Python API: + +```python +from cyclone.parsers.parse_999 import parse as parse_999 +result = parse_999(open("path.999.x12").read()) +``` + +### Switching from manual → real (and back) + +When this host's IP is whitelisted with Gainwell's MFT admin, the +SP25 procedure above flips the block. The CLI one-liner: + +```bash +export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS" +.venv/bin/python -c " +from cyclone import db, store +from cyclone.providers import Clearhouse, SftpBlock +db.init_db() +ch = store.store.get_clearhouse() +sb = ch.sftp_block.model_dump() +sb['stub'] = False +sb['auth'] = {'password_keychain_account': 'sftp.gainwell.password'} +new = Clearhouse( + id=1, name=ch.name, tpid=ch.tpid, + submitter_id_qual=ch.submitter_id_qual, + submitter_name=ch.submitter_name, + submitter_contact_name=ch.submitter_contact_name, + submitter_contact_email=ch.submitter_contact_email, + filename_block=ch.filename_block, + sftp_block=SftpBlock.model_validate(sb, strict=True), + updated_at=ch.updated_at, +) +print('updated:', store.store.update_clearhouse(new).sftp_block.stub) +" +``` + +Hot-reload via `PATCH /api/clearhouse` (preferred over direct DB +write — it also calls `scheduler.reconfigure_scheduler` so the running +daemon picks up the new block without a restart): + +```bash +curl -s -b cookies.txt http://127.0.0.1:8000/api/clearhouse > /tmp/ch.json +jq '.sftp_block.stub = false + | .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \ + /tmp/ch.json > /tmp/ch-patched.json +curl -X PATCH http://127.0.0.1:8000/api/clearhouse \ + -H 'Content-Type: application/json' -b cookies.txt \ + --data @/tmp/ch-patched.json +``` + +To revert, set `stub: true` and `auth: {"method": "keychain", "secret_ref": "sftp.gainwell.password"}`.