docs: document manual SFTP mode for this host (post-SP36)
- .env.example: add GAINWELL_SFTP_* mirror vars with bridge instructions - .gitignore: exclude ingest/ + macOS AppleDouble residue - CLAUDE.md: 'Production SFTP posture' section (env bridge, inbound drop zone, auth caveat, ingestion paths, daemon hot-reload caveat) - docs/RUNBOOK.md: 'Manual SFTP mode' section (daily flow, backfill, parse-CLI caveat, manual→real switch procedure) Recalibration after the auth-failed incident from 103.14.26.95 — local SFTP uploads won't work until Gainwell whitelists this IP, so the canonical flow is now manual file copy + cyclone pull-inbound.
This commit is contained in:
Nora
+107
@@ -138,3 +138,110 @@ started without the secret mount), the scheduler surfaces a
|
||||
`RuntimeError` at the next tick that names the env var and the
|
||||
missing path — this is intentional, so a silent fall-through doesn't
|
||||
mask a real misconfiguration.
|
||||
|
||||
## Manual SFTP mode (this box's current posture)
|
||||
|
||||
Use this when the host's IP isn't whitelisted with Gainwell's MFT, when
|
||||
you don't want a daemon polling every minute, or when you prefer to
|
||||
drag-drop files with FileZilla / WinSCP. The seeded `dzinesco`
|
||||
clearhouse ships in this mode (`sftp_block.stub: true`); no daemon
|
||||
changes required.
|
||||
|
||||
**Posture.** `SftpClient` reads/writes to local staging instead of
|
||||
`mft.gainwelltechnologies.com`:
|
||||
|
||||
- Outbound (`write_file`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/ToHPE/`
|
||||
- Inbound (`list_inbound`): `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
|
||||
|
||||
(Paths are relative to the backend cwd; create the directories if
|
||||
absent.)
|
||||
|
||||
### Daily flow (manual mode)
|
||||
|
||||
1. **Pull inbound from Gainwell** with your SFTP client:
|
||||
`/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
|
||||
2. **Stage locally** in `./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/`
|
||||
(or, equivalently, drop them in `/home/tyler/dev/cyclone/ingest/`
|
||||
and copy in).
|
||||
3. **Process** with `pull-inbound` (writes acks/835s to DB, dedupes
|
||||
via `processed_inbound_files`):
|
||||
|
||||
```bash
|
||||
cd /home/tyler/dev/cyclone/backend
|
||||
mkdir -p "./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE"
|
||||
cp /home/tyler/dev/cyclone/ingest/*.x12 \
|
||||
"./var/sftp/staging/CO XIX/PROD/coxix_prod_11525703/FromHPE/"
|
||||
.venv/bin/python -m cyclone pull-inbound --date 20260701
|
||||
.venv/bin/python -m cyclone pull-inbound --date 20260702
|
||||
.venv/bin/python -m cyclone pull-inbound --date 20260703
|
||||
```
|
||||
|
||||
4. **Submit outbound** with `POST /api/clearhouse/submit` (writes
|
||||
serialized 837P files into the stub staging dir; you drag-drop
|
||||
them to your SFTP client's `/CO XIX/PROD/coxix_prod_11525703/ToHPE/`).
|
||||
|
||||
### Backfill a backlog
|
||||
|
||||
`ingest/` often holds days of unprocessed acks. The copy + per-date
|
||||
`pull-inbound` flow above clears it. Files already in
|
||||
`processed_inbound_files` are skipped automatically — to re-process,
|
||||
delete the row first (`DELETE FROM processed_inbound_files WHERE name = ...`).
|
||||
|
||||
### Note on per-file parse CLIs
|
||||
|
||||
`parse-837` and `parse-835` exist as CLIs but only emit JSON files to
|
||||
`--output-dir`; they do NOT write to the DB. There is no
|
||||
`parse-999` / `parse-ta1` / `parse-277ca` CLI in this version of
|
||||
cyclone — the canonical 999/TA1/277CA/835 ingestion path is
|
||||
`pull-inbound` → `Scheduler.process_inbound_files`. For inspection of
|
||||
a single file without DB writes, use the Python API:
|
||||
|
||||
```python
|
||||
from cyclone.parsers.parse_999 import parse as parse_999
|
||||
result = parse_999(open("path.999.x12").read())
|
||||
```
|
||||
|
||||
### Switching from manual → real (and back)
|
||||
|
||||
When this host's IP is whitelisted with Gainwell's MFT admin, the
|
||||
SP25 procedure above flips the block. The CLI one-liner:
|
||||
|
||||
```bash
|
||||
export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
|
||||
.venv/bin/python -c "
|
||||
from cyclone import db, store
|
||||
from cyclone.providers import Clearhouse, SftpBlock
|
||||
db.init_db()
|
||||
ch = store.store.get_clearhouse()
|
||||
sb = ch.sftp_block.model_dump()
|
||||
sb['stub'] = False
|
||||
sb['auth'] = {'password_keychain_account': 'sftp.gainwell.password'}
|
||||
new = Clearhouse(
|
||||
id=1, name=ch.name, tpid=ch.tpid,
|
||||
submitter_id_qual=ch.submitter_id_qual,
|
||||
submitter_name=ch.submitter_name,
|
||||
submitter_contact_name=ch.submitter_contact_name,
|
||||
submitter_contact_email=ch.submitter_contact_email,
|
||||
filename_block=ch.filename_block,
|
||||
sftp_block=SftpBlock.model_validate(sb, strict=True),
|
||||
updated_at=ch.updated_at,
|
||||
)
|
||||
print('updated:', store.store.update_clearhouse(new).sftp_block.stub)
|
||||
"
|
||||
```
|
||||
|
||||
Hot-reload via `PATCH /api/clearhouse` (preferred over direct DB
|
||||
write — it also calls `scheduler.reconfigure_scheduler` so the running
|
||||
daemon picks up the new block without a restart):
|
||||
|
||||
```bash
|
||||
curl -s -b cookies.txt http://127.0.0.1:8000/api/clearhouse > /tmp/ch.json
|
||||
jq '.sftp_block.stub = false
|
||||
| .sftp_block.auth = {"password_keychain_account": "sftp.gainwell.password"}' \
|
||||
/tmp/ch.json > /tmp/ch-patched.json
|
||||
curl -X PATCH http://127.0.0.1:8000/api/clearhouse \
|
||||
-H 'Content-Type: application/json' -b cookies.txt \
|
||||
--data @/tmp/ch-patched.json
|
||||
```
|
||||
|
||||
To revert, set `stub: true` and `auth: {"method": "keychain", "secret_ref": "sftp.gainwell.password"}`.
|
||||
|
||||
Reference in New Issue
Block a user