Nora
3ba5ca0849
After the 8-commit SP23 implementation landed, kicking the tires on
`docker compose build && docker compose up -d` (the gated DOCKER_TESTS=1
live test) surfaced five real bugs that don't show up in unit tests:
1. Backend wheel was built from the stub `__init__.py`, not the real
source. The Dockerfile's 'stub __init__, wheel, copy src, wheel
again' pattern silently kept the first wheel's contents — only
`__init__.py` got re-stubbed. The installed package had an empty
`__init__.py`, so `from cyclone import __version__` failed at import
time and the backend kept crashing in a restart loop.
Fix: single `COPY src/` + single `pip wheel`. Comment explains
why the stub trick is gone for good.
2. Backend binds to 127.0.0.1 (intentional — local-only by design,
see CLAUDE.md). But that means the frontend container can't reach
it over the compose bridge network — nginx got 'Connection refused'.
Fix: `CYCLONE_HOST` env var, defaults to 127.0.0.1 (preserves local
posture for non-Docker runs), set to 0.0.0.0 by the docker-compose
backend service. Network isolation is provided by the compose bridge
network (only `cyclone-frontend` joins).
3. Healthcheck probed `/api/healthz` (404 — the route is `/api/health`).
Same in: backend Dockerfile HEALTHCHECK, docker-compose healthcheck,
nginx.conf doesn't have one (frontend proxies through), RUNBOOK.md,
scripts/post-deploy.sh, scripts/smoke.sh.
Fix: `/api/healthz` → `/api/health` everywhere SP23 owns.
4. The auth matrix in `cyclone.auth.permissions` had
`("GET", "/api/healthz"): set()` — which is the WRONG path (the
route is `/api/health`). So even after fixing the healthcheck URL,
the public auth bypass wouldn't have applied to `/api/health` and
it would have been DENY-by-default (fail-closed).
Fix: matrix entry updated to `/api/health`.
5. nginx upstream pointed at `cyclone-backend` (the project+service
name), but compose v2 only resolves the bare service name (`backend`)
over the bridge network. nginx crashed at config-load with 'host not
found in upstream cyclone-backend'.
Fix: `cyclone-backend:8000` → `backend:8000` in nginx.conf + spec
+ plan.
6. Frontend HEALTHCHECK used `http://localhost:8080/`. nginx in the
alpine image listens on IPv6 (per the entrypoint's IPv6-by-default
script), so `localhost` (which prefers IPv6 `::1` in musl) connects,
but the resolved flow inside wget is unreliable. `127.0.0.1` works.
Fix: HEALTHCHECK uses `http://127.0.0.1:8080/`.
Also moves `frontend/Dockerfile` → `Dockerfile.frontend` and
`frontend/nginx.conf` → `nginx.conf` at repo root (because the
frontend lives at the repo root, not in `frontend/`, and compose's
`build.context: .` needs them at the same root as compose.yml).
The frontend's pre-existing `.dockerignore` was empty/unused, so it's
dropped — the root `.dockerignore` covers it.
Adds `docker-compose.override.yml` for local bring-up testing on a
host without sudo. Production uses `/etc/cyclone/secrets/` directly.
Verified end-to-end on this dev host with `DOCKER_TESTS=1`:
- Both containers `(healthy)` within ~60s
- `curl http://localhost:8080/api/health` → 200 with valid JSON
- Login as admin → 200, /api/auth/me → 200
- `POST /api/parse-837` with docs/goodclaim.x12 → 200, batch created
- Full backend test suite: 1014 passed, 9 skipped (prodfiles gitignored)
66 lines
3.8 KiB
Markdown
66 lines
3.8 KiB
Markdown
# Cyclone Operator Runbook
|
|
|
|
Production operations for a single-operator Cyclone deploy on Ubuntu Linux. Assumes the box was bootstrapped via `scripts/cyclone-init.sh` and the stack is up via `docker compose up -d`.
|
|
|
|
## Daily
|
|
|
|
- [ ] Confirm the host healthcheck cron hasn't emailed. It pings `http://localhost:8080/api/health` every 5 minutes.
|
|
- [ ] `docker compose ps` — both services `healthy`.
|
|
- [ ] `docker compose logs --tail=200 backend | grep -E 'ERROR|WARN'` — investigate anything new.
|
|
|
|
## Weekly
|
|
|
|
- [ ] `curl -fsS http://localhost:8080/api/admin/audit-log -b cookies.txt | jq '.events[] | select(.event | test("login_failed|backup.failed"))'` — review failed logins + backup failures.
|
|
- [ ] Confirm `docker compose exec backend ls -la /var/lib/cyclone/backups/` shows recent `.bin` files (within 25h of now).
|
|
|
|
## Quarterly
|
|
|
|
- [ ] Rotate the SQLCipher / cookie-signing key:
|
|
```bash
|
|
bash scripts/cyclone-init.sh --force # overwrites /etc/cyclone/secrets/db.key
|
|
docker compose restart backend # picks up the new key
|
|
```
|
|
Old `.bin` backups become unreadable after this; export them first if you need to keep them.
|
|
|
|
## As needed
|
|
|
|
- **Add an operator.** Log in as admin → `/admin/users` → Create user. Roles: `admin` / `user` / `viewer`.
|
|
- **Reset a password.** Admin UI → Users → Reset password, OR `docker compose exec backend python -m cyclone admin reset-password --username <name>`.
|
|
- **Restore from backup.** Admin UI → Backups → pick the snapshot → Initiate restore → Confirm. The backend will restart automatically.
|
|
- **Roll back the code (not the schema).** `TAG=0.0.9 docker compose up -d`. The previous image stays in the local Docker cache for one cycle.
|
|
- **Pull a new `:stable`.**
|
|
```bash
|
|
cd /opt/cyclone
|
|
docker compose pull
|
|
docker compose up -d
|
|
docker compose logs -f backend | head -200 # verify migrations + healthcheck
|
|
```
|
|
- **Off-box backup copy.** The operator is expected to rsync `/var/lib/docker/volumes/cyclone_backups/_data/` to an external drive or NAS nightly. The `.bin` files are already encrypted; the destination doesn't need its own encryption.
|
|
- **Inspect the DB.** `docker compose exec backend sqlite3 /var/lib/cyclone/db/cyclone.db ".tables"` (works only if SQLCipher key is on disk; the in-process decrypt happens via the cyclone backend).
|
|
|
|
## Annual
|
|
|
|
- [ ] Rotate the admin password (force re-login for everyone).
|
|
- [ ] Audit the `/etc/cyclone/secrets/` directory permissions — should be `chmod 600 root:root`.
|
|
- [ ] Review the audit log for stale admin sessions.
|
|
|
|
## Emergency
|
|
|
|
- **Backend won't start.** `docker compose logs --tail=300 backend`. Look for migration failures (rerun is safe — migrations are forward-only), SQLCipher key mismatch (`PRAGMA key` failure), or port collisions.
|
|
- **Frontend won't serve.** `docker compose logs --tail=100 frontend`. Usually nginx config drift; `docker compose restart frontend`.
|
|
- **Both unhealthy after a host reboot.** Docker may have come up before the named volumes did. `docker compose down && docker compose up -d`.
|
|
- **Suspected key compromise.** Rotate immediately (see Quarterly above). All active sessions are invalidated.
|
|
|
|
## Where things live
|
|
|
|
| Asset | Path |
|
|
|---|---|
|
|
| Docker compose file | `/opt/cyclone/docker-compose.yml` |
|
|
| Secrets | `/etc/cyclone/secrets/{db.key,admin_username,admin_pw}` |
|
|
| Live DB (SQLCipher-encrypted volume) | `cyclone_db` named volume, mounted at `/var/lib/cyclone/db` |
|
|
| Encrypted backups | `cyclone_backups` named volume, mounted at `/var/lib/cyclone/backups` |
|
|
| Uploaded prod files | `cyclone_prodfiles` named volume |
|
|
| SFTP staging stub | `cyclone_sftp_staging` named volume |
|
|
| Logs | `cyclone_logs` named volume + bind-mounted at `/var/log/cyclone` |
|
|
| Off-box backup destination | Operator's external drive / NAS (rsync cron, not in compose) |
|