feat(sp23): operator scripts (cyclone-init, post-deploy, smoke) + RUNBOOK

cyclone-init.sh generates /etc/cyclone/secrets/{db.key,admin_username,admin_pw} with openssl rand, prints the admin password once. post-deploy.sh installs logrotate.d/cyclone and a 5-minute healthcheck cron. smoke.sh brings up + logs in + parses a sample 837 end-to-end. RUNBOOK.md covers daily/weekly/quarterly/annual ops procedures plus the emergency runbook.
2026-06-23 17:22:44 -06:00
parent d42fbc8c1b
commit 256ddfa5fa
4 changed files with 240 additions and 0 deletions
@@ -0,0 +1,65 @@
+# Cyclone Operator Runbook
+
+Production operations for a single-operator Cyclone deploy on Ubuntu Linux. Assumes the box was bootstrapped via `scripts/cyclone-init.sh` and the stack is up via `docker compose up -d`.
+
+## Daily
+
+- [ ] Confirm the host healthcheck cron hasn't emailed. It pings `http://localhost:8080/api/healthz` every 5 minutes.
+- [ ] `docker compose ps` — both services `healthy`.
+- [ ] `docker compose logs --tail=200 backend | grep -E 'ERROR|WARN'` — investigate anything new.
+
+## Weekly
+
+- [ ] `curl -fsS http://localhost:8080/api/admin/audit-log -b cookies.txt | jq '.events[] | select(.event | test("login_failed|backup.failed"))'` — review failed logins + backup failures.
+- [ ] Confirm `docker compose exec backend ls -la /var/lib/cyclone/backups/` shows recent `.bin` files (within 25h of now).
+
+## Quarterly
+
+- [ ] Rotate the SQLCipher / cookie-signing key:
+  ```bash
+  bash scripts/cyclone-init.sh --force   # overwrites /etc/cyclone/secrets/db.key
+  docker compose restart backend          # picks up the new key
+  ```
+  Old `.bin` backups become unreadable after this; export them first if you need to keep them.
+
+## As needed
+
+- **Add an operator.** Log in as admin → `/admin/users` → Create user. Roles: `admin` / `user` / `viewer`.
+- **Reset a password.** Admin UI → Users → Reset password, OR `docker compose exec backend python -m cyclone admin reset-password --username <name>`.
+- **Restore from backup.** Admin UI → Backups → pick the snapshot → Initiate restore → Confirm. The backend will restart automatically.
+- **Roll back the code (not the schema).** `TAG=0.0.9 docker compose up -d`. The previous image stays in the local Docker cache for one cycle.
+- **Pull a new `:stable`.**
+  ```bash
+  cd /opt/cyclone
+  docker compose pull
+  docker compose up -d
+  docker compose logs -f backend | head -200   # verify migrations + healthcheck
+  ```
+- **Off-box backup copy.** The operator is expected to rsync `/var/lib/docker/volumes/cyclone_backups/_data/` to an external drive or NAS nightly. The `.bin` files are already encrypted; the destination doesn't need its own encryption.
+- **Inspect the DB.** `docker compose exec backend sqlite3 /var/lib/cyclone/db/cyclone.db ".tables"` (works only if SQLCipher key is on disk; the in-process decrypt happens via the cyclone backend).
+
+## Annual
+
+- [ ] Rotate the admin password (force re-login for everyone).
+- [ ] Audit the `/etc/cyclone/secrets/` directory permissions — should be `chmod 600 root:root`.
+- [ ] Review the audit log for stale admin sessions.
+
+## Emergency
+
+- **Backend won't start.** `docker compose logs --tail=300 backend`. Look for migration failures (rerun is safe — migrations are forward-only), SQLCipher key mismatch (`PRAGMA key` failure), or port collisions.
+- **Frontend won't serve.** `docker compose logs --tail=100 frontend`. Usually nginx config drift; `docker compose restart frontend`.
+- **Both unhealthy after a host reboot.** Docker may have come up before the named volumes did. `docker compose down && docker compose up -d`.
+- **Suspected key compromise.** Rotate immediately (see Quarterly above). All active sessions are invalidated.
+
+## Where things live
+
+| Asset | Path |
+|---|---|
+| Docker compose file | `/opt/cyclone/docker-compose.yml` |
+| Secrets | `/etc/cyclone/secrets/{db.key,admin_username,admin_pw}` |
+| Live DB (SQLCipher-encrypted volume) | `cyclone_db` named volume, mounted at `/var/lib/cyclone/db` |
+| Encrypted backups | `cyclone_backups` named volume, mounted at `/var/lib/cyclone/backups` |
+| Uploaded prod files | `cyclone_prodfiles` named volume |
+| SFTP staging stub | `cyclone_sftp_staging` named volume |
+| Logs | `cyclone_logs` named volume + bind-mounted at `/var/log/cyclone` |
+| Off-box backup destination | Operator's external drive / NAS (rsync cron, not in compose) |