# Cyclone Operator Runbook

Production operations for a single-operator Cyclone deploy on Ubuntu Linux. Assumes the box was bootstrapped via `scripts/cyclone-init.sh` and the stack is up via `docker compose up -d`.

## Daily

- [ ] Confirm the host healthcheck cron hasn't emailed. It pings `http://localhost:8080/api/healthz` every 5 minutes.
- [ ] `docker compose ps` — both services `healthy`.
- [ ] `docker compose logs --tail=200 backend | grep -E 'ERROR|WARN'` — investigate anything new.

## Weekly

- [ ] `curl -fsS http://localhost:8080/api/admin/audit-log -b cookies.txt | jq '.events[] | select(.event | test("login_failed|backup.failed"))'` — review failed logins + backup failures.
- [ ] Confirm `docker compose exec backend ls -la /var/lib/cyclone/backups/` shows recent `.bin` files (within 25h of now).

## Quarterly

- [ ] Rotate the SQLCipher / cookie-signing key:
  ```bash
  bash scripts/cyclone-init.sh --force   # overwrites /etc/cyclone/secrets/db.key
  docker compose restart backend          # picks up the new key
  ```
  Old `.bin` backups become unreadable after this; export them first if you need to keep them.

## As needed

- **Add an operator.** Log in as admin → `/admin/users` → Create user. Roles: `admin` / `user` / `viewer`.
- **Reset a password.** Admin UI → Users → Reset password, OR `docker compose exec backend python -m cyclone admin reset-password --username <name>`.
- **Restore from backup.** Admin UI → Backups → pick the snapshot → Initiate restore → Confirm. The backend will restart automatically.
- **Roll back the code (not the schema).** `TAG=0.0.9 docker compose up -d`. The previous image stays in the local Docker cache for one cycle.
- **Pull a new `:stable`.**
  ```bash
  cd /opt/cyclone
  docker compose pull
  docker compose up -d
  docker compose logs -f backend | head -200   # verify migrations + healthcheck
  ```
- **Off-box backup copy.** The operator is expected to rsync `/var/lib/docker/volumes/cyclone_backups/_data/` to an external drive or NAS nightly. The `.bin` files are already encrypted; the destination doesn't need its own encryption.
- **Inspect the DB.** `docker compose exec backend sqlite3 /var/lib/cyclone/db/cyclone.db ".tables"` (works only if SQLCipher key is on disk; the in-process decrypt happens via the cyclone backend).

## Annual

- [ ] Rotate the admin password (force re-login for everyone).
- [ ] Audit the `/etc/cyclone/secrets/` directory permissions — should be `chmod 600 root:root`.
- [ ] Review the audit log for stale admin sessions.

## Emergency

- **Backend won't start.** `docker compose logs --tail=300 backend`. Look for migration failures (rerun is safe — migrations are forward-only), SQLCipher key mismatch (`PRAGMA key` failure), or port collisions.
- **Frontend won't serve.** `docker compose logs --tail=100 frontend`. Usually nginx config drift; `docker compose restart frontend`.
- **Both unhealthy after a host reboot.** Docker may have come up before the named volumes did. `docker compose down && docker compose up -d`.
- **Suspected key compromise.** Rotate immediately (see Quarterly above). All active sessions are invalidated.

## Where things live

| Asset | Path |
|---|---|
| Docker compose file | `/opt/cyclone/docker-compose.yml` |
| Secrets | `/etc/cyclone/secrets/{db.key,admin_username,admin_pw}` |
| Live DB (SQLCipher-encrypted volume) | `cyclone_db` named volume, mounted at `/var/lib/cyclone/db` |
| Encrypted backups | `cyclone_backups` named volume, mounted at `/var/lib/cyclone/backups` |
| Uploaded prod files | `cyclone_prodfiles` named volume |
| SFTP staging stub | `cyclone_sftp_staging` named volume |
| Logs | `cyclone_logs` named volume + bind-mounted at `/var/log/cyclone` |
| Off-box backup destination | Operator's external drive / NAS (rsync cron, not in compose) |