cyclone/RUNBOOK.md

Cyclone Operator Runbook

Production operations for a single-operator Cyclone deploy on Ubuntu Linux. Assumes the box was bootstrapped via scripts/cyclone-init.sh and the stack is up via docker compose up -d.

Daily

• Confirm the host healthcheck cron hasn't emailed. It pings http://localhost:8080/api/healthz every 5 minutes.
• docker compose ps — both services healthy.
• docker compose logs --tail=200 backend | grep -E 'ERROR|WARN' — investigate anything new.

Weekly

• curl -fsS http://localhost:8080/api/admin/audit-log -b cookies.txt | jq '.events[] | select(.event | test("login_failed|backup.failed"))' — review failed logins + backup failures.
• Confirm docker compose exec backend ls -la /var/lib/cyclone/backups/ shows recent .bin files (within 25h of now).

Quarterly

• Rotate the SQLCipher / cookie-signing key:

  bash scripts/cyclone-init.sh --force   # overwrites /etc/cyclone/secrets/db.key
  docker compose restart backend          # picks up the new key
  

  Old .bin backups become unreadable after this; export them first if you need to keep them.

As needed

• Add an operator. Log in as admin → /admin/users → Create user. Roles: admin / user / viewer.
• Reset a password. Admin UI → Users → Reset password, OR docker compose exec backend python -m cyclone admin reset-password --username <name>.
• Restore from backup. Admin UI → Backups → pick the snapshot → Initiate restore → Confirm. The backend will restart automatically.
• Roll back the code (not the schema). TAG=0.0.9 docker compose up -d. The previous image stays in the local Docker cache for one cycle.
• Pull a new :stable.

  cd /opt/cyclone
  docker compose pull
  docker compose up -d
  docker compose logs -f backend | head -200   # verify migrations + healthcheck
  

• Off-box backup copy. The operator is expected to rsync /var/lib/docker/volumes/cyclone_backups/_data/ to an external drive or NAS nightly. The .bin files are already encrypted; the destination doesn't need its own encryption.
• Inspect the DB. docker compose exec backend sqlite3 /var/lib/cyclone/db/cyclone.db ".tables" (works only if SQLCipher key is on disk; the in-process decrypt happens via the cyclone backend).

Annual

• Rotate the admin password (force re-login for everyone).
• Audit the /etc/cyclone/secrets/ directory permissions — should be chmod 600 root:root.
• Review the audit log for stale admin sessions.

Emergency

• Backend won't start. docker compose logs --tail=300 backend. Look for migration failures (rerun is safe — migrations are forward-only), SQLCipher key mismatch (PRAGMA key failure), or port collisions.
• Frontend won't serve. docker compose logs --tail=100 frontend. Usually nginx config drift; docker compose restart frontend.
• Both unhealthy after a host reboot. Docker may have come up before the named volumes did. docker compose down && docker compose up -d.
• Suspected key compromise. Rotate immediately (see Quarterly above). All active sessions are invalidated.

Where things live

Asset	Path
Docker compose file	/opt/cyclone/docker-compose.yml
Secrets	/etc/cyclone/secrets/{db.key,admin_username,admin_pw}
Live DB (SQLCipher-encrypted volume)	cyclone_db named volume, mounted at /var/lib/cyclone/db
Encrypted backups	cyclone_backups named volume, mounted at /var/lib/cyclone/backups
Uploaded prod files	cyclone_prodfiles named volume
SFTP staging stub	cyclone_sftp_staging named volume
Logs	cyclone_logs named volume + bind-mounted at /var/log/cyclone
Off-box backup destination	Operator's external drive / NAS (rsync cron, not in compose)