diff --git a/docs/superpowers/plans/2026-06-23-cyclone-ubuntu-docker-deployment.md b/docs/superpowers/plans/2026-06-23-cyclone-ubuntu-docker-deployment.md new file mode 100644 index 0000000..d633e3a --- /dev/null +++ b/docs/superpowers/plans/2026-06-23-cyclone-ubuntu-docker-deployment.md @@ -0,0 +1,1154 @@ +# Ubuntu Docker Deployment Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use +> superpowers:subagent-driven-development (recommended) or +> superpowers:executing-plans to implement this plan task-by-task. Steps +> use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Wrap the existing Cyclone stack (auth already on `main` + the FastAPI backend + the React SPA) in a two-container Docker Compose deployment for production on a single Ubuntu Linux server, with daily encrypted backups, SQLCipher-at-rest with a Docker-secret key, LAN-bind, container healthcheck + restart, an operator runbook, and reproducible bootstrap + smoke scripts. + +**Architecture:** `docker-compose.yml` at repo root orchestrates two containers — `cyclone-backend` (FastAPI on `python:3.11-slim-bookworm`, internal port 8000) and `cyclone-frontend` (nginx:1.27-alpine serving the built SPA + reverse-proxying `/api/*` to the backend on the compose-managed bridge, host port 8080). Named Docker volumes hold the SQLCipher-encrypted DB, encrypted backups, uploaded prod files, SFTP staging, and JSON logs. Host-managed secrets at `/etc/cyclone/secrets/` are mounted as Docker secrets into the backend. The existing `BackupService` (SP17) is wired to autostart via `CYCLONE_BACKUP_AUTOSTART=1` on container boot with 24h interval and 14-day retention. Bootstrap is `git clone` → `bash scripts/cyclone-init.sh` → `docker compose build` → `docker compose up -d`. + +**Tech Stack:** Docker (multi-stage builds), Docker Compose v2, `python:3.11-slim-bookworm`, `nginx:1.27-alpine`, `node:20-alpine` (build stage only), bash for ops scripts, SQLCipher (`libsqlcipher-dev`), tini (PID 1), curl + jq in smoke script. + +**Spec:** [`docs/superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md`](../specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md) + +**Depends on auth-on-main:** Auth (bcrypt + HttpOnly session cookie + `matrix_gate` + `Login.tsx` + `AuthProvider`) is already on `main` (2026-06-23). SP23 does **not** re-implement auth; it wires the Docker posture around it. See spec §1.1 for the full delta vs. main. + +--- + +## File structure + +New files: +- `backend/Dockerfile` — multi-stage build for the FastAPI service +- `backend/.dockerignore` — exclude `.venv`, tests, fixtures from build context +- `frontend/Dockerfile` — multi-stage build (node:20-alpine → nginx:1.27-alpine) +- `frontend/nginx.conf` — SPA routing + reverse proxy to backend +- `frontend/.dockerignore` — exclude node_modules, src/, test files +- `docker-compose.yml` — two-service orchestration + named volumes + secrets +- `.dockerignore` — repo-root ignore for compose build contexts +- `scripts/cyclone-init.sh` — generate `/etc/cyclone/secrets/*` + print admin password +- `scripts/post-deploy.sh` — set up logrotate + healthcheck cron on host +- `scripts/smoke.sh` — end-to-end bring-up + login + parse + export test +- `RUNBOOK.md` — operator daily / weekly / quarterly / as-needed procedures +- `backend/tests/test_docker.py` — `docker compose config` validation + Dockerfile-build smoke tests + +Modified files: +- `README.md` (root, if present) or `docs/ARCHITECTURE.md` — link to `RUNBOOK.md` from the deployment section +- `.gitignore` — add `dist/` (already there), `/run/secrets/*` no-op (Docker secrets are not on host) + +No new SQL migration — auth tables (`users`, `sessions`) already exist at `0013_auth_users_and_sessions.sql` + `0014_audit_log_user_id.sql`. + +--- + +## Task 0: Worktree + branch (already done) + +- [x] Worktree at `.worktrees/sp23-ubuntu-docker-deployment` on branch `sp23-ubuntu-docker-deployment`, off `main@c398aa7`. +- [x] Spec patch committed (`35c561d`): status `Approved` + §1.1 Delta vs. main. + +This task is pre-completed by the dispatcher. Listed for completeness. + +--- + +## Task 1: Backend `Dockerfile` + `backend/.dockerignore` + +**Files:** +- Create: `backend/Dockerfile` +- Create: `backend/.dockerignore` + +- [ ] **Step 1: Write `backend/.dockerignore`** + +Create `backend/.dockerignore` with this exact content: + +``` +.venv/ +venv/ +__pycache__/ +*.py[cod] +*.egg-info/ +.pytest_cache/ +.ruff_cache/ +tests/ +docs/prodfiles/ +*.production.txt +.git/ +.github/ +``` + +- [ ] **Step 2: Write the multi-stage `backend/Dockerfile`** + +Create `backend/Dockerfile`: + +```dockerfile +# syntax=docker/dockerfile:1.7 +# Builder stage — installs cyclone + sqlcipher extra into a wheel dir. +FROM python:3.11-slim-bookworm AS builder + +ENV PIP_NO_CACHE_DIR=1 \ + PIP_DISABLE_PIP_VERSION_CHECK=1 \ + PYTHONDONTWRITEBYTECODE=1 + +RUN apt-get update && apt-get install -y --no-install-recommends \ + build-essential \ + libffi-dev \ + libsqlcipher-dev \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /build + +# Copy ONLY the build manifest first so this layer caches across source edits. +COPY pyproject.toml ./ + +# Stub package so `pip wheel` can resolve `cyclone` without the source tree yet. +RUN mkdir -p src/cyclone && touch src/cyclone/__init__.py +RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher]' + +COPY src/ ./src/ +RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher]' + +# Runtime stage — slim base, non-root user, healthcheck, tini PID 1. +FROM python:3.11-slim-bookworm + +ENV PYTHONUNBUFFERED=1 \ + PYTHONDONTWRITEBYTECODE=1 + +RUN apt-get update && apt-get install -y --no-install-recommends \ + libsqlcipher-dev \ + curl \ + tini \ + && rm -rf /var/lib/apt/lists/* \ + && useradd --create-home --uid 1000 --shell /bin/bash cyclone + +WORKDIR /app + +COPY --from=builder /wheels /wheels +RUN pip install --no-cache-dir --no-index --find-links /wheels cyclone \ + && rm -rf /wheels + +COPY --chown=cyclone:cyclone src/ /app/src/ + +USER cyclone + +EXPOSE 8000 + +HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \ + CMD curl -fs http://localhost:8000/api/healthz || exit 1 + +ENTRYPOINT ["tini", "--"] +CMD ["python", "-m", "cyclone", "serve"] +``` + +- [ ] **Step 3: Verify `backend/Dockerfile` parses** + +Run from repo root: + +```bash +docker build --check -f backend/Dockerfile backend/ +``` + +Expected: exit code 0, no errors. (Older Docker engines may not have `--check`; fall back to `docker build --target builder backend/` if needed.) + +- [ ] **Step 4: Commit** + +```bash +git add backend/Dockerfile backend/.dockerignore +git commit -m "feat(sp23): backend Dockerfile (multi-stage, sqlcipher, non-root, healthcheck)" +``` + +--- + +## Task 2: Frontend `Dockerfile` + `frontend/nginx.conf` + `frontend/.dockerignore` + +**Files:** +- Create: `frontend/Dockerfile` +- Create: `frontend/nginx.conf` +- Create: `frontend/.dockerignore` + +- [ ] **Step 1: Write `frontend/.dockerignore`** + +``` +node_modules/ +dist/ +build/ +*.tsbuildinfo +coverage/ +src/**/*.test.ts +src/**/*.test.tsx +src/test/ +.git/ +.github/ +``` + +- [ ] **Step 2: Write `frontend/nginx.conf`** + +```nginx +server { + listen 8080; + server_name _; + + root /usr/share/nginx/html; + index index.html; + + # Long-lived connections for live-tail NDJSON streams. + proxy_read_timeout 300s; + proxy_send_timeout 300s; + client_max_body_size 50m; + + # API + auth: proxy to backend over the compose-managed bridge network. + location /api/ { + proxy_pass http://cyclone-backend:8000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + # Cookie comes back from backend on /api/; rewrite path to / so the + # browser sends it on every subsequent request to the SPA. + proxy_cookie_path /api/ /; + } + + # SPA fallback: serve index.html for any non-/api route. + location / { + try_files $uri $uri/ /index.html; + } + + # Don't cache the index.html — the SPA references hashed asset filenames. + location = /index.html { + add_header Cache-Control "no-cache, no-store, must-revalidate" always; + expires off; + } +} +``` + +- [ ] **Step 3: Write `frontend/Dockerfile`** + +```dockerfile +# syntax=docker/dockerfile:1.7 +# Build stage. +FROM node:20-alpine AS builder + +WORKDIR /build + +COPY package.json package-lock.json* ./ +RUN npm ci + +COPY . . +RUN npm run build + +# Runtime stage — nginx serving the built SPA. +FROM nginx:1.27-alpine + +RUN rm -f /etc/nginx/conf.d/default.conf + +COPY nginx.conf /etc/nginx/conf.d/default.conf +COPY --from=builder /build/dist /usr/share/nginx/html + +HEALTHCHECK --interval=30s --timeout=5s --retries=3 \ + CMD wget -qO- http://localhost:8080/ >/dev/null || exit 1 + +EXPOSE 8080 +``` + +- [ ] **Step 4: Verify `frontend/Dockerfile` parses** + +```bash +docker build --check -f frontend/Dockerfile frontend/ +``` + +Expected: exit code 0. (Fall back to building a stub `dist/` first if `--check` is unavailable.) + +- [ ] **Step 5: Commit** + +```bash +git add frontend/Dockerfile frontend/nginx.conf frontend/.dockerignore +git commit -m "feat(sp23): frontend Dockerfile + nginx.conf (SPA + reverse proxy)" +``` + +--- + +## Task 3: `docker-compose.yml` at repo root + +**Files:** +- Create: `docker-compose.yml` +- Create: `.dockerignore` (repo root, protects compose build contexts from accidental host pollution) + +- [ ] **Step 1: Write the repo-root `.dockerignore`** + +``` +.worktrees/ +.git/ +.github/ +docs/prodfiles/ +*.production.txt +node_modules/ +dist/ +.venv/ +.superpowers/brainstorm/ +``` + +- [ ] **Step 2: Write `docker-compose.yml`** + +```yaml +name: cyclone + +services: + backend: + image: cyclone-backend:${TAG:-stable} + build: + context: ./backend + restart: unless-stopped + environment: + CYCLONE_DB_URL: "sqlite:////var/lib/cyclone/db/cyclone.db" + CYCLONE_BACKUP_AUTOSTART: "1" + CYCLONE_BACKUP_INTERVAL_HOURS: "24" + CYCLONE_BACKUP_RETENTION_DAYS: "14" + CYCLONE_LOG_LEVEL: "INFO" + CYCLONE_LOG_FILE: "/var/log/cyclone/cyclone.log" + CYCLONE_LOG_JSON: "1" + CYCLONE_SESSION_LIFETIME_HOURS: "24" + # The cookie signing key is the SQLCipher key by default; override + # CYCLONE_SECRET_KEY_FILE below to split them in production. + CYCLONE_SECRET_KEY_FILE: "/run/secrets/cyclone_db_key" + CYCLONE_COOKIE_SECURE: "1" + # First-admin bootstrap — set by cyclone-init.sh from /etc/cyclone/secrets/admin_pw. + CYCLONE_ADMIN_USERNAME_FILE: "/run/secrets/cyclone_admin_username" + CYCLONE_ADMIN_PASSWORD_FILE: "/run/secrets/cyclone_admin_password" + secrets: + - cyclone_db_key + - cyclone_admin_username + - cyclone_admin_password + volumes: + - cyclone_db:/var/lib/cyclone/db + - cyclone_backups:/var/lib/cyclone/backups + - cyclone_prodfiles:/var/lib/cyclone/prodfiles + - cyclone_sftp_staging:/var/lib/cyclone/sftp_staging + - cyclone_logs:/var/log/cyclone + healthcheck: + test: ["CMD", "curl", "-fs", "http://localhost:8000/api/healthz"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 30s + networks: + - cyclone_network + + frontend: + image: cyclone-frontend:${TAG:-stable} + build: + context: ./frontend + restart: unless-stopped + ports: + - "8080:8080" + depends_on: + backend: + condition: service_healthy + networks: + - cyclone_network + +secrets: + cyclone_db_key: + file: /etc/cyclone/secrets/db.key + cyclone_admin_username: + file: /etc/cyclone/secrets/admin_username + cyclone_admin_password: + file: /etc/cyclone/secrets/admin_pw + +volumes: + cyclone_db: + cyclone_backups: + cyclone_prodfiles: + cyclone_sftp_staging: + cyclone_logs: + +networks: + cyclone_network: + driver: bridge +``` + +- [ ] **Step 3: Validate compose** + +```bash +docker compose config --quiet +``` + +Expected: exit code 0, no output. Fix any schema warnings before continuing. + +- [ ] **Step 4: Commit** + +```bash +git add docker-compose.yml .dockerignore +git commit -m "feat(sp23): docker-compose.yml with backend + frontend, named volumes, secrets" +``` + +--- + +## Task 4: `scripts/cyclone-init.sh` + +**Files:** +- Create: `scripts/cyclone-init.sh` + +- [ ] **Step 1: Write the bootstrap script** + +Create `scripts/cyclone-init.sh`: + +```bash +#!/usr/bin/env bash +# cyclone-init.sh — first-time host bootstrap for a production Cyclone deploy. +# +# Generates /etc/cyclone/secrets/{db.key,admin_username,admin_pw} with +# openssl rand -hex 32, sets ownership + permissions, and prints the +# admin password ONCE for the operator to capture. +# +# Idempotent: refuses to overwrite existing secrets unless --force. +# +# Usage: +# bash scripts/cyclone-init.sh # generate if missing +# bash scripts/cyclone-init.sh --force # overwrite (destroys old data!) +set -euo pipefail + +SECRETS_DIR="${CYCLONE_SECRETS_DIR:-/etc/cyclone/secrets}" +FORCE=0 +[[ "${1:-}" == "--force" ]] && FORCE=1 + +if [[ $EUID -ne 0 ]]; then + echo "ERROR: cyclone-init.sh must run as root (it writes to ${SECRETS_DIR})." >&2 + exit 1 +fi + +if [[ -f "${SECRETS_DIR}/db.key" && $FORCE -eq 0 ]]; then + echo "Secrets already exist at ${SECRETS_DIR}. Re-run with --force to overwrite." >&2 + exit 0 +fi + +mkdir -p "${SECRETS_DIR}" +chmod 700 "${SECRETS_DIR}" + +# 64-hex-char (256-bit) random key for SQLCipher + cookie signing. +openssl rand -hex 32 > "${SECRETS_DIR}/db.key" + +# Admin username defaults to "admin" unless CYCLONE_ADMIN_USERNAME is set. +ADMIN_USER="${CYCLONE_ADMIN_USERNAME:-admin}" +printf '%s' "${ADMIN_USER}" > "${SECRETS_DIR}/admin_username" + +# Memorable-but-random admin password — printed once. +ADMIN_PW="$(openssl rand -base64 18 | tr -d '/+=' | head -c 20)Aa1!" +printf '%s' "${ADMIN_PW}" > "${SECRETS_DIR}/admin_pw" + +chmod 600 "${SECRETS_DIR}"/* +chown -R root:root "${SECRETS_DIR}" + +cat <:8080/login and log in as '${ADMIN_USER}'. +Change the password from /admin/users immediately. +================================================================ +EOF +``` + +- [ ] **Step 2: Make it executable** + +```bash +chmod +x scripts/cyclone-init.sh +``` + +- [ ] **Step 3: Syntax check** + +```bash +bash -n scripts/cyclone-init.sh +``` + +Expected: exit code 0, no output. + +- [ ] **Step 4: Commit** + +```bash +git add scripts/cyclone-init.sh +git commit -m "feat(sp23): scripts/cyclone-init.sh generates /etc/cyclone/secrets/*" +``` + +--- + +## Task 5: `scripts/post-deploy.sh` + +**Files:** +- Create: `scripts/post-deploy.sh` + +- [ ] **Step 1: Write the post-deploy script** + +Create `scripts/post-deploy.sh`: + +```bash +#!/usr/bin/env bash +# post-deploy.sh — set up host-side logrotate and healthcheck cron for a +# production Cyclone deploy. Run once after `docker compose up -d`. +set -euo pipefail + +LOG_DIR="/var/log/cyclone" +HEALTHCHECK_URL="http://localhost:8080/api/healthz" +HEALTHCHECK_EMAIL="${CYCLONE_HEALTHCHECK_EMAIL:-root}" +CRON_USER="${CYCLONE_CRON_USER:-root}" + +if [[ $EUID -ne 0 ]]; then + echo "ERROR: post-deploy.sh must run as root." >&2 + exit 1 +fi + +# 1. Logrotate — keeps 14 days of JSON logs compressed. +cat > /etc/logrotate.d/cyclone <<'LOGROTATE' +/var/log/cyclone/*.log { + daily + rotate 14 + compress + delaycompress + missingok + notifempty + copytruncate + dateext + dateformat -%Y%m%d +} +LOGROTATE + +mkdir -p "${LOG_DIR}" +chmod 755 "${LOG_DIR}" + +# 2. Healthcheck cron — every 5 minutes, email if down. +CRON_LINE="*/5 * * * * curl -fsS --max-time 10 ${HEALTHCHECK_URL} >/dev/null 2>&1 || echo 'Cyclone DOWN at $(date)' | mail -s 'Cyclone healthz FAIL' ${HEALTHCHECK_EMAIL}" +TMP_CRON="$(mktemp)" +crontab -u "${CRON_USER}" -l 2>/dev/null > "${TMP_CRON}" || true +# Remove any existing cyclone healthcheck line, then re-add. +grep -v -F "cyclone healthz" "${TMP_CRON}" > "${TMP_CRON}.new" || cp "${TMP_CRON}" "${TMP_CRON}.new" +echo "${CRON_LINE}" >> "${TMP_CRON}.new" +crontab -u "${CRON_USER}" "${TMP_CRON}.new" +rm -f "${TMP_CRON}" "${TMP_CRON}.new" + +cat < waiting for healthz..." +for i in {1..30}; do + if curl -fsS "${BASE_URL}/api/healthz" >/dev/null 2>&1; then break; fi + sleep 2 + [[ $i -eq 30 ]] && { echo "FAIL: healthz never came up"; exit 1; } +done +echo " ok" + +echo "==> logging in as admin..." +HTTP_CODE=$(curl -sS -o /dev/null -w '%{http_code}' \ + -c "${COOKIE_JAR}" \ + -H 'Content-Type: application/json' \ + -X POST "${BASE_URL}/api/auth/login" \ + -d "{\"username\":\"admin\",\"password\":\"${ADMIN_PW}\"}") +[[ "${HTTP_CODE}" == "200" ]] || { echo "FAIL: login returned ${HTTP_CODE}"; exit 1; } +echo " ok (cookie set)" + +echo "==> /api/auth/me..." +ME=$(curl -fsS -b "${COOKIE_JAR}" "${BASE_URL}/api/auth/me") +echo " ${ME}" + +echo "==> parsing sample 837 (${SAMPLE_837})..." +[[ -f "${SAMPLE_837}" ]] || { echo "skip: sample not found"; exit 0; } +PARSE=$(curl -fsS -b "${COOKIE_JAR}" -X POST "${BASE_URL}/api/parse-837" \ + -F "file=@${SAMPLE_837}") +echo " ${PARSE}" | head -c 200 +echo + +echo "==> listing batches..." +BATCHES=$(curl -fsS -b "${COOKIE_JAR}" "${BASE_URL}/api/batches") +echo " ${BATCHES}" | head -c 200 +echo + +echo "smoke: OK" +``` + +- [ ] **Step 2: Make it executable + syntax check** + +```bash +chmod +x scripts/smoke.sh +bash -n scripts/smoke.sh +``` + +- [ ] **Step 3: Commit** + +```bash +git add scripts/smoke.sh +git commit -m "feat(sp23): scripts/smoke.sh brings up + logs in + parses 837 end-to-end" +``` + +--- + +## Task 7: `RUNBOOK.md` + +**Files:** +- Create: `RUNBOOK.md` + +- [ ] **Step 1: Write the runbook** + +Create `RUNBOOK.md`: + +```markdown +# Cyclone Operator Runbook + +Production operations for a single-operator Cyclone deploy on Ubuntu Linux. Assumes the box was bootstrapped via `scripts/cyclone-init.sh` and the stack is up via `docker compose up -d`. + +## Daily + +- [ ] Confirm the host healthcheck cron hasn't emailed. It pings `http://localhost:8080/api/healthz` every 5 minutes. +- [ ] `docker compose ps` — both services `healthy`. +- [ ] `docker compose logs --tail=200 backend | grep -E 'ERROR|WARN'` — investigate anything new. + +## Weekly + +- [ ] `curl -fsS http://localhost:8080/api/admin/audit-log -b cookies.txt | jq '.events[] | select(.event | test("login_failed|backup.failed"))'` — review failed logins + backup failures. +- [ ] Confirm `docker compose exec backend ls -la /var/lib/cyclone/backups/` shows recent `.bin` files (within 25h of now). + +## Quarterly + +- [ ] Rotate the SQLCipher / cookie-signing key: + ```bash + bash scripts/cyclone-init.sh --force # overwrites /etc/cyclone/secrets/db.key + docker compose restart backend # picks up the new key + ``` + Old `.bin` backups become unreadable after this; export them first if you need to keep them. + +## As needed + +- **Add an operator.** Log in as admin → `/admin/users` → Create user. Roles: `admin` / `user` / `viewer`. +- **Reset a password.** Admin UI → Users → Reset password, OR `docker compose exec backend python -m cyclone admin reset-password --username `. +- **Restore from backup.** Admin UI → Backups → pick the snapshot → Initiate restore → Confirm. The backend will restart automatically. +- **Roll back the code (not the schema).** `TAG=0.0.9 docker compose up -d`. The previous image stays in the local Docker cache for one cycle. +- **Pull a new `:stable`.** + ```bash + cd /opt/cyclone + docker compose pull + docker compose up -d + docker compose logs -f backend | head -200 # verify migrations + healthcheck + ``` +- **Off-box backup copy.** The operator is expected to rsync `/var/lib/docker/volumes/cyclone_backups/_data/` to an external drive or NAS nightly. The `.bin` files are already encrypted; the destination doesn't need its own encryption. +- **Inspect the DB.** `docker compose exec backend sqlite3 /var/lib/cyclone/db/cyclone.db ".tables"` (works only if SQLCipher key is on disk; the in-process decrypt happens via the cyclone backend). + +## Annual + +- [ ] Rotate the admin password (force re-login for everyone). +- [ ] Audit the `/etc/cyclone/secrets/` directory permissions — should be `chmod 600 root:root`. +- [ ] Review the audit log for stale admin sessions. + +## Emergency + +- **Backend won't start.** `docker compose logs --tail=300 backend`. Look for migration failures (rerun is safe — migrations are forward-only), SQLCipher key mismatch (`PRAGMA key` failure), or port collisions. +- **Frontend won't serve.** `docker compose logs --tail=100 frontend`. Usually nginx config drift; `docker compose restart frontend`. +- **Both unhealthy after a host reboot.** Docker may have come up before the named volumes did. `docker compose down && docker compose up -d`. +- **Suspected key compromise.** Rotate immediately (see Quarterly above). All active sessions are invalidated. + +## Where things live + +| Asset | Path | +|---|---| +| Docker compose file | `/opt/cyclone/docker-compose.yml` | +| Secrets | `/etc/cyclone/secrets/{db.key,admin_username,admin_pw}` | +| Live DB (SQLCipher-encrypted volume) | `cyclone_db` named volume, mounted at `/var/lib/cyclone/db` | +| Encrypted backups | `cyclone_backups` named volume, mounted at `/var/lib/cyclone/backups` | +| Uploaded prod files | `cyclone_prodfiles` named volume | +| SFTP staging stub | `cyclone_sftp_staging` named volume | +| Logs | `cyclone_logs` named volume + bind-mounted at `/var/log/cyclone` | +| Off-box backup destination | Operator's external drive / NAS (rsync cron, not in compose) | +``` + +- [ ] **Step 2: Commit** + +```bash +git add RUNBOOK.md +git commit -m "docs(sp23): RUNBOOK.md with daily/weekly/quarterly/emergency ops" +``` + +--- + +## Task 8: `backend/tests/test_docker.py` + +**Files:** +- Create: `backend/tests/test_docker.py` + +- [ ] **Step 1: Write the tests** + +```python +"""Dockerfile + compose-config smoke tests for SP23. + +These tests do NOT require a running Docker daemon (the compose-up test +is gated on DOCKER_TESTS=1 so it can be skipped on bare CI without +Docker). They validate that: + +* `docker-compose.yml` at the repo root is syntactically valid and that + the shape we expect (services, secrets, volumes, networks) is present. +* Both Dockerfiles parse with `docker build --check` if Docker is on PATH. +* The named-volume mount paths match what `cyclone.db` + the BackupService + expect at runtime. +""" +from __future__ import annotations + +import os +import shutil +import subprocess +from pathlib import Path + +import pytest +import yaml + +REPO_ROOT = Path(__file__).resolve().parents[2] +COMPOSE_FILE = REPO_ROOT / "docker-compose.yml" +BACKEND_DOCKERFILE = REPO_ROOT / "backend" / "Dockerfile" +FRONTEND_DOCKERFILE = REPO_ROOT / "frontend" / "Dockerfile" + + +def _has_docker() -> bool: + return shutil.which("docker") is not None + + +def _has_docker_compose() -> bool: + return shutil.which("docker") is not None and ( + subprocess.run( + ["docker", "compose", "version"], + capture_output=True, + check=False, + ).returncode + == 0 + ) + + +def test_compose_file_exists(): + assert COMPOSE_FILE.exists(), f"missing {COMPOSE_FILE}" + + +def test_compose_config_validates(): + """`docker compose config` should exit 0 with no stderr.""" + if not _has_docker_compose(): + pytest.skip("docker compose not on PATH") + result = subprocess.run( + ["docker", "compose", "-f", str(COMPOSE_FILE), "config", "--quiet"], + capture_output=True, + text=True, + ) + assert result.returncode == 0, ( + f"compose config failed: stderr={result.stderr!r}" + ) + + +def test_compose_declares_required_services(): + compose = yaml.safe_load(COMPOSE_FILE.read_text()) + services = compose.get("services", {}) + assert "backend" in services, "compose must declare a 'backend' service" + assert "frontend" in services, "compose must declare a 'frontend' service" + + backend = services["backend"] + assert "cyclone_db" in backend.get("volumes", []) or any( + "cyclone_db" in (v if isinstance(v, str) else v.get("source", "")) + for v in backend.get("volumes", []) + ), "backend must mount cyclone_db volume" + assert backend.get("restart") == "unless-stopped" + assert "healthcheck" in backend, "backend must have a healthcheck" + + frontend = services["frontend"] + assert "8080:8080" in frontend.get("ports", []), ( + "frontend must publish 8080:8080 for LAN access" + ) + assert frontend.get("depends_on", {}).get("backend", {}).get( + "condition" + ) == "service_healthy", "frontend must wait for backend healthy" + + +def test_compose_declares_required_secrets_and_volumes(): + compose = yaml.safe_load(COMPOSE_FILE.read_text()) + secrets = compose.get("secrets", {}) + for required in ("cyclone_db_key", "cyclone_admin_password"): + assert required in secrets, f"compose must declare secret {required!r}" + volumes = compose.get("volumes", {}) + for required in ( + "cyclone_db", + "cyclone_backups", + "cyclone_prodfiles", + "cyclone_sftp_staging", + "cyclone_logs", + ): + assert required in volumes, f"compose must declare volume {required!r}" + + +@pytest.mark.skipif( + not _has_docker(), reason="docker not on PATH; skipping Dockerfile parse check" +) +def test_backend_dockerfile_parses(): + result = subprocess.run( + ["docker", "build", "--check", "-f", str(BACKEND_DOCKERFILE), str(REPO_ROOT / "backend")], + capture_output=True, + text=True, + ) + assert result.returncode == 0, ( + f"backend Dockerfile failed to parse: stderr={result.stderr!r}" + ) + + +@pytest.mark.skipif( + not _has_docker(), reason="docker not on PATH; skipping Dockerfile parse check" +) +def test_frontend_dockerfile_parses(): + result = subprocess.run( + [ + "docker", + "build", + "--check", + "-f", + str(FRONTEND_DOCKERFILE), + str(REPO_ROOT / "frontend"), + ], + capture_output=True, + text=True, + ) + assert result.returncode == 0, ( + f"frontend Dockerfile failed to parse: stderr={result.stderr!r}" + ) + + +@pytest.mark.skipif( + not os.environ.get("DOCKER_TESTS") or not _has_docker_compose(), + reason="DOCKER_TESTS=1 + docker compose required for live bring-up", +) +def test_compose_up_brings_up_healthy_stack(): + """Gated live test — only runs when DOCKER_TESTS=1 and docker is available.""" + subprocess.run( + ["docker", "compose", "-f", str(COMPOSE_FILE), "up", "-d", "--build"], + check=True, + cwd=REPO_ROOT, + ) + try: + # Wait up to 90s for backend healthcheck. + for _ in range(45): + ps = subprocess.run( + [ + "docker", + "compose", + "-f", + str(COMPOSE_FILE), + "ps", + "--format", + "json", + ], + capture_output=True, + text=True, + cwd=REPO_ROOT, + ) + if "healthy" in ps.stdout: + return + import time + + time.sleep(2) + pytest.fail("compose stack did not become healthy within 90s") + finally: + subprocess.run( + ["docker", "compose", "-f", str(COMPOSE_FILE), "down", "-v"], + check=False, + cwd=REPO_ROOT, + ) +``` + +- [ ] **Step 2: Add `pyyaml` to dev deps if not already there** + +```bash +grep -q '^pyyaml' backend/pyproject.toml || \ + .venv/bin/pip install pyyaml +``` + +If `pyyaml` is not yet a project dep (check `backend/pyproject.toml` `[project.optional-dependencies] dev` section), add it: + +```toml +[project.optional-dependencies] +dev = [ + "pytest>=8", + "pytest-asyncio>=0.23", + "pyyaml>=6", # <-- add + "ruff>=0.5", +] +``` + +- [ ] **Step 3: Run the tests** + +```bash +cd backend && .venv/bin/pytest tests/test_docker.py -v +``` + +Expected: tests 1-5 pass. Test 6 (live bring-up) skips unless `DOCKER_TESTS=1`. + +- [ ] **Step 4: Commit** + +```bash +git add backend/tests/test_docker.py backend/pyproject.toml backend/uv.lock +git commit -m "feat(sp23): tests/test_docker.py validates compose config + Dockerfile parse" +``` + +--- + +## Task 9: Wire auth env vars (small backend addition) + +The compose env block references `CYCLONE_ADMIN_USERNAME_FILE` + `CYCLONE_ADMIN_PASSWORD_FILE`. Auth on main reads `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD` directly. We add minimal support for the `_FILE` variants so Docker secrets can be mounted as files (the standard pattern). + +**Files:** +- Modify: `backend/src/cyclone/auth/bootstrap.py` + +- [ ] **Step 1: Add `_FILE` env var support** + +In `backend/src/cyclone/auth/bootstrap.py`, replace the env-var read block (currently lines 45-46) with: + +```python +def _read_secret(env_var: str, file_var: str) -> str | None: + """Read a secret from a `_FILE` env var (Docker secret pattern) first, + falling back to the plain env var. Returns None if neither is set. + """ + file_path = os.environ.get(file_var) + if file_path: + try: + return Path(file_path).read_text().strip() + except OSError as exc: + raise RuntimeError(f"failed to read {file_var}={file_path}: {exc}") + return os.environ.get(env_var) + + +# ...inside run(), replace: +# username = os.environ.get("CYCLONE_ADMIN_USERNAME") +# password = os.environ.get("CYCLONE_ADMIN_PASSWORD") +# with: + username = _read_secret("CYCLONE_ADMIN_USERNAME", "CYCLONE_ADMIN_USERNAME_FILE") + password = _read_secret("CYCLONE_ADMIN_PASSWORD", "CYCLONE_ADMIN_PASSWORD_FILE") +``` + +Add the import at the top of the file alongside the existing `import os`: + +```python +from pathlib import Path +``` + +- [ ] **Step 2: Add a test for the `_FILE` env var path** + +Create `backend/tests/test_auth_bootstrap_file.py`: + +```python +"""Test that auth bootstrap reads from CYCLONE_ADMIN_*_FILE env vars when set. + +Mirrors the Docker-secret posture in docker-compose.yml: secrets mounted +at /run/secrets/ with the `_FILE` env var pointing at the path. +""" +from __future__ import annotations + +import os +from pathlib import Path + +import pytest + +from cyclone.auth import bootstrap +from cyclone.db import SessionLocal + + +@pytest.fixture +def tmp_secrets(tmp_path: Path) -> tuple[str, str]: + user_file = tmp_path / "admin_username" + pw_file = tmp_path / "admin_pw" + user_file.write_text("deploy-admin\n") + pw_file.write_text("super-secret-password-123\n") + return str(user_file), str(pw_file) + + +def test_bootstrap_reads_from_file_env_vars(tmp_path, monkeypatch, tmp_secrets): + user_file, pw_file = tmp_secrets + db_path = tmp_path / "test.db" + monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{db_path}") + monkeypatch.delenv("CYCLONE_ADMIN_USERNAME", raising=False) + monkeypatch.delenv("CYCLONE_ADMIN_PASSWORD", raising=False) + monkeypatch.setenv("CYCLONE_ADMIN_USERNAME_FILE", user_file) + monkeypatch.setenv("CYCLONE_ADMIN_PASSWORD_FILE", pw_file) + + bootstrap.run() + + # Verify the user was created with the password from the file. + from cyclone.auth.permissions import Role + from cyclone.auth import users + from cyclone.db import User + + with SessionLocal()() as db: + user = db.query(User).filter(User.username == "deploy-admin").one() + assert user.role == Role.ADMIN.value + # Verify the password actually verifies. + assert users.verify(db, "deploy-admin", "super-secret-password-123") +``` + +- [ ] **Step 3: Run tests** + +```bash +cd backend && .venv/bin/pytest tests/test_auth_bootstrap_file.py -v +``` + +Expected: PASS. + +- [ ] **Step 4: Commit** + +```bash +git add backend/src/cyclone/auth/bootstrap.py backend/tests/test_auth_bootstrap_file.py +git commit -m "feat(sp23): auth bootstrap reads CYCLONE_ADMIN_*_FILE for Docker secrets" +``` + +--- + +## Task 10: Final verification + lint + typecheck + +- [ ] **Step 1: Run full backend test suite** + +```bash +cd backend && .venv/bin/pytest -v +``` + +Expected: all tests pass, including the new `test_docker.py` + `test_auth_bootstrap_file.py` tests. + +- [ ] **Step 2: Run frontend typecheck + lint + build** + +```bash +npm run typecheck +npm run lint +npm run build +``` + +Expected: zero errors. `npm run build` produces `frontend/dist/`. + +- [ ] **Step 3: Verify `docker compose config` is clean** + +```bash +docker compose config --quiet +``` + +Expected: exit code 0. + +- [ ] **Step 4: No-commit check** + +```bash +cd /home/tyler/dev/cyclone/.worktrees/sp23-ubuntu-docker-deployment +git status +``` + +Expected: clean working tree (except for any pre-existing operator edits in the main checkout, which are out of scope for SP23). + +--- + +## Task 11: Open the PR + +- [ ] **Step 1: Push the branch** + +```bash +git -C /home/tyler/dev/cyclone/.worktrees/sp23-ubuntu-docker-deployment \ + push -u origin sp23-ubuntu-docker-deployment +``` + +- [ ] **Step 2: Open the PR with title `SP23 Ubuntu Docker Deployment`** + +Body should reference the spec + plan paths and call out the auth-on-main delta + the SQLCipher/Keychain split for Docker secrets. + +--- + +## Task 12: Single atomic merge into `main` + +- [ ] **Step 1: After review approval, merge with `git merge --no-ff`** + +```bash +git checkout main +git merge --no-ff sp23-ubuntu-docker-deployment -m "merge: SP23 Ubuntu Docker Deployment into main" +git push origin main +``` + +- [ ] **Step 2: Tag the release** + +```bash +git tag -a v0.23.0 -m "SP23 — Ubuntu Docker Deployment" +git push origin v0.23.0 +``` + +- [ ] **Step 3: Smoke the merged main** + +```bash +docker compose pull +docker compose up -d +bash scripts/smoke.sh +``` + +Expected: smoke: OK. + +--- + +## Out of scope (explicit) + +Per spec §14 — not done in this plan: + +- Real SFTP wire (paramiko). Operators continue to use the Gainwell MFT UI. +- Public TLS / Caddy reverse proxy. LAN-bind only; VPN handles outside access. +- Multi-host HA / DB replication / load balancing. +- Prometheus / Grafana / real alerting. v1 uses `curl healthz | mail` cron. +- Off-box automated backup shipping. Operator's responsibility, documented in RUNBOOK. +- 2FA / SSO / OAuth. +- Self-service password reset. Admin CLI only. +- Per-file encryption of prod files / sftp_staging. +- LUKS full-disk encryption. Operator-level concern. +- Watchtower / automatic updates. Manual `docker compose pull`. +- Linting / pre-commit / dev tooling polish. +- Component / E2E browser tests (Playwright).