cyclone/docs/superpowers/plans/2026-06-23-cyclone-ubuntu-docker-deployment.md

# Ubuntu Docker Deployment Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use
> superpowers:subagent-driven-development (recommended) or
> superpowers:executing-plans to implement this plan task-by-task. Steps
> use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Wrap the existing Cyclone stack (auth already on `main` + the FastAPI backend + the React SPA) in a two-container Docker Compose deployment for production on a single Ubuntu Linux server, with daily encrypted backups, SQLCipher-at-rest with a Docker-secret key, LAN-bind, container healthcheck + restart, an operator runbook, and reproducible bootstrap + smoke scripts.

**Architecture:** `docker-compose.yml` at repo root orchestrates two containers — `cyclone-backend` (FastAPI on `python:3.11-slim-bookworm`, internal port 8000) and `cyclone-frontend` (nginx:1.27-alpine serving the built SPA + reverse-proxying `/api/*` to the backend on the compose-managed bridge, host port 8080). Named Docker volumes hold the SQLCipher-encrypted DB, encrypted backups, uploaded prod files, SFTP staging, and JSON logs. Host-managed secrets at `/etc/cyclone/secrets/` are mounted as Docker secrets into the backend. The existing `BackupService` (SP17) is wired to autostart via `CYCLONE_BACKUP_AUTOSTART=1` on container boot with 24h interval and 14-day retention. Bootstrap is `git clone` → `bash scripts/cyclone-init.sh` → `docker compose build` → `docker compose up -d`.

**Tech Stack:** Docker (multi-stage builds), Docker Compose v2, `python:3.11-slim-bookworm`, `nginx:1.27-alpine`, `node:20-alpine` (build stage only), bash for ops scripts, SQLCipher (`libsqlcipher-dev`), tini (PID 1), curl + jq in smoke script.

**Spec:** [`docs/superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md`](../specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md)

**Depends on auth-on-main:** Auth (bcrypt + HttpOnly session cookie + `matrix_gate` + `Login.tsx` + `AuthProvider`) is already on `main` (2026-06-23). SP23 does **not** re-implement auth; it wires the Docker posture around it. See spec §1.1 for the full delta vs. main.

---

## File structure

New files:
- `backend/Dockerfile` — multi-stage build for the FastAPI service
- `backend/.dockerignore` — exclude `.venv`, tests, fixtures from build context
- `frontend/Dockerfile` — multi-stage build (node:20-alpine → nginx:1.27-alpine)
- `frontend/nginx.conf` — SPA routing + reverse proxy to backend
- `frontend/.dockerignore` — exclude node_modules, src/, test files
- `docker-compose.yml` — two-service orchestration + named volumes + secrets
- `.dockerignore` — repo-root ignore for compose build contexts
- `scripts/cyclone-init.sh` — generate `/etc/cyclone/secrets/*` + print admin password
- `scripts/post-deploy.sh` — set up logrotate + healthcheck cron on host
- `scripts/smoke.sh` — end-to-end bring-up + login + parse + export test
- `RUNBOOK.md` — operator daily / weekly / quarterly / as-needed procedures
- `backend/tests/test_docker.py` — `docker compose config` validation + Dockerfile-build smoke tests

Modified files:
- `README.md` (root, if present) or `docs/ARCHITECTURE.md` — link to `RUNBOOK.md` from the deployment section
- `.gitignore` — add `dist/` (already there), `/run/secrets/*` no-op (Docker secrets are not on host)

No new SQL migration — auth tables (`users`, `sessions`) already exist at `0013_auth_users_and_sessions.sql` + `0014_audit_log_user_id.sql`.

---

## Task 0: Worktree + branch (already done)

- [x] Worktree at `.worktrees/sp23-ubuntu-docker-deployment` on branch `sp23-ubuntu-docker-deployment`, off `main@c398aa7`.
- [x] Spec patch committed (`35c561d`): status `Approved` + §1.1 Delta vs. main.

This task is pre-completed by the dispatcher. Listed for completeness.

---

## Task 1: Backend `Dockerfile` + `backend/.dockerignore`

**Files:**
- Create: `backend/Dockerfile`
- Create: `backend/.dockerignore`

- [ ] **Step 1: Write `backend/.dockerignore`**

Create `backend/.dockerignore` with this exact content:

```
.venv/
venv/
__pycache__/
*.py[cod]
*.egg-info/
.pytest_cache/
.ruff_cache/
tests/
docs/prodfiles/
*.production.txt
.git/
.github/
```

- [ ] **Step 2: Write the multi-stage `backend/Dockerfile`**

Create `backend/Dockerfile`:

```dockerfile
# syntax=docker/dockerfile:1.7
# Builder stage — installs cyclone + sqlcipher extra into a wheel dir.
FROM python:3.11-slim-bookworm AS builder

ENV PIP_NO_CACHE_DIR=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=1 \
    PYTHONDONTWRITEBYTECODE=1

RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        libffi-dev \
        libsqlcipher-dev \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /build

# Copy ONLY the build manifest first so this layer caches across source edits.
COPY pyproject.toml ./

# Stub package so `pip wheel` can resolve `cyclone` without the source tree yet.
RUN mkdir -p src/cyclone && touch src/cyclone/__init__.py
RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher]'

COPY src/ ./src/
RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher]'

# Runtime stage — slim base, non-root user, healthcheck, tini PID 1.
FROM python:3.11-slim-bookworm

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1

RUN apt-get update && apt-get install -y --no-install-recommends \
        libsqlcipher-dev \
        curl \
        tini \
    && rm -rf /var/lib/apt/lists/* \
    && useradd --create-home --uid 1000 --shell /bin/bash cyclone

WORKDIR /app

COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir --no-index --find-links /wheels cyclone \
    && rm -rf /wheels

COPY --chown=cyclone:cyclone src/ /app/src/

USER cyclone

EXPOSE 8000

HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
    CMD curl -fs http://localhost:8000/api/healthz || exit 1

ENTRYPOINT ["tini", "--"]
CMD ["python", "-m", "cyclone", "serve"]
```

- [ ] **Step 3: Verify `backend/Dockerfile` parses**

Run from repo root:

```bash
docker build --check -f backend/Dockerfile backend/
```

Expected: exit code 0, no errors. (Older Docker engines may not have `--check`; fall back to `docker build --target builder backend/` if needed.)

- [ ] **Step 4: Commit**

```bash
git add backend/Dockerfile backend/.dockerignore
git commit -m "feat(sp23): backend Dockerfile (multi-stage, sqlcipher, non-root, healthcheck)"
```

---

## Task 2: Frontend `Dockerfile` + `frontend/nginx.conf` + `frontend/.dockerignore`

**Files:**
- Create: `frontend/Dockerfile`
- Create: `frontend/nginx.conf`
- Create: `frontend/.dockerignore`

- [ ] **Step 1: Write `frontend/.dockerignore`**

```
node_modules/
dist/
build/
*.tsbuildinfo
coverage/
src/**/*.test.ts
src/**/*.test.tsx
src/test/
.git/
.github/
```

- [ ] **Step 2: Write `frontend/nginx.conf`**

```nginx
server {
    listen 8080;
    server_name _;

    root /usr/share/nginx/html;
    index index.html;

    # Long-lived connections for live-tail NDJSON streams.
    proxy_read_timeout 300s;
    proxy_send_timeout 300s;
    client_max_body_size 50m;

    # API + auth: proxy to backend over the compose-managed bridge network.
    location /api/ {
        proxy_pass http://backend:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        # Cookie comes back from backend on /api/; rewrite path to / so the
        # browser sends it on every subsequent request to the SPA.
        proxy_cookie_path /api/ /;
    }

    # SPA fallback: serve index.html for any non-/api route.
    location / {
        try_files $uri $uri/ /index.html;
    }

    # Don't cache the index.html — the SPA references hashed asset filenames.
    location = /index.html {
        add_header Cache-Control "no-cache, no-store, must-revalidate" always;
        expires off;
    }
}
```

- [ ] **Step 3: Write `frontend/Dockerfile`**

```dockerfile
# syntax=docker/dockerfile:1.7
# Build stage.
FROM node:20-alpine AS builder

WORKDIR /build

COPY package.json package-lock.json* ./
RUN npm ci

COPY . .
RUN npm run build

# Runtime stage — nginx serving the built SPA.
FROM nginx:1.27-alpine

RUN rm -f /etc/nginx/conf.d/default.conf

COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY --from=builder /build/dist /usr/share/nginx/html

HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
    CMD wget -qO- http://127.0.0.1:8080/ >/dev/null || exit 1

EXPOSE 8080
```

- [ ] **Step 4: Verify `frontend/Dockerfile` parses**

```bash
docker build --check -f frontend/Dockerfile frontend/
```

Expected: exit code 0. (Fall back to building a stub `dist/` first if `--check` is unavailable.)

- [ ] **Step 5: Commit**

```bash
git add frontend/Dockerfile frontend/nginx.conf frontend/.dockerignore
git commit -m "feat(sp23): frontend Dockerfile + nginx.conf (SPA + reverse proxy)"
```

---

## Task 3: `docker-compose.yml` at repo root

**Files:**
- Create: `docker-compose.yml`
- Create: `.dockerignore` (repo root, protects compose build contexts from accidental host pollution)

- [ ] **Step 1: Write the repo-root `.dockerignore`**

```
.worktrees/
.git/
.github/
docs/prodfiles/
*.production.txt
node_modules/
dist/
.venv/
.superpowers/brainstorm/
```

- [ ] **Step 2: Write `docker-compose.yml`**

```yaml
name: cyclone

services:
  backend:
    image: cyclone-backend:${TAG:-stable}
    build:
      context: ./backend
    restart: unless-stopped
    environment:
      CYCLONE_DB_URL: "sqlite:////var/lib/cyclone/db/cyclone.db"
      CYCLONE_BACKUP_AUTOSTART: "1"
      CYCLONE_BACKUP_INTERVAL_HOURS: "24"
      CYCLONE_BACKUP_RETENTION_DAYS: "14"
      CYCLONE_LOG_LEVEL: "INFO"
      CYCLONE_LOG_FILE: "/var/log/cyclone/cyclone.log"
      CYCLONE_LOG_JSON: "1"
      CYCLONE_SESSION_LIFETIME_HOURS: "24"
      # The cookie signing key is the SQLCipher key by default; override
      # CYCLONE_SECRET_KEY_FILE below to split them in production.
      CYCLONE_SECRET_KEY_FILE: "/run/secrets/cyclone_db_key"
      CYCLONE_COOKIE_SECURE: "1"
      # First-admin bootstrap — set by cyclone-init.sh from /etc/cyclone/secrets/admin_pw.
      CYCLONE_ADMIN_USERNAME_FILE: "/run/secrets/cyclone_admin_username"
      CYCLONE_ADMIN_PASSWORD_FILE: "/run/secrets/cyclone_admin_password"
    secrets:
      - cyclone_db_key
      - cyclone_admin_username
      - cyclone_admin_password
    volumes:
      - cyclone_db:/var/lib/cyclone/db
      - cyclone_backups:/var/lib/cyclone/backups
      - cyclone_prodfiles:/var/lib/cyclone/prodfiles
      - cyclone_sftp_staging:/var/lib/cyclone/sftp_staging
      - cyclone_logs:/var/log/cyclone
    healthcheck:
      test: ["CMD", "curl", "-fs", "http://localhost:8000/api/healthz"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 30s
    networks:
      - cyclone_network

  frontend:
    image: cyclone-frontend:${TAG:-stable}
    build:
      context: ./frontend
    restart: unless-stopped
    ports:
      - "8080:8080"
    depends_on:
      backend:
        condition: service_healthy
    networks:
      - cyclone_network

secrets:
  cyclone_db_key:
    file: /etc/cyclone/secrets/db.key
  cyclone_admin_username:
    file: /etc/cyclone/secrets/admin_username
  cyclone_admin_password:
    file: /etc/cyclone/secrets/admin_pw

volumes:
  cyclone_db:
  cyclone_backups:
  cyclone_prodfiles:
  cyclone_sftp_staging:
  cyclone_logs:

networks:
  cyclone_network:
    driver: bridge
```

- [ ] **Step 3: Validate compose**

```bash
docker compose config --quiet
```

Expected: exit code 0, no output. Fix any schema warnings before continuing.

- [ ] **Step 4: Commit**

```bash
git add docker-compose.yml .dockerignore
git commit -m "feat(sp23): docker-compose.yml with backend + frontend, named volumes, secrets"
```

---

## Task 4: `scripts/cyclone-init.sh`

**Files:**
- Create: `scripts/cyclone-init.sh`

- [ ] **Step 1: Write the bootstrap script**

Create `scripts/cyclone-init.sh`:

```bash
#!/usr/bin/env bash
# cyclone-init.sh — first-time host bootstrap for a production Cyclone deploy.
#
# Generates /etc/cyclone/secrets/{db.key,admin_username,admin_pw} with
# openssl rand -hex 32, sets ownership + permissions, and prints the
# admin password ONCE for the operator to capture.
#
# Idempotent: refuses to overwrite existing secrets unless --force.
#
# Usage:
#   bash scripts/cyclone-init.sh            # generate if missing
#   bash scripts/cyclone-init.sh --force    # overwrite (destroys old data!)
set -euo pipefail

SECRETS_DIR="${CYCLONE_SECRETS_DIR:-/etc/cyclone/secrets}"
FORCE=0
[[ "${1:-}" == "--force" ]] && FORCE=1

if [[ $EUID -ne 0 ]]; then
    echo "ERROR: cyclone-init.sh must run as root (it writes to ${SECRETS_DIR})." >&2
    exit 1
fi

if [[ -f "${SECRETS_DIR}/db.key" && $FORCE -eq 0 ]]; then
    echo "Secrets already exist at ${SECRETS_DIR}. Re-run with --force to overwrite." >&2
    exit 0
fi

mkdir -p "${SECRETS_DIR}"
chmod 700 "${SECRETS_DIR}"

# 64-hex-char (256-bit) random key for SQLCipher + cookie signing.
openssl rand -hex 32 > "${SECRETS_DIR}/db.key"

# Admin username defaults to "admin" unless CYCLONE_ADMIN_USERNAME is set.
ADMIN_USER="${CYCLONE_ADMIN_USERNAME:-admin}"
printf '%s' "${ADMIN_USER}" > "${SECRETS_DIR}/admin_username"

# Memorable-but-random admin password — printed once.
ADMIN_PW="$(openssl rand -base64 18 | tr -d '/+=' | head -c 20)Aa1!"
printf '%s' "${ADMIN_PW}" > "${SECRETS_DIR}/admin_pw"

chmod 600 "${SECRETS_DIR}"/*
chown -R root:root "${SECRETS_DIR}"

cat <<EOF
================================================================
Cyclone secrets generated at ${SECRETS_DIR}.

  db.key            SQLCipher key + cookie signing (64 hex chars)
  admin_username    ${ADMIN_USER}
  admin_pw          ${ADMIN_PW}    <-- capture this NOW; it won't be shown again.

Next steps:
  cd /opt/cyclone
  docker compose pull          # or: docker compose build
  docker compose up -d
  bash scripts/post-deploy.sh  # logrotate + healthcheck cron
  bash scripts/smoke.sh        # end-to-end smoke test

Then open http://<lan-ip>:8080/login and log in as '${ADMIN_USER}'.
Change the password from /admin/users immediately.
================================================================
EOF
```

- [ ] **Step 2: Make it executable**

```bash
chmod +x scripts/cyclone-init.sh
```

- [ ] **Step 3: Syntax check**

```bash
bash -n scripts/cyclone-init.sh
```

Expected: exit code 0, no output.

- [ ] **Step 4: Commit**

```bash
git add scripts/cyclone-init.sh
git commit -m "feat(sp23): scripts/cyclone-init.sh generates /etc/cyclone/secrets/*"
```

---

## Task 5: `scripts/post-deploy.sh`

**Files:**
- Create: `scripts/post-deploy.sh`

- [ ] **Step 1: Write the post-deploy script**

Create `scripts/post-deploy.sh`:

```bash
#!/usr/bin/env bash
# post-deploy.sh — set up host-side logrotate and healthcheck cron for a
# production Cyclone deploy. Run once after `docker compose up -d`.
set -euo pipefail

LOG_DIR="/var/log/cyclone"
HEALTHCHECK_URL="http://127.0.0.1:8080/api/healthz"
HEALTHCHECK_EMAIL="${CYCLONE_HEALTHCHECK_EMAIL:-root}"
CRON_USER="${CYCLONE_CRON_USER:-root}"

if [[ $EUID -ne 0 ]]; then
    echo "ERROR: post-deploy.sh must run as root." >&2
    exit 1
fi

# 1. Logrotate — keeps 14 days of JSON logs compressed.
cat > /etc/logrotate.d/cyclone <<'LOGROTATE'
/var/log/cyclone/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
    dateext
    dateformat -%Y%m%d
}
LOGROTATE

mkdir -p "${LOG_DIR}"
chmod 755 "${LOG_DIR}"

# 2. Healthcheck cron — every 5 minutes, email if down.
CRON_LINE="*/5 * * * * curl -fsS --max-time 10 ${HEALTHCHECK_URL} >/dev/null 2>&1 || echo 'Cyclone DOWN at $(date)' | mail -s 'Cyclone healthz FAIL' ${HEALTHCHECK_EMAIL}"
TMP_CRON="$(mktemp)"
crontab -u "${CRON_USER}" -l 2>/dev/null > "${TMP_CRON}" || true
# Remove any existing cyclone healthcheck line, then re-add.
grep -v -F "cyclone healthz" "${TMP_CRON}" > "${TMP_CRON}.new" || cp "${TMP_CRON}" "${TMP_CRON}.new"
echo "${CRON_LINE}" >> "${TMP_CRON}.new"
crontab -u "${CRON_USER}" "${TMP_CRON}.new"
rm -f "${TMP_CRON}" "${TMP_CRON}.new"

cat <<EOF
post-deploy.sh complete:
  - logrotate installed at /etc/logrotate.d/cyclone
  - healthcheck cron for ${HEALTHCHECK_URL} installed for user '${CRON_USER}'
  - logs rotate daily, 14 days retention

Verify with:
  logrotate -d /etc/logrotate.d/cyclone
  crontab -u ${CRON_USER} -l | grep cyclone
EOF
```

- [ ] **Step 2: Make it executable + syntax check**

```bash
chmod +x scripts/post-deploy.sh
bash -n scripts/post-deploy.sh
```

- [ ] **Step 3: Commit**

```bash
git add scripts/post-deploy.sh
git commit -m "feat(sp23): scripts/post-deploy.sh installs logrotate + healthcheck cron"
```

---

## Task 6: `scripts/smoke.sh`

**Files:**
- Create: `scripts/smoke.sh`

- [ ] **Step 1: Write the smoke script**

Create `scripts/smoke.sh`:

```bash
#!/usr/bin/env bash
# smoke.sh — end-to-end bring-up + login + parse + export smoke test.
# Assumes docker compose is up and /etc/cyclone/secrets/admin_pw exists.
set -euo pipefail

BASE_URL="${CYCLONE_SMOKE_URL:-http://localhost:8080}"
SECRETS_DIR="${CYCLONE_SECRETS_DIR:-/etc/cyclone/secrets}"
ADMIN_PW="$(cat "${SECRETS_DIR}/admin_pw")"
COOKIE_JAR="$(mktemp)"
SAMPLE_837="${CYCLONE_SMOKE_SAMPLE:-docs/prodfiles/co_medicaid/sample_837p.txt}"

cleanup() { rm -f "${COOKIE_JAR}" /tmp/cyclone_smoke_export.zip; }
trap cleanup EXIT

echo "==> waiting for healthz..."
for i in {1..30}; do
    if curl -fsS "${BASE_URL}/api/healthz" >/dev/null 2>&1; then break; fi
    sleep 2
    [[ $i -eq 30 ]] && { echo "FAIL: healthz never came up"; exit 1; }
done
echo "    ok"

echo "==> logging in as admin..."
HTTP_CODE=$(curl -sS -o /dev/null -w '%{http_code}' \
    -c "${COOKIE_JAR}" \
    -H 'Content-Type: application/json' \
    -X POST "${BASE_URL}/api/auth/login" \
    -d "{\"username\":\"admin\",\"password\":\"${ADMIN_PW}\"}")
[[ "${HTTP_CODE}" == "200" ]] || { echo "FAIL: login returned ${HTTP_CODE}"; exit 1; }
echo "    ok (cookie set)"

echo "==> /api/auth/me..."
ME=$(curl -fsS -b "${COOKIE_JAR}" "${BASE_URL}/api/auth/me")
echo "    ${ME}"

echo "==> parsing sample 837 (${SAMPLE_837})..."
[[ -f "${SAMPLE_837}" ]] || { echo "skip: sample not found"; exit 0; }
PARSE=$(curl -fsS -b "${COOKIE_JAR}" -X POST "${BASE_URL}/api/parse-837" \
    -F "file=@${SAMPLE_837}")
echo "    ${PARSE}" | head -c 200
echo

echo "==> listing batches..."
BATCHES=$(curl -fsS -b "${COOKIE_JAR}" "${BASE_URL}/api/batches")
echo "    ${BATCHES}" | head -c 200
echo

echo "smoke: OK"
```

- [ ] **Step 2: Make it executable + syntax check**

```bash
chmod +x scripts/smoke.sh
bash -n scripts/smoke.sh
```

- [ ] **Step 3: Commit**

```bash
git add scripts/smoke.sh
git commit -m "feat(sp23): scripts/smoke.sh brings up + logs in + parses 837 end-to-end"
```

---

## Task 7: `RUNBOOK.md`

**Files:**
- Create: `RUNBOOK.md`

- [ ] **Step 1: Write the runbook**

Create `RUNBOOK.md`:

```markdown
# Cyclone Operator Runbook

Production operations for a single-operator Cyclone deploy on Ubuntu Linux. Assumes the box was bootstrapped via `scripts/cyclone-init.sh` and the stack is up via `docker compose up -d`.

## Daily

- [ ] Confirm the host healthcheck cron hasn't emailed. It pings `http://127.0.0.1:8080/api/healthz` every 5 minutes.
- [ ] `docker compose ps` — both services `healthy`.
- [ ] `docker compose logs --tail=200 backend | grep -E 'ERROR|WARN'` — investigate anything new.

## Weekly

- [ ] `curl -fsS http://127.0.0.1:8080/api/admin/audit-log -b cookies.txt | jq '.events[] | select(.event | test("login_failed|backup.failed"))'` — review failed logins + backup failures.
- [ ] Confirm `docker compose exec backend ls -la /var/lib/cyclone/backups/` shows recent `.bin` files (within 25h of now).

## Quarterly

- [ ] Rotate the SQLCipher / cookie-signing key:
  ```bash
  bash scripts/cyclone-init.sh --force   # overwrites /etc/cyclone/secrets/db.key
  docker compose restart backend          # picks up the new key
  ```
  Old `.bin` backups become unreadable after this; export them first if you need to keep them.

## As needed

- **Add an operator.** Log in as admin → `/admin/users` → Create user. Roles: `admin` / `user` / `viewer`.
- **Reset a password.** Admin UI → Users → Reset password, OR `docker compose exec backend python -m cyclone admin reset-password --username <name>`.
- **Restore from backup.** Admin UI → Backups → pick the snapshot → Initiate restore → Confirm. The backend will restart automatically.
- **Roll back the code (not the schema).** `TAG=0.0.9 docker compose up -d`. The previous image stays in the local Docker cache for one cycle.
- **Pull a new `:stable`.**
  ```bash
  cd /opt/cyclone
  docker compose pull
  docker compose up -d
  docker compose logs -f backend | head -200   # verify migrations + healthcheck
  ```
- **Off-box backup copy.** The operator is expected to rsync `/var/lib/docker/volumes/cyclone_backups/_data/` to an external drive or NAS nightly. The `.bin` files are already encrypted; the destination doesn't need its own encryption.
- **Inspect the DB.** `docker compose exec backend sqlite3 /var/lib/cyclone/db/cyclone.db ".tables"` (works only if SQLCipher key is on disk; the in-process decrypt happens via the cyclone backend).

## Annual

- [ ] Rotate the admin password (force re-login for everyone).
- [ ] Audit the `/etc/cyclone/secrets/` directory permissions — should be `chmod 600 root:root`.
- [ ] Review the audit log for stale admin sessions.

## Emergency

- **Backend won't start.** `docker compose logs --tail=300 backend`. Look for migration failures (rerun is safe — migrations are forward-only), SQLCipher key mismatch (`PRAGMA key` failure), or port collisions.
- **Frontend won't serve.** `docker compose logs --tail=100 frontend`. Usually nginx config drift; `docker compose restart frontend`.
- **Both unhealthy after a host reboot.** Docker may have come up before the named volumes did. `docker compose down && docker compose up -d`.
- **Suspected key compromise.** Rotate immediately (see Quarterly above). All active sessions are invalidated.

## Where things live

| Asset | Path |
|---|---|
| Docker compose file | `/opt/cyclone/docker-compose.yml` |
| Secrets | `/etc/cyclone/secrets/{db.key,admin_username,admin_pw}` |
| Live DB (SQLCipher-encrypted volume) | `cyclone_db` named volume, mounted at `/var/lib/cyclone/db` |
| Encrypted backups | `cyclone_backups` named volume, mounted at `/var/lib/cyclone/backups` |
| Uploaded prod files | `cyclone_prodfiles` named volume |
| SFTP staging stub | `cyclone_sftp_staging` named volume |
| Logs | `cyclone_logs` named volume + bind-mounted at `/var/log/cyclone` |
| Off-box backup destination | Operator's external drive / NAS (rsync cron, not in compose) |
```

- [ ] **Step 2: Commit**

```bash
git add RUNBOOK.md
git commit -m "docs(sp23): RUNBOOK.md with daily/weekly/quarterly/emergency ops"
```

---

## Task 8: `backend/tests/test_docker.py`

**Files:**
- Create: `backend/tests/test_docker.py`

- [ ] **Step 1: Write the tests**

```python
"""Dockerfile + compose-config smoke tests for SP23.

These tests do NOT require a running Docker daemon (the compose-up test
is gated on DOCKER_TESTS=1 so it can be skipped on bare CI without
Docker). They validate that:

* `docker-compose.yml` at the repo root is syntactically valid and that
  the shape we expect (services, secrets, volumes, networks) is present.
* Both Dockerfiles parse with `docker build --check` if Docker is on PATH.
* The named-volume mount paths match what `cyclone.db` + the BackupService
  expect at runtime.
"""
from __future__ import annotations

import os
import shutil
import subprocess
from pathlib import Path

import pytest
import yaml

REPO_ROOT = Path(__file__).resolve().parents[2]
COMPOSE_FILE = REPO_ROOT / "docker-compose.yml"
BACKEND_DOCKERFILE = REPO_ROOT / "backend" / "Dockerfile"
FRONTEND_DOCKERFILE = REPO_ROOT / "frontend" / "Dockerfile"


def _has_docker() -> bool:
    return shutil.which("docker") is not None


def _has_docker_compose() -> bool:
    return shutil.which("docker") is not None and (
        subprocess.run(
            ["docker", "compose", "version"],
            capture_output=True,
            check=False,
        ).returncode
        == 0
    )


def test_compose_file_exists():
    assert COMPOSE_FILE.exists(), f"missing {COMPOSE_FILE}"


def test_compose_config_validates():
    """`docker compose config` should exit 0 with no stderr."""
    if not _has_docker_compose():
        pytest.skip("docker compose not on PATH")
    result = subprocess.run(
        ["docker", "compose", "-f", str(COMPOSE_FILE), "config", "--quiet"],
        capture_output=True,
        text=True,
    )
    assert result.returncode == 0, (
        f"compose config failed: stderr={result.stderr!r}"
    )


def test_compose_declares_required_services():
    compose = yaml.safe_load(COMPOSE_FILE.read_text())
    services = compose.get("services", {})
    assert "backend" in services, "compose must declare a 'backend' service"
    assert "frontend" in services, "compose must declare a 'frontend' service"

    backend = services["backend"]
    assert "cyclone_db" in backend.get("volumes", []) or any(
        "cyclone_db" in (v if isinstance(v, str) else v.get("source", ""))
        for v in backend.get("volumes", [])
    ), "backend must mount cyclone_db volume"
    assert backend.get("restart") == "unless-stopped"
    assert "healthcheck" in backend, "backend must have a healthcheck"

    frontend = services["frontend"]
    assert "8080:8080" in frontend.get("ports", []), (
        "frontend must publish 8080:8080 for LAN access"
    )
    assert frontend.get("depends_on", {}).get("backend", {}).get(
        "condition"
    ) == "service_healthy", "frontend must wait for backend healthy"


def test_compose_declares_required_secrets_and_volumes():
    compose = yaml.safe_load(COMPOSE_FILE.read_text())
    secrets = compose.get("secrets", {})
    for required in ("cyclone_db_key", "cyclone_admin_password"):
        assert required in secrets, f"compose must declare secret {required!r}"
    volumes = compose.get("volumes", {})
    for required in (
        "cyclone_db",
        "cyclone_backups",
        "cyclone_prodfiles",
        "cyclone_sftp_staging",
        "cyclone_logs",
    ):
        assert required in volumes, f"compose must declare volume {required!r}"


@pytest.mark.skipif(
    not _has_docker(), reason="docker not on PATH; skipping Dockerfile parse check"
)
def test_backend_dockerfile_parses():
    result = subprocess.run(
        ["docker", "build", "--check", "-f", str(BACKEND_DOCKERFILE), str(REPO_ROOT / "backend")],
        capture_output=True,
        text=True,
    )
    assert result.returncode == 0, (
        f"backend Dockerfile failed to parse: stderr={result.stderr!r}"
    )


@pytest.mark.skipif(
    not _has_docker(), reason="docker not on PATH; skipping Dockerfile parse check"
)
def test_frontend_dockerfile_parses():
    result = subprocess.run(
        [
            "docker",
            "build",
            "--check",
            "-f",
            str(FRONTEND_DOCKERFILE),
            str(REPO_ROOT / "frontend"),
        ],
        capture_output=True,
        text=True,
    )
    assert result.returncode == 0, (
        f"frontend Dockerfile failed to parse: stderr={result.stderr!r}"
    )


@pytest.mark.skipif(
    not os.environ.get("DOCKER_TESTS") or not _has_docker_compose(),
    reason="DOCKER_TESTS=1 + docker compose required for live bring-up",
)
def test_compose_up_brings_up_healthy_stack():
    """Gated live test — only runs when DOCKER_TESTS=1 and docker is available."""
    subprocess.run(
        ["docker", "compose", "-f", str(COMPOSE_FILE), "up", "-d", "--build"],
        check=True,
        cwd=REPO_ROOT,
    )
    try:
        # Wait up to 90s for backend healthcheck.
        for _ in range(45):
            ps = subprocess.run(
                [
                    "docker",
                    "compose",
                    "-f",
                    str(COMPOSE_FILE),
                    "ps",
                    "--format",
                    "json",
                ],
                capture_output=True,
                text=True,
                cwd=REPO_ROOT,
            )
            if "healthy" in ps.stdout:
                return
            import time

            time.sleep(2)
        pytest.fail("compose stack did not become healthy within 90s")
    finally:
        subprocess.run(
            ["docker", "compose", "-f", str(COMPOSE_FILE), "down", "-v"],
            check=False,
            cwd=REPO_ROOT,
        )
```

- [ ] **Step 2: Add `pyyaml` to dev deps if not already there**

```bash
grep -q '^pyyaml' backend/pyproject.toml || \
    .venv/bin/pip install pyyaml
```

If `pyyaml` is not yet a project dep (check `backend/pyproject.toml` `[project.optional-dependencies] dev` section), add it:

```toml
[project.optional-dependencies]
dev = [
    "pytest>=8",
    "pytest-asyncio>=0.23",
    "pyyaml>=6",        # <-- add
    "ruff>=0.5",
]
```

- [ ] **Step 3: Run the tests**

```bash
cd backend && .venv/bin/pytest tests/test_docker.py -v
```

Expected: tests 1-5 pass. Test 6 (live bring-up) skips unless `DOCKER_TESTS=1`.

- [ ] **Step 4: Commit**

```bash
git add backend/tests/test_docker.py backend/pyproject.toml backend/uv.lock
git commit -m "feat(sp23): tests/test_docker.py validates compose config + Dockerfile parse"
```

---

## Task 9: Wire auth env vars (small backend addition)

The compose env block references `CYCLONE_ADMIN_USERNAME_FILE` + `CYCLONE_ADMIN_PASSWORD_FILE`. Auth on main reads `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD` directly. We add minimal support for the `_FILE` variants so Docker secrets can be mounted as files (the standard pattern).

**Files:**
- Modify: `backend/src/cyclone/auth/bootstrap.py`

- [ ] **Step 1: Add `_FILE` env var support**

In `backend/src/cyclone/auth/bootstrap.py`, replace the env-var read block (currently lines 45-46) with:

```python
def _read_secret(env_var: str, file_var: str) -> str | None:
    """Read a secret from a `_FILE` env var (Docker secret pattern) first,
    falling back to the plain env var. Returns None if neither is set.
    """
    file_path = os.environ.get(file_var)
    if file_path:
        try:
            return Path(file_path).read_text().strip()
        except OSError as exc:
            raise RuntimeError(f"failed to read {file_var}={file_path}: {exc}")
    return os.environ.get(env_var)


# ...inside run(), replace:
#     username = os.environ.get("CYCLONE_ADMIN_USERNAME")
#     password = os.environ.get("CYCLONE_ADMIN_PASSWORD")
# with:
    username = _read_secret("CYCLONE_ADMIN_USERNAME", "CYCLONE_ADMIN_USERNAME_FILE")
    password = _read_secret("CYCLONE_ADMIN_PASSWORD", "CYCLONE_ADMIN_PASSWORD_FILE")
```

Add the import at the top of the file alongside the existing `import os`:

```python
from pathlib import Path
```

- [ ] **Step 2: Add a test for the `_FILE` env var path**

Create `backend/tests/test_auth_bootstrap_file.py`:

```python
"""Test that auth bootstrap reads from CYCLONE_ADMIN_*_FILE env vars when set.

Mirrors the Docker-secret posture in docker-compose.yml: secrets mounted
at /run/secrets/<name> with the `_FILE` env var pointing at the path.
"""
from __future__ import annotations

import os
from pathlib import Path

import pytest

from cyclone.auth import bootstrap
from cyclone.db import SessionLocal


@pytest.fixture
def tmp_secrets(tmp_path: Path) -> tuple[str, str]:
    user_file = tmp_path / "admin_username"
    pw_file = tmp_path / "admin_pw"
    user_file.write_text("deploy-admin\n")
    pw_file.write_text("super-secret-password-123\n")
    return str(user_file), str(pw_file)


def test_bootstrap_reads_from_file_env_vars(tmp_path, monkeypatch, tmp_secrets):
    user_file, pw_file = tmp_secrets
    db_path = tmp_path / "test.db"
    monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{db_path}")
    monkeypatch.delenv("CYCLONE_ADMIN_USERNAME", raising=False)
    monkeypatch.delenv("CYCLONE_ADMIN_PASSWORD", raising=False)
    monkeypatch.setenv("CYCLONE_ADMIN_USERNAME_FILE", user_file)
    monkeypatch.setenv("CYCLONE_ADMIN_PASSWORD_FILE", pw_file)

    bootstrap.run()

    # Verify the user was created with the password from the file.
    from cyclone.auth.permissions import Role
    from cyclone.auth import users
    from cyclone.db import User

    with SessionLocal()() as db:
        user = db.query(User).filter(User.username == "deploy-admin").one()
        assert user.role == Role.ADMIN.value
        # Verify the password actually verifies.
        assert users.verify(db, "deploy-admin", "super-secret-password-123")
```

- [ ] **Step 3: Run tests**

```bash
cd backend && .venv/bin/pytest tests/test_auth_bootstrap_file.py -v
```

Expected: PASS.

- [ ] **Step 4: Commit**

```bash
git add backend/src/cyclone/auth/bootstrap.py backend/tests/test_auth_bootstrap_file.py
git commit -m "feat(sp23): auth bootstrap reads CYCLONE_ADMIN_*_FILE for Docker secrets"
```

---

## Task 10: Final verification + lint + typecheck

- [ ] **Step 1: Run full backend test suite**

```bash
cd backend && .venv/bin/pytest -v
```

Expected: all tests pass, including the new `test_docker.py` + `test_auth_bootstrap_file.py` tests.

- [ ] **Step 2: Run frontend typecheck + lint + build**

```bash
npm run typecheck
npm run lint
npm run build
```

Expected: zero errors. `npm run build` produces `frontend/dist/`.

- [ ] **Step 3: Verify `docker compose config` is clean**

```bash
docker compose config --quiet
```

Expected: exit code 0.

- [ ] **Step 4: No-commit check**

```bash
cd /home/tyler/dev/cyclone/.worktrees/sp23-ubuntu-docker-deployment
git status
```

Expected: clean working tree (except for any pre-existing operator edits in the main checkout, which are out of scope for SP23).

---

## Task 11: Open the PR

- [ ] **Step 1: Push the branch**

```bash
git -C /home/tyler/dev/cyclone/.worktrees/sp23-ubuntu-docker-deployment \
    push -u origin sp23-ubuntu-docker-deployment
```

- [ ] **Step 2: Open the PR with title `SP23 Ubuntu Docker Deployment`**

Body should reference the spec + plan paths and call out the auth-on-main delta + the SQLCipher/Keychain split for Docker secrets.

---

## Task 12: Single atomic merge into `main`

- [ ] **Step 1: After review approval, merge with `git merge --no-ff`**

```bash
git checkout main
git merge --no-ff sp23-ubuntu-docker-deployment -m "merge: SP23 Ubuntu Docker Deployment into main"
git push origin main
```

- [ ] **Step 2: Tag the release**

```bash
git tag -a v0.23.0 -m "SP23 — Ubuntu Docker Deployment"
git push origin v0.23.0
```

- [ ] **Step 3: Smoke the merged main**

```bash
docker compose pull
docker compose up -d
bash scripts/smoke.sh
```

Expected: smoke: OK.

---

## Out of scope (explicit)

Per spec §14 — not done in this plan:

- Real SFTP wire (paramiko). Operators continue to use the Gainwell MFT UI.
- Public TLS / Caddy reverse proxy. LAN-bind only; VPN handles outside access.
- Multi-host HA / DB replication / load balancing.
- Prometheus / Grafana / real alerting. v1 uses `curl healthz | mail` cron.
- Off-box automated backup shipping. Operator's responsibility, documented in RUNBOOK.
- 2FA / SSO / OAuth.
- Self-service password reset. Admin CLI only.
- Per-file encryption of prod files / sftp_staging.
- LUKS full-disk encryption. Operator-level concern.
- Watchtower / automatic updates. Manual `docker compose pull`.
- Linting / pre-commit / dev tooling polish.
- Component / E2E browser tests (Playwright).

---

## Live verification (post-implementation, gated `DOCKER_TESTS=1`)

Before opening the PR, the full `docker compose up -d` stack was brought up on the dev host with `DOCKER_TESTS=1`. Six bugs surfaced that don't show up in unit tests — all fixed in the followup commit `3ba5ca0`:

1. **Wheel built from stub `__init__.py`** — the "stub init, wheel, copy src, wheel again" pattern silently kept the first wheel's contents. The installed package had an empty `__init__.py`, so `from cyclone import __version__` crashed at import time. Single `COPY src/` + single `pip wheel` is correct.
2. **Backend binds to `127.0.0.1`** — intentional (local-only by design, see CLAUDE.md). But that means the frontend container can't reach it over the compose bridge. Added `CYCLONE_HOST` env var, defaults to `127.0.0.1`, set to `0.0.0.0` in compose.
3. **Healthcheck probed `/api/healthz` (404)** — actual route is `/api/health`. Updated Dockerfile, compose, RUNBOOK, post-deploy.sh, smoke.sh.
4. **Auth matrix had `("/api/healthz"): set()`** — wrong path; would have deny-by-defaulted the real health endpoint. Updated to `/api/health`.
5. **nginx upstream `cyclone-backend:8000`** — compose v2 only resolves the bare service name over the bridge network. Updated to `backend:8000`.
6. **Frontend healthcheck `http://localhost:8080/`** — alpine nginx listens on IPv6 by default; `localhost` is unreliable. Updated to `http://127.0.0.1:8080/`.

Live verification results (with `/tmp/cyclone-test-secrets/` + override, nocodb stopped):
- Both containers `(healthy)` within ~60s
- `curl http://localhost:8080/api/health` → 200, valid JSON
- `POST /api/auth/login` as admin → 200, cookie set
- `GET /api/auth/me` → 200 with admin user
- `POST /api/parse-837` with `docs/goodclaim.x12` → 200, batch `6252a9265ea943039d363bbca2c16059` created
- Full backend test suite (`DOCKER_TESTS=1`): **1026 passed, 9 skipped** (skips are prodfile corpus, gitignored)

Plus a followup commit `aecf831` to the live bring-up test itself: uses the override file when present, skips cleanly when port 8080 is bound by another service on a dev workstation.