route-commerce/docs/superpowers/plans/2026-06-prod-db-schema-migration-reliability.md

Production DB Schema Migration Reliability Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:systematic-debugging (completed root cause), superpowers:writing-plans (this), superpowers:test-driven-development where code changes have tests, superpowers:verification-before-completion, and superpowers:executing-plans or subagent-driven-development to implement task-by-task. Steps use checkbox syntax.

Goal: Ensure that every production deploy (push to main) results in the full schema from db/migrations/0001_init.sql (including the admin_users and admin_user_brands tables required by getAdminUser()) being present in the DATABASE_URL that the running app connects to via its pool / drizzle client. Eliminate "relation does not exist" errors and the resulting Access Denied screen for properly provisioned Neon Auth users.

Architecture:

• Make the CI "Run migrations" step a hard gate (fatal on failure, plus explicit post-migrate verification that critical tables exist).
• Ship the minimal migration runner + SQL files as part of the deploy artifact so the target server has a recovery path.
• Add a lightweight post-deploy / startup verification in the app or deploy script (fail fast with clear message instead of silent 500s on first admin request).
• Keep the "migrate from full source locally" path working for initial prod DB bootstrap and emergencies.
• Do not change the core migration logic or 0001_init.sql in this plan (that would be a separate architectural change if the double-BEGIN wrapping proves fragile).

Tech Stack: Gitea Actions (YAML), Node 22 + pg + drizzle on target, Next.js standalone output, pm2 on Ubuntu server, Neon Postgres (with neon_auth schema).

Root Cause (from systematic-debugging Phase 1): The prod runtime DB lacked the admin_users table because (1) the migration step in .gitea/workflows/deploy.yml used || echo making any failure (connection, SQL error in the huge 0001 file, FK to neon_auth.user, tx nesting from the file's BEGIN + script's BEGIN) non-fatal, (2) only .next/, public/, package.json (and optional next.config) are scp'd — scripts/migrate.js and db/migrations/ are never on the server, (3) no verification after "migrate" or at app startup that the tables the admin permission layer depends on actually exist, (4) the .env.production written from the same secret as the CI migrate step was used, but the apply didn't happen or was skipped due to _migrations state or partial rollback.

Evidence Gathered:

• deploy.yml: Run migrations step, limited scp, .env.production printf, pm2 "npm start".
• scripts/migrate.js: dotenv .env.local + env override, _migrations tracking, per-file client.query(sql) inside script tx, re-throw on error.
• db/migrations/0001_init.sql: explicit CREATE TABLE admin_users (with FK to neon_auth.user), admin_user_brands, brands; file starts with BEGIN;.
• db/client.ts + src/lib/admin-permissions.ts: withPlatformAdmin → drizzle select on adminUsers from schema (the exact query that 42P01s).
• next.config.ts: output: 'standalone' (explains pm2 warning).
• Runtime logs: the repeated "Database query failed" + "relation does not exist", app starts fine.
• Git history: recent deploy "fixes" focused on SSH/env writing, not migration reliability.

Files to touch (decomposition by responsibility):

• .gitea/workflows/deploy.yml (CI pipeline gates + artifact contents)
• scripts/migrate.js (minor hardening if needed for verification hook)
• src/app/api/health/route.ts or similar (new, for startup/schema check — or add to existing)
• CLAUDE.md + PRODUCTION_DEPLOYMENT_CHECKLIST.md (docs)
• Possibly a small scripts/verify-prod-schema.js helper

---

Task 1: Make CI migration step a hard failure + add explicit verification for admin_users

Files:

• Modify: .gitea/workflows/deploy.yml:23-27 (the Run migrations step and surrounding)

• Step 1.1: Replace the non-fatal migration line with a strict block that fails the job if migrate fails or the critical table is missing after.

- name: Run migrations
  env:
    DATABASE_URL: ${{ secrets.DATABASE_URL }}
  run: |
    set -e
    echo "=== Running migrations against prod DB ==="
    npm run migrate:one
    echo "=== Verifying critical schema (admin_users) ==="
    node -e '
      const {Client} = require("pg");
      const c = new Client({connectionString: process.env.DATABASE_URL});
      c.connect().then(() => c.query("SELECT 1 FROM admin_users LIMIT 1")).then(() => {
        console.log("✓ admin_users table exists");
        return c.end();
      }).then(() => process.exit(0)).catch(e => {
        console.error("✗ admin_users missing or inaccessible:", e.message);
        process.exit(1);
      });
    '

• Step 1.2: Run a local simulation or note that the Gitea runner will now fail the whole deploy if the secret DB is missing the table (good — forces the bootstrap to happen before code that depends on it ships).

• Step 1.3: Commit the yml change with message referencing the root cause (missing table in prod due to masked migration).

Task 2: Ship migration capability in the deploy artifact so server has a recovery path

Files:

• Modify: .gitea/workflows/deploy.yml in the "Deploy" step (the scp and ssh sections)

• Step 2.1: Add scp for the migration assets (after the existing public/.next scp):

echo "Copying migration runner and SQL..."
scp -o ConnectTimeout=15 -o StrictHostKeyChecking=no -r scripts/migrate.js tyler@...:$APP_DIR/scripts/ || true
scp -o ConnectTimeout=15 -o StrictHostKeyChecking=no -r db/migrations tyler@...:$APP_DIR/db/ || true

• Step 2.2: Update the server ssh install/restart line to also ensure the scripts dir has the right perms if needed, but mainly document that now node scripts/migrate.js will work on the server with the .env.production.

• Step 2.3: In the same Deploy step, after writing .env.production and before or after the pm2 restart, optionally run the migrate on the server as a belt-and-suspenders (using the just-written .env):

ssh ... "cd $APP_DIR && source .env.production 2>/dev/null || export \$(grep DATABASE_URL .env.production); node scripts/migrate.js || echo 'migrate on server completed or not needed'"

(Keep it non-fatal on server for now; the CI gate is the hard one.)

• Step 2.4: Test the scp paths in a dry-run or note the change.

Task 3: Add a minimal runtime / startup guard (fail fast with clear message)

Files:

• Create: src/app/api/health/db-schema/route.ts (or add to an existing health if present)

• Or simpler: in the admin layout or a top level, but a dedicated health is better for PM2/docker.

• Step 3.1: Create a tiny health endpoint that does the same check the CI verification does (SELECT 1 FROM admin_users) using the existing pool or withDb, returns 200 or 503 with message "Schema not applied - run migrations".

• Step 3.2: Wire it so the deploy can curl it after restart as a final gate (in the workflow ssh step).

• Step 3.3: (Optional but recommended per defense-in-depth) Call a similar check early in getAdminUser or the admin layout and log a very loud message + return a better error than generic "does not have admin access" when the table is literally missing.

Task 4: Update documentation and bootstrap instructions (so humans know the right sequence)

Files:

• Modify: CLAUDE.md (the Commands and Important File Locations + Gotchas sections)

• Modify or create: PRODUCTION_DEPLOYMENT_CHECKLIST.md or a new docs/PROD_BOOTSTRAP.md

• Step 4.1: In CLAUDE.md under "Commands" and "Adding a New Brand" / auth section, add a "First production deploy / new prod DB bootstrap" subsection:

  1. Ensure the Neon project has neon_auth enabled and the DATABASE_URL secret in Gitea points to it.
  2. (Before first code push that depends on admin) Locally or in a throwaway runner: DATABASE_URL=prod... node scripts/migrate.js
  3. Then DATABASE_URL=prod... npx tsx scripts/provision-admin.ts you@real.com platform_admin (after signing in on the prod URL).
  4. Push; the CI gate + shipped runner will keep it healthy on future deploys.
  5. If you ever see "relation admin_users does not exist" in prod logs, the DB the app is talking to is not the one that had migrate run.

• Step 4.2: Add a note about the || echo anti-pattern that was removed and why the new verification step exists.

• Step 4.3: Mention the standalone vs npm start issue (already in logs) and that the start command on server should eventually be updated to node .next/standalone/server.js -p 3100 (can be a follow-up task).

Task 5: Verification before claiming success (use the dedicated skill)

Files: (none new, just process)

• Step 5.1: Before merging the plan changes, use superpowers:verification-before-completion checklist: the change makes a fresh DB get the table, an "already applied" DB is a no-op, a deploy with missing table now fails the job early with clear output, a manual server migrate works because the files are there, the runtime health returns 200 when table present.

• Step 5.2: After the PR is on a branch, trigger a deploy to a staging or the real prod (with a test DB first if possible), capture the CI log showing the new verification passing, and the app logs showing no more "Database query failed" on /admin.

• Step 5.3: Run the provision script as the final user-visible test; confirm the Access Denied with email message is gone and the platform_admin can see the UI.

• Step 5.4: Document the before/after in the plan or a memory file.

Task 6: (Stretch / follow-up) Improve the migrate script's resilience for huge init files (if the double tx ever bites again)

Files:

• Modify: scripts/migrate.js

• Only if during verification the 0001 apply is flaky: change the per-file execution to not wrap the file's own BEGIN/COMMIT, or use a separate connection, or exec psql -f (but keep node/pg for consistency). Add a comment explaining the previous fragility.

Rollback / emergency: If a deploy breaks because of this, the server now has the scripts + db/migrations copied, so SSH + source .env.production; node scripts/migrate.js is the recovery (exactly what the user was trying to do manually).

Success criteria:

• A brand new prod DB + push to main results in a green deploy + working /admin after provision.
• The error "relation "admin_users" does not exist" no longer appears in prod pm2 logs for normal admin flows.
• The pipeline fails loudly (with the table name in the error) instead of shipping a broken app that only shows "Access Denied".