fix(deploy): make 0001_init.sql re-runnable and repair historical _migrations tracking
Deploy to route.crispygoat.com / deploy (push) Failing after 1m12s

The "Run migrations" job (and thus the whole deploy) was failing on every push after the first successful bootstrap with:

  ✗ 0001_init.sql failed: relation "admin_users" already exists

Root causes:
- 0001_init.sql used plain CREATE TABLE/INDEX/TRIGGER despite the header comment claiming "CREATE TABLE IF NOT EXISTS".
- The _migrations tracking row for 0001_init.sql was never recorded on the prod DATABASE_URL (tracking logic landed after initial apply, or apply happened outside the runner).

Fix (3 layers of defense):
* db/migrations/0001_init.sql: every CREATE TABLE now uses IF NOT EXISTS (63 tables), CREATE INDEX uses IF NOT EXISTS (49), all CREATE TRIGGER wrapped in safe DO $$ IF NOT EXISTS (pg_trigger join check) THEN ... END IF; $$, removed the file's own BEGIN/COMMIT so the runner's transaction is authoritative. RLS drop+create and CREATE OR REPLACE FUNCTION were already safe.
* scripts/migrate.js: added ensureTracked() right after loading the applied set. For 0001/0002, if the core objects (admin_users table, password_hash column) already exist in information_schema but the tracking row is absent, INSERT the filename (ON CONFLICT DO NOTHING) and log "repaired tracking". Then the normal skip path handles it.
* .gitea/workflows/deploy.yml: added an inline node -e pre-repair (same admin_users existence check → force _migrations row) immediately before `npm run migrate:one`. This protects even on an older checkout of the runner. The neon_auth preflight and post-apply "admin_users must exist" hard gate are untouched.
* Minor: cleaned 0002_admin_password.sql tx wrapper for consistency; updated MEMORY.md with the incident + resolution.

Result: deploys (and any server-side `node scripts/migrate.js` recovery runs) on an already-initialized prod DB will repair/skip 0001 cleanly, pass the verification query, and continue to build/deploy. The scp of scripts/ + db/migrations/ in the Deploy step now carries the fixed versions.

See the plan note in deploy.yml and MEMORY.md for background.
This commit is contained in:
Tyler
2026-06-09 17:55:42 -06:00
parent 0db1609c89
commit 1d4300d505
5 changed files with 487 additions and 152 deletions
+28
View File
@@ -48,6 +48,34 @@ jobs:
.catch(e => { console.error("Pre-flight check failed:", e.message); process.exit(1); }); .catch(e => { console.error("Pre-flight check failed:", e.message); process.exit(1); });
' '
echo "=== Running migrations against DATABASE_URL (masked value for logs) ===" echo "=== Running migrations against DATABASE_URL (masked value for logs) ==="
# The migrate runner (scripts/migrate.js) now includes repair logic for
# DBs that had 0001_init.sql applied before _migrations tracking was added.
# We also run a tiny pre-repair here so the step is resilient even if an
# older copy of the runner is present in the checkout.
node -e '
const { Client } = require("pg");
const url = process.env.DATABASE_URL;
if (!url) { process.exit(0); }
const c = new Client({ connectionString: url });
c.connect()
.then(() => c.query("SELECT 1 FROM information_schema.tables WHERE table_name = '\''admin_users'\'' LIMIT 1"))
.then(async (res) => {
if (res.rows.length > 0) {
await c.query(`
CREATE TABLE IF NOT EXISTS _migrations (
filename TEXT PRIMARY KEY,
applied_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
INSERT INTO _migrations (filename)
VALUES ('\''0001_init.sql'\'')
ON CONFLICT (filename) DO NOTHING;
`);
console.log("✓ 0001_init.sql tracking repaired (admin_users already present)");
}
return c.end();
})
.catch(() => { /* best effort */ });
'
npm run migrate:one npm run migrate:one
echo "=== Post-migration verification: critical table admin_users must exist ===" echo "=== Post-migration verification: critical table admin_users must exist ==="
node -e ' node -e '
+29 -1
View File
@@ -2,7 +2,35 @@
This file captures key context, decisions, fixes, and state from recent work so it survives across conversations. This file captures key context, decisions, fixes, and state from recent work so it survives across conversations.
**Last updated:** 2026-06-06 (Supabase → Postgres pivot) **Last updated:** 2026-06 (migration reliability + Google sign-in work)
## 2026-06: CI migration failures on re-deploy (0001_init.sql "already exists")
Prod DATABASE_URL already had the schema from the first successful bootstrap.
The deploy workflow runs `npm run migrate:one` on every push (after neon_auth preflight).
`scripts/migrate.js` has `_migrations` tracking + skip, but the row for `0001_init.sql` was never recorded (the tracking logic landed after the initial apply, or an apply happened outside the runner).
`db/migrations/0001_init.sql` header *claimed* "CREATE TABLE IF NOT EXISTS" but the actual statements were plain `CREATE TABLE`, plain `CREATE INDEX`, and unguarded `CREATE TRIGGER`.
Result: every subsequent deploy hit `relation "admin_users" already exists` (and would have hit index/trigger dups too) inside the runner's BEGIN, causing ROLLBACK + failure of the whole "Run migrations" job.
### Fixes applied
- Made `0001_init.sql` truly re-runnable:
- All `CREATE TABLE``CREATE TABLE IF NOT EXISTS`
- All `CREATE INDEX` / `CREATE UNIQUE INDEX``... IF NOT EXISTS`
- Every `CREATE TRIGGER` wrapped in a `DO $$ IF NOT EXISTS (pg_trigger check) THEN CREATE TRIGGER ... END IF; $$` guard
- Removed the file-level `BEGIN; ... COMMIT;` (the runner owns the tx; this also prevents inner-COMMIT from ending the runner tx early).
- `0002_admin_password.sql` had its tx wrapper removed for consistency (its ALTER was already `IF NOT EXISTS`).
- Hardened `scripts/migrate.js`:
- Added `ensureTracked()` repair: for 0001/0002, if the core objects (admin_users table, or the password_hash column) already exist in the target DB but the tracking row is absent, we INSERT the row (ON CONFLICT DO NOTHING) and skip the file. Logs "repaired tracking".
- Hardened `.gitea/workflows/deploy.yml` "Run migrations" step:
- Added an inline pre-repair node snippet (same idea) right before `npm run migrate:one`. This protects even if an older runner is checked out.
- The existing neon_auth preflight + post `admin_users` verification remain as the hard gate.
- Updated header comments and docs in the files.
After this, `npm run migrate` / deploys on an already-initialized DB will log the "repaired" or "already applied" lines for 0001 and proceed cleanly. The shipped `scripts/migrate.js` + `db/migrations/` on the target server also benefit for emergency recovery runs.
See also the plan doc referenced in deploy.yml for the broader reliability work.
--- ---
File diff suppressed because it is too large Load Diff
+2 -3
View File
@@ -10,12 +10,11 @@
-- `src/lib/auth.ts` — it queries this column and runs `verifyPassword` -- `src/lib/auth.ts` — it queries this column and runs `verifyPassword`
-- (see `src/lib/passwords.ts`) before returning the user. -- (see `src/lib/passwords.ts`) before returning the user.
BEGIN; -- Transaction is managed by the migrate runner (scripts/migrate.js).
-- File kept small and re-runnable via IF NOT EXISTS on the ALTER.
ALTER TABLE users ALTER TABLE users
ADD COLUMN IF NOT EXISTS password_hash TEXT; ADD COLUMN IF NOT EXISTS password_hash TEXT;
-- Update the updated_at trigger tracking — no new triggers needed since -- Update the updated_at trigger tracking — no new triggers needed since
-- `users` already has `set_updated_at` from migration 0001. -- `users` already has `set_updated_at` from migration 0001.
COMMIT;
+39 -6
View File
@@ -1,14 +1,16 @@
#!/usr/bin/env node #!/usr/bin/env node
/** /**
* Apply Postgres migrations from `db/migrations/*.sql` in lexical order. * Apply Postgres migrations from `db/migrations/*.sql` in lexical order.
* Wraps the whole thing in a transaction; tracks applied files in * Wraps each new file in a transaction; tracks applied files in
* `_migrations` so re-runs are safe. * `_migrations` so re-runs are safe and idempotent.
*
* The 0001_init.sql (and 0002) are now fully re-runnable (IF NOT EXISTS +
* trigger guards) + this script has repair logic for DBs that were
* initialized before tracking was introduced.
* *
* Usage: * Usage:
* npm run db:migrate * npm run migrate
* * npm run migrate:one # same as above (applies all pending)
* Replaces the old `supabase/push-migrations.js` — that script was
* hardcoded to a Supabase URL. This one reads `DATABASE_URL` directly.
*/ */
require("dotenv").config({ path: ".env.local" }); require("dotenv").config({ path: ".env.local" });
@@ -51,6 +53,37 @@ async function main() {
); );
const appliedSet = new Set(applied.map((r) => r.filename)); const appliedSet = new Set(applied.map((r) => r.filename));
// Repair for historical DBs (applied before _migrations tracking existed,
// or via direct psql / earlier tooling). If the core objects are present
// we record the filename so future deploys (and server-side recovery runs)
// treat 0001/0002 as done without re-executing the large init script.
async function ensureTracked(filename, existenceCheckSql) {
if (appliedSet.has(filename)) return;
try {
const { rows } = await client.query(existenceCheckSql);
if (rows.length > 0) {
await client.query(
`INSERT INTO _migrations (filename) VALUES ($1) ON CONFLICT (filename) DO NOTHING`,
[filename]
);
console.log(`${filename} (objects present in DB; repaired tracking)`);
appliedSet.add(filename);
}
} catch (e) {
// Non-fatal: the check may fail on a brand-new DB or with limited perms.
// We'll let the normal apply path handle it.
}
}
await ensureTracked(
"0001_init.sql",
"SELECT 1 FROM information_schema.tables WHERE table_name = 'admin_users' LIMIT 1"
);
await ensureTracked(
"0002_admin_password.sql",
"SELECT 1 FROM information_schema.columns WHERE table_name = 'users' AND column_name = 'password_hash' LIMIT 1"
);
let appliedNow = 0; let appliedNow = 0;
for (const file of files) { for (const file of files) {
if (appliedSet.has(file)) { if (appliedSet.has(file)) {