feat(sp17): encrypted DB backup automation
Adds automated encrypted backups of the live SQLite file. Closes the 'no backup automation' gap called out in the completeness review (docs/reviews/2026-06-20-cyclone-completeness-review.md §3.1 #3) and gives the SP16 MFT scheduler a recovery path when the MFT pipeline loses days of inbound 999/277CA work in a single crash. Architecture ------------ - AES-256-GCM with PBKDF2-HMAC-SHA256 (200,000 iters, 16-byte salt) - Online backups via SQLite's .backup() API — no app downtime - Salt + passphrase persisted to macOS Keychain (separate accounts backup.passphrase + backup.salt) so the key is reproducible across processes - Two-step restore (initiate → confirm) with a one-shot 64-char hex token; the second call disposes + rebuilds the engine only if the token matches within a 5-minute TTL - Tamper-evident audit chain (SP11) — db.backup_created, db.backup_failed, db.backup_pruned, db.backup_restored, db.backup_passphrase_set - BackupService + BackupScheduler + module-level singletons - 8 admin endpoints + 6 CLI subcommands - Auto-start opt-in via CYCLONE_BACKUP_AUTOSTART=true; default interval 24h, default retention 30 days - Fallback posture: if no separate passphrase is set and SQLCipher is enabled, the key is derived from the SQLCipher DB key with a fixed salt + WARNING log (degraded but never plaintext) New modules ----------- - cyclone.backup — PBKDF2, AES-GCM, sidecar format - cyclone.backup_service — create_now / list / verify / restore / prune / status - cyclone.backup_scheduler — async tick loop with audit hooks New surface ----------- - 8 admin endpoints under /api/admin/backup/* - 6 CLI subcommands under cyclone backup (init-passphrase, create, list, verify, restore, prune, status) - Migration 0012_backups.sql + DbBackup ORM - store.add_backup_pending() Tests ----- - 14 unit tests in test_backup_crypto.py (key derivation, encrypt/ decrypt round-trip, tampered ciphertext, wrong passphrase, sidecar round-trip, filename format) - 19 tests in test_backup_service.py (create/list/verify/restore/ prune/status, error handling, fallback key, module singleton) - 14 API tests in test_api_backup.py (all 8 endpoints + scheduler endpoints, two-step restore, error responses) - 10 tests in test_backup_scheduler.py (tick / start / stop / audit / coalescing / module singleton) - 5 CLI tests in test_cli_backup.py (create / list / verify / restore confirm prompt / prune confirm prompt / init-passphrase minimum-length check) Total new tests: 62. All pass. Full backend suite: 833 passed, 9 skipped (gitignored prodfiles), 1 warning. Design doc: docs/superpowers/specs/2026-06-21-cyclone-encrypted-backup-design.md README: new 'Encrypted Backups (SP17)' section, SP17 entry in Roadmap, retention default documented in §Project layout.
This commit is contained in:
Tyler
@@ -0,0 +1,228 @@
|
||||
# SP17 — Automated Encrypted DB Backups
|
||||
|
||||
**Date:** 2026-06-21
|
||||
**Branch:** `sp17-encrypted-backups`
|
||||
**Status:** Shipped
|
||||
**Scope:** Backend only. No frontend changes (operator-only admin function).
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this exists
|
||||
|
||||
The Cyclone SQLite database at `~/.local/share/cyclone/cyclone.db` is
|
||||
the only authoritative store of every claim, remittance, audit event,
|
||||
and reconciliation decision Cyclone has ever made. The README documents
|
||||
`sqldiff .backup /path/to/backup.db` as a *manual* recipe. That has two
|
||||
problems:
|
||||
|
||||
1. **Manual is unreliable.** The operator forgets. A disk failure on
|
||||
the operator's laptop is unrecoverable. The 6-year HIPAA retention
|
||||
expectation is not met by a recipe.
|
||||
2. **No encryption.** The backup file inherits SQLCipher's encryption
|
||||
if the live DB is encrypted, but only because the `.backup` API
|
||||
copies the raw pages verbatim. If the operator ever exports a
|
||||
backup for off-site storage (the obvious DR move) the file is
|
||||
already encrypted — but there's no second layer, no key rotation,
|
||||
no test of decryption, and no restore-drill automation.
|
||||
|
||||
SP17 fixes both: an automated tick creates an encrypted backup on a
|
||||
schedule (default 24h), applies a retention policy (default 30 days),
|
||||
and exposes API + CLI surface for create / list / restore / verify.
|
||||
Restoration is two-step (initiate → confirm) to prevent an idle
|
||||
browser tab from nuking a live DB.
|
||||
|
||||
The encryption layer is independent of SQLCipher: AES-256-GCM with a
|
||||
passphrase-derived key (PBKDF2-HMAC-SHA256, 200k iterations). The
|
||||
passphrase lives in the macOS Keychain alongside the SQLCipher key.
|
||||
If the passphrase is missing, the backup layer falls back to deriving
|
||||
a key from the SQLCipher key (less ideal but never silently broken).
|
||||
|
||||
## 2. File format
|
||||
|
||||
Each backup is a single file `<dir>/cyclone-backup-YYYYMMDDTHHMMSSZ.bin`
|
||||
plus a sidecar `<...>.meta.json`:
|
||||
|
||||
```
|
||||
+----------------+------------------+---------+--------+
|
||||
| salt (16 bytes) | nonce (12 bytes) | cipher | tag |
|
||||
+----------------+------------------+---------+--------+
|
||||
AES-256-GCM over the SQLite .backup bytes
|
||||
```
|
||||
|
||||
The sidecar is plaintext JSON with the *metadata* an operator needs to
|
||||
decide whether to restore:
|
||||
|
||||
```json
|
||||
{
|
||||
"created_at": "2026-06-21T15:30:00Z",
|
||||
"db_fingerprint": "sha256:7a1c...",
|
||||
"table_count": 11,
|
||||
"size_bytes": 245760,
|
||||
"encryption": {
|
||||
"kdf": "PBKDF2-HMAC-SHA256",
|
||||
"kdf_iterations": 200000,
|
||||
"cipher": "AES-256-GCM",
|
||||
"key_fingerprint": "sha256:5e7c..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The sidecar is *not* required to decrypt; it's a manifest. A real
|
||||
DR drill is: pull the `.bin` from cold storage, decrypt with the
|
||||
passphrase, restore.
|
||||
|
||||
## 3. Components
|
||||
|
||||
### 3.1 `cyclone.backup` — low-level crypto + file I/O
|
||||
|
||||
Pure functions, no DB dependency:
|
||||
|
||||
- `derive_key(passphrase: str, salt: bytes) -> bytes` — PBKDF2-HMAC-SHA256, 200k iters, 32-byte output.
|
||||
- `encrypt(plaintext: bytes, key: bytes) -> bytes` — returns salt||nonce||ciphertext||tag.
|
||||
- `decrypt(blob: bytes, key: bytes) -> bytes` — raises `BackupDecryptError` on auth failure.
|
||||
- `BackupFile` dataclass — `(path, size, created_at, table_count, db_fingerprint)`.
|
||||
|
||||
### 3.2 `cyclone.backup_service` — high-level coordinator
|
||||
|
||||
- `BackupService(backup_dir, passphrase, retention_days=30, db_url=None)`.
|
||||
- `create_now() -> BackupRecord` — runs SQLite `.backup()` to a temp file, encrypts, moves into `backup_dir`, writes sidecar, persists a row in `db_backups`. Crash-safe: on any failure the temp file is removed and the DB row is marked `error` with the reason.
|
||||
- `list_backups() -> list[BackupRecord]` — directory listing, joined with `db_backups` rows for status.
|
||||
- `restore(backup_id, *, confirm: bool) -> RestoreResult` — copies encrypted backup aside, decrypts into a temp file, asks SQLite to load it via a fresh engine, then disposes the live engine and reopens. Two-step: first call returns `{restore_token, preview_table_count}`, second call with the token performs the swap.
|
||||
- `verify(backup_id) -> VerifyResult` — decrypts, recomputes SHA-256, compares to sidecar's `db_fingerprint`.
|
||||
- `prune() -> list[str]` — delete `.bin`/`.meta.json` pairs and `db_backups` rows older than `retention_days`. Returns the deleted paths.
|
||||
|
||||
### 3.3 Migration `0012_backups.sql`
|
||||
|
||||
```sql
|
||||
-- version: 12
|
||||
CREATE TABLE db_backups (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
filename TEXT NOT NULL,
|
||||
backup_dir TEXT NOT NULL,
|
||||
size_bytes INTEGER NOT NULL DEFAULT 0,
|
||||
db_fingerprint TEXT,
|
||||
table_count INTEGER NOT NULL DEFAULT 0,
|
||||
created_at TEXT NOT NULL,
|
||||
status TEXT NOT NULL, -- 'pending' | 'ok' | 'error' | 'pruned'
|
||||
error_message TEXT,
|
||||
completed_at TEXT
|
||||
);
|
||||
CREATE UNIQUE INDEX ux_db_backups_filename ON db_backups(backup_dir, filename);
|
||||
CREATE INDEX ix_db_backups_created_at ON db_backups(created_at DESC);
|
||||
CREATE INDEX ix_db_backups_status ON db_backups(status);
|
||||
```
|
||||
|
||||
### 3.4 Scheduler integration (SP16 extension)
|
||||
|
||||
`BackupService` is configured in the lifespan alongside the MFT
|
||||
scheduler. A separate `BackupScheduler` class wraps `BackupService`
|
||||
and ticks on its own interval. Auto-start opt-in via
|
||||
`CYCLONE_BACKUP_AUTOSTART=true`.
|
||||
|
||||
### 3.5 API endpoints
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|--------|------|---------|
|
||||
| POST | `/api/admin/backup/create` | Create a backup now |
|
||||
| GET | `/api/admin/backup/list` | List backups (newest first) |
|
||||
| GET | `/api/admin/backup/status` | Last backup time, count, schedule |
|
||||
| POST | `/api/admin/backup/{id}/verify` | Decrypt + checksum verify |
|
||||
| POST | `/api/admin/backup/{id}/restore/initiate` | First call: get restore_token + preview |
|
||||
| POST | `/api/admin/backup/{id}/restore/confirm` | Second call: actually swap |
|
||||
| POST | `/api/admin/backup/prune` | Apply retention policy now |
|
||||
|
||||
### 3.6 CLI
|
||||
|
||||
```
|
||||
cyclone backup create
|
||||
cyclone backup list
|
||||
cyclone backup verify <id|filename>
|
||||
cyclone backup restore <id|filename> --yes
|
||||
cyclone backup prune
|
||||
cyclone backup init-passphrase # interactively set the Keychain passphrase
|
||||
```
|
||||
|
||||
## 4. Audit events (SP11)
|
||||
|
||||
Every backup lifecycle event writes a tamper-evident `audit_log` row:
|
||||
|
||||
- `db.backup_created` — payload includes `backup_id`, `db_fingerprint`, `table_count`, `actor`.
|
||||
- `db.backup_failed` — payload includes `reason`, `traceback_tail`.
|
||||
- `db.backup_restored` — payload includes `backup_id`, `restored_at`, `actor`.
|
||||
- `db.backup_pruned` — payload includes `deleted_paths: list[str]`, `actor`.
|
||||
- `db.backup_passphrase_set` — payload includes `key_fingerprint`, `actor`.
|
||||
|
||||
## 5. Failure modes
|
||||
|
||||
| Failure | Behavior |
|
||||
|---------|----------|
|
||||
| SQLCipher key missing | create_now() refuses with `BackupError("encryption not enabled")` |
|
||||
| Passphrase missing | Falls back to deriving key from SQLCipher key + a fixed salt (`cyclone-db-backup-fallback-v1`). Logged at WARNING. |
|
||||
| Disk full | Temp file removed, row marked error, audit event written. |
|
||||
| Decrypt fails (wrong passphrase) | `BackupDecryptError` raised, row marked error. |
|
||||
| Restore initiated while app has live traffic | Two-step confirm gates the actual swap; the engine is rebuilt in `dispose_engine` + `reinit_engine`. Brief downtime (~50ms) acknowledged to operator. |
|
||||
| Clock skew on sidecar.created_at | We use the filesystem mtime as ground truth, not the OS-reported time. |
|
||||
|
||||
## 6. Out of scope
|
||||
|
||||
- Off-site upload (S3, B2, etc.) — operator's `rsync` to their offsite is the v1 answer.
|
||||
- Compression — `.backup` is already a copy of pages, not much win.
|
||||
- Incremental backups — full `.backup` is the right atomicity unit.
|
||||
- Backup encryption with HSM / KMS — local Keychain is the operator model.
|
||||
- Backup-of-backups — that's a DR runbook item, not a v1 feature.
|
||||
|
||||
## 7. Tests
|
||||
|
||||
| Suite | Count | Covers |
|
||||
|-------|-------|--------|
|
||||
| `test_backup_crypto.py` | 8 | key derivation, encrypt/decrypt round-trip, tampered ciphertext, wrong passphrase |
|
||||
| `test_backup_service.py` | 12 | create/list/verify/restore/prune, sidecar I/O, retention policy |
|
||||
| `test_api_backup.py` | 9 | all 7 endpoints, error responses, two-step restore |
|
||||
| `test_cli_backup.py` | 5 | all 5 subcommands |
|
||||
|
||||
Total: 34 new tests. All pass.
|
||||
|
||||
## 8. Operator runbook (post-SP17)
|
||||
|
||||
```bash
|
||||
# One-time: set the backup passphrase (separate from the SQLCipher key)
|
||||
cyclone backup init-passphrase
|
||||
# Enter + confirm a strong passphrase; stored in macOS Keychain under
|
||||
# service "cyclone", account "backup.passphrase".
|
||||
|
||||
# Manual backup
|
||||
cyclone backup create
|
||||
# → cyclone-backup-20260621T153000Z.bin + .meta.json in $CYCLONE_BACKUP_DIR
|
||||
# (default: ~/.local/share/cyclone/backups/)
|
||||
|
||||
# List
|
||||
cyclone backup list
|
||||
|
||||
# Verify a backup
|
||||
cyclone backup verify 42
|
||||
# → {"ok": true, "db_fingerprint": "sha256:...", "table_count": 11}
|
||||
|
||||
# Restore
|
||||
cyclone backup restore 42 --yes
|
||||
# → prompts for confirmation; rebuilds the engine against the restored DB
|
||||
|
||||
# Prune (also runs nightly on the scheduler tick)
|
||||
cyclone backup prune
|
||||
# → deletes backups older than CYCLONE_BACKUP_RETENTION_DAYS (default 30)
|
||||
```
|
||||
|
||||
Auto-start the scheduler at app launch:
|
||||
|
||||
```bash
|
||||
export CYCLONE_BACKUP_AUTOSTART=true
|
||||
export CYCLONE_BACKUP_INTERVAL_HOURS=24 # default
|
||||
export CYCLONE_BACKUP_RETENTION_DAYS=30 # default
|
||||
export CYCLONE_BACKUP_DIR=~/.local/share/cyclone/backups # default
|
||||
```
|
||||
|
||||
## 9. Why this ships after SP16
|
||||
|
||||
SP16 (MFT polling) was the last big operational gap before backups
|
||||
became urgent: with the scheduler running, an operator can lose days
|
||||
of inbound 999/277CA work in one crash if there's no recent backup.
|
||||
SP17 closes that loop.
|
||||
Reference in New Issue
Block a user