feat(sp17): encrypted DB backup automation

Adds automated encrypted backups of the live SQLite file. Closes the
'no backup automation' gap called out in the completeness review
(docs/reviews/2026-06-20-cyclone-completeness-review.md §3.1 #3) and
gives the SP16 MFT scheduler a recovery path when the MFT pipeline
loses days of inbound 999/277CA work in a single crash.

Architecture
------------
- AES-256-GCM with PBKDF2-HMAC-SHA256 (200,000 iters, 16-byte salt)
- Online backups via SQLite's .backup() API — no app downtime
- Salt + passphrase persisted to macOS Keychain (separate accounts
  backup.passphrase + backup.salt) so the key is reproducible across
  processes
- Two-step restore (initiate → confirm) with a one-shot 64-char hex
  token; the second call disposes + rebuilds the engine only if the
  token matches within a 5-minute TTL
- Tamper-evident audit chain (SP11) — db.backup_created,
  db.backup_failed, db.backup_pruned, db.backup_restored,
  db.backup_passphrase_set
- BackupService + BackupScheduler + module-level singletons
- 8 admin endpoints + 6 CLI subcommands
- Auto-start opt-in via CYCLONE_BACKUP_AUTOSTART=true; default
  interval 24h, default retention 30 days
- Fallback posture: if no separate passphrase is set and SQLCipher
  is enabled, the key is derived from the SQLCipher DB key with a
  fixed salt + WARNING log (degraded but never plaintext)

New modules
-----------
- cyclone.backup          — PBKDF2, AES-GCM, sidecar format
- cyclone.backup_service  — create_now / list / verify / restore / prune / status
- cyclone.backup_scheduler — async tick loop with audit hooks

New surface
-----------
- 8 admin endpoints under /api/admin/backup/*
- 6 CLI subcommands under cyclone backup (init-passphrase, create,
  list, verify, restore, prune, status)
- Migration 0012_backups.sql + DbBackup ORM
- store.add_backup_pending()

Tests
-----
- 14 unit tests in test_backup_crypto.py (key derivation, encrypt/
  decrypt round-trip, tampered ciphertext, wrong passphrase, sidecar
  round-trip, filename format)
- 19 tests in test_backup_service.py (create/list/verify/restore/
  prune/status, error handling, fallback key, module singleton)
- 14 API tests in test_api_backup.py (all 8 endpoints + scheduler
  endpoints, two-step restore, error responses)
- 10 tests in test_backup_scheduler.py (tick / start / stop /
  audit / coalescing / module singleton)
- 5 CLI tests in test_cli_backup.py (create / list / verify /
  restore confirm prompt / prune confirm prompt /
  init-passphrase minimum-length check)

Total new tests: 62. All pass. Full backend suite: 833 passed,
9 skipped (gitignored prodfiles), 1 warning.

Design doc: docs/superpowers/specs/2026-06-21-cyclone-encrypted-backup-design.md
README: new 'Encrypted Backups (SP17)' section, SP17 entry in
Roadmap, retention default documented in §Project layout.
This commit is contained in:
Tyler
2026-06-21 09:43:51 -06:00
parent 7119b7a2b1
commit f003c1f73a
16 changed files with 3735 additions and 6 deletions
+98 -1
View File
@@ -357,6 +357,70 @@ operator has created the Keychain entry on first run. See
for the one-time setup recipe and the HIPAA Security Rule §164.312(a)(2)(iv)
mapping.
### Key rotation (SP15)
`POST /api/admin/db/rotate-key` re-encrypts the SQLite file in place
with a fresh SQLCipher key via `PRAGMA rekey`, then updates the
Keychain so subsequent connections open with the new key. The
rotation holds a module-level `threading.Lock` (so two concurrent
requests can't race), disposes + rebuilds the SQLAlchemy engine with
`NullPool` (so SQLCipher's thread affinity is honored), and writes a
tamper-evident `db.key_rotated` audit event with old + new
fingerprints and the post-rotation table count. The old key is
retained in the `cyclone.db.key.previous` Keychain account for a
grace period so a botched rotation can be rolled back by hand.
## Encrypted Backups (SP17)
The BackupService takes an online consistent snapshot of the live
SQLite file via SQLite's `.backup()` API, encrypts the bytes with
AES-256-GCM, and writes a `.bin` + `.meta.json` pair into the backup
directory (default `~/.local/share/cyclone/backups/`). The encryption
key is derived from a separate passphrase in the macOS Keychain
(PBKDF2-HMAC-SHA256, 200,000 iterations, 16-byte salt persisted to
Keychain) — so a SQLCipher DB-key compromise does not unlock the
backups, and a backup-passphrase compromise does not unlock the live
DB. If neither is set, the service refuses (`BackupError`) rather than
silently writing plaintext.
| Method | Path | Purpose |
| ------ | ---- | ------- |
| POST | `/api/admin/backup/create` | Take an encrypted backup now. |
| GET | `/api/admin/backup/list` | List `db_backups` rows (newest first, filterable). |
| GET | `/api/admin/backup/status` | Counts, disk usage, last-run timestamp, scheduler snapshot. |
| POST | `/api/admin/backup/{id}/verify` | Decrypt + SHA-256 verify against the sidecar. |
| POST | `/api/admin/backup/{id}/restore/initiate` | First step: get `restore_token` + preview (fingerprints of backup vs live). |
| POST | `/api/admin/backup/{id}/restore/confirm` | Second step: dispose engine, copy decrypted DB, rebuild engine. |
| POST | `/api/admin/backup/prune` | Apply retention policy now. |
| POST | `/api/admin/backup/scheduler/{start,stop,tick}` | Operate the backup scheduler. |
Restore is two-step by design: an idle browser tab can't nuke the
live DB. The first call returns a one-shot 64-char hex
`restore_token` plus a side-by-side preview (`backup_db_fingerprint`,
`backup_table_count`, `current_db_fingerprint`, `current_table_count`).
The second call swaps the live engine only if the token matches
within a 5-minute TTL.
The scheduler (auto-start opt-in via `CYCLONE_BACKUP_AUTOSTART`)
ticks every `CYCLONE_BACKUP_INTERVAL_HOURS` (default 24), runs
`create_now` + `prune`, and writes audit events for each outcome
(`db.backup_created`, `db.backup_failed`, `db.backup_pruned`,
`db.backup_restored`). The CLI mirrors the API surface:
```bash
cyclone backup init-passphrase # one-time; interactive
cyclone backup create
cyclone backup list
cyclone backup verify <id>
cyclone backup restore <id> --yes
cyclone backup prune --yes
cyclone backup status
```
Retention defaults to 30 days (`CYCLONE_BACKUP_RETENTION_DAYS`). The
retention policy is best-effort: an operator who runs `cyclone
backup create` manually retains full control.
## SFTP Wire-Up (paramiko)
The `clearhouse.submit` endpoint uses `paramiko` to push a batch of
@@ -473,7 +537,7 @@ backup API).
## Roadmap
Sub-projects 2 through 13 are **shipped**. See the [completeness
Sub-projects 2 through 17 are **shipped**. See the [completeness
review](docs/reviews/2026-06-20-cyclone-completeness-review.md) for
the honest gap analysis against the industry definition of a HIPAA
clearinghouse — the short version is that the local-only,
@@ -484,6 +548,39 @@ scope.
Shipped sub-projects (most recent first):
- **Sub-project 17 (shipped) — Encrypted DB backups.** Automated
encrypted backups via AES-256-GCM (PBKDF2-HMAC-SHA256, 200k iters).
The operator sets a separate passphrase in the macOS Keychain
(`cyclone backup init-passphrase`); if missing, the service falls
back to deriving from the SQLCipher DB key with a WARNING. Online
backups via SQLite `.backup()`, two-step restore (`initiate`
`confirm` with one-shot 64-char hex token), retention pruning with
a 30-day default, and a tamper-evident audit chain (`db.backup_created`,
`db.backup_failed`, `db.backup_pruned`, `db.backup_restored`,
`db.backup_passphrase_set`). Backup scheduler ticks every 24h
(configurable); auto-start opt-in via `CYCLONE_BACKUP_AUTOSTART`.
Seven admin endpoints + six CLI subcommands. See
[Encrypted Backups](#encrypted-backups) below.
- **Sub-project 16 (shipped) — Live MFT polling scheduler.** asyncio
background loop polls the Gainwell MFT inbound path, downloads
new files, and routes them through the right parser (999 / 835 /
277CA / TA1). Idempotent (re-ticks skip already-processed files
via the new `processed_inbound_files` table). Crash-safe (per-file
try/except so a bad file doesn't stop the loop). Five admin
endpoints (`/api/admin/scheduler/{status,start,stop,tick,processed-files}`).
- **Sub-project 15 (shipped) — SQLCipher key rotation.** In-place
rotation via `PRAGMA rekey`, serialized through a module-level
`threading.Lock` and a SQLAlchemy `NullPool` to keep SQLCipher
thread-affine under FastAPI's per-request threadpool. Writes a
`db.key_rotated` audit event with old + new key fingerprints and
post-rotation `table_count`. See
[Encryption at Rest — Key rotation](#key-rotation).
- **Sub-project 14 (shipped) — 5-lane Inbox UI.** The Payer-Rejected
lane is now rendered in the Inbox alongside Rejected / Candidates /
Unmatched / Done today. New bulk action
`POST /api/inbox/payer-rejected/acknowledge` drops claims from the
working surface without erasing the original 277CA rejection event
(audit log stays intact, SP11).
- **Sub-project 13 (shipped) — SFTP wire-up.** `paramiko`-backed
`SftpClient` replaces the SP9 stub. The clearhouse.submit endpoint
actually pushes to