feat(sp17): encrypted DB backup automation
Adds automated encrypted backups of the live SQLite file. Closes the 'no backup automation' gap called out in the completeness review (docs/reviews/2026-06-20-cyclone-completeness-review.md §3.1 #3) and gives the SP16 MFT scheduler a recovery path when the MFT pipeline loses days of inbound 999/277CA work in a single crash. Architecture ------------ - AES-256-GCM with PBKDF2-HMAC-SHA256 (200,000 iters, 16-byte salt) - Online backups via SQLite's .backup() API — no app downtime - Salt + passphrase persisted to macOS Keychain (separate accounts backup.passphrase + backup.salt) so the key is reproducible across processes - Two-step restore (initiate → confirm) with a one-shot 64-char hex token; the second call disposes + rebuilds the engine only if the token matches within a 5-minute TTL - Tamper-evident audit chain (SP11) — db.backup_created, db.backup_failed, db.backup_pruned, db.backup_restored, db.backup_passphrase_set - BackupService + BackupScheduler + module-level singletons - 8 admin endpoints + 6 CLI subcommands - Auto-start opt-in via CYCLONE_BACKUP_AUTOSTART=true; default interval 24h, default retention 30 days - Fallback posture: if no separate passphrase is set and SQLCipher is enabled, the key is derived from the SQLCipher DB key with a fixed salt + WARNING log (degraded but never plaintext) New modules ----------- - cyclone.backup — PBKDF2, AES-GCM, sidecar format - cyclone.backup_service — create_now / list / verify / restore / prune / status - cyclone.backup_scheduler — async tick loop with audit hooks New surface ----------- - 8 admin endpoints under /api/admin/backup/* - 6 CLI subcommands under cyclone backup (init-passphrase, create, list, verify, restore, prune, status) - Migration 0012_backups.sql + DbBackup ORM - store.add_backup_pending() Tests ----- - 14 unit tests in test_backup_crypto.py (key derivation, encrypt/ decrypt round-trip, tampered ciphertext, wrong passphrase, sidecar round-trip, filename format) - 19 tests in test_backup_service.py (create/list/verify/restore/ prune/status, error handling, fallback key, module singleton) - 14 API tests in test_api_backup.py (all 8 endpoints + scheduler endpoints, two-step restore, error responses) - 10 tests in test_backup_scheduler.py (tick / start / stop / audit / coalescing / module singleton) - 5 CLI tests in test_cli_backup.py (create / list / verify / restore confirm prompt / prune confirm prompt / init-passphrase minimum-length check) Total new tests: 62. All pass. Full backend suite: 833 passed, 9 skipped (gitignored prodfiles), 1 warning. Design doc: docs/superpowers/specs/2026-06-21-cyclone-encrypted-backup-design.md README: new 'Encrypted Backups (SP17)' section, SP17 entry in Roadmap, retention default documented in §Project layout.
This commit is contained in:
@@ -357,6 +357,70 @@ operator has created the Keychain entry on first run. See
|
||||
for the one-time setup recipe and the HIPAA Security Rule §164.312(a)(2)(iv)
|
||||
mapping.
|
||||
|
||||
### Key rotation (SP15)
|
||||
|
||||
`POST /api/admin/db/rotate-key` re-encrypts the SQLite file in place
|
||||
with a fresh SQLCipher key via `PRAGMA rekey`, then updates the
|
||||
Keychain so subsequent connections open with the new key. The
|
||||
rotation holds a module-level `threading.Lock` (so two concurrent
|
||||
requests can't race), disposes + rebuilds the SQLAlchemy engine with
|
||||
`NullPool` (so SQLCipher's thread affinity is honored), and writes a
|
||||
tamper-evident `db.key_rotated` audit event with old + new
|
||||
fingerprints and the post-rotation table count. The old key is
|
||||
retained in the `cyclone.db.key.previous` Keychain account for a
|
||||
grace period so a botched rotation can be rolled back by hand.
|
||||
|
||||
## Encrypted Backups (SP17)
|
||||
|
||||
The BackupService takes an online consistent snapshot of the live
|
||||
SQLite file via SQLite's `.backup()` API, encrypts the bytes with
|
||||
AES-256-GCM, and writes a `.bin` + `.meta.json` pair into the backup
|
||||
directory (default `~/.local/share/cyclone/backups/`). The encryption
|
||||
key is derived from a separate passphrase in the macOS Keychain
|
||||
(PBKDF2-HMAC-SHA256, 200,000 iterations, 16-byte salt persisted to
|
||||
Keychain) — so a SQLCipher DB-key compromise does not unlock the
|
||||
backups, and a backup-passphrase compromise does not unlock the live
|
||||
DB. If neither is set, the service refuses (`BackupError`) rather than
|
||||
silently writing plaintext.
|
||||
|
||||
| Method | Path | Purpose |
|
||||
| ------ | ---- | ------- |
|
||||
| POST | `/api/admin/backup/create` | Take an encrypted backup now. |
|
||||
| GET | `/api/admin/backup/list` | List `db_backups` rows (newest first, filterable). |
|
||||
| GET | `/api/admin/backup/status` | Counts, disk usage, last-run timestamp, scheduler snapshot. |
|
||||
| POST | `/api/admin/backup/{id}/verify` | Decrypt + SHA-256 verify against the sidecar. |
|
||||
| POST | `/api/admin/backup/{id}/restore/initiate` | First step: get `restore_token` + preview (fingerprints of backup vs live). |
|
||||
| POST | `/api/admin/backup/{id}/restore/confirm` | Second step: dispose engine, copy decrypted DB, rebuild engine. |
|
||||
| POST | `/api/admin/backup/prune` | Apply retention policy now. |
|
||||
| POST | `/api/admin/backup/scheduler/{start,stop,tick}` | Operate the backup scheduler. |
|
||||
|
||||
Restore is two-step by design: an idle browser tab can't nuke the
|
||||
live DB. The first call returns a one-shot 64-char hex
|
||||
`restore_token` plus a side-by-side preview (`backup_db_fingerprint`,
|
||||
`backup_table_count`, `current_db_fingerprint`, `current_table_count`).
|
||||
The second call swaps the live engine only if the token matches
|
||||
within a 5-minute TTL.
|
||||
|
||||
The scheduler (auto-start opt-in via `CYCLONE_BACKUP_AUTOSTART`)
|
||||
ticks every `CYCLONE_BACKUP_INTERVAL_HOURS` (default 24), runs
|
||||
`create_now` + `prune`, and writes audit events for each outcome
|
||||
(`db.backup_created`, `db.backup_failed`, `db.backup_pruned`,
|
||||
`db.backup_restored`). The CLI mirrors the API surface:
|
||||
|
||||
```bash
|
||||
cyclone backup init-passphrase # one-time; interactive
|
||||
cyclone backup create
|
||||
cyclone backup list
|
||||
cyclone backup verify <id>
|
||||
cyclone backup restore <id> --yes
|
||||
cyclone backup prune --yes
|
||||
cyclone backup status
|
||||
```
|
||||
|
||||
Retention defaults to 30 days (`CYCLONE_BACKUP_RETENTION_DAYS`). The
|
||||
retention policy is best-effort: an operator who runs `cyclone
|
||||
backup create` manually retains full control.
|
||||
|
||||
## SFTP Wire-Up (paramiko)
|
||||
|
||||
The `clearhouse.submit` endpoint uses `paramiko` to push a batch of
|
||||
@@ -473,7 +537,7 @@ backup API).
|
||||
|
||||
## Roadmap
|
||||
|
||||
Sub-projects 2 through 13 are **shipped**. See the [completeness
|
||||
Sub-projects 2 through 17 are **shipped**. See the [completeness
|
||||
review](docs/reviews/2026-06-20-cyclone-completeness-review.md) for
|
||||
the honest gap analysis against the industry definition of a HIPAA
|
||||
clearinghouse — the short version is that the local-only,
|
||||
@@ -484,6 +548,39 @@ scope.
|
||||
|
||||
Shipped sub-projects (most recent first):
|
||||
|
||||
- **Sub-project 17 (shipped) — Encrypted DB backups.** Automated
|
||||
encrypted backups via AES-256-GCM (PBKDF2-HMAC-SHA256, 200k iters).
|
||||
The operator sets a separate passphrase in the macOS Keychain
|
||||
(`cyclone backup init-passphrase`); if missing, the service falls
|
||||
back to deriving from the SQLCipher DB key with a WARNING. Online
|
||||
backups via SQLite `.backup()`, two-step restore (`initiate` →
|
||||
`confirm` with one-shot 64-char hex token), retention pruning with
|
||||
a 30-day default, and a tamper-evident audit chain (`db.backup_created`,
|
||||
`db.backup_failed`, `db.backup_pruned`, `db.backup_restored`,
|
||||
`db.backup_passphrase_set`). Backup scheduler ticks every 24h
|
||||
(configurable); auto-start opt-in via `CYCLONE_BACKUP_AUTOSTART`.
|
||||
Seven admin endpoints + six CLI subcommands. See
|
||||
[Encrypted Backups](#encrypted-backups) below.
|
||||
- **Sub-project 16 (shipped) — Live MFT polling scheduler.** asyncio
|
||||
background loop polls the Gainwell MFT inbound path, downloads
|
||||
new files, and routes them through the right parser (999 / 835 /
|
||||
277CA / TA1). Idempotent (re-ticks skip already-processed files
|
||||
via the new `processed_inbound_files` table). Crash-safe (per-file
|
||||
try/except so a bad file doesn't stop the loop). Five admin
|
||||
endpoints (`/api/admin/scheduler/{status,start,stop,tick,processed-files}`).
|
||||
- **Sub-project 15 (shipped) — SQLCipher key rotation.** In-place
|
||||
rotation via `PRAGMA rekey`, serialized through a module-level
|
||||
`threading.Lock` and a SQLAlchemy `NullPool` to keep SQLCipher
|
||||
thread-affine under FastAPI's per-request threadpool. Writes a
|
||||
`db.key_rotated` audit event with old + new key fingerprints and
|
||||
post-rotation `table_count`. See
|
||||
[Encryption at Rest — Key rotation](#key-rotation).
|
||||
- **Sub-project 14 (shipped) — 5-lane Inbox UI.** The Payer-Rejected
|
||||
lane is now rendered in the Inbox alongside Rejected / Candidates /
|
||||
Unmatched / Done today. New bulk action
|
||||
`POST /api/inbox/payer-rejected/acknowledge` drops claims from the
|
||||
working surface without erasing the original 277CA rejection event
|
||||
(audit log stays intact, SP11).
|
||||
- **Sub-project 13 (shipped) — SFTP wire-up.** `paramiko`-backed
|
||||
`SftpClient` replaces the SP9 stub. The clearhouse.submit endpoint
|
||||
actually pushes to
|
||||
|
||||
Reference in New Issue
Block a user