tyler/cyclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(architecture): add top-level technical design doc

Browse Source 
Closes round 4 of doc-prep. ARCHITECTURE.md is the day-1 read for
engineers joining Cyclone: process & network topology, tech stack
(backend + frontend + external), backend package layout with
one-line module responsibility map, frontend architecture (route
map, state management, live tail), data model (migrations 0001-0012,
ERD, 7-state claim lifecycle, audit chain), data flow (manual upload,
SFTP polling, auto-match, outbound 837P, ack ingestion), API surface,
cross-cutting concerns (logging, audit, PHI/PII, error model),
operational concerns (startup order, schedulers, health probe,
shutdown), deployment, out-of-scope, and references.

759 lines, 13 top-level sections. Cross-linked from REQUIREMENTS.md
sister-docs block and from README.md Roadmap section.
This commit is contained in:
Tyler
2026-06-23 11:50:12 -06:00
parent cad104a38c
commit 1148a03a43
3 changed files with 765 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+5 -1
View File
@@ -801,7 +801,11 @@ backup API).
## Roadmap
> **Cross-cutting requirements, architecture, traceability, and definition of done live in [`docs/REQUIREMENTS.md`](docs/REQUIREMENTS.md).** Read that first if you are new — it is the single index that ties the 22 shipped sub-projects to the 38 functional requirements, the 18 non-functional requirements, the test strategy, and the project-level DoD.
> **Read order for new engineers:**
> 1. [`docs/REQUIREMENTS.md`](docs/REQUIREMENTS.md) — what Cyclone does (FRs + NFRs + DoD + traceability). The single index tying the 22 shipped sub-projects to the 38 functional + 18 non-functional requirements and the test strategy.
> 2. [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — how it fits together (process topology, package layout, data flow, lifecycle, operational concerns).
> 3. The per-SP spec under [`docs/superpowers/specs/`](docs/superpowers/specs/) for whatever you're touching.
> 4. The per-SP plan under [`docs/superpowers/plans/`](docs/superpowers/plans/) if you're implementing.
Sub-projects 2 through 19 are **shipped**. See the [completeness
review](docs/reviews/2026-06-20-cyclone-completeness-review.md) for
docs/ARCHITECTURE.md
+759
View File
@@ -0,0 +1,759 @@
# Cyclone — Technical Architecture
> **Day-1 read for engineers joining Cyclone.** This is the top-level technical design: how the system fits together, what the boundaries are, and why it looks the way it does.
>
> **Companion docs:**
> - **Requirements:** [`docs/REQUIREMENTS.md`](REQUIREMENTS.md) — what Cyclone does (FRs + NFRs + DoD + traceability).
> - **Specs:** [`docs/superpowers/specs/`](superpowers/specs/) — design specs per sub-project (22 specs).
> - **Plans:** [`docs/superpowers/plans/`](superpowers/plans/) — TDD-shaped implementation plans per sub-project (27 plans).
>
> **Scope of this doc:** architecture and design. The spec files own per-SP design; the plan files own the task order. This doc explains the *whole* — the boundaries between modules, the data flow across them, the lifecycle of an inbound file from upload to reconciled claim.
---
## 1. System overview
Cyclone is a self-hosted X12 EDI claims-management suite for a single billing office. The first deployment target is Colorado Medicaid (one operator, one trading partner). The system is local-only on purpose: the backend binds to `127.0.0.1` and has no authentication because the threat model is a stolen or imaged drive, not a remote attacker.
Cyclone processes three X12 transaction pairs end-to-end:
| Inbound | Outbound | Use |
|---|---|---|
| **837P** (005010X222A1) | 837P (resubmit) | Professional claim submission, acknowledgment tracking |
| **835** (005010X221A1) | — | ERA remittance ingestion, auto-reconciliation |
| **999 / TA1 / 270 / 271 / 277CA** | 999 (after submit) | Acknowledgments, eligibility, claim status |
The authoritative store is a single SQLite file at `~/.local/share/cyclone/cyclone.db` (or, with `sqlcipher3` installed and a Keychain entry, a SQLCipher-encrypted file at the same path). Every claim, remittance, audit event, and reconciliation decision is in that one file. The 6-year HIPAA retention expectation is met by automated daily encrypted backups (SP17) and a tamper-evident hash-chained audit log (SP11).
The whole stack is a single Python process (FastAPI + uvicorn on port 8000) plus a single Node process in dev (Vite on port 5173). No message broker, no separate worker, no database server. The v1 deployment model tolerates this because the operator is one person, the host is one machine, and the failure mode is "open the Activity log and check what broke."
---
## 2. Process & network topology
```
+---------------------+ +-----------------------+
| Browser | HTTP | Vite dev server |
| (localhost:5173) | <----> | (Node 20+) |
| | Vite | |
| React 18 + TanStack | proxy | src/ |
| Query, Zustand, | /api | - pages, components, |
| Radix UI, Tailwind | -----> | hooks, lib |
+---------------------+ +-----------------------+
|
| HTTP (localhost)
v
+----------------------------------------------------------+
| Python process: `python -m cyclone serve` |
| |
| FastAPI app (uvicorn, 127.0.0.1:8000) |
| +-----------------------------------------------------+ |
| | cyclone.api:app (3,548 LOC) | |
| | - routers: claims, remits, providers, acks, | |
| | activity, inbox, upload, download, batches, | |
| | admin, scheduler, health, ta1_acks | |
| | - live-tail: NDJSON pubsub (pubsub.py) | |
| | - lifespan: init db -> seed SP9 -> start SP16 + | |
| | SP17 schedulers -> load PayerConfig cache | |
| +-----------------------------------------------------+ |
| | cyclone.store (facade, 2,423 LOC) | |
| | - add/get/list/manual_match/iter_claims/... | |
| | - delegates to: db.SessionLocal() | |
| +-----------------------------------------------------+ |
| | cyclone.parsers.* (X12 837P/835/999/TA1/270/271/ | |
| | 277CA: tokenize -> segmentize -> model -> validate | |
| | -> write to store) | |
| +-----------------------------------------------------+ |
| | cyclone.reconcile, scoring, batch_diff, audit_log, | |
| | backup_service, scheduler, db_crypto, secrets | |
| +-----------------------------------------------------+ |
+----------------------------------------------------------+
| | |
v v v
+----------------+ +-------------------+ +----------------+
| SQLite / | | macOS Keychain | | Trading |
| SQLCipher | | (keyring) | | partner SFTP |
| cyclone.db | | - SQLCipher key | | (paramiko, SP13)|
| (12 migrations)| | - SFTP password | | |
| | | - backup pass | | |
+----------------+ +-------------------+ +----------------+
```
**One process, one file, three side-channels.** The Keychain (secrets) and SFTP (outbound to the payer) are side-channels; the database is the central state.
The `cyclone-pipeline/` sibling project (not in this repo) is a separate Python CLI that drives the full 7-phase round-trip: preflight → browser upload → parse verification → SFTP submit → TA1 wait → 999 wait → scan + report. It is not part of the in-process architecture; it sits alongside the backend and uses its API endpoints.
---
## 3. Tech stack
### 3.1 Backend
| Layer | Technology | Version | Notes |
|---|---|---|---|
| Language | Python | 3.11+ (3.13 in dev) | `requires-python = ">=3.11"` |
| Web framework | FastAPI | ≥0.110, <1 | `python -m cyclone serve` boots uvicorn |
| ASGI server | uvicorn[standard] | ≥0.27, <1 | `--host 127.0.0.1 --port 8000` |
| ORM | SQLAlchemy | ≥2.0, <3 | 2.x style; engine + SessionLocal() function-accessor |
| Validation | Pydantic | ≥2.6, <3 | v2 model_validator + ConfigDict |
| CLI | Click | ≥8.1, <9 | `cyclone` console script |
| YAML | PyYAML | ≥6.0, <7 | PayerConfig (SP9) |
| Secrets | keyring | ≥25.0, <26 | macOS Keychain on darwin |
| Crypto | cryptography | ≥49.0, <50 | AES-256-GCM for SP17 backups (hard dep) |
| SQLCipher (opt) | sqlcipher3 | ≥0.6, <1 | `pip install -e .[sqlcipher]` |
| SFTP (opt) | paramiko | ≥3.4, <6 | `pip install -e .[sftp]` |
| Tests | pytest + pytest-cov + pytest-asyncio + pytest-randomly | latest | 964 tests, 2026-06-23 |
| Test client | httpx | ≥0.27, <1 | dev only |
### 3.2 Frontend
| Layer | Technology | Version | Notes |
|---|---|---|---|
| Build | Vite | latest | `npm run dev` (port 5173) |
| Language | TypeScript | latest | `tsc -b --noEmit` is the typecheck script |
| UI | React | 18.3.1 | function components + hooks |
| Routing | react-router-dom | 6.27.0 | SPA, 11 top-level routes |
| Data | @tanstack/react-query | 5.101+ | server-state cache |
| Client state | Zustand | 4.5.5 | small, ephemeral state (live tail, drill stack) |
| UI primitives | @radix-ui/* | latest | dialog, label, select, slot, tabs |
| Styling | Tailwind CSS | latest | utility-first |
| Icons | lucide-react | 0.453+ | |
| Toasts | sonner | 1.5+ | |
| Test runtime | vitest | 4.1.9 | Node 18+ has Response/fetch natively |
| Test DOM | happy-dom | 20.10+ | not jsdom (NFR-2) |
| Component tests | @testing-library/react | 16.3+ | 73 test files |
### 3.3 External / operational
- **macOS Keychain** — secret store for SQLCipher key, SFTP password, backup passphrase. No secrets on disk in plaintext (NFR-4).
- **X12 005010 implementation guides** — the parsers follow the CMS / X12 published guides. The 837P parser is the most heavily tested; 835 is the second; 999/TA1/270/271/277CA are lighter.
- **One trading partner** — Colorado Medicaid via the Gainwell SFTP endpoint. Other payers are configured via `cyclone/providers.py` `PayerConfig` rows (SP9).
---
## 4. Backend architecture
### 4.1 Package layout
The `cyclone` package is a single namespace rooted at `backend/src/cyclone/`. 22 top-level modules + 4 subpackages. The store (SP21 in-flight split) is the largest; everything else is small and focused.
```
backend/src/cyclone/
├── __main__.py — entry point; `python -m cyclone` → CLI, `python -m cyclone serve` → uvicorn
├── __init__.py — `__version__`, package init
├── api.py — FastAPI app, route includes, lifespan (3,548 LOC — the only large file)
├── api_helpers.py — request/response formatters, error → HTTP mapping
├── api_routers/
│ ├── __init__.py
│ ├── acks.py — 999/TA1/271/277CA acknowledgments
│ ├── admin.py — admin-only endpoints (validate-provider, scheduler, backup)
│ ├── health.py — GET /api/health (SP19 deep snapshot)
│ └── ta1_acks.py — TA1-specific (split from acks for the SP3 lifecycle)
├── audit_log.py — SHA-256 hash-chained append-only log (SP11)
├── backup.py — SP17 primitives: derive_key, encrypt, decrypt, BackupFile
├── backup_scheduler.py — SP17 BackupScheduler wrapper
├── backup_service.py — SP17 coordinator: create/list/restore/verify/prune
├── batch_diff.py — 837 ↔ 835 diff (SP4)
├── clearhouse/
│ └── __init__.py — Clearhouse, SftpClient (SP9, SP13)
├── cli.py — Click CLI: `cyclone serve`, `cyclone validate-npi`, `cyclone backup`, etc.
├── db.py — SQLAlchemy engine, SessionLocal, ORM models, init_db, reinit_engine
├── db_crypto.py — SP12 SQLCipher connect-creator + is_encryption_enabled()
├── db_migrate.py — runs the 12 SQL migrations in order
├── edi/
│ └── filenames.py — X12 filename convention helpers (SP9)
├── inbox_lanes.py — 5-lane Inbox computation (SP14)
├── inbox_state.py — claim ↔ inbox state mapping helpers
├── inbox_state_277ca.py — SP10 277CA inbox state integration
├── logging_config.py — SP18 structured JSON + PII scrubber
├── migrations/ — 12 SQL files (0001_initial through 0012_backups)
├── npi.py — SP20 NPI Luhn + Tax ID validation
├── parsers/ — X12 parser subpackage
│ ├── parse_837.py — 837P ingest
│ ├── parse_835.py — 835 ingest
│ ├── parse_999.py / parse_ta1.py
│ ├── parse_270.py / parse_271.py
│ ├── parse_277ca.py — SP10 277CA ingest
│ ├── models*.py — one Pydantic model module per transaction type
│ ├── segments.py — X12 segment tokenizer
│ ├── validator.py — 837P rule registry (R-codes)
│ ├── validator_835.py — 835 validator
│ ├── serialize_837.py — SP8 outbound 837P serializer
│ ├── serialize_270.py / serialize_999.py
│ ├── cas_codes.py — CAS reason-code table
│ ├── batch_ack_builder.py
│ ├── writer.py / writer_835.py
│ └── exceptions.py
├── payers.py — Payer ORM row accessor (SP9)
├── providers.py — Payer / Clearhouse / Provider ORM row DTOs (SP9)
├── pubsub.py — NDJSON live-tail EventBus
├── reconcile.py — 835 → 837 auto-reconciliation engine
├── scheduler.py — SP16 MFT polling scheduler
├── scoring.py — 4-field lane scoring (40/25/20/15, hard-coded — backlog item)
├── secrets.py — Keychain access wrapper (SP9)
├── security.py — security headers, HealthSnapshot, deep health probe (SP19)
└── store.py — CycloneStore facade (2,423 LOC; SP21 in-flight split)
```
### 4.2 Module responsibility map (one line each)
| Module | Responsibility |
|---|---|
| `__main__` | Dispatch `serve` vs CLI; honor `CYCLONE_PORT` / `CYCLONE_RELOAD` |
| `api` | FastAPI app, lifespan, route includes, 999/270/271 endpoints, exception → HTTP |
| `api_helpers` | Error → status code mapping, common response shapes |
| `audit_log` | Append + verify a SHA-256 hash-chained log (SP11) |
| `backup` | Pure crypto + file I/O for SP17 backup format |
| `backup_scheduler` | Wrap `BackupService` in a `tick()` loop (SP17) |
| `backup_service` | Create / list / restore / verify / prune (SP17) |
| `batch_diff` | Pairwise diff of two 837 batches (SP4) |
| `clearhouse` | `Clearhouse` + `SftpClient` (SP9 stub, SP13 paramiko) |
| `cli` | `cyclone` console script: serve, validate-npi, backup, list-batches, … |
| `db` | Engine factory, `SessionLocal()`, all ORM rows, `init_db()`, `reinit_engine()` |
| `db_crypto` | SQLCipher branch + `is_encryption_enabled()` (SP12) |
| `db_migrate` | Walk `migrations/*.sql` in order, apply pending |
| `edi.filenames` | Parse / construct X12 file names (SP9) |
| `inbox_lanes` | Compute the 5 lanes (rejected, payer_rejected, candidates, unmatched, done_today) |
| `inbox_state` | Pure helpers for state ↔ lane mapping |
| `inbox_state_277ca` | SP10 277CA state integration (monotonic rule) |
| `logging_config` | SP18 JSON formatter + PII scrubber |
| `npi` | SP20 pure validators (Luhn over 80840+body, EIN prefix check) |
| `parsers.*` | X12 transaction set parsers (one per type) |
| `payers` / `providers` | SP9 ORM row DTOs |
| `pubsub` | In-process NDJSON EventBus for live tail |
| `reconcile` | 835 → 837 auto-match + per-line CAS reconciliation |
| `scheduler` | SP16 MFT polling scheduler (per-payer tick) |
| `scoring` | 4-field lane scoring weights (40/25/20/15) |
| `secrets` | Keychain get/set/delete via `keyring` |
| `security` | Security headers, deep `HealthSnapshot` (SP19) |
| `store` | `CycloneStore` facade: `add` / `get` / `list` / `manual_match` / `iter_*` / `recent_activity` |
### 4.3 The `CycloneStore` facade
The store is the single read/write surface for the database. Every endpoint that mutates state goes through `cyclone.store` (eventually the `cyclone.store.*` subpackage after SP21 lands). The facade pattern preserves public API:
- `store.add(record)` — add a parsed batch
- `store.get(batch_id)`, `get_batch(batch_id)`, `list(limit)`, `all()`
- `store.manual_match(claim_id, remit_id)` — raises `AlreadyMatchedError` (→ 409)
- `store.manual_unmatch(claim_id)` — raises `NotMatchedError` (→ 409)
- `store.iter_claims(...)`, `iter_remittances(...)`
- `store.distinct_providers()`, `recent_activity(limit)`
- `store.list_unmatched(kind="both")`
- `store.export_*()` — CSV/JSON dumpers
All persistence flows through SQLAlchemy sessions via `db.SessionLocal()()`. The engine is single-connection-per-request via FastAPI dependency injection. SP21 splits `store.py` (2,423 LOC) into a `cyclone/store/` subpackage with one module per domain (batches, inbox, claim_detail, acks, backups, providers, etc.) and a thin facade that delegates every method to its domain module. After SP21, the public API of `cyclone.store` is unchanged; the only thing that changes is the file layout.
### 4.4 The parser pipeline
All inbound files follow the same 5-stage pipeline:
```
X12 .txt on disk
↓
[1] tokenize (parsers/segments.py: split on ~, then ^, then *:)
↓
[2] segmentize (build Segment records with elements + repeats)
↓
[3] model (parsers/models_*.py: Pydantic models per transaction type)
↓
[4] validate (parsers/validator.py: R-codes; raise ParseError on fatal)
↓
[5] write (store.add(record); record → DB rows)
```
The 837P validator has the most rules (R020, R021, R034, R035, R200, R210 — see [837P parser spec](superpowers/specs/2026-06-19-cyclone-837p-parser-design.md) §2.4 for the registry). The 835 validator is simpler (no R-codes, just a model completeness check + a status_code allowlist). 999/TA1/270/271/277CA have no validators — they trust the upstream.
The writer is responsible for: (a) creating the `batches` row, (b) creating N `claims` / `remittances` / `cas_adjustments` rows, (c) emitting an `activity_events` row, (d) for 837P, the synthetic claim-status from the 999 envelope, (e) for 835, triggering `reconcile.apply_payment` to auto-match. If any step raises, the entire batch is rolled back (single SQLAlchemy transaction).
### 4.5 The reconciliation engine
`cyclone.reconcile` is a pure-functions module. The store calls into it for every 835 ingest:
- `apply_payment(claim, remit, strategy) -> ApplyIntent` — happy path; returns the new `Match` row + state transition
- `apply_reversal(...)` — handles a reversal 835 (the original payment + the reversal both stay in the DB; the `is_reversal` flag on the `Match` row disambiguates)
- `recompute_line_totals(claim)` — for SP7's per-line audit, when an 835 carries CAS adjustments
The engine is pure (no DB access). The store orchestrates the SQL writes after a successful `apply_*` returns. The 4-field scoring (40/25/20/15 for charge amount / NPI / patient ID / service date) lives in `cyclone.scoring` and is invoked by `inbox_lanes.compute_lanes` for the `candidates` and `unmatched` lanes.
### 4.6 The audit chain
`cyclone.audit_log` is a separate, append-only log table (`audit_log` — distinct from `activity_events` which is the un-chained `/activity` feed). Every event type the system can emit has a key in the chain:
- `claim.submitted`, `claim.rejected`, `claim.matched`, `claim.unmatched`
- `claim.payer_rejected`, `claim.payer_rejected_acknowledged`
- `clearhouse.submitted`
- `key.rotated` (SP15), `backup.created`, `backup.restored`, `backup.verified` (SP17)
- `api.request_rejected` (SP19)
Each row carries the SHA-256 of `(prev_hash || kind || actor || payload_json || ts)`. The first row has `prev_hash = 0x00 * 32`. `audit_log.verify_chain()` walks the table top-to-bottom, recomputing each hash, and raises `AuditChainError` on the first mismatch with the row id and the expected vs. actual hash. SP19's deep health probe calls `verify_chain()` on every `/api/health` request so a broken chain surfaces in the operator's status badge within one tick.
---
## 5. Frontend architecture
### 5.1 Tech stack
See §3.2. The frontend is a Vite SPA with TypeScript strict mode, Tailwind utility classes, and Radix primitives. There is no SSR, no Next.js, no router data-loading — every page is a function component that calls `useQuery` and renders.
### 5.2 Route map
| Path | Page | Purpose |
|---|---|---|
| `/` | `Dashboard` | KPI tiles + ticker tape + recent activity |
| `/claims` | `Claims` | Paginated claim list, filters, drill to claim drawer |
| `/remittances` | `Remittances` | Paginated remit list, drill to remit drawer |
| `/providers` | `Providers` | Provider rollup (NPIs, charge totals) |
| `/activity` | `ActivityLog` | Append-only `activity_events` feed (NDJSON) |
| `/upload` | `Upload` | Drag-drop or click to upload an 837P / 835 / 999 / 277CA |
| `/reconciliation` | `ReconciliationPage` | Cross-claim reconciliation view (4-field score) |
| `/inbox` | `Inbox` | 5-lane inbox (SP14) with BulkBar |
| `/acks` | `Acks` | 999/TA1/270/271/277CA acknowledgment list |
| `/batches` | `Batches` | Batch list with totals + validation summary |
| `/batch-diff` | `BatchDiff` | Pairwise 837 batch diff |
| `*` | `NotFound` | 404 |
### 5.3 State management
Two layers:
- **Server state** — `@tanstack/react-query`. Every page uses one or more `useQuery` calls. The query key shape is `["domain", id?, filters?]`. Mutations use `useMutation` and invalidate the relevant keys. There is no global refetch strategy; each page controls its own stale time.
- **Client state** — `zustand`. Two stores:
- `tail-store.ts` — the live-tail connection state (reconnecting, last event id, subscriber counts). Powers the bottom-right TailStatusPill.
- `DrillStackProvider` (in `src/components/drill/`) — the SP22 universal-drilldown stack. A LIFO of drill contexts (claim → claim line → match → CAS adjustment) so the back button works across drawers.
There is no Redux, no Context for global state, no MobX. The combination of react-query for server state + zustand for client state is intentional — see [universal-drilldown spec](superpowers/specs/2026-06-21-cyclone-universal-drilldown-design.md) for the rationale.
### 5.4 Live tail (NDJSON pubsub)
The Activity feed and the TailStatusPill share a single connection: `GET /api/tail?since=<last_event_id>`. The server returns `Content-Type: application/x-ndjson` and streams one JSON object per line. Each line is one of:
- `{kind: "heartbeat", ts: "..."}` — every 5s when no events
- `{kind: "event", id: <int>, event_type: "...", payload: {...}}` — when something happens
- `{kind: "error", message: "..."}` — server-initiated close (rare)
The client uses `fetch` + `ReadableStream` to consume the body and dispatches each line to the relevant `useQuery` cache via `queryClient.setQueryData`. The connection auto-reconnects on `error` with exponential backoff. See [live-tail spec](superpowers/specs/2026-06-20-cyclone-live-tail-design.md) for the wire format and reconnection policy.
---
## 6. Data model
### 6.1 Migrations 0001–0012
The schema evolves in 12 ordered SQL files under `backend/src/cyclone/migrations/`. Each file starts with `-- version: N` and is applied in order by `db_migrate.py`. There is no migration framework — `db_migrate.py` is a 30-line file that checks the `schema_version` row and applies everything newer.
| # | File | What it adds |
|---|---|---|
| 1 | `0001_initial.sql` | `batches`, `claims`, `remittances`, `cas_adjustments`, `matches`, `activity_events` (6 tables) |
| 2 | `0002_acks.sql` | `acks` table (999/271) |
| 3 | `0003_drop_claims_remits_unique_constraints.sql` | Spec-bug fix: drop the UNIQUE constraints that prevented reversal flows |
| 4 | `0004_rejections_and_state_history.sql` | `rejections` table + `claim_state_history` |
| 5 | `0005_create_ta1_acks.sql` | `ta1_acks` table |
| 6 | `0006_line_reconciliation.sql` | `line_reconciliations` (SP7) |
| 7 | `0007_providers_payers_clearhouse.sql` | `providers`, `payers`, `clearhouses` (SP9) |
| 8 | `0008_payer_rejected_columns.sql` | `claim.payer_rejected_*` columns (SP10) |
| 9 | `0009_audit_log.sql` | `audit_log` hash-chained table (SP11) |
| 10 | `0010_payer_rejected_acknowledged.sql` | `claim.payer_rejected_acknowledged_*` columns (SP14 hand-off) |
| 11 | `0011_processed_inbound_files.sql` | `processed_inbound_files` idempotency table (SP16) |
| 12 | `0012_backups.sql` | `db_backups` table (SP17) |
A 13th (`0013_claims_unique_constraint.sql`) and 14th (`0014_409_ux.sql`) are in-flight on the `claims-unique-fix` worktree but not yet on `main`.
### 6.2 ERD (high level)
```
batches (1) ─┬─< (N) claims ─┬─< (N) matches >─ (N) ─┬─ (1) remittances ─< (N) cas_adjustments
│ │ │
│ └─< (N) line_reconciliations (SP7)
│ └─< (N) rejections / claim_state_history
│
├─< (N) acks (999/271)
├─< (N) ta1_acks
└─< (N) activity_events
claims (1) ─< (N) line_reconciliations ─> (1) cas_adjustments
audit_log (no FKs — append-only, hash-chained)
db_backups (no FKs — directory + filename, status row)
processed_inbound_files (no FKs — path + hash, idempotency key)
providers (1) ─< (N) claims
payers (1) ─< (N) claims
clearhouses (1) ─< (N) payers
```
The `batches` row is the parent of every parsed X12 file. A batch is the unit of atomicity for parsing: either every claim/remit/ack in the file lands, or the file rolls back. The `claims.state` column drives the lifecycle (§6.3); the `matches` table is the join that says "this 835 paid this 837" (with a `strategy` field for audit: auto vs. manual).
### 6.3 Claim lifecycle (7 states)
The `claims.state` column is the canonical state of a claim in the operator's workflow:
```
submitted ──> accepted ──> paid
│ │ │
│ │ └─> reversed
│ │
│ └─> denied ──> appealed ──> paid
│
└─> rejected (envelope 999)
│
└─> payer_rejected (277CA A4/A6/A7)
```
- `submitted` — 837P ingested, awaiting 999 ACK
- `accepted` — 999 ACK came back clean; sitting in the trading partner's queue
- `paid` — 835 matched, payment posted
- `reversed` — 835 reversal applied (the original payment + the reversal are both preserved; the `state` reflects the most recent)
- `denied` — 835 came back with a denial status code (CAS group CO)
- `appealed` — operator flagged for appeal workflow
- `rejected` — envelope-level 999 rejection (syntactic / structural)
- `payer_rejected` — semantic 277CA rejection (A4/A6/A7 — claim accepted by envelope but rejected at the line level). SP10 introduced this state and the `claim.payer_rejected_*` columns.
State transitions are explicit functions in `cyclone.store` (e.g. `mark_accepted`, `mark_paid`, `mark_rejected`). Each transition emits an `activity_events` row and, for the security-relevant ones, an `audit_log` row.
### 6.4 Audit chain integrity
The `audit_log` table is a separate chain from `activity_events`:
- `activity_events` — un-chained, powers the `/activity` feed. The activity feed can be repopulated from the underlying state if rows are missing; integrity is informational.
- `audit_log` — SHA-256 hash-chained, powers compliance / forensics. A row cannot be deleted without breaking the chain. `audit_log.verify_chain()` is the integrity check; the SP19 deep health probe runs it on every `/api/health` call.
The two are intentionally separate because they have different retention and integrity requirements. The convention-vs-trigger tradeoff is documented in the [SP11 spec](superpowers/specs/2026-06-23-cyclone-sp11-hash-chained-audit-design.md) §7.
---
## 7. Data flow
### 7.1 File ingest (manual upload)
```
Operator ──drag-drop──> /upload (React)
│
│ POST /api/upload (multipart, .txt file)
v
FastAPI route
│
│ save to /tmp/cyclone-uploads/<uuid>.txt
v
parser.dispatch(kind) (auto-detect 837/835/999/TA1/270/271/277CA by ISA segment)
│
│ (the 5-stage pipeline from §4.4)
v
CycloneStore.add(record) (single SQLAlchemy transaction)
│
├─> emit activity_events row
├─> for 835: trigger reconcile.apply_payment
├─> for 999/TA1/277CA: mark the source claim
└─> commit
│
│ emit NDJSON event on the pubsub bus
v
live-tail subscribers (Activity feed, TailStatusPill)
```
The HTTP response is the new `batch_id` + a counts summary (e.g. `{"batch_id": "...", "claims": 12, "remittances": 0, "validation_issues": 0}`). The 999 is the only case where the response carries an ack payload — the operator can read the parsed 999 directly without a second request.
### 7.2 File ingest (SFTP polling — SP16)
```
every N seconds (default 30s, per-payer config):
Scheduler.tick()
│
├─> for each clearhouse:
│ ├─> SftpClient.list_inbound() (paramiko, SP13)
│ ├─> for each new .x12 file:
│ │ ├─> check processed_inbound_files (idempotency)
│ │ ├─> SftpClient.read_file(path) (to /tmp/...)
│ │ ├─> parser.dispatch(...)
│ │ ├─> CycloneStore.add(record)
│ │ ├─> INSERT processed_inbound_files (path, sha256, ts)
│ │ └─> emit activity_events row
│ └─> log tick result (count, errors, ms)
└─> heartbeat to /api/health (SP19 reports last tick ts)
```
The idempotency table is the linchpin: a crash between "read file" and "INSERT processed" results in the same file being processed again on the next tick. The 5-step ordering (SELECT → INSERT side-effects → INSERT idempotency; IntegrityError as the correctness backstop) is detailed in the [SP16 spec](superpowers/specs/2026-06-23-cyclone-sp16-mft-polling-design.md) §3.4.
### 7.3 File ingest (manual upload, replayed by pipeline)
`cyclone-pipeline/` (sibling project) does the same as the operator dragging-and-dropping — it `POST`s to `/api/upload` with the file. There's no separate "pipeline" ingestion path. The pipeline agent is a thin wrapper that adds preflight + SFTP submit + wait-for-999 around the Cyclone API.
### 7.4 Auto-match on 835 ingest
When an 835 lands, the writer calls `reconcile.apply_payment(claim, remit, strategy)` for each `(claim, remit)` pair in the file. The strategy is one of:
- `auto:pcn_npi_charge` — patient_control_number + provider_npi + charge_amount match (the strongest)
- `auto:pcn_charge` — patient_control_number + charge_amount (the most common in practice)
- `manual` — the operator uses `/reconciliation` to pair by hand; the strategy is recorded for audit
Each auto-match creates a `matches` row + transitions the claim state + emits an `activity_events` row + emits an `audit_log` row. The match rate is visible in the Dashboard KPI tile.
### 7.5 Outbound 837P (resubmit)
Two paths:
- **Single claim** — `GET /api/claims/{id}/serialize-837` (SP8). The serializer round-trips the claim through `serialize_837.py`, validates with the R-code registry, and returns the X12 text. The Claim drawer's "Download 837" button calls this and triggers a browser download.
- **Bundle ZIP** — `POST /api/claims/resubmit?download=zip` (SP8). The serializer rebuilds N 837P files (one per claim in the request), zips them with a manifest, and returns the ZIP. The Inbox BulkBar's "Resubmit selected" calls this.
Both paths are read-only against the DB (they don't mutate the claim state). The operator is expected to download the regenerated file, then submit it manually via SFTP (or via `cyclone-pipeline run`).
### 7.6 Acknowledgement ingestion (999, TA1, 277CA)
These are the three "ack" transaction types the operator's trading partner sends back after a submission:
- **999** — functional ack (envelope-level). Triggers `claim.state = accepted | rejected`.
- **TA1** — interchange ack. Triggers an activity_events row; no state change.
- **277CA** — claim status (line-level). Triggers `claim.payer_rejected` if any line is A4/A6/A7. The `claim.payer_rejected_*` columns carry the status code + the 277CA's batch id for the audit trail.
All three are ingested via the same `/api/upload` path; the parser auto-detects the transaction type from the ISA segment. The SP10 spec describes the monotonic rule: a later 277CA cannot clear a previous rejection. This is the regulatory guarantee the SP11 audit chain relies on.
---
## 8. API surface
### 8.1 REST endpoints (by domain)
The full list is in [REQUIREMENTS.md §6.3 traceability matrix](REQUIREMENTS.md). The structure is:
```
/api/health (SP19 deep health probe)
/api/tail?since=<id> (NDJSON live tail)
/api/upload (multipart, any X12 type)
/api/parse-837 (parse + return Pydantic, do not persist — pipeline agent)
/api/batches (list, with totals + validation)
/api/batches/{id} (detail)
/api/batches/{id}/diff?against={other_id} (batch_diff)
/api/batches/{id}/serialize-837 (SP8 — round-trip)
/api/claims (list, paginated, filterable)
/api/claims/{id} (detail, with remits + matches + line_recons)
/api/claims/{id}/serialize-837 (SP8)
/api/claims/resubmit?download=zip (SP8 bulk)
/api/remittances (list)
/api/remittances/{id} (detail)
/api/providers (rollup)
/api/providers/{npi} (single provider, all claims)
/api/activity?since=<id> (un-chained feed, paginated)
/api/inbox (SP14 5-lane view, filterable)
/api/inbox/payer-rejected/acknowledge (SP10 + SP14)
/api/acks (999/271/277CA list)
/api/acks/{id} (detail)
/api/ta1-acks (TA1 list)
/api/admin/validate-provider?npi=&tax_id= (SP20)
/api/admin/db/rotate-key (SP15)
/api/admin/db/fingerprint (SP12)
/api/admin/backup/create (SP17)
/api/admin/backup/list (SP17)
/api/admin/backup/{id}/restore?confirm=true (SP17, two-step)
/api/admin/backup/{id}/verify (SP17)
/api/admin/scheduler/status (SP16)
/api/admin/scheduler/tick (SP16 — manual trigger)
/api/admin/scheduler/start (SP16)
/api/admin/scheduler/stop (SP16)
```
### 8.2 Live-tail NDJSON format
`GET /api/tail?since=<last_event_id>` returns `Content-Type: application/x-ndjson` and streams:
```
{"kind":"event","id":1234,"event_type":"claim.submitted","payload":{"claim_id":"...","batch_id":"...","ts":"2026-06-23T..."}}
{"kind":"event","id":1235,"event_type":"claim.matched","payload":{"claim_id":"...","remit_id":"...","strategy":"auto:pcn_npi_charge"}}
{"kind":"heartbeat","ts":"2026-06-23T..."}
{"kind":"event","id":1236,"event_type":"claim.payer_rejected","payload":{"claim_id":"...","status_code":"A7","source_277ca_id":"..."}}
```
Heartbeats every 5s when no events. The client reconnects on `error` with exponential backoff capped at 30s. See [live-tail spec](superpowers/specs/2026-06-20-cyclone-live-tail-design.md) for the full wire format.
---
## 9. Cross-cutting concerns
### 9.1 Logging (SP18)
`cyclone.logging_config` configures a single JSON formatter that:
- Emits one JSON object per line to stdout (12-factor)
- Includes `ts`, `level`, `logger`, `msg`, `module`, `line`, `process`, `thread`
- Scrubs PHI/PII fields by key name (`npi`, `member_id`, `patient_name`, `ssn_last4`, `charge_amount`, `paid_amount`, `tax_id`) — replaced with `***`
- Redacts secrets by string match (anything matching `password=...&` or `Bearer ...`)
The PII scrubber is a best-effort defense-in-depth, not a security boundary. Real PHI protection comes from the SQLCipher layer (SP12) and the Keychain (no secrets in env). The scrubber exists so an operator tailing logs to debug doesn't accidentally paste a log line into a chat.
### 9.2 Audit chain (SP11)
See §4.6. The chain is checked on every `/api/health` request and exposes `status="degraded"` if broken. The operator sees the status in the TailStatusPill and the Dashboard KPI tile.
### 9.3 PHI/PII handling
- **Storage** — SQLCipher at rest (opt-in, SP12). Daily encrypted backups (SP17). Tamper-evident audit chain (SP11). 6-year retention via automated backup rotation.
- **In transit (SFTP)** — TLS to the trading partner; no plaintext EDI over the wire.
- **In the UI** — no PHI is shown on the Dashboard or Inbox (only counts + KPIs). The Claim drawer and Remit drawer show full PHI; these pages are not in the route map's preview and require an explicit navigation click.
- **In logs** — scrubbed by key name (see §9.1).
- **In analytics** — no analytics. No third-party tracking. No telemetry. The frontend makes no requests outside `127.0.0.1:8000`.
### 9.4 Error model
Every API error is a JSON object with `{"error": "<code>", "message": "<human>", "detail": <optional>}`. The mapping is in `api_helpers.py`:
| Error class | HTTP status | `error` code |
|---|---|---|
| `AlreadyMatchedError` | 409 | `already_matched` |
| `NotMatchedError` | 409 | `not_matched` |
| `InvalidStateError` | 409 | `invalid_state` |
| `ParseError` | 422 | `parse_error` |
| `ValidationError` (R-codes) | 422 | `validation_failed` (issues in `detail`) |
| `FileNotFoundError` | 404 | `not_found` |
| `ValueError` | 400 | `bad_request` |
| `PermissionError` | 403 | `forbidden` (reserved — no auth today) |
| (anything else) | 500 | `internal` (with a request id for log correlation) |
The 409-vs-422-vs-400 split is intentional: 409 for state-machine violations (the operator's input is well-formed but conflicts with current state), 422 for parse / validation, 400 for bad input shape.
---
## 10. Operational concerns
### 10.1 Startup order
`python -m cyclone serve` → `cyclone.api.lifespan` runs:
1. `db.init_db()` — apply pending migrations (0001-0012, plus 0013/0014 on the `claims-unique-fix` worktree)
2. Seed `cyclone.providers` with the SP9 defaults if the table is empty (PayerConfig for Colorado Medicaid + Gainwell)
3. Configure `BackupService` (SP17) — open backup dir, read passphrase from Keychain
4. Configure `Scheduler` (SP16) — wire MFT polling for each clearhouse
5. If `CYCLONE_SCHEDULER_AUTOSTART=true` (default), start both schedulers
6. Mount the FastAPI app and start uvicorn
The ordering is load-bearing: the store facade cannot be used until `init_db` finishes, and the schedulers cannot query the DB until the seed step populates the clearhouse rows.
### 10.2 MFT scheduler (SP16)
`cyclone.scheduler.Scheduler` runs a per-clearhouse tick on a configurable interval (default 30s, per `CYCLONE_SCHEDULER_INTERVAL` env var). The tick is described in §7.2. The scheduler is opt-in via `CYCLONE_SCHEDULER_AUTOSTART`; the operator can start/stop it from the admin endpoints or via `POST /api/admin/scheduler/tick` for a manual tick.
The 4-step idempotency sequence (SELECT → INSERT side-effects → INSERT idempotency; IntegrityError as the correctness backstop) is the linchpin. The scheduler's `_already_processed` + `_record` pair wraps the INSERT to handle the case where two ticks race (an extremely rare event given the 30s interval, but possible during operator-driven manual ticks).
### 10.3 Backup scheduler (SP17)
`cyclone.backup_scheduler.BackupScheduler` wraps `BackupService` and ticks on its own interval (default 24h, per `CYCLONE_BACKUP_INTERVAL` env var). On each tick: create a backup, prune anything older than `retention_days` (default 30). Auto-start opt-in via `CYCLONE_BACKUP_AUTOSTART`.
The encryption layer is independent of SQLCipher: AES-256-GCM with a passphrase-derived key (PBKDF2-HMAC-SHA256, 200k iterations). The passphrase lives in the macOS Keychain. If the passphrase is missing, the backup layer falls back to deriving a key from the SQLCipher key (less ideal but never silently broken — the operator's `/api/health` reports `status="degraded"` with a `backup.passphrase_missing` flag).
### 10.4 Health probe (SP19)
`GET /api/health` is a deep snapshot:
- **db** — open a session, `SELECT 1`
- **scheduler** — last tick ts, is alive
- **pubsub** — current subscriber counts per event kind
- **batch** — most recent batch id + ts
- **audit_log** — `verify_chain()` result (deep check on every call; 1ms on a healthy chain, fails loudly on a broken one)
- **backup** — last backup ts, last verify result, passphrase presence
- **encryption** — `is_encryption_enabled()` boolean
Returns `status="ok"` only when every subsystem is healthy. `status="degraded"` if any subsystem is unhappy but the API itself is responsive. The status is shown in the TailStatusPill (bottom-right) and the Dashboard KPI tile.
### 10.5 Shutdown
`SIGTERM` (or Ctrl-C) → uvicorn drains in-flight requests → cancels the MFT scheduler (in-flight tick completes via `asyncio.to_thread`) → cancels the backup scheduler (in-flight tick completes) → closes the SQLAlchemy engine → exits 0. There is no checkpoint; the in-flight tick is allowed to complete naturally because the idempotency table makes a partial-tick restart safe.
---
## 11. Deployment
### 11.1 Local dev (default)
The default deployment. Two terminals:
```bash
# Terminal 1 — backend on 127.0.0.1:8000
cd backend
.venv/bin/python -m cyclone serve
# Terminal 2 — frontend on localhost:5173
cd ..
npm run dev
```
The frontend reads its backend URL from `VITE_API_BASE_URL` (default empty). Create `.env.local` at the repo root with `VITE_API_BASE_URL=http://127.0.0.1:8000`. Without it, the UI falls back to its in-memory sample store via the `data` adapter (parses are disabled).
### 11.2 Optional Docker (SP23 — product fork)
A Ubuntu + Docker + auth + RBAC + LAN-bind spec exists at [`docs/superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md`](superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md). This is a **product fork** awaiting user decision (per [REQUIREMENTS.md §6.2](REQUIREMENTS.md)). The v1 single-host single-operator posture assumes local-only; SP23 changes the threat model to "remote operator on LAN."
### 11.3 Single-host "production"
For a single operator on a single machine, the dev deployment IS the production deployment. There is no separate build / deploy pipeline; the v1 distribution model is `git clone && pip install -e '.[dev]' && npm install && python -m cyclone serve`. A release-time gate (not yet implemented — backlog item) would run `pytest` (backend) and `npm test` (frontend) and fail if any test is red.
---
## 12. Out of scope (v1)
The following are explicitly NOT built in v1, by design:
- **Multi-user / multi-host** — no auth, no LAN-bind, no Docker, no reverse proxy. The operator is one person, the host is one machine. (SP23 fork if this changes.)
- **Other trading partners** — only Colorado Medicaid via Gainwell is configured. The `PayerConfig` mechanism (SP9) supports adding more; the spec/plan/CLI work for that is on the backlog.
- **837I / 837D / 834 / 820 / 275 / 278 / 276-277** — only 837P and 835 are first-class. Other transaction types are listed in [REQUIREMENTS.md §6.2](REQUIREMENTS.md) as backlog items.
- **NPPES real-time lookup** — only local NPI checksum validation (SP20). The operator who wants real registry lookup has to wire it in later.
- **EHR / PM integration** — no webhook in/out, no callable `POST /api/parse-837` (the pipeline agent uses `/api/upload` for now).
- **COB / secondary claims** — no automatic coordination-of-benefits generator. The operator generates a secondary 837P by hand from the CAS adjustments.
- **Real-time eligibility** — no SFTP pickup + SOAP + 271 round-trip. The 270/271 parsers exist; the live cycle doesn't.
Each of these is a separate SP. The "v1 done" bar is the 22 shipped SPs (SP1–SP22) on `main` plus the [REQUIREMENTS.md §8 DoD](REQUIREMENTS.md).
---
## 13. References
### 13.1 Top-level docs
- [Requirements](REQUIREMENTS.md) — FRs, NFRs, DoD, traceability
- [README](../../README.md) — install + dev + pipeline agent
### 13.2 Specs (per sub-project)
All in `docs/superpowers/specs/`:
- SP1 production-readiness: `2026-06-19-cyclone-production-readiness-design.md`
- SP2 DB + reconciliation: `2026-06-19-cyclone-db-reconciliation-design.md`
- SP3 EDI features: `2026-06-20-cyclone-edi-features-design.md`
- SP4 claim drawer: `2026-06-20-cyclone-claim-drawer-design.md`
- SP5 live tail: `2026-06-20-cyclone-live-tail-design.md`
- SP7 line reconciliation: `2026-06-20-cyclone-line-reconciliation-design.md`
- SP8 serialize 837: `2026-06-20-cyclone-serialize-837-design.md`
- SP9 multi-payer / NPI / SFTP stub: `2026-06-20-cyclone-multi-payer-npi-sftp-design.md`
- SP10 277CA: `2026-06-23-cyclone-sp10-277ca-payer-rejected-design.md`
- SP11 audit log: `2026-06-23-cyclone-sp11-hash-chained-audit-design.md`
- SP12 SQLCipher: `2026-06-23-cyclone-sp12-sqlcipher-encryption-design.md`
- SP13 SFTP client: `2026-06-23-cyclone-sp13-sftp-client-design.md`
- SP14 5-lane Inbox: `2026-06-23-cyclone-sp14-inbox-5lane-design.md`
- SP15 key rotation: `2026-06-23-cyclone-sp15-key-rotation-design.md`
- SP16 MFT scheduler: `2026-06-23-cyclone-sp16-mft-polling-design.md`
- SP17 encrypted backup: `2026-06-21-cyclone-encrypted-backup-design.md`
- SP18 structured logging: `2026-06-21-cyclone-structured-logging-design.md`
- SP19 security hardening: `2026-06-21-cyclone-security-hardening-design.md`
- SP20 NPI validation: `2026-06-21-cyclone-npi-validation-design.md`
- SP21 store split: `2026-06-21-cyclone-store-split-design.md`; universal drilldown: `2026-06-21-cyclone-universal-drilldown-design.md`
- SP22 parse-decide: `2026-06-21-cyclone-parse-decide-workflow-design.md`; pipeline agent: `2026-06-21-cyclone-pipeline-agent-design.md`
- SP23 (fork) Ubuntu + Docker: `2026-06-22-cyclone-ubuntu-docker-deployment-design.md`
### 13.3 Plans
All in `docs/superpowers/plans/`. 27 files — one per spec (plus the round-2 backfill for SP9–SP20 in the `2026-06-23-cyclone-sp*` naming).
### 13.4 Reference docs
- `docs/reference/co-medicaid.md` — operator setup for SQLCipher + SFTP + the Gainwell trading-partner details (KP-specific).
### 13.5 Sibling project
- `cyclone-pipeline/` — round-trip agent (sibling, not in this repo). See [README](../../README.md) §"Pipeline automation agent" for install + usage.
docs/REQUIREMENTS.md
+1 -1
View File
@@ -4,7 +4,7 @@
**Status:** Round 2 (post-review corrections — ready for verification)
**Audience:** New engineer or AI agent who needs to understand Cyclone's contract, the work that's already shipped, the work that's not, and what "done" means before they write a single line of code.
**Scope:** Local-only, single-operator, single-host EDI claims-management suite for one billing office, currently trading with Colorado Medicaid (CO XIX).
**Sister docs:** [`README.md`](../README.md) is the operator-facing entry point; [`docs/superpowers/specs/`](superpowers/specs/) holds per-SP design specs; [`docs/superpowers/plans/`](superpowers/plans/) holds per-SP implementation plans; [`docs/reviews/2026-06-20-cyclone-completeness-review.md`](reviews/2026-06-20-cyclone-completeness-review.md) is the industry-scope gap analysis; [`docs/reference/`](reference/) holds condensed X12 / payer notes.
**Sister docs:** [`README.md`](../README.md) is the operator-facing entry point; [`docs/ARCHITECTURE.md`](ARCHITECTURE.md) is the top-level technical design (process topology, package layout, data flow, lifecycle); [`docs/superpowers/specs/`](superpowers/specs/) holds per-SP design specs; [`docs/superpowers/plans/`](superpowers/plans/) holds per-SP implementation plans; [`docs/reviews/2026-06-20-cyclone-completeness-review.md`](reviews/2026-06-20-cyclone-completeness-review.md) is the industry-scope gap analysis; [`docs/reference/`](reference/) holds condensed X12 / payer notes.
---