diff --git a/docs/superpowers/plans/2026-06-29-cyclone-store-split-resume.md b/docs/superpowers/plans/2026-06-29-cyclone-store-split-resume.md new file mode 100644 index 0000000..86cbfd4 --- /dev/null +++ b/docs/superpowers/plans/2026-06-29-cyclone-store-split-resume.md @@ -0,0 +1,1061 @@ +# CycloneStore split (Step 4) — Implementation Plan (Resumed) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Behaviour-preserving structural split of `backend/src/cyclone/store.py` (2,995 lines) into a `cyclone/store/` subpackage with **14 modules** and a thin facade. Zero public API changes, zero test changes, single atomic commit at the end. + +**Architecture:** Module functions for each domain (`batches.add_record`, `inbox.manual_match`, `kpis.dashboard_kpis`, etc.); `CycloneStore` class in `__init__.py` keeps its current method signatures as 1-line delegations. The facade re-exports every currently-importable name (public symbols + 3 private helpers used by tests) so no caller changes. + +**Tech Stack:** Python 3.11+, SQLAlchemy 2.x, Pydantic v2, pytest. Existing test infrastructure (no new tests). + +**Spec:** [`docs/superpowers/specs/2026-06-21-cyclone-store-split-design.md`](../specs/2026-06-21-cyclone-store-split-design.md) (resumed 2026-06-29). + +**Supersedes:** [`docs/superpowers/plans/2026-06-21-cyclone-store-split.md`](2026-06-21-cyclone-store-split.md) — preserved as historical record. The original plan targeted 13 modules / 2,412 LOC / line-number-anchored code copies; the resumed store.py is 583 LOC larger, has 3 new top-level symbols, and uses function-name references rather than line numbers (which drift with every edit). + +--- + +## Delta vs. 2026-06-21 plan + +| | 2026-06-21 | 2026-06-29 (this plan) | +|---|---|---| +| `store.py` size | 2,412 LOC | 2,995 LOC | +| Target modules | 13 | **14** (new `kpis.py`) | +| Test baseline | ~712 / ~31 / ~16 | **1,176 / 1 (isolation flake) / 10** | +| Per-task code copies | yes (every function pasted) | no (function-name references; read store.py for the source) | +| New symbols to extract | — | `dashboard_kpis` + `_claim_state_str` → `kpis.py`; `check_matched_pair_drift` → `claim_detail.py` | +| Private-helper re-exports | `_claim_status_from_validation` | + `_persist_835_remit` + `_remittance_835_row` | + +**How to use this plan alongside the 2026-06-21 plan:** Treat the 2026-06-21 plan as the historical record of how the pattern was designed. This plan is what to execute today. Tasks 0-13 below correspond to Tasks 0-12 in the original with: (a) `check_matched_pair_drift` folded into Task 8 (claim_detail), (b) a new Task 9 (kpis), (c) original Tasks 9-12 renumbered to 10-13. + +--- + +## File structure (target) + +``` +backend/src/cyclone/ +├── store.py ← DELETED in Task 13 +└── store/ + ├── __init__.py ← Task 1 — facade: re-exports + CycloneStore + store + utcnow + ├── exceptions.py ← Task 2 + ├── records.py ← Task 3 + ├── orm_builders.py ← Task 4 + ├── ui.py ← Task 5 + ├── write.py ← Task 6 + ├── batches.py ← Task 7 + ├── claim_detail.py ← Task 8 (now also hosts check_matched_pair_drift) + ├── kpis.py ← Task 9 — NEW: dashboard_kpis + _claim_state_str + ├── acks.py ← Task 10 + ├── backups.py ← Task 11 + ├── inbox.py ← Task 12 + └── providers.py ← Task 12 +``` + +Each module has one clear responsibility. Files that change together live together. + +--- + +## Task 0: Pre-flight — capture baseline + audit private-helper leaks + +**Goal:** Snapshot the test suite baseline. Find every private-helper import leak (imports of `_`-prefixed names from `cyclone.store`) so the facade re-exports cover all callers. + +**Files:** +- Read: `backend/src/cyclone/store.py` +- Write: `/tmp/cyclone-baseline.txt` (test output snapshot) +- Write: `/tmp/cyclone-test-private-imports.txt` +- Write: `/tmp/cyclone-src-private-imports.txt` + +- [ ] **Step 1: Capture pytest baseline** + +```bash +cd backend && python -m pytest tests/ --tb=line -q 2>&1 | tee /tmp/cyclone-baseline.txt | tail -5 +``` + +Expected: `1 failed, 1176 passed, 10 skipped in `. The 1 failure is `tests/test_provider_extended_response.py::test_provider_detail_includes_orphan_remit_received` — known test-isolation flake (passes solo, fails after other tests due to shared state). Record the exact 5-line tail. + +- [ ] **Step 2: Audit private-helper imports in tests** + +```bash +grep -rn "from cyclone.store import _" backend/tests/ | tee /tmp/cyclone-test-private-imports.txt +``` + +Expected (verified 2026-06-29): + +``` +backend/tests/test_service_line_payments.py:23:from cyclone.store import _persist_835_remit, _remittance_835_row +backend/tests/test_api_line_reconciliation.py:21:from cyclone.store import _persist_835_remit, _remittance_835_row +backend/tests/test_reconcile_line_level.py:25:from cyclone.store import _persist_835_remit, _remittance_835_row +backend/tests/test_inbox_endpoints_sp7.py:22:from cyclone.store import _persist_835_remit, _remittance_835_row +``` + +Both `_persist_835_remit` and `_remittance_835_row` must be re-exported from the facade (in addition to `_claim_status_from_validation` from the original audit). + +- [ ] **Step 3: Audit private-helper imports in source** + +```bash +grep -rn "from cyclone.store import _" backend/src/ | tee /tmp/cyclone-src-private-imports.txt +``` + +Expected (verified 2026-06-29): + +``` +backend/src/cyclone/batch_diff.py:32:from cyclone.store import BatchRecord, BatchRecord835, BatchRecord837, _claim_status_from_validation +``` + +`_claim_status_from_validation` must be re-exported from the facade (already documented in original spec). + +- [ ] **Step 4: Verify package discovery on the current tree** + +```bash +cd backend && pip install -e . --quiet 2>&1 | tail -3 +python -c "import cyclone.store; print('OK:', cyclone.store.__file__)" +``` + +Expected: prints the absolute path to `backend/src/cyclone/store.py`. + +- [ ] **Step 5: Verify no `store*` subpackage exists yet** + +```bash +find backend/src/cyclone -maxdepth 1 -name "store*" +``` + +Expected: only `backend/src/cyclone/store.py` (no `store/` directory). + +No commit. Pre-flight only. + +--- + +## Task 1: Create the package skeleton (atomic move, no commit yet) + +The trick: copy `store.py` content into `store/__init__.py`, delete `store.py`, verify everything still works via the package. The CycloneStore class and `store` singleton live in `__init__.py` for now; they'll be delegated to module functions in Tasks 2-12. The working tree stays dirty through Tasks 2-12 — the single atomic commit happens at the end of Task 13. + +**Files:** +- Create: `backend/src/cyclone/store/__init__.py` (verbatim copy of store.py with updated docstring) +- Delete: `backend/src/cyclone/store.py` + +- [ ] **Step 1: Read the current store.py** + +```bash +wc -l backend/src/cyclone/store.py +``` + +Expected: `2995`. + +- [ ] **Step 2: Create the package directory and copy** + +```bash +mkdir -p backend/src/cyclone/store +cp backend/src/cyclone/store.py backend/src/cyclone/store/__init__.py +``` + +- [ ] **Step 3: Update the module docstring** + +Read the first ~30 lines of `backend/src/cyclone/store/__init__.py` (the existing docstring). Replace the existing module docstring with this updated text that names all 14 sibling modules and notes the 3 private-helper re-exports: + +```python +"""SQLAlchemy-backed batch store for parsed X12 files. + +The package exposes a single ``CycloneStore`` class and a module-level +singleton (``store``). All persistence flows through SQLAlchemy +sessions via ``db.SessionLocal()()`` (the double-paren function-style +accessor). + +This ``__init__.py`` is the facade — it re-exports every public name +plus the 3 private helpers imported by tests (``_claim_status_from_validation``, +``_persist_835_remit``, ``_remittance_835_row``) from the sibling modules: + + exceptions AlreadyMatchedError, NotMatchedError, InvalidStateError + records BatchKind, BatchRecord, BatchRecord837, BatchRecord835 + orm_builders ORM row builders + the 3 private-helper re-exports + ui UI serializers (to_ui_*) + write add_record + private event/reconcile helpers + batches Batch reads + _BatchesShim (test-cleanup shim) + claim_detail Single-claim reads + matched-pair drift audit + kpis Dashboard aggregation + acks 999 / TA1 / 277CA ACK persistence + backups Backup-pending marker inserts + inbox Manual match/unmatch + lane listing + providers Provider/payer/clearhouse config + +The ``CycloneStore`` class below keeps its current method signatures as +thin delegations for back-compat with existing callers and tests. + +Public API (preserved verbatim): + CycloneStore, store, BatchRecord, BatchRecord837, BatchRecord835, + BatchKind, AlreadyMatchedError, NotMatchedError, InvalidStateError, + utcnow, dashboard_kpis, check_matched_pair_drift + +Backward-compat shims for tests: + ``_lock`` — a threading.RLock used by 7 test files for cleanup. + ``_batches.clear()`` — wipes all rows from the DB tables; the + ``_BatchesShim`` class lives in ``batches.py``. +""" +``` + +- [ ] **Step 4: Delete the old store.py** + +```bash +rm backend/src/cyclone/store.py +``` + +- [ ] **Step 5: Verify imports still work** + +```bash +python -c " +from cyclone.store import ( + CycloneStore, store, BatchRecord, BatchRecord837, BatchRecord835, BatchKind, + AlreadyMatchedError, NotMatchedError, InvalidStateError, utcnow, + dashboard_kpis, check_matched_pair_drift, + _claim_status_from_validation, _persist_835_remit, _remittance_835_row, +) +print('all imports OK') +" +``` + +Expected: `all imports OK`. + +- [ ] **Step 6: Verify test suite matches baseline** + +```bash +cd backend && python -m pytest tests/ --tb=line -q 2>&1 | tail -5 +``` + +Expected: identical 5-line tail to `/tmp/cyclone-baseline.txt`. Any deviation is a regression — halt and investigate before continuing. + +**No commit yet.** The final atomic commit happens in Task 13. + +--- + +## Task 2: Extract `exceptions.py` + +**Files:** +- Create: `backend/src/cyclone/store/exceptions.py` +- Modify: `backend/src/cyclone/store/__init__.py` + +- [ ] **Step 1: Create `exceptions.py`** + +Read the 3 exception class definitions in `backend/src/cyclone/store/__init__.py` (the 3 classes with names starting at column 0: `AlreadyMatchedError`, `NotMatchedError`, `InvalidStateError`). Copy them verbatim into `backend/src/cyclone/store/exceptions.py` with this module docstring: + +```python +"""Exception types raised by CycloneStore and its domain modules. + +These are part of the public API — callers (API endpoints) catch them +and translate to HTTP 409 Conflict responses. +""" +``` + +- [ ] **Step 2: Update `__init__.py` to re-export** + +Remove the 3 class definitions from `__init__.py`. Add at the top of `__init__.py` (after `from cyclone import db`): + +```python +from .exceptions import AlreadyMatchedError, NotMatchedError, InvalidStateError +``` + +- [ ] **Step 3: Verify tests still pass** + +```bash +cd backend && python -m pytest tests/ --tb=line -q 2>&1 | tail -5 +``` + +Expected: identical to baseline. + +--- + +## Task 3: Extract `records.py` + +**Files:** +- Create: `backend/src/cyclone/store/records.py` +- Modify: `backend/src/cyclone/store/__init__.py` + +- [ ] **Step 1: Create `records.py`** + +Read the Pydantic-model block in `__init__.py` (the 4 definitions: `BatchKind = Literal[...]`, `class BatchRecord`, `class BatchRecord837`, `class BatchRecord835`). Copy them verbatim into `backend/src/cyclone/store/records.py` with this module docstring: + +```python +"""Pydantic models for parsed-batch records. + +``BatchRecord`` is the union type; ``BatchRecord837`` and ``BatchRecord835`` +narrow ``result`` to the parser-specific output types. Construction of +``BatchRecord(kind="837p", ...)`` dispatches to ``BatchRecord837`` via +``__new__`` so isinstance checks downstream narrow ``result`` correctly. +""" +``` + +- [ ] **Step 2: Update `__init__.py`** + +Remove the 4 definitions from `__init__.py`. Add to imports: + +```python +from .records import BatchKind, BatchRecord, BatchRecord837, BatchRecord835 +``` + +- [ ] **Step 3: Verify tests still pass** + +```bash +cd backend && python -m pytest tests/ --tb=line -q 2>&1 | tail -5 +``` + +Expected: identical to baseline. + +--- + +## Task 4: Extract `orm_builders.py` + +**Files:** +- Create: `backend/src/cyclone/store/orm_builders.py` +- Modify: `backend/src/cyclone/store/__init__.py` + +- [ ] **Step 1: Create `orm_builders.py`** + +Read the ORM row-builder block in `__init__.py` (the 6 module-level helpers named with leading underscore: `_service_dates_from_claim`, `_claim_837_row`, `_remittance_835_row`, `_cas_adjustment_row`, `_persist_835_remit`, `_claim_status_from_validation`). Copy them verbatim into `backend/src/cyclone/store/orm_builders.py` with this module docstring: + +```python +"""ORM row builders — convert parser output into SQLAlchemy ORM rows. + +Each function returns an unsaved ORM object (or inserts related rows +within an existing session). They never commit or close the session — +the caller (``write.add_record``, ``acks.add_ack``, etc.) owns the +transaction boundary. +""" +``` + +- [ ] **Step 2: Update `__init__.py`** + +Remove the 6 helpers from `__init__.py`. Add to imports — re-export the 3 private helpers that tests + `batch_diff.py` import: + +```python +from .orm_builders import ( + _claim_status_from_validation, + _persist_835_remit, + _remittance_835_row, +) +``` + +The other 3 (`_service_dates_from_claim`, `_claim_837_row`, `_cas_adjustment_row`) are module-private; siblings import them directly via `from .orm_builders import ...`. + +- [ ] **Step 3: Verify tests still pass** + +```bash +cd backend && python -m pytest tests/test_service_line_payments.py tests/test_api_line_reconciliation.py tests/test_reconcile_line_level.py tests/test_inbox_endpoints_sp7.py tests/test_batch_diff.py tests/ --tb=line -q 2>&1 | tail -5 +``` + +Expected: identical to baseline. (The 4 files importing the private helpers will fail loud if the re-export is wrong.) + +--- + +## Task 5: Extract `ui.py` + +**Files:** +- Create: `backend/src/cyclone/store/ui.py` +- Modify: `backend/src/cyclone/store/__init__.py` + +- [ ] **Step 1: Create `ui.py`** + +Read the UI-serializer block in `__init__.py`: every top-level function whose name starts with `to_ui_` or `_iso_z` / `_address_to_ui` / `_validation_issues_to_ui` / `_svc_to_wire_dict` / `_date_in_bounds` / `_provider_orm_to_dict` / `_payer_orm_to_dict`, plus the constant `CLAIM_DETAIL_HISTORY_LIMIT`. Also include the `utcnow()` function. Copy them verbatim into `backend/src/cyclone/store/ui.py` with this module docstring: + +```python +"""UI serialization helpers — convert ORM rows to API-shaped dicts. + +These are the boundary between the persistence layer and the wire format +consumed by the React frontend. Keep them stable: any rename or +shape change ripples to every API consumer. + +Also hosts ``utcnow()`` (the tz-aware UTC now) because it's a small +free function that several modules want and there's no better home. +""" +``` + +- [ ] **Step 2: Update `__init__.py`** + +Remove the UI serializer functions AND `utcnow()` from `__init__.py`. Add to imports: + +```python +from .ui import utcnow +``` + +`utcnow` is the only one re-exported (for `api_helpers.py` docstring reference and any external use). + +- [ ] **Step 3: Verify tests still pass** + +```bash +cd backend && python -m pytest tests/ --tb=line -q 2>&1 | tail -5 +``` + +Expected: identical to baseline. + +--- + +## Task 6: Extract `write.py` + +**Files:** +- Create: `backend/src/cyclone/store/write.py` +- Modify: `backend/src/cyclone/store/__init__.py` + +- [ ] **Step 1: Create `write.py`** + +Read the `CycloneStore.add` method body, plus the 3 private methods `_publish_events_sync`, `_sync_publish`, `_run_reconcile` in `__init__.py`. Copy them into `backend/src/cyclone/store/write.py` as module functions with these renames: + +| Class method | Module function | +|---|---| +| `CycloneStore.add` | `add_record(record, *, event_bus=None)` | +| `CycloneStore._publish_events_sync` | `publish_events_sync(event_bus, record, claim_ids, remit_ids)` | +| `CycloneStore._sync_publish` | `_sync_publish(event_bus, kind, payload)` | +| `CycloneStore._run_reconcile` | `run_reconcile(batch_id)` | + +In each function body: +- Strip the `self.` prefix from all attribute accesses (but no other refactoring). +- `self._publish_events_sync(...)` → `publish_events_sync(...)` +- `self._run_reconcile(...)` → `run_reconcile(...)` +- `self._sync_publish(...)` → `_sync_publish(...)` + +Module docstring: + +```python +"""Write path for parsed batches — insert, reconcile, publish. + +The single entry point is ``add_record(record, *, event_bus=None)``, +which replaces the body of the previous ``CycloneStore.add`` method. +It owns its own SQLAlchemy session, runs idempotency checks, persists +the batch + child rows, then (for 835 batches) triggers reconciliation +and (if ``event_bus`` is provided) publishes live-tail events. + +Reconciliation runs OUTSIDE the persistence session, fail-soft: errors +are logged but do not roll back the persisted batch. +""" +``` + +- [ ] **Step 2: Update `__init__.py` CycloneStore class** + +Add import: + +```python +from . import write +``` + +Replace the 4 method bodies with 1-line delegations: + +```python + def add(self, record, *, event_bus=None): + """Persist a parsed batch (837P or 835).""" + return write.add_record(record, event_bus=event_bus) + + def _publish_events_sync(self, event_bus, record, claim_ids, remit_ids): + return write.publish_events_sync(event_bus, record, claim_ids, remit_ids) + + def _run_reconcile(self, batch_id): + return write.run_reconcile(batch_id) + + @staticmethod + def _sync_publish(event_bus, kind, payload): + return write._sync_publish(event_bus, kind, payload) +``` + +- [ ] **Step 3: Verify hot-path tests pass** + +```bash +cd backend && python -m pytest tests/test_parse_999.py tests/test_parse_277ca.py tests/test_api_999.py tests/test_api_277ca.py tests/test_reconcile.py tests/test_reconcile_line_level.py tests/test_api_line_reconciliation.py --tb=short -q +``` + +Expected: 100% pass on these 7 files. + +- [ ] **Step 4: Verify full suite matches baseline** + +```bash +cd backend && python -m pytest tests/ --tb=line -q 2>&1 | tail -5 +``` + +Expected: identical to baseline. + +--- + +## Task 7: Extract `batches.py` + +**Files:** +- Create: `backend/src/cyclone/store/batches.py` +- Modify: `backend/src/cyclone/store/__init__.py` + +- [ ] **Step 1: Create `batches.py`** + +Read in `__init__.py`: +- The `_BatchesShim` class definition (verbatim — the test-cleanup shim) +- The `_row_to_record` helper +- The 5 `CycloneStore` methods: `get_batch`, `get`, `list`, `all`, `load_two_for_diff` + +Copy into `backend/src/cyclone/store/batches.py` as module functions with these renames: + +| Class method | Module function | +|---|---| +| `CycloneStore.get_batch` | `get_batch(batch_id)` | +| `CycloneStore.get` | `get_record(batch_id)` (rename avoids builtin shadow) | +| `CycloneStore.list` | `list_batches(*, limit=100)` | +| `CycloneStore.all` | `all_batches()` | +| `CycloneStore.load_two_for_diff` | `load_two_for_diff(batch_ids)` | + +Strip `self` from each. Module docstring: + +```python +"""Batch read APIs and the legacy _BatchesShim. + +The ``_BatchesShim`` class is retained for back-compat with the +``with store._lock: store._batches.clear()`` test-cleanup idiom used +in 7 test files. Its ``clear()`` method wipes all rows from the DB +tables so tests that depended on a fresh in-memory list per-test get +a fresh DB state per-test. +""" +``` + +- [ ] **Step 2: Update `__init__.py` CycloneStore class** + +Add import: + +```python +from .batches import ( + _BatchesShim, _row_to_record, + get_batch, get_record, list_batches, all_batches, load_two_for_diff, +) +``` + +Replace the 5 method bodies with 1-line delegations. (Note: `get_batch` is a top-level function in `batches.py`; the class method also called `get_batch` is a thin wrapper — Python scope resolves correctly.) + +```python + def get_batch(self, batch_id): + return get_batch(batch_id) + + def get(self, batch_id): + return get_record(batch_id) + + def list(self, *, limit=100): + return list_batches(limit=limit) + + def all(self): + return all_batches() + + def load_two_for_diff(self, batch_ids): + return load_two_for_diff(batch_ids) +``` + +- [ ] **Step 3: Verify tests pass** + +```bash +cd backend && python -m pytest tests/test_api_gets.py tests/test_api_parse_persists.py tests/test_api_parse_persists_ack.py tests/test_api_claim_detail.py tests/test_api_999.py tests/test_api_ta1.py tests/test_store.py --tb=short -q +``` + +Expected: 100% pass on these 7 files. + +--- + +## Task 8: Extract `claim_detail.py` (also hosts `check_matched_pair_drift`) + +**Files:** +- Create: `backend/src/cyclone/store/claim_detail.py` +- Modify: `backend/src/cyclone/store/__init__.py` + +- [ ] **Step 1: Create `claim_detail.py`** + +Read in `__init__.py`: +- The 6 `CycloneStore` methods: `get_remittance`, `get_claim_detail`, `iter_claims`, `iter_remittances`, `distinct_providers`, `recent_activity` +- The module-level function `check_matched_pair_drift()` (added by SP27; ~110 LOC, returns `int` count of drifted pairs) + +Copy into `backend/src/cyclone/store/claim_detail.py` as module functions: + +| Class method / function | Module function | +|---|---| +| `CycloneStore.get_remittance` | `get_remittance(remittance_id)` | +| `CycloneStore.get_claim_detail` | `get_claim_detail(claim_id)` | +| `CycloneStore.iter_claims` | `iter_claims(...)` (preserve generator pattern) | +| `CycloneStore.iter_remittances` | `iter_remittances(...)` (preserve generator pattern) | +| `CycloneStore.distinct_providers` | `distinct_providers()` | +| `CycloneStore.recent_activity` | `recent_activity(*, limit=200)` | +| module-level `check_matched_pair_drift` | `check_matched_pair_drift()` (no rename) | + +For the 2 generators (`iter_claims`, `iter_remittances`), each must open its own `with db.SessionLocal()() as s:` and `yield` inside; the session lifetime tracks the generator's. Preserve the existing pattern verbatim. + +Module docstring: + +```python +"""Claim- and remittance-detail read APIs. + +Includes the only large non-write query in the store: +``get_claim_detail`` which joins Claim + CasAdjustment + ActivityEvent +history for the right-drawer UI. + +Also hosts ``check_matched_pair_drift()`` — the SP27 startup invariant +audit that returns the count of Claim↔Remit pairs where +``matched_remittance_id`` doesn't agree with the FK in ``remittances.claim_id``. +It lives here (not in its own module) because it reads the same pair of +tables as ``get_claim_detail``'s matched-remittance summary. +""" +``` + +- [ ] **Step 2: Update `__init__.py` CycloneStore class** + +Add import: + +```python +from .claim_detail import ( + get_remittance, get_claim_detail, + iter_claims, iter_remittances, + distinct_providers, recent_activity, + check_matched_pair_drift, +) +``` + +Replace the 6 method bodies with 1-line delegations: + +```python + def get_remittance(self, remittance_id): + return get_remittance(remittance_id) + + def get_claim_detail(self, claim_id): + return get_claim_detail(claim_id) + + def iter_claims(self, *args, **kwargs): + return iter_claims(*args, **kwargs) + + def iter_remittances(self, *args, **kwargs): + return iter_remittances(*args, **kwargs) + + def distinct_providers(self): + return distinct_providers() + + def recent_activity(self, *, limit=200): + return recent_activity(limit=limit) +``` + +- [ ] **Step 3: Verify tests pass** + +```bash +cd backend && python -m pytest tests/test_api_claim_detail.py tests/test_store_claim_detail.py tests/test_store_match_invariant.py --tb=short -q +``` + +Expected: 100% pass on these 3 files (the drift-audit tests live in `test_store_match_invariant.py`). + +--- + +## Task 9: Extract `kpis.py` (NEW MODULE from SP27) + +**Files:** +- Create: `backend/src/cyclone/store/kpis.py` +- Modify: `backend/src/cyclone/store/__init__.py` + +- [ ] **Step 1: Create `kpis.py`** + +Read in `__init__.py`: +- The module-level function `dashboard_kpis()` (~150 LOC, returns a dict with `claims_total`, `remittances_total`, `matched_total`, `unmatched_total`, `paid_total_cents`, `adjustments_total_cents`, `recent_activity`) +- The module-level helper `_claim_state_str()` (used internally by `dashboard_kpis`) + +Copy them verbatim into `backend/src/cyclone/store/kpis.py`. Module docstring: + +```python +"""Dashboard KPI aggregation — cross-table reads consumed by /api/dashboard/kpis. + +``dashboard_kpis()`` returns a single dict that the React Dashboard page +consumes as the source of truth for the 4 KPI tiles + the recent-activity +panel. It reads from claims + remittances + batches in a single pass. + +``_claim_state_str`` is a small mapping helper that translates ORM +status enum values to UI-friendly strings. It's private to this module. +""" +``` + +- [ ] **Step 2: Update `__init__.py`** + +Remove `dashboard_kpis` and `_claim_state_str` from `__init__.py`. Add to imports: + +```python +from .kpis import dashboard_kpis +``` + +(`_claim_state_str` is module-private; not re-exported.) + +- [ ] **Step 3: Verify tests pass** + +```bash +cd backend && python -m pytest tests/test_dashboard_kpis.py tests/ --tb=line -q 2>&1 | tail -5 +``` + +Expected: identical to baseline. + +--- + +## Task 10: Extract `acks.py` + +**Files:** +- Create: `backend/src/cyclone/store/acks.py` +- Modify: `backend/src/cyclone/store/__init__.py` + +- [ ] **Step 1: Create `acks.py`** + +Read the 9 `CycloneStore` ACK methods in `__init__.py`: `add_ack`, `list_acks`, `get_ack`, `add_ta1_ack`, `list_ta1_acks`, `get_ta1_ack`, `add_277ca_ack`, `list_277ca_acks`, `get_277ca_ack`. + +Copy into `backend/src/cyclone/store/acks.py` as module functions with these renames for clarity: + +| Class method | Module function | +|---|---| +| `CycloneStore.add_ack` | `add_999_ack(...)` | +| `CycloneStore.list_acks` | `list_acks()` | +| `CycloneStore.get_ack` | `get_ack(ack_id)` | +| `CycloneStore.add_ta1_ack` | `add_ta1_ack(...)` | +| `CycloneStore.list_ta1_acks` | `list_ta1_acks()` | +| `CycloneStore.get_ta1_ack` | `get_ta1_ack(ack_id)` | +| `CycloneStore.add_277ca_ack` | `add_277ca_ack(...)` | +| `CycloneStore.list_277ca_acks` | `list_277ca_acks()` | +| `CycloneStore.get_277ca_ack` | `get_277ca_ack(ack_id)` | + +Strip `self` from each. Module docstring: + +```python +"""ACK persistence — 999 / TA1 / 277CA acknowledgements. + +Each ACK type has its own ORM row (Ack / Ta1Ack / Two77caAck). The +write methods are simple inserts; the list/get methods are simple +queries. Fail-soft: persistence errors are logged but do not block +parsing. +""" +``` + +- [ ] **Step 2: Update `__init__.py` CycloneStore class** + +Add import: + +```python +from .acks import ( + add_999_ack, list_acks, get_ack, + add_ta1_ack, list_ta1_acks, get_ta1_ack, + add_277ca_ack, list_277ca_acks, get_277ca_ack, +) +``` + +Replace the 9 method bodies with 1-line delegations. Method names on the class stay the same as today; only the module-function names use the type prefix for clarity. + +```python + def add_ack(self, **kwargs): + return add_999_ack(**kwargs) + + def list_acks(self): + return list_acks() + + def get_ack(self, ack_id): + return get_ack(ack_id) + + def add_ta1_ack(self, **kwargs): + return add_ta1_ack(**kwargs) + + def list_ta1_acks(self): + return list_ta1_acks() + + def get_ta1_ack(self, ack_id): + return get_ta1_ack(ack_id) + + def add_277ca_ack(self, **kwargs): + return add_277ca_ack(**kwargs) + + def list_277ca_acks(self): + return list_277ca_acks() + + def get_277ca_ack(self, ack_id): + return get_277ca_ack(ack_id) +``` + +- [ ] **Step 3: Verify tests pass** + +```bash +cd backend && python -m pytest tests/test_api_parse_persists_ack.py tests/test_api_ta1.py tests/test_acks.py tests/test_api_999.py tests/test_apply_277ca_rejections.py --tb=short -q +``` + +Expected: 100% pass on these 5 ACK-related test files. + +--- + +## Task 11: Extract `backups.py` + +**Files:** +- Create: `backend/src/cyclone/store/backups.py` +- Modify: `backend/src/cyclone/store/__init__.py` + +- [ ] **Step 1: Create `backups.py`** + +Read `CycloneStore.add_backup_pending` in `__init__.py`. Copy into `backend/src/cyclone/store/backups.py` as a module function `add_backup_pending(*, filename, backup_dir)`. Strip `self`. Module docstring: + +```python +"""Backup-pending marker persistence. + +Used by ``cyclone.backup_service`` to mark files that need backing up +after parsing. Simple insert into the DbBackup table. +""" +``` + +- [ ] **Step 2: Update `__init__.py` CycloneStore class** + +Add import: + +```python +from .backups import add_backup_pending +``` + +Replace the method body with a 1-line delegation: + +```python + def add_backup_pending(self, *, filename, backup_dir): + return add_backup_pending(filename=filename, backup_dir=backup_dir) +``` + +- [ ] **Step 3: Verify tests pass** + +```bash +cd backend && python -m pytest tests/test_backup_service.py tests/test_api_backup.py --tb=short -q +``` + +Expected: 100% pass. + +--- + +## Task 12: Extract `inbox.py` + `providers.py` + +**Files:** +- Create: `backend/src/cyclone/store/inbox.py` +- Create: `backend/src/cyclone/store/providers.py` +- Modify: `backend/src/cyclone/store/__init__.py` + +- [ ] **Step 1: Create `inbox.py`** + +Read in `__init__.py`: +- `CycloneStore.list_unmatched` +- `CycloneStore.manual_match` +- `CycloneStore.manual_unmatch` + +Copy into `backend/src/cyclone/store/inbox.py` as module functions with the same names (no renames needed). Strip `self`. Module docstring: + +```python +"""Inbox lane state and manual match/unmatch operations. + +``list_unmatched`` returns the 5-lane inbox view (unmatched claims + +unmatched remits). ``manual_match`` and ``manual_unmatch`` invoke the +reconcile pure functions and persist the resulting Match row (or +remove it). Invalid state transitions surface as ``InvalidStateError``. +""" +``` + +Add at the top of `inbox.py`: + +```python +from cyclone import db, reconcile +from .exceptions import InvalidStateError +``` + +- [ ] **Step 2: Create `providers.py`** + +Read in `__init__.py`: +- The 7 `CycloneStore` methods: `list_providers`, `get_provider`, `upsert_provider`, `list_payers`, `get_payer_config`, `get_clearhouse`, `ensure_clearhouse_seeded` +- The 2 module-level helpers `_provider_orm_to_dict` and `_payer_orm_to_dict` + +Copy into `backend/src/cyclone/store/providers.py` as module functions, stripping `self`. Module docstring: + +```python +"""Provider, payer, and clearhouse configuration reads and upserts. + +The ``providers`` and ``payers`` modules in ``cyclone`` provide the +ORM-row DTOs; this module is the read/upsert surface over them. +``ensure_clearhouse_seeded`` is called at startup to insert the +default Clearhouse row if missing. +""" +``` + +Add at the top of `providers.py`: + +```python +from cyclone import db +from cyclone.providers import Clearhouse, Payer, Provider +``` + +- [ ] **Step 3: Update `__init__.py` CycloneStore class** + +Add imports: + +```python +from .inbox import list_unmatched, manual_match, manual_unmatch +from .providers import ( + list_providers, get_provider, upsert_provider, list_payers, + get_payer_config, get_clearhouse, ensure_clearhouse_seeded, +) +``` + +Replace the 10 method bodies with 1-line delegations: + +```python + def list_unmatched(self, *, kind="both"): + return list_unmatched(kind=kind) + + def manual_match(self, claim_id, remit_id): + return manual_match(claim_id, remit_id) + + def manual_unmatch(self, claim_id): + return manual_unmatch(claim_id) + + def list_providers(self, *, is_active=True): + return list_providers(is_active=is_active) + + def get_provider(self, npi): + return get_provider(npi) + + def upsert_provider(self, provider): + return upsert_provider(provider) + + def list_payers(self, *, is_active=True): + return list_payers(is_active=is_active) + + def get_payer_config(self, payer_id, transaction_type): + return get_payer_config(payer_id, transaction_type) + + def get_clearhouse(self): + return get_clearhouse() + + def ensure_clearhouse_seeded(self): + return ensure_clearhouse_seeded() +``` + +- [ ] **Step 4: Verify hot-path tests pass** + +```bash +cd backend && python -m pytest tests/test_inbox_state.py tests/test_inbox_lanes.py tests/test_inbox_endpoints.py tests/test_inbox_endpoints_sp7.py tests/test_payer_rejected_acknowledge.py tests/test_store_reconcile.py tests/test_reconcile.py tests/test_reconcile_line_level.py tests/test_api_line_reconciliation.py tests/test_providers_seed.py --tb=short -q +``` + +Expected: 100% pass on these 10 hot-path files. + +--- + +## Task 13: Final verification + single atomic commit + +- [ ] **Step 1: Run full test suite, compare to baseline** + +```bash +cd backend && python -m pytest tests/ --tb=line -q 2>&1 | tail -5 +``` + +Expected: identical 5-line tail to `/tmp/cyclone-baseline.txt`. If any deviation, halt and investigate before committing. + +- [ ] **Step 2: Import sanity check** + +```bash +python -c " +from cyclone.store import ( + CycloneStore, store, BatchRecord, BatchRecord837, BatchRecord835, BatchKind, + AlreadyMatchedError, NotMatchedError, InvalidStateError, utcnow, + dashboard_kpis, check_matched_pair_drift, + _claim_status_from_validation, _persist_835_remit, _remittance_835_row, +) +print('all public imports OK') + +import inspect +store_methods = [m for m in dir(store) if not m.startswith('__')] +expected = { + # batch reads + 'add', 'get', 'get_batch', 'list', 'all', 'load_two_for_diff', + # claim-detail reads + 'get_remittance', 'get_claim_detail', 'iter_claims', 'iter_remittances', + 'distinct_providers', 'recent_activity', + # KPI + 'summarize_remittances', + # ACKs + 'add_ack', 'list_acks', 'get_ack', + 'add_ta1_ack', 'list_ta1_acks', 'get_ta1_ack', + 'add_277ca_ack', 'list_277ca_acks', 'get_277ca_ack', + # backups + 'add_backup_pending', + # inbox + 'list_unmatched', 'manual_match', 'manual_unmatch', + # providers + 'list_providers', 'get_provider', 'upsert_provider', + 'list_payers', 'get_payer_config', 'get_clearhouse', 'ensure_clearhouse_seeded', +} +missing = expected - set(store_methods) +assert not missing, f'missing methods: {missing}' +print('all expected methods present on singleton') + +# Back-compat shims +assert hasattr(store, '_lock'), 'store._lock missing' +assert hasattr(store, '_batches'), 'store._batches missing' +print('back-compat shims intact') +" +``` + +Expected: `all public imports OK`, `all expected methods present on singleton`, `back-compat shims intact`. + +- [ ] **Step 3: Live smoke test on 8 endpoints** + +Run these 8 curls against a live `python -m cyclone serve` instance and confirm no 500s: + +| Endpoint | Expected | +|---|---| +| `POST /api/parse-999` (valid file) | 200, `ack_code=A` | +| `POST /api/parse-277ca` (valid file) | 200, 3 claim statuses | +| `GET /api/inbox?lane=unmatched` | 200, list shape | +| `POST /api/inbox/match` (happy path) | 200 | +| `POST /api/inbox/match` (already matched) | 409 (AlreadyMatchedError) | +| `POST /api/inbox/unmatch` (happy path) | 200 | +| `POST /api/inbox/unmatch` (not matched) | 409 (NotMatchedError) | +| `GET /api/batches` + `GET /api/batches/{id}` | 200 | +| `GET /api/claims/{id}` | 200, claim detail shape | +| `GET /api/999-acks/{id}` | 200 | +| `GET /api/277ca-acks/{id}` | 200 | + +Smoke output is documented in the commit message. + +- [ ] **Step 4: Single atomic commit** + +```bash +git add -A backend/src/cyclone/store/ backend/src/cyclone/store.py +git status # confirm: store.py deleted; store/ created with 14 modules +git commit -m "refactor(sp21): split store.py into cyclone/store/ subpackage (14 modules) + +Behaviour-preserving structural split of the 2,995-LOC store.py into +a 14-module subpackage with a thin facade. CycloneStore class keeps +its full method surface as 1-line delegations to module functions. + +14 modules: + exceptions, records, orm_builders, ui, write, + batches, claim_detail, kpis, acks, + backups, inbox, providers + (+ __init__.py facade) + +Facade re-exports 12 public symbols (CycloneStore, store, BatchRecord*, +BatchKind, AlreadyMatchedError/NotMatchedError/InvalidStateError, utcnow, +dashboard_kpis, check_matched_pair_drift) + 3 private helpers +(_claim_status_from_validation, _persist_835_remit, _remittance_835_row) +to preserve the 4 test files that import them. + +Zero public API changes, zero test changes, zero importer changes. +CycloneStore._lock and CycloneStore._batches.clear() remain intact for +the 7 test files still using the cleanup idiom. + +Verified: 1,176 / 1 (pre-existing isolation flake) / 10 tests pass — +identical to baseline. Smoke-tested 11 endpoints, no 500s." +``` + +Expected: 1 commit on `sp21-store-split`. Working tree clean. + +- [ ] **Step 5: Push branch + open PR (or wait for user direction)** + +Per the SP-N conventions, the single atomic commit on `sp21-store-split` lands via a single atomic merge into `main` (no squash, no rebase). Either: +- Push the branch and let the user trigger the merge, OR +- Run the merge immediately if the user has approved + +Default: stop here and report the commit hash + branch state. + +--- + +## Out of scope (explicit) + +- Renaming any public symbol +- Changing any method signature +- Refactoring `manual_match` / `manual_unmatch` business logic +- Refactoring `add()` body or its idempotency strategy +- Refactoring `dashboard_kpis` / `check_matched_pair_drift` aggregation logic +- Adding new tests +- Modifying DB models in `cyclone.db` +- Updating SQLCipher key rotation flow +- Touching any of the 7 test files that use `_lock` / `_batches.clear()` +- Touching any of the 4 test files that import `_persist_835_remit` / `_remittance_835_row` +- Touching any importer in `cyclone.api`, `cyclone.api_routers.*`, `cyclone.batch_diff`, `cyclone.backup_service`, `cyclone.scheduler`, `cyclone.handlers.*`, `cyclone.parsers.validator`, `cyclone.api_helpers` + +If any of these surface during implementation, they get deferred to follow-up tasks — never silently included. \ No newline at end of file