diff --git a/docs/superpowers/specs/2026-07-02-cyclone-ack-live-tail-design.md b/docs/superpowers/specs/2026-07-02-cyclone-ack-live-tail-design.md new file mode 100644 index 0000000..d70e7cd --- /dev/null +++ b/docs/superpowers/specs/2026-07-02-cyclone-ack-live-tail-design.md @@ -0,0 +1,100 @@ +# SP25 — Ack Live-Tail: Design Spec + +**Date:** 2026-07-02 +**Status:** Draft, awaiting user sign-off +**Branch:** `sp25-ack-live-tail` +**Aesthetic direction:** No new visual language. The Acks page already has its dark-instrument chrome (hero / KPI strip / register / TA1 section / footer). We add a single `` next to the existing "Envelope accepted / Mixed envelope" badge in the hero — same component, same semantic states, used identically on Claims / Remittances / Activity. No new color, type, layout, or motion decisions. + +## 1. Scope + +When the SFTP poller (or a manual upload) drops a TA1 or 999 file, every copy of the `/acks` page should reflect the new row within one heartbeat — no manual refresh, no polling. The mechanism is the same one Claims / Remittances / Activity already use: the store publishes a per-resource event after the row commits, a streaming NDJSON endpoint fans that event out, the page's `useTailStream` hook appends it to a zustand slice, and `useMergedTail` dedups against the page's initial JSON fetch. + +The store already does this for `claim_written` / `remittance_written` / `activity_recorded`. SP25 extends the same pattern to the three ack tables (999, TA1, 277CA). 277CA is a no-UI side effect — the store still publishes the event so any future 277CA page can subscribe without a backend change. 999 and TA1 also get stream endpoints and frontend wiring because the Acks page already renders both. + +The handler-side `if event_bus is not None: publish(...)` block — the broken stub that started this work — disappears along with the `event_bus` kwarg from `handle_999` / `handle_ta1` / `handle_277ca` / `handle_835`. The TODO(sp27-task-6) comments go away because the design they were deferring is no longer the design. + +Out of scope (explicit, each is its own future increment if requested): +- A new 277CA page and `useTwo77caAcks` hook. The store publishes the event for future use; no page subscribes today. +- Backfilling historical acks into the live store. The snapshot-on-reconnect already covers "first time the page mounts" so the operator never sees an empty page when the DB has rows. +- Reformatting the Acks page hero, KPI strip, or sectioning. We add a pill; we do not redesign the page. +- A unified "all acks" stream endpoint. The page renders 999 and TA1 in distinct sections with different shapes; one stream per resource is the right call, mirroring the existing pattern. +- Any scheduler change. The scheduler already calls the store; once the store publishes, the scheduler benefits without further code. +- A 277CA audit-event publish path. `claim.payer_rejected` and the SP27 Task 13 follow-up are out of scope. + +## 2. Goals + +1. **The Acks page reflects new acks within one heartbeat.** The operator sees the same "appears on its own" behavior they already get on Claims / Remittances / Activity. +2. **The store owns the publish, not the handler.** Mirrors the existing three live resources. Removes the sync-handler-meets-async-EventBus design friction that was the source of the original bug. +3. **The event payload matches the list-endpoint shape.** The stream-endpoint snapshot and the live-event item are interchangeable; a downstream consumer that snapshots the page over HTTP and one that subscribes to the stream see the same row shape for the same row id. +4. **Both write paths fire the event.** The SFTP poller AND the manual `/api/parse-999` / `/api/parse-ta1` / `/api/parse-277ca` endpoints both end up in the store's `add_*_ack` methods, so both fire the event without any caller-specific plumbing. +5. **A failed publish never rolls back the row.** Persisted state is the source of truth; the event is the cache-invalidation hint. Match the existing `claim_written` semantics. +6. **No DB migration.** The events are pure pubsub; no new tables, no new columns, no new indexes. + +## 3. Threat model + +The auth boundary is HTTP (login required, bcrypt + HttpOnly session cookie); file-system threats remain the local-only threat model (SQLCipher at rest, macOS Keychain). The new stream endpoints sit behind `Depends(matrix_gate)` like every other `/api/*` endpoint — operators must be authenticated to receive live ack updates. The store's `add_*_ack` methods take `event_bus` as an explicit parameter (not a global) so the publish path is unit-testable without a real bus, and so a future move of the bus out of process does not require touching the store API. + +## 4. Locked decisions + +### 4.1 Publish from the store, not the handler + +The handler-side `if event_bus is not None: publish(...)` block is removed. The store's `add_999_ack` / `add_ta1_ack` / `add_277ca_ack` methods take `event_bus: EventBus | None` and call `_sync_publish(...)` after the row commits, exactly the way `store.write.add_record` does for the three existing live resources. The handlers stop accepting `event_bus`; they just call the store and return. The `TODO(sp27-task-6)` comments go away because the sync/async gap they were deferring no longer exists at the call site. + +Rationale: the handler is `def` (sync) and `EventBus.publish` is `async`. The current stub is a silent bug — `publish(...)` returns an unawaited coroutine and the warning gets swallowed by the `try/except`. Publishing from the store uses `_sync_publish`, which is the project's blessed bridge between sync write paths and the async bus. One change, one place. + +### 4.2 Three event names, full `to_ui_*` payload + +The three new event names are `ack_received` (999), `ta1_ack_received` (TA1), and `two77ca_ack_received` (277CA). Each payload is the full `to_ui_*` row shape that the matching list endpoint would return for the same row id — `to_ui_ack` for 999, `to_ui_ta1_ack` for TA1. (277CA's payload is the same shape its list endpoint at `GET /api/277ca-acks` returns; that serializer already lives in `api.py` near the endpoint and moves to `store/ui.py` along with the other two for ownership consistency.) + +Rationale: the existing `_written` events carry the full row shape, and `cyclone-store` makes the event-payload-must-match-list-endpoint contract explicit. Two acks in flight through the same channel become trivially comparable; a future live-tail consumer that does its own dedup does not need a separate refetch to see the canonical row. + +### 4.3 Two new stream endpoints, registered in `api.py` + +`/api/acks/stream` subscribes to `["ack_received"]`. `/api/ta1-acks/stream` subscribes to `["ta1_ack_received"]`. Both follow the same wire format as the three existing live-tail endpoints (`application/x-ndjson`, snapshot → `snapshot_end` → live items, 15-second heartbeats, 30-second stall threshold). Both sit behind `Depends(matrix_gate)`. The new endpoints live in `api.py` next to `/api/claims/stream` / `/api/remittances/stream` / `/api/activity/stream` because the live-tail triplet is currently inlined there; moving the acks streams into the `api_routers/acks.py` / `api_routers/ta1_acks.py` modules is a separate refactor. + +### 4.4 `_to_ui` mappers move to `store/ui.py` + +`_ack_to_ui` (currently in `api_routers/acks.py`) becomes `to_ui_ack` in `store/ui.py`. `_ta1_to_ui` (currently in `api_routers/ta1_acks.py`) becomes `to_ui_ta1_ack` in `store/ui.py`. The 277CA serializer moves alongside. `store/ui.py` already houses `to_ui_claim_from_orm` / `to_ui_remittance_from_orm` / `to_ui_provider` per the SP21 split map; the ack serializers are the obvious home. The api_routers/ modules import the renamed mappers from the store. The store's `add_*_ack` methods import them from the same module. One source of truth. + +### 4.5 Frontend `TailResource` gains two members + +`TailResource` in `src/lib/tail-stream.ts` becomes `"claims" | "remittances" | "activity" | "acks" | "ta1_acks"`. `useTailStream` gains two dispatch cases in its resource switch. `useTailStore` gains two keyed-by-id slices (`acks`, `ta1_acks`) with the same FIFO cap as `claims` / `remittances`. `useMergedTail` adds two slice selectors that read in arrival order from `ackOrder` / `ta1AckOrder` arrays. The 277CA case is intentionally absent — no page consumes it today, and YAGNI says we add it when the page exists. + +### 4.6 Acks page subscribes via the same hook triplet + +`Acks.tsx` calls `useTailStream("acks")` and `useMergedTail("acks", data?.items ?? [])` for the 999 register, then `useTailStream("ta1_acks")` and `useMergedTail("ta1_acks", ta1Items)` inside the `Ta1AcksSection`. Each stream mounts its own `` so the operator sees a `live` / `connecting` / `stalled` indicator independently for 999s and TA1s. The 999 pill sits in the hero next to the existing "Envelope accepted / Mixed envelope" badge (the two read on the same scanline so the operator's eye lands on both at once). The TA1 pill sits in the TA1 section header next to the "Envelope acks" eyebrow. + +The page continues to use the existing `useAcks` / `useTa1Acks` TanStack Query hooks for the initial JSON fetch — the tail subscription is an additive append layer, never a replacement (per the original live-tail spec's locked decisions). + +### 4.7 Handler `event_bus` kwarg removed; parse endpoints thread `event_bus` to the store + +`handle_999` / `handle_ta1` / `handle_277ca` / `handle_835` drop the `event_bus: Optional[object] = None` parameter. They drop the `if event_bus is not None: ...` block too. The handlers do the parse + persist as today; the store does the publish. + +The FastAPI `/api/parse-999` / `/api/parse-ta1` / `/api/parse-277ca` endpoints (which today inline the parse + persist logic) keep their inline code but gain `event_bus=request.app.state.event_bus` on the call to `store.add_*_ack(...)`. The migration of those inline bodies to call the handler functions is out of scope for SP25 — it is the SP27 Task 6 follow-up and is independent of this bug fix. + +## 5. Test impact + +Per the `cyclone-tests` skill, every backend change ships a pytest case and every frontend change ships a vitest sibling. + +Backend: +- `backend/tests/test_store_acks_publish.py` (new) — three red-green-red tests asserting `add_999_ack` / `add_ta1_ack` / `add_277ca_ack` publish the right event kind with a payload that round-trips through `to_ui_ack` / `to_ui_ta1_ack` / the 277CA serializer. Includes a "publish failure does not roll back the row" test. +- `backend/tests/test_api_ack_stream.py` (new) — integration tests for `/api/acks/stream` and `/api/ta1-acks/stream`: snapshot shape, `snapshot_end` ordering, live event lands within one cycle, heartbeat cadence, 404 / 401 paths. +- `backend/tests/test_api_999.py` and `test_api_ta1.py` — existing tests keep passing (regression). A new case asserts that a successful parse fires the event on `app.state.event_bus`. + +Frontend: +- `src/hooks/useTailStream.test.ts` (new) — extends the existing dispatch test with cases for `"acks"` and `"ta1_acks"`. +- `src/hooks/useMergedTail.test.ts` (new or extended) — slice-selector tests for the new resources. +- `src/store/tail-store.test.ts` (new) — `addAck` / `addTa1Ack` FIFO eviction, dedup, and the `reset("acks")` / `reset("ta1_acks")` paths. +- `src/pages/Acks.test.tsx` — existing tests keep passing. Two new cases: "live event lands a new row in the 999 table" and "live event lands a new row in the TA1 table" (both mock `streamTail` via `vi.mock("@/lib/tail-stream", ...)`). +- `src/lib/tail-stream.test.ts` (new or extended) — `KNOWN_TYPES` includes the new event types (or, more precisely, the wire format parser already treats `item` / `snapshot_end` / `heartbeat` / `item_dropped` / `error` symmetrically across resources, so the new resource URLs are the only thing that needs to land). + +## 6. Branch / commit / merge shape + +Per the SP-N increment flow (see `cyclone-spec`): + +- **Branch:** `sp25-ack-live-tail` off `main`. +- **Spec:** this file, committed as `docs(spec): design for SP25 ack live-tail`. +- **Plan:** `docs/superpowers/plans/2026-07-02-cyclone-ack-live-tail.md`, committed as `docs(plan): plan for SP25 ack live-tail`. +- **Implementation commits:** all `feat(sp25): ...` on the branch. +- **PR title:** `SP25 Ack live-tail`. +- **Merge:** single atomic merge commit `merge: SP25 ack live-tail into main` (no squash, no rebase).