docs(sp42): skills cyclone-store/tail — refresh module list, event kinds, streaming endpoint count

This commit is contained in:
Nora
2026-07-08 23:20:42 -06:00
parent 96ed8f5f64
commit 542a55fdbe
2 changed files with 50 additions and 51 deletions
+29 -31
View File
@@ -6,17 +6,9 @@ description: "Cyclone store write-paths, the pubsub event contract (claim_writte
# cyclone-store # cyclone-store
Cyclone persists every parsed X12 batch through one facade, Cyclone persists every parsed X12 batch through one facade,
`CycloneStore` (`backend/src/cyclone/store.py:882`). Every write `CycloneStore` (now exposed via `backend/src/cyclone/store/__init__.py` after the SP21 split landed; the public import surface `from cyclone.store import …` is unchanged). Every write inserts the row AND publishes a pubsub event on the in-process `EventBus` (`backend/src/cyclone/pubsub.py:20`) so live-tail pages see new rows the moment they land. The event contract is the seam between persistence and streaming — a wrong event name silently goes stale.
inserts the row AND publishes a pubsub event on the in-process
`EventBus` (`backend/src/cyclone/pubsub.py:20`) so live-tail pages
see new rows the moment they land. The event contract is the seam
between persistence and streaming — a wrong event name silently
goes stale.
As of this writing: the store is a **single 2412-line module** and As of this writing: the store is a **16-module subpackage** under `backend/src/cyclone/store/` (SP21 split landed). Event kinds in use today: `claim_written`, `remittance_written`, `activity_recorded`, plus the SP-acks-tail additions `ack_written` and `ta1_ack_written` for the `/api/acks/stream` and `/api/ta1-acks/stream` endpoints. Next: **SP43** (after the SP42 doc-pass merges).
the SP21 split into `backend/src/cyclone/store/` is in progress.
Three event kinds: `claim_written`, `remittance_written`,
`activity_recorded`. Next: **SP22**.
## When to use ## When to use
@@ -45,12 +37,11 @@ Three event kinds: `claim_written`, `remittance_written`,
2. **Every write publishes an event.** The event name matches the 2. **Every write publishes an event.** The event name matches the
entity: `claim_written`, `remittance_written`, entity: `claim_written`, `remittance_written`,
`activity_recorded` (the trailing `_recorded` signals a `activity_recorded` (the trailing `_recorded` signals a
non-canonical row — activity events are derived, not first-class; non-canonical row — activity events are derived, not first-class),
current names at `store.py:1072,1081,1096`). New entities get plus `ack_written` and `ta1_ack_written` for the SP-acks-tail
`<entity>_written`; activity-style side rows get streams. New entities get `<entity>_written`; activity-style side
`<entity>_recorded`. Publish is **best-effort** failures are rows get `<entity>_recorded`. Publish is **best-effort**
logged but never roll back the persisted batch failures are logged but never roll back the persisted batch.
(`store.py:1097-1098`).
3. **Snapshot shape.** Each entity has a `to_ui_<entity>` serializer 3. **Snapshot shape.** Each entity has a `to_ui_<entity>` serializer
(plain Python function returning a dict) — currently (plain Python function returning a dict) — currently
`to_ui_claim`, `to_ui_remittance`, `to_ui_claim_from_orm`, `to_ui_claim`, `to_ui_remittance`, `to_ui_claim_from_orm`,
@@ -61,13 +52,15 @@ Three event kinds: `claim_written`, `remittance_written`,
that row (`store.py:1052-1054`). that row (`store.py:1052-1054`).
4. **SP21 boundaries.** Post-split, each domain lives in its own 4. **SP21 boundaries.** Post-split, each domain lives in its own
module under `backend/src/cyclone/store/`: `__init__.py` (facade module under `backend/src/cyclone/store/`: `__init__.py` (facade
+ `CycloneStore` class), `exceptions.py`, `records.py`, + `CycloneStore` class), `acks.py`, `backfill.py`, `backups.py`,
`orm_builders.py`, `ui.py`, `write.py`, `batches.py`, `batches.py`, `claim_acks.py`, `claim_detail.py`, `exceptions.py`,
`claim_detail.py`, `acks.py`, `backups.py`, `inbox.py`, `inbox.py`, `kpis.py`, `orm_builders.py`, `providers.py`,
`providers.py`. Cross-module writes go through `CycloneStore` `records.py`, `resubmissions.py`, `submission_dedup.py`, `ui.py`,
facade methods, not direct module access. The facade re-exports `write.py` — 16 modules total. Cross-module writes go through
every name callers currently import from `cyclone.store`. Full `CycloneStore` facade methods, not direct module access. The
list: `docs/superpowers/plans/2026-06-21-cyclone-store-split.md:25-37`. facade re-exports every name callers currently import from
`cyclone.store`. Full split plan:
`docs/superpowers/plans/2026-06-21-cyclone-store-split.md`.
5. **No business logic in route handlers.** A route handler validates 5. **No business logic in route handlers.** A route handler validates
input, calls `store.<method>(...)`, passes the `event_bus`, input, calls `store.<method>(...)`, passes the `event_bus`,
returns the serialized result. Reconciliation, idempotency returns the serialized result. Reconciliation, idempotency
@@ -77,7 +70,7 @@ Three event kinds: `claim_written`, `remittance_written`,
### A `CycloneStore.add` write — publishes events from inserted rows ### A `CycloneStore.add` write — publishes events from inserted rows
Taken from `backend/src/cyclone/store.py:898-1107`. The method opens Taken from `backend/src/cyclone/store/write.py` (post-SP21). The method opens
a session, inserts rows, then runs a sync `_publish_events_sync` a session, inserts rows, then runs a sync `_publish_events_sync`
after commit so subscribers can immediately re-fetch consistent data. after commit so subscribers can immediately re-fetch consistent data.
@@ -118,12 +111,13 @@ FastAPI handlers to await.
### Backend `/api/<resource>/stream` endpoint — subscribes to the event ### Backend `/api/<resource>/stream` endpoint — subscribes to the event
Taken from `backend/src/cyclone/api.py:1380-1401`. Two phases — eager Streaming endpoints live in `backend/src/cyclone/api_routers/<topic>.py`
post-SP36 (e.g. `api_routers/claims.py`). Two phases — eager
snapshot, then live subscription — wrapped in `StreamingResponse` snapshot, then live subscription — wrapped in `StreamingResponse`
with `media_type="application/x-ndjson"`. with `media_type="application/x-ndjson"`.
```python ```python
@app.get("/api/claims/stream") @router.get("/api/claims/stream")
async def claims_stream(request: Request, ...) -> StreamingResponse: async def claims_stream(request: Request, ...) -> StreamingResponse:
bus: EventBus = request.app.state.event_bus bus: EventBus = request.app.state.event_bus
@@ -139,6 +133,10 @@ async def claims_stream(request: Request, ...) -> StreamingResponse:
return StreamingResponse(gen(), media_type="application/x-ndjson") return StreamingResponse(gen(), media_type="application/x-ndjson")
``` ```
The same shape is repeated in `api_routers/remittances.py`,
`api_routers/activity.py`, `api_routers/acks.py`, and
`api_routers/ta1_acks.py` (five streaming endpoints total).
`_ndjson_line` and `_tail_events` live in `_ndjson_line` and `_tail_events` live in
`backend/src/cyclone/api_helpers.py`. The `["remittance_written"]` `backend/src/cyclone/api_helpers.py`. The `["remittance_written"]`
and `["activity_recorded"]` subscriptions are at `api.py:1891,2002`. and `["activity_recorded"]` subscriptions are at `api.py:1891,2002`.
@@ -172,11 +170,11 @@ def test_publishes_claim_written_event(client: TestClient) -> None:
equivalent `get_*` facade method). equivalent `get_*` facade method).
- **Don't introduce a new event name without updating the subscriber - **Don't introduce a new event name without updating the subscriber
list.** Today every `<entity>_written` event has exactly one list.** Today every `<entity>_written` event has exactly one
consumer — the matching `/api/<resource>/stream` endpoint, consumer — the matching `/api/<resource>/stream` endpoint, now
currently in `backend/src/cyclone/api.py` (claims at `:1398`, split across `api_routers/{claims,remittances,activity,acks,ta1_acks}.py`
remittances at `:1891`, activity at `:2002`). Any future split (post-SP36 split). Any new event must wire its subscription in
into `backend/src/cyclone/api_routers/` must wire the same the matching router and add the resource name to the hook
subscription. triplet in the corresponding page.
- **Don't merge a write method with its event publication into - **Don't merge a write method with its event publication into
separate places.** `_publish_events_sync` lives next to `add` in separate places.** `_publish_events_sync` lives next to `add` in
`store.py:1042-1107` so reviewers see both halves of the contract `store.py:1042-1107` so reviewers see both halves of the contract
+21 -20
View File
@@ -5,14 +5,14 @@ description: "Cyclone live-tail streaming wire format and the useTailStream / us
# cyclone-tail # cyclone-tail
Cyclone keeps the Claims, Remittances, and Activity pages live without Cyclone keeps the Claims, Remittances, Activity, Acks, and Ta1Acks pages live without
polling: every store write publishes an internal EventBus event, the polling: every store write publishes an internal EventBus event, the
page opens a `GET /api/<resource>/stream` HTTP/1.1 chunked-NDJSON page opens a `GET /api/<resource>/stream` HTTP/1.1 chunked-NDJSON
connection, and new rows land in the table the moment they hit the connection, and new rows land in the table the moment they hit the
database. This skill codifies the wire format, the hook triplet, and database. This skill codifies the wire format, the hook triplet, and
the backoff/stall machinery so additions stay consistent with the the backoff/stall machinery so additions stay consistent with the
three streaming pages already shipped (`Claims`, `Remittances`, five streaming pages already shipped (`Claims`, `Remittances`,
`ActivityLog`). `ActivityLog`, `Acks`, `Ta1Acks`).
## When to use ## When to use
@@ -42,8 +42,8 @@ three streaming pages already shipped (`Claims`, `Remittances`,
`item_dropped` (`{"id": "..."}` queue-overflow notice), `error` `item_dropped` (`{"id": "..."}` queue-overflow notice), `error`
(`{"message": "..."}` promoted to a thrown error by the hook). (`{"message": "..."}` promoted to a thrown error by the hook).
Defined at `src/lib/tail-stream.ts:22-44`; emitted by Defined at `src/lib/tail-stream.ts:22-44`; emitted by
`backend/src/cyclone/api.py:1357-2006`. See the streaming routers in `backend/src/cyclone/api_routers/{claims,remittances,activity,acks,ta1_acks}.py`
`references/wire-format.md`. (post-SP36 split). See `references/wire-format.md`.
2. **Hook triplet.** Streaming pages compose three pieces: 2. **Hook triplet.** Streaming pages compose three pieces:
`useTailStream(resource)` (`src/hooks/useTailStream.ts:80` — opens `useTailStream(resource)` (`src/hooks/useTailStream.ts:80` — opens
the stream, drives the backoff/stall state machine, dispatches the stream, drives the backoff/stall state machine, dispatches
@@ -52,8 +52,9 @@ three streaming pages already shipped (`Claims`, `Remittances`,
(`src/hooks/useMergedTail.ts:25` — returns (`src/hooks/useMergedTail.ts:25` — returns
`baseItems + tailSlice` dedup'd against `baseItems`), and a `baseItems + tailSlice` dedup'd against `baseItems`), and a
per-resource initial-fetch hook (`useClaims`, `useRemittances`, per-resource initial-fetch hook (`useClaims`, `useRemittances`,
`useActivity`). The page wires all three — see `useActivity`, `useAcks`, `useTa1Acks`). The page wires all three —
`src/pages/Claims.tsx:87-89`. see `src/pages/Claims.tsx:87-89`. The five resources share the
same triplet; only the resource string differs.
3. **Stall threshold.** 30 seconds of total silence — heartbeats 3. **Stall threshold.** 30 seconds of total silence — heartbeats
included — flips status to `stalled` and surfaces the included — flips status to `stalled` and surfaces the
`↻ Reconnect` button on `<TailStatusPill>`. Constant: `↻ Reconnect` button on `<TailStatusPill>`. Constant:
@@ -67,8 +68,8 @@ three streaming pages already shipped (`Claims`, `Remittances`,
`<TailStatusPill>` from `connecting` to `live` and to reset the `<TailStatusPill>` from `connecting` to `live` and to reset the
reconnect backoff counter (`useTailStream.ts:174-180`). reconnect backoff counter (`useTailStream.ts:174-180`).
5. **Content-Type.** Stream endpoints respond with 5. **Content-Type.** Stream endpoints respond with
`media_type="application/x-ndjson"` — see `media_type="application/x-ndjson"` — set in each
`backend/src/cyclone/api.py:1401,1894,2005`. Frontend sets `api_routers/<resource>.py` stream handler. Frontend sets
`Accept: application/x-ndjson` at `src/lib/tail-stream.ts:67-70`. `Accept: application/x-ndjson` at `src/lib/tail-stream.ts:67-70`.
Never `application/json` for a stream endpoint. Never `application/json` for a stream endpoint.
6. **Backoff.** Transient errors retry with `1s → 2s → 4s → 8s → 16s 6. **Backoff.** Transient errors retry with `1s → 2s → 4s → 8s → 16s
@@ -108,7 +109,7 @@ export function ClaimsPage() {
### Backend `/api/foo/stream` endpoint ### Backend `/api/foo/stream` endpoint
Pattern from `backend/src/cyclone/api.py:1357-1401`. Register BEFORE Pattern from `backend/src/cyclone/api_routers/claims.py`. Register BEFORE
`/api/foo/{foo_id}` so the literal `stream` segment doesn't match as `/api/foo/{foo_id}` so the literal `stream` segment doesn't match as
an id. Two phases — eager snapshot, then live subscription — wrapped an id. Two phases — eager snapshot, then live subscription — wrapped
in `StreamingResponse` with `media_type="application/x-ndjson"`. in `StreamingResponse` with `media_type="application/x-ndjson"`.
@@ -160,7 +161,7 @@ takes `status` + `lastEventAt` from `useTailStream` plus
- **Don't change the wire format on one endpoint without updating - **Don't change the wire format on one endpoint without updating
the others.** The parser (`src/lib/tail-stream.ts`) and the hook the others.** The parser (`src/lib/tail-stream.ts`) and the hook
dispatch switch (`useTailStream.ts:173-195`) are shared across all dispatch switch (`useTailStream.ts:173-195`) are shared across all
three live streams. Adding a new event type means updating five live streams. Adding a new event type means updating
`TailEvent`, `KNOWN_TYPES`, the dispatch `case`, the `armStall` `TailEvent`, `KNOWN_TYPES`, the dispatch `case`, the `armStall`
re-arm list, and the backend emitter — in that order. re-arm list, and the backend emitter — in that order.
- **Don't emit `item` events before `snapshot_end`.** The hook uses - **Don't emit `item` events before `snapshot_end`.** The hook uses
@@ -168,7 +169,7 @@ takes `status` + `lastEventAt` from `useTailStream` plus
`connecting` to `live` and to reset the reconnect backoff counter. `connecting` to `live` and to reset the reconnect backoff counter.
Emitting `item`s first lands rows in `useTailStore` but the UI Emitting `item`s first lands rows in `useTailStore` but the UI
still reads "Connecting" — operators see a flash of stale state. still reads "Connecting" — operators see a flash of stale state.
Emit snapshot, then `snapshot_end`, then live (`api.py:1386-1401`). Emit snapshot, then `snapshot_end`, then live.
- **Don't change the 30s stall threshold without updating the README - **Don't change the 30s stall threshold without updating the README
"Status pill" table.** The constant "Status pill" table.** The constant
(`STALL_TIMEOUT_MS = 30_000` at `useTailStream.ts:53`) is the (`STALL_TIMEOUT_MS = 30_000` at `useTailStream.ts:53`) is the
@@ -176,7 +177,7 @@ takes `status` + `lastEventAt` from `useTailStream` plus
edit at `README.md:118`. edit at `README.md:118`.
- **Don't add `useTailStream` calls from `useFoo.ts` hooks.** - **Don't add `useTailStream` calls from `useFoo.ts` hooks.**
Streaming connections belong on the page (`src/pages/Claims.tsx`, Streaming connections belong on the page (`src/pages/Claims.tsx`,
`Remittances.tsx`, `ActivityLog.tsx`). Hoisting into `useFoo` `Remittances.tsx`, `ActivityLog.tsx`, `Acks.tsx`, `Ta1Acks.tsx`). Hoisting into `useFoo`
couples the lifecycle to whoever mounts it and breaks couples the lifecycle to whoever mounts it and breaks
one-resource-one-page ownership. one-resource-one-page ownership.
@@ -185,16 +186,16 @@ takes `status` + `lastEventAt` from `useTailStream` plus
- **`cyclone-frontend-page`** — page components live in `src/pages/`; - **`cyclone-frontend-page`** — page components live in `src/pages/`;
load when adding or refactoring a streaming page to confirm the load when adding or refactoring a streaming page to confirm the
route + `PageHeader` + table conventions. route + `PageHeader` + table conventions.
- **`cyclone-api-router`** — endpoint conventions (`api_routers/` for - **`cyclone-api-router`** — endpoint conventions (`api_routers/`
resource-group routers, `api.py` for the live-tail endpoints); load hosts the streaming endpoints post-SP36, alongside the other
when adding or changing an HTTP endpoint that surfaces streamed or resource-group routers); load when adding or changing an HTTP
paginated data. endpoint that surfaces streamed or paginated data.
- **`cyclone-store`** — write-path conventions in `store.py` and the - **`cyclone-store`** — write-path conventions in `store.py` and the
pubsub event contract (`claim_written`, `remittance_written`, pubsub event contract (`claim_written`, `remittance_written`,
`activity_recorded`); load when adding a new entity whose writes `activity_recorded`); load when adding a new entity whose writes
should fan out to a stream endpoint. should fan out to a stream endpoint.
- **`cyclone-tests`** — frontend `*.test.tsx` siblings cover - **`cyclone-tests`** — frontend `*.test.tsx` siblings cover
`useTailStream`, `useMergedTail`, `TailStatusPill`; backend `useTailStream`, `useMergedTail`, `TailStatusPill`; backend
`test_api_stream_live.py` covers the three live-tail endpoints; `test_api_stream_live.py` covers the five live-tail endpoints
load when the increment changes wire-format behavior or adds a (claims, remittances, activity, acks, ta1-acks); load when the
streaming hook. increment changes wire-format behavior or adds a streaming hook.