docs(sp42): skills cyclone-store/tail — refresh module list, event kinds, streaming endpoint count
This commit is contained in:
@@ -6,17 +6,9 @@ description: "Cyclone store write-paths, the pubsub event contract (claim_writte
|
|||||||
# cyclone-store
|
# cyclone-store
|
||||||
|
|
||||||
Cyclone persists every parsed X12 batch through one facade,
|
Cyclone persists every parsed X12 batch through one facade,
|
||||||
`CycloneStore` (`backend/src/cyclone/store.py:882`). Every write
|
`CycloneStore` (now exposed via `backend/src/cyclone/store/__init__.py` after the SP21 split landed; the public import surface `from cyclone.store import …` is unchanged). Every write inserts the row AND publishes a pubsub event on the in-process `EventBus` (`backend/src/cyclone/pubsub.py:20`) so live-tail pages see new rows the moment they land. The event contract is the seam between persistence and streaming — a wrong event name silently goes stale.
|
||||||
inserts the row AND publishes a pubsub event on the in-process
|
|
||||||
`EventBus` (`backend/src/cyclone/pubsub.py:20`) so live-tail pages
|
|
||||||
see new rows the moment they land. The event contract is the seam
|
|
||||||
between persistence and streaming — a wrong event name silently
|
|
||||||
goes stale.
|
|
||||||
|
|
||||||
As of this writing: the store is a **single 2412-line module** and
|
As of this writing: the store is a **16-module subpackage** under `backend/src/cyclone/store/` (SP21 split landed). Event kinds in use today: `claim_written`, `remittance_written`, `activity_recorded`, plus the SP-acks-tail additions `ack_written` and `ta1_ack_written` for the `/api/acks/stream` and `/api/ta1-acks/stream` endpoints. Next: **SP43** (after the SP42 doc-pass merges).
|
||||||
the SP21 split into `backend/src/cyclone/store/` is in progress.
|
|
||||||
Three event kinds: `claim_written`, `remittance_written`,
|
|
||||||
`activity_recorded`. Next: **SP22**.
|
|
||||||
|
|
||||||
## When to use
|
## When to use
|
||||||
|
|
||||||
@@ -45,12 +37,11 @@ Three event kinds: `claim_written`, `remittance_written`,
|
|||||||
2. **Every write publishes an event.** The event name matches the
|
2. **Every write publishes an event.** The event name matches the
|
||||||
entity: `claim_written`, `remittance_written`,
|
entity: `claim_written`, `remittance_written`,
|
||||||
`activity_recorded` (the trailing `_recorded` signals a
|
`activity_recorded` (the trailing `_recorded` signals a
|
||||||
non-canonical row — activity events are derived, not first-class;
|
non-canonical row — activity events are derived, not first-class),
|
||||||
current names at `store.py:1072,1081,1096`). New entities get
|
plus `ack_written` and `ta1_ack_written` for the SP-acks-tail
|
||||||
`<entity>_written`; activity-style side rows get
|
streams. New entities get `<entity>_written`; activity-style side
|
||||||
`<entity>_recorded`. Publish is **best-effort** — failures are
|
rows get `<entity>_recorded`. Publish is **best-effort** —
|
||||||
logged but never roll back the persisted batch
|
failures are logged but never roll back the persisted batch.
|
||||||
(`store.py:1097-1098`).
|
|
||||||
3. **Snapshot shape.** Each entity has a `to_ui_<entity>` serializer
|
3. **Snapshot shape.** Each entity has a `to_ui_<entity>` serializer
|
||||||
(plain Python function returning a dict) — currently
|
(plain Python function returning a dict) — currently
|
||||||
`to_ui_claim`, `to_ui_remittance`, `to_ui_claim_from_orm`,
|
`to_ui_claim`, `to_ui_remittance`, `to_ui_claim_from_orm`,
|
||||||
@@ -61,13 +52,15 @@ Three event kinds: `claim_written`, `remittance_written`,
|
|||||||
that row (`store.py:1052-1054`).
|
that row (`store.py:1052-1054`).
|
||||||
4. **SP21 boundaries.** Post-split, each domain lives in its own
|
4. **SP21 boundaries.** Post-split, each domain lives in its own
|
||||||
module under `backend/src/cyclone/store/`: `__init__.py` (facade
|
module under `backend/src/cyclone/store/`: `__init__.py` (facade
|
||||||
+ `CycloneStore` class), `exceptions.py`, `records.py`,
|
+ `CycloneStore` class), `acks.py`, `backfill.py`, `backups.py`,
|
||||||
`orm_builders.py`, `ui.py`, `write.py`, `batches.py`,
|
`batches.py`, `claim_acks.py`, `claim_detail.py`, `exceptions.py`,
|
||||||
`claim_detail.py`, `acks.py`, `backups.py`, `inbox.py`,
|
`inbox.py`, `kpis.py`, `orm_builders.py`, `providers.py`,
|
||||||
`providers.py`. Cross-module writes go through `CycloneStore`
|
`records.py`, `resubmissions.py`, `submission_dedup.py`, `ui.py`,
|
||||||
facade methods, not direct module access. The facade re-exports
|
`write.py` — 16 modules total. Cross-module writes go through
|
||||||
every name callers currently import from `cyclone.store`. Full
|
`CycloneStore` facade methods, not direct module access. The
|
||||||
list: `docs/superpowers/plans/2026-06-21-cyclone-store-split.md:25-37`.
|
facade re-exports every name callers currently import from
|
||||||
|
`cyclone.store`. Full split plan:
|
||||||
|
`docs/superpowers/plans/2026-06-21-cyclone-store-split.md`.
|
||||||
5. **No business logic in route handlers.** A route handler validates
|
5. **No business logic in route handlers.** A route handler validates
|
||||||
input, calls `store.<method>(...)`, passes the `event_bus`,
|
input, calls `store.<method>(...)`, passes the `event_bus`,
|
||||||
returns the serialized result. Reconciliation, idempotency
|
returns the serialized result. Reconciliation, idempotency
|
||||||
@@ -77,7 +70,7 @@ Three event kinds: `claim_written`, `remittance_written`,
|
|||||||
|
|
||||||
### A `CycloneStore.add` write — publishes events from inserted rows
|
### A `CycloneStore.add` write — publishes events from inserted rows
|
||||||
|
|
||||||
Taken from `backend/src/cyclone/store.py:898-1107`. The method opens
|
Taken from `backend/src/cyclone/store/write.py` (post-SP21). The method opens
|
||||||
a session, inserts rows, then runs a sync `_publish_events_sync`
|
a session, inserts rows, then runs a sync `_publish_events_sync`
|
||||||
after commit so subscribers can immediately re-fetch consistent data.
|
after commit so subscribers can immediately re-fetch consistent data.
|
||||||
|
|
||||||
@@ -118,12 +111,13 @@ FastAPI handlers to await.
|
|||||||
|
|
||||||
### Backend `/api/<resource>/stream` endpoint — subscribes to the event
|
### Backend `/api/<resource>/stream` endpoint — subscribes to the event
|
||||||
|
|
||||||
Taken from `backend/src/cyclone/api.py:1380-1401`. Two phases — eager
|
Streaming endpoints live in `backend/src/cyclone/api_routers/<topic>.py`
|
||||||
|
post-SP36 (e.g. `api_routers/claims.py`). Two phases — eager
|
||||||
snapshot, then live subscription — wrapped in `StreamingResponse`
|
snapshot, then live subscription — wrapped in `StreamingResponse`
|
||||||
with `media_type="application/x-ndjson"`.
|
with `media_type="application/x-ndjson"`.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
@app.get("/api/claims/stream")
|
@router.get("/api/claims/stream")
|
||||||
async def claims_stream(request: Request, ...) -> StreamingResponse:
|
async def claims_stream(request: Request, ...) -> StreamingResponse:
|
||||||
bus: EventBus = request.app.state.event_bus
|
bus: EventBus = request.app.state.event_bus
|
||||||
|
|
||||||
@@ -139,6 +133,10 @@ async def claims_stream(request: Request, ...) -> StreamingResponse:
|
|||||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The same shape is repeated in `api_routers/remittances.py`,
|
||||||
|
`api_routers/activity.py`, `api_routers/acks.py`, and
|
||||||
|
`api_routers/ta1_acks.py` (five streaming endpoints total).
|
||||||
|
|
||||||
`_ndjson_line` and `_tail_events` live in
|
`_ndjson_line` and `_tail_events` live in
|
||||||
`backend/src/cyclone/api_helpers.py`. The `["remittance_written"]`
|
`backend/src/cyclone/api_helpers.py`. The `["remittance_written"]`
|
||||||
and `["activity_recorded"]` subscriptions are at `api.py:1891,2002`.
|
and `["activity_recorded"]` subscriptions are at `api.py:1891,2002`.
|
||||||
@@ -172,11 +170,11 @@ def test_publishes_claim_written_event(client: TestClient) -> None:
|
|||||||
equivalent `get_*` facade method).
|
equivalent `get_*` facade method).
|
||||||
- **Don't introduce a new event name without updating the subscriber
|
- **Don't introduce a new event name without updating the subscriber
|
||||||
list.** Today every `<entity>_written` event has exactly one
|
list.** Today every `<entity>_written` event has exactly one
|
||||||
consumer — the matching `/api/<resource>/stream` endpoint,
|
consumer — the matching `/api/<resource>/stream` endpoint, now
|
||||||
currently in `backend/src/cyclone/api.py` (claims at `:1398`,
|
split across `api_routers/{claims,remittances,activity,acks,ta1_acks}.py`
|
||||||
remittances at `:1891`, activity at `:2002`). Any future split
|
(post-SP36 split). Any new event must wire its subscription in
|
||||||
into `backend/src/cyclone/api_routers/` must wire the same
|
the matching router and add the resource name to the hook
|
||||||
subscription.
|
triplet in the corresponding page.
|
||||||
- **Don't merge a write method with its event publication into
|
- **Don't merge a write method with its event publication into
|
||||||
separate places.** `_publish_events_sync` lives next to `add` in
|
separate places.** `_publish_events_sync` lives next to `add` in
|
||||||
`store.py:1042-1107` so reviewers see both halves of the contract
|
`store.py:1042-1107` so reviewers see both halves of the contract
|
||||||
|
|||||||
@@ -5,14 +5,14 @@ description: "Cyclone live-tail streaming wire format and the useTailStream / us
|
|||||||
|
|
||||||
# cyclone-tail
|
# cyclone-tail
|
||||||
|
|
||||||
Cyclone keeps the Claims, Remittances, and Activity pages live without
|
Cyclone keeps the Claims, Remittances, Activity, Acks, and Ta1Acks pages live without
|
||||||
polling: every store write publishes an internal EventBus event, the
|
polling: every store write publishes an internal EventBus event, the
|
||||||
page opens a `GET /api/<resource>/stream` HTTP/1.1 chunked-NDJSON
|
page opens a `GET /api/<resource>/stream` HTTP/1.1 chunked-NDJSON
|
||||||
connection, and new rows land in the table the moment they hit the
|
connection, and new rows land in the table the moment they hit the
|
||||||
database. This skill codifies the wire format, the hook triplet, and
|
database. This skill codifies the wire format, the hook triplet, and
|
||||||
the backoff/stall machinery so additions stay consistent with the
|
the backoff/stall machinery so additions stay consistent with the
|
||||||
three streaming pages already shipped (`Claims`, `Remittances`,
|
five streaming pages already shipped (`Claims`, `Remittances`,
|
||||||
`ActivityLog`).
|
`ActivityLog`, `Acks`, `Ta1Acks`).
|
||||||
|
|
||||||
## When to use
|
## When to use
|
||||||
|
|
||||||
@@ -42,8 +42,8 @@ three streaming pages already shipped (`Claims`, `Remittances`,
|
|||||||
`item_dropped` (`{"id": "..."}` queue-overflow notice), `error`
|
`item_dropped` (`{"id": "..."}` queue-overflow notice), `error`
|
||||||
(`{"message": "..."}` promoted to a thrown error by the hook).
|
(`{"message": "..."}` promoted to a thrown error by the hook).
|
||||||
Defined at `src/lib/tail-stream.ts:22-44`; emitted by
|
Defined at `src/lib/tail-stream.ts:22-44`; emitted by
|
||||||
`backend/src/cyclone/api.py:1357-2006`. See
|
the streaming routers in `backend/src/cyclone/api_routers/{claims,remittances,activity,acks,ta1_acks}.py`
|
||||||
`references/wire-format.md`.
|
(post-SP36 split). See `references/wire-format.md`.
|
||||||
2. **Hook triplet.** Streaming pages compose three pieces:
|
2. **Hook triplet.** Streaming pages compose three pieces:
|
||||||
`useTailStream(resource)` (`src/hooks/useTailStream.ts:80` — opens
|
`useTailStream(resource)` (`src/hooks/useTailStream.ts:80` — opens
|
||||||
the stream, drives the backoff/stall state machine, dispatches
|
the stream, drives the backoff/stall state machine, dispatches
|
||||||
@@ -52,8 +52,9 @@ three streaming pages already shipped (`Claims`, `Remittances`,
|
|||||||
(`src/hooks/useMergedTail.ts:25` — returns
|
(`src/hooks/useMergedTail.ts:25` — returns
|
||||||
`baseItems + tailSlice` dedup'd against `baseItems`), and a
|
`baseItems + tailSlice` dedup'd against `baseItems`), and a
|
||||||
per-resource initial-fetch hook (`useClaims`, `useRemittances`,
|
per-resource initial-fetch hook (`useClaims`, `useRemittances`,
|
||||||
`useActivity`). The page wires all three — see
|
`useActivity`, `useAcks`, `useTa1Acks`). The page wires all three —
|
||||||
`src/pages/Claims.tsx:87-89`.
|
see `src/pages/Claims.tsx:87-89`. The five resources share the
|
||||||
|
same triplet; only the resource string differs.
|
||||||
3. **Stall threshold.** 30 seconds of total silence — heartbeats
|
3. **Stall threshold.** 30 seconds of total silence — heartbeats
|
||||||
included — flips status to `stalled` and surfaces the
|
included — flips status to `stalled` and surfaces the
|
||||||
`↻ Reconnect` button on `<TailStatusPill>`. Constant:
|
`↻ Reconnect` button on `<TailStatusPill>`. Constant:
|
||||||
@@ -67,8 +68,8 @@ three streaming pages already shipped (`Claims`, `Remittances`,
|
|||||||
`<TailStatusPill>` from `connecting` to `live` and to reset the
|
`<TailStatusPill>` from `connecting` to `live` and to reset the
|
||||||
reconnect backoff counter (`useTailStream.ts:174-180`).
|
reconnect backoff counter (`useTailStream.ts:174-180`).
|
||||||
5. **Content-Type.** Stream endpoints respond with
|
5. **Content-Type.** Stream endpoints respond with
|
||||||
`media_type="application/x-ndjson"` — see
|
`media_type="application/x-ndjson"` — set in each
|
||||||
`backend/src/cyclone/api.py:1401,1894,2005`. Frontend sets
|
`api_routers/<resource>.py` stream handler. Frontend sets
|
||||||
`Accept: application/x-ndjson` at `src/lib/tail-stream.ts:67-70`.
|
`Accept: application/x-ndjson` at `src/lib/tail-stream.ts:67-70`.
|
||||||
Never `application/json` for a stream endpoint.
|
Never `application/json` for a stream endpoint.
|
||||||
6. **Backoff.** Transient errors retry with `1s → 2s → 4s → 8s → 16s
|
6. **Backoff.** Transient errors retry with `1s → 2s → 4s → 8s → 16s
|
||||||
@@ -108,7 +109,7 @@ export function ClaimsPage() {
|
|||||||
|
|
||||||
### Backend `/api/foo/stream` endpoint
|
### Backend `/api/foo/stream` endpoint
|
||||||
|
|
||||||
Pattern from `backend/src/cyclone/api.py:1357-1401`. Register BEFORE
|
Pattern from `backend/src/cyclone/api_routers/claims.py`. Register BEFORE
|
||||||
`/api/foo/{foo_id}` so the literal `stream` segment doesn't match as
|
`/api/foo/{foo_id}` so the literal `stream` segment doesn't match as
|
||||||
an id. Two phases — eager snapshot, then live subscription — wrapped
|
an id. Two phases — eager snapshot, then live subscription — wrapped
|
||||||
in `StreamingResponse` with `media_type="application/x-ndjson"`.
|
in `StreamingResponse` with `media_type="application/x-ndjson"`.
|
||||||
@@ -160,7 +161,7 @@ takes `status` + `lastEventAt` from `useTailStream` plus
|
|||||||
- **Don't change the wire format on one endpoint without updating
|
- **Don't change the wire format on one endpoint without updating
|
||||||
the others.** The parser (`src/lib/tail-stream.ts`) and the hook
|
the others.** The parser (`src/lib/tail-stream.ts`) and the hook
|
||||||
dispatch switch (`useTailStream.ts:173-195`) are shared across all
|
dispatch switch (`useTailStream.ts:173-195`) are shared across all
|
||||||
three live streams. Adding a new event type means updating
|
five live streams. Adding a new event type means updating
|
||||||
`TailEvent`, `KNOWN_TYPES`, the dispatch `case`, the `armStall`
|
`TailEvent`, `KNOWN_TYPES`, the dispatch `case`, the `armStall`
|
||||||
re-arm list, and the backend emitter — in that order.
|
re-arm list, and the backend emitter — in that order.
|
||||||
- **Don't emit `item` events before `snapshot_end`.** The hook uses
|
- **Don't emit `item` events before `snapshot_end`.** The hook uses
|
||||||
@@ -168,7 +169,7 @@ takes `status` + `lastEventAt` from `useTailStream` plus
|
|||||||
`connecting` to `live` and to reset the reconnect backoff counter.
|
`connecting` to `live` and to reset the reconnect backoff counter.
|
||||||
Emitting `item`s first lands rows in `useTailStore` but the UI
|
Emitting `item`s first lands rows in `useTailStore` but the UI
|
||||||
still reads "Connecting" — operators see a flash of stale state.
|
still reads "Connecting" — operators see a flash of stale state.
|
||||||
Emit snapshot, then `snapshot_end`, then live (`api.py:1386-1401`).
|
Emit snapshot, then `snapshot_end`, then live.
|
||||||
- **Don't change the 30s stall threshold without updating the README
|
- **Don't change the 30s stall threshold without updating the README
|
||||||
"Status pill" table.** The constant
|
"Status pill" table.** The constant
|
||||||
(`STALL_TIMEOUT_MS = 30_000` at `useTailStream.ts:53`) is the
|
(`STALL_TIMEOUT_MS = 30_000` at `useTailStream.ts:53`) is the
|
||||||
@@ -176,7 +177,7 @@ takes `status` + `lastEventAt` from `useTailStream` plus
|
|||||||
edit at `README.md:118`.
|
edit at `README.md:118`.
|
||||||
- **Don't add `useTailStream` calls from `useFoo.ts` hooks.**
|
- **Don't add `useTailStream` calls from `useFoo.ts` hooks.**
|
||||||
Streaming connections belong on the page (`src/pages/Claims.tsx`,
|
Streaming connections belong on the page (`src/pages/Claims.tsx`,
|
||||||
`Remittances.tsx`, `ActivityLog.tsx`). Hoisting into `useFoo`
|
`Remittances.tsx`, `ActivityLog.tsx`, `Acks.tsx`, `Ta1Acks.tsx`). Hoisting into `useFoo`
|
||||||
couples the lifecycle to whoever mounts it and breaks
|
couples the lifecycle to whoever mounts it and breaks
|
||||||
one-resource-one-page ownership.
|
one-resource-one-page ownership.
|
||||||
|
|
||||||
@@ -185,16 +186,16 @@ takes `status` + `lastEventAt` from `useTailStream` plus
|
|||||||
- **`cyclone-frontend-page`** — page components live in `src/pages/`;
|
- **`cyclone-frontend-page`** — page components live in `src/pages/`;
|
||||||
load when adding or refactoring a streaming page to confirm the
|
load when adding or refactoring a streaming page to confirm the
|
||||||
route + `PageHeader` + table conventions.
|
route + `PageHeader` + table conventions.
|
||||||
- **`cyclone-api-router`** — endpoint conventions (`api_routers/` for
|
- **`cyclone-api-router`** — endpoint conventions (`api_routers/`
|
||||||
resource-group routers, `api.py` for the live-tail endpoints); load
|
hosts the streaming endpoints post-SP36, alongside the other
|
||||||
when adding or changing an HTTP endpoint that surfaces streamed or
|
resource-group routers); load when adding or changing an HTTP
|
||||||
paginated data.
|
endpoint that surfaces streamed or paginated data.
|
||||||
- **`cyclone-store`** — write-path conventions in `store.py` and the
|
- **`cyclone-store`** — write-path conventions in `store.py` and the
|
||||||
pubsub event contract (`claim_written`, `remittance_written`,
|
pubsub event contract (`claim_written`, `remittance_written`,
|
||||||
`activity_recorded`); load when adding a new entity whose writes
|
`activity_recorded`); load when adding a new entity whose writes
|
||||||
should fan out to a stream endpoint.
|
should fan out to a stream endpoint.
|
||||||
- **`cyclone-tests`** — frontend `*.test.tsx` siblings cover
|
- **`cyclone-tests`** — frontend `*.test.tsx` siblings cover
|
||||||
`useTailStream`, `useMergedTail`, `TailStatusPill`; backend
|
`useTailStream`, `useMergedTail`, `TailStatusPill`; backend
|
||||||
`test_api_stream_live.py` covers the three live-tail endpoints;
|
`test_api_stream_live.py` covers the five live-tail endpoints
|
||||||
load when the increment changes wire-format behavior or adds a
|
(claims, remittances, activity, acks, ta1-acks); load when the
|
||||||
streaming hook.
|
increment changes wire-format behavior or adds a streaming hook.
|
||||||
Reference in New Issue
Block a user