Compare commits
9 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Tyler
|
804e557a49 | ||
|
Tyler
|
ff43f90156 | ||
|
Tyler
|
ea64e6e0f0 | ||
|
Tyler
|
6ce638553b | ||
|
Tyler
|
e63be87ba9 | ||
|
Tyler
|
6aa440d3e4 | ||
|
Tyler
|
45cafbdb32 | ||
|
Tyler
|
bbf89c9dd8 | ||
|
Tyler
|
d25f00ac58 |
@@ -1,13 +0,0 @@
|
||||
# Repo-root .dockerignore — applies to `docker compose build` (which builds
|
||||
# both backend and frontend contexts from the repo root). Excludes anything
|
||||
# that should never end up in a build context.
|
||||
|
||||
.worktrees/
|
||||
.git/
|
||||
.github/
|
||||
docs/prodfiles/
|
||||
*.production.txt
|
||||
node_modules/
|
||||
dist/
|
||||
.venv/
|
||||
.superpowers/brainstorm/
|
||||
+4
-17
@@ -1,20 +1,7 @@
|
||||
# Cyclone — environment configuration
|
||||
# Copy this file to `.env.local` and fill in values for your environment.
|
||||
|
||||
# Required on first boot. Cyclone refuses to start without these unless
|
||||
# at least one user already exists (e.g. seeded via `python -m cyclone users create`).
|
||||
# Min 12 chars for password.
|
||||
CYCLONE_ADMIN_USERNAME=admin
|
||||
CYCLONE_ADMIN_PASSWORD=change-me-to-a-strong-password-min-12-chars
|
||||
|
||||
# Base URL for the Python (FastAPI) backend. Leave empty for the
|
||||
# Docker deployment (nginx proxies /api/* to backend on compose network).
|
||||
VITE_API_BASE_URL=
|
||||
|
||||
# Optional. Set to 1 if you're behind an HTTPS reverse proxy and want
|
||||
# the session cookie to include the Secure flag.
|
||||
# CYCLONE_BEHIND_HTTPS=1
|
||||
|
||||
# Optional. Set to 1 to disable auth entirely (DEV ONLY). When set,
|
||||
# the backend auto-grants admin access without checking credentials.
|
||||
# CYCLONE_AUTH_DISABLED=0
|
||||
# Base URL for the Python (FastAPI) backend that powers the Upload page and
|
||||
# the real /api/parse-837 + /api/parse-835 endpoints. Leave empty to keep
|
||||
# the in-memory sample data store and disable real EDI parsing.
|
||||
VITE_API_BASE_URL=http://localhost:8000
|
||||
|
||||
+2
-7
@@ -35,10 +35,5 @@ claims_output/
|
||||
# Worktrees (subagent-driven development)
|
||||
.worktrees/
|
||||
|
||||
# Brainstorm session artifacts (visual companion mockups, events, server state).
|
||||
# Skills under .superpowers/skills/ are committed project-scoped guidance.
|
||||
.superpowers/brainstorm/
|
||||
|
||||
# SP33+ scratch / production-data ingest. Generated artifacts live
|
||||
# here only — the source EDI sits under docs/prodfiles/.
|
||||
ingest/
|
||||
# Brainstorm session artifacts (visual companion mockups, events, server state)
|
||||
.superpowers/
|
||||
|
||||
@@ -1,193 +0,0 @@
|
||||
---
|
||||
name: cyclone-api-router
|
||||
description: "Cyclone FastAPI router conventions (api_routers/, api_helpers.py, response shapes, error envelopes). Use when: adding or changing an HTTP endpoint, splitting a route out of api.py, or wiring a new helper into api_helpers.py."
|
||||
---
|
||||
|
||||
# cyclone-api-router
|
||||
|
||||
Cyclone splits its FastAPI surface two ways: small resource-group
|
||||
routers live in `backend/src/cyclone/api_routers/<topic>.py` and are
|
||||
mounted bare into `api.py`; the high-traffic streaming and parse
|
||||
endpoints still live as top-level decorators in `api.py` itself. This
|
||||
skill codifies the conventions so additions stay consistent with the
|
||||
four routers already shipped (`acks`, `admin`, `health`, `ta1_acks`).
|
||||
|
||||
As of this writing: **4 router modules** under
|
||||
`backend/src/cyclone/api_routers/`, **one shared helpers module** at
|
||||
`backend/src/cyclone/api_helpers.py` (248 lines, NDJSON primitives +
|
||||
content negotiation + `tail_events`), and ~30 routes still inlined
|
||||
in `backend/src/cyclone/api.py`. The next refactor target is the
|
||||
parse endpoints.
|
||||
|
||||
## Auth gate (SP24)
|
||||
|
||||
Every router declared in `backend/src/cyclone/api_routers/` **must** carry `dependencies=[Depends(matrix_gate)]` at the `APIRouter(...)` declaration — not on each individual endpoint. The gate lives at `backend/src/cyclone/auth/deps.py:107` and the role matrix is at `backend/src/cyclone/auth/permissions.py`. The roles are `admin / user / viewer`; `matrix_gate` returns 401 when there's no session and 403 when the role is below the endpoint's required role. When `AUTH_DISABLED` is True (conftest autouse fixture flips it; `CYCLONE_AUTH_DISABLED=1` in prod-by-mistake), the gate short-circuits to a synthetic admin — see the SP24 spec for the threat-model implications. New routers get the gate by default; the auth-aware convention is `router = APIRouter(dependencies=[Depends(matrix_gate)])`.
|
||||
|
||||
## When to use
|
||||
|
||||
- **Adding an endpoint.** You're adding a new GET / POST handler —
|
||||
you need to know whether it belongs in `api.py` (parse / streaming)
|
||||
or in a new router under `api_routers/`, and what the response
|
||||
shape and test file conventions look like.
|
||||
- **Splitting a route.** You're moving a route out of `api.py` into a
|
||||
dedicated `api_routers/<topic>.py` module and need the import /
|
||||
mounting rules (`from cyclone.api_routers import <topic>` then
|
||||
`app.include_router(<topic>.router)`).
|
||||
- **Adding a helper.** You're wiring a new function into
|
||||
`api_helpers.py` (NDJSON primitive, content-negotiation probe,
|
||||
tail-event helper) and need to keep it private to the API layer.
|
||||
- **Defining an error response.** You're raising from a route handler
|
||||
and need the conventional `HTTPException(status_code=..., detail=...)`
|
||||
shape used everywhere else in the API surface.
|
||||
|
||||
## Conventions
|
||||
|
||||
1. **No new top-level routes in `api.py` for resource groups.** Any
|
||||
endpoint grouped under a resource (`/api/<resource>` and its
|
||||
`/{id}` detail) lives in `backend/src/cyclone/api_routers/<topic>.py`
|
||||
as an `APIRouter`. The streaming list endpoints (`/api/claims/stream`,
|
||||
`/api/remittances/stream`, `/api/activity/stream`) and the parse
|
||||
endpoints (`/api/parse-*`) currently stay in `api.py` because they
|
||||
span multiple store modules — don't move them unless you're also
|
||||
restructuring the store split.
|
||||
2. **Reuse `api_helpers.py`.** NDJSON primitives (`ndjson_line`,
|
||||
`ndjson_stream_list`, `ndjson_stream_837`, `ndjson_stream_835`),
|
||||
content negotiation (`client_wants_json`, `wants_ndjson`), the
|
||||
strict / `raw_segments` rewrites (`strict_rewrite_837`,
|
||||
`strict_rewrite_835`, `drop_raw_segments_837`, `drop_raw_segments_835`),
|
||||
and the shared live-tail generator (`tail_events`,
|
||||
`heartbeat_seconds`) all live there. Don't duplicate them in a
|
||||
router. The module's docstring (`api_helpers.py:1-18`) declares it
|
||||
private to the API layer — no business logic, no DB writes.
|
||||
3. **Response shape.** Every successful response is a **plain dict**
|
||||
produced by a per-router `<entity>_to_ui(row)` helper (see
|
||||
`_ack_to_ui` at `api_routers/acks.py:29-48` and `_ta1_to_ui` at
|
||||
`api_routers/ta1_acks.py:23-37`). This dict shape **must** match
|
||||
the matching `<entity>_written` event payload so live-tail pages
|
||||
don't drift (see `cyclone-store` for the serializer contract).
|
||||
Errors use FastAPI's `HTTPException` with a `detail` dict of the
|
||||
form `{"error": "<Title>", "detail": "<message>"}` —
|
||||
`acks.py:85-88`, `ta1_acks.py:72`. There is **no** shared
|
||||
`ErrorEnvelope` Pydantic model; the `detail` dict is the contract.
|
||||
4. **Mounting.** Routers are mounted bare in `api.py:251-256` —
|
||||
`app.include_router(<name>.router)` with **no** `prefix=`
|
||||
argument. Each `@router.<verb>` decorator carries the **full**
|
||||
`/api/<resource>` path itself (see `acks.py:51,75`,
|
||||
`ta1_acks.py:49,67`, `admin.py:23`, `health.py:28`). The
|
||||
`router = APIRouter()` declaration carries no `tags=` either —
|
||||
keep it minimal.
|
||||
5. **Streaming endpoints.** Use
|
||||
`StreamingResponse(media_type="application/x-ndjson")` and feed it
|
||||
either `ndjson_stream_list(items, total, returned, has_more)` (for
|
||||
list pages) or `tail_events(request, bus, kinds)` (for live-tail
|
||||
pages). See `acks.py:62-66` for the list-stream skeleton and
|
||||
`api.py:1357-1401` for the full live-tail pattern (snapshot →
|
||||
`snapshot_end` → subscription → heartbeats). See `cyclone-tail`
|
||||
for the wire format.
|
||||
6. **Tests.** Every new endpoint gets a `test_api_<topic>_<verb>.py`
|
||||
under `backend/tests/` (see `cyclone-tests` for the naming
|
||||
convention + autouse `conftest.py`). Existing examples:
|
||||
`test_api_validate_provider.py` (admin),
|
||||
`test_api_parse_persists_ack.py` (acks).
|
||||
|
||||
## Patterns
|
||||
|
||||
### A new `APIRouter` skeleton
|
||||
|
||||
Pattern from `backend/src/cyclone/api_routers/acks.py:1-72`. Module
|
||||
docstring names the resource, imports the shared helpers, declares
|
||||
`router = APIRouter()` with no prefix, and defines a `_foo_to_ui(row)`
|
||||
mapper at module scope.
|
||||
|
||||
```python
|
||||
"""``/api/foo`` — list & detail endpoints for <topic>."""
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from cyclone.api_helpers import ndjson_stream_list, wants_ndjson
|
||||
from cyclone.store import store
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _foo_to_ui(row) -> dict:
|
||||
"""Map a Foo ORM row to the UI shape used by ``/api/foo``."""
|
||||
return {"id": row.id, "name": row.name}
|
||||
|
||||
|
||||
@router.get("/api/foo")
|
||||
def list_foo(
|
||||
request: Request,
|
||||
limit: int = Query(100, ge=1, le=1000),
|
||||
):
|
||||
"""Return the list of persisted Foo rows, newest first."""
|
||||
rows = store.list_foo()
|
||||
items = [_foo_to_ui(r) for r in rows[:limit]]
|
||||
total = len(rows)
|
||||
returned = len(items)
|
||||
has_more = total > returned
|
||||
if wants_ndjson(request):
|
||||
return StreamingResponse(
|
||||
ndjson_stream_list(items, total, returned, has_more),
|
||||
media_type="application/x-ndjson",
|
||||
)
|
||||
return {"items": items, "total": total, "returned": returned, "has_more": has_more}
|
||||
```
|
||||
|
||||
### A `get_<topic>` detail endpoint with 404
|
||||
|
||||
Pattern from `api_routers/acks.py:75-104` and `ta1_acks.py:67-76`.
|
||||
Path param is `<entity>_id` (not `id`) so it doesn't shadow
|
||||
FastAPI's internal `id` and the OpenAPI docs stay self-describing.
|
||||
|
||||
```python
|
||||
@router.get("/api/foo/{foo_id}")
|
||||
def get_foo(foo_id: int) -> dict:
|
||||
"""Return one persisted Foo row with its parsed detail.
|
||||
|
||||
Path param is ``foo_id`` (not ``id``) to avoid shadowing
|
||||
FastAPI's internal ``id`` name and to keep OpenAPI docs
|
||||
self-describing. Returns 404 when the row is missing — never 500.
|
||||
"""
|
||||
row = store.get_foo(foo_id)
|
||||
if row is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={"error": "Not found", "detail": f"Foo {foo_id} not found"},
|
||||
)
|
||||
return _foo_to_ui(row)
|
||||
```
|
||||
|
||||
### Mounting in `api.py`
|
||||
|
||||
Pattern from `backend/src/cyclone/api.py:246-256`. The block lives
|
||||
just after middleware registration and just before the first
|
||||
`@app.<verb>` decorator.
|
||||
|
||||
```python
|
||||
# Resource-group routers. Each module owns its own APIRouter and is
|
||||
# registered below. New resources go in `cyclone.api_routers.<name>`
|
||||
# and are wired in here.
|
||||
from cyclone.api_routers import acks, admin, health, ta1_acks # noqa: E402
|
||||
|
||||
app.include_router(health.router)
|
||||
app.include_router(acks.router)
|
||||
app.include_router(ta1_acks.router)
|
||||
app.include_router(admin.router)
|
||||
```
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Don't import from `cyclone.api` into a router — the dependency runs the other way.** Routers are mounted *into* `api.py` (`api_routers/acks.py` etc. know nothing about `cyclone.api`). A circular import would silently break the `from cyclone.api_routers import ...` block at `api.py:251`.
|
||||
- **Don't introduce Pydantic response models where the codebase returns dicts.** Every existing list / detail endpoint returns a plain dict produced by a `<entity>_to_ui(row)` helper (see `acks.py:29-48`, `ta1_acks.py:23-37`). That dict shape **is** the event payload that `<entity>_written` carries — see `cyclone-store`. Introducing a Pydantic model on one side drifts the event payload from the list shape and silently breaks live-tail dedup.
|
||||
- **Don't bypass `CycloneStore` to query the ORM directly from a route.** Always call `store.<method>(...)` (`store.list_acks()`, `store.get_ta1_ack(ack_id)`) so the read path picks up the same session + snapshot serializer as the live-tail subscriber. A raw `with db.SessionLocal()() as s: s.get(Foo, foo_id)` in a handler bypasses the serializer contract and breaks the event-payload match. See `cyclone-store`.
|
||||
- **Don't set `prefix=` on `APIRouter`.** Mount the router bare (`app.include_router(<name>.router)`) and put the full `/api/<resource>` path in the decorator. Mixing the two styles scatters the URL across two files and breaks `grep "/api/foo"` audits.
|
||||
|
||||
## Related skills
|
||||
|
||||
- **`cyclone-store`** — most routes call `store.<method>(...)` and the dict payload **is** the `<entity>_written` event payload; load when adding a route so the read path stays aligned with the pubsub contract.
|
||||
- **`cyclone-tail`** — streaming endpoints (`/api/<resource>/stream`) and the NDJSON wire format; load when adding a live-tail route or changing the wire format.
|
||||
- **`cyclone-edi`** — parse endpoints (`/api/parse-837`, `/api/parse-835`, `/api/parse-999`, `/api/parse-ta1`, `/api/parse-277ca`) currently live in `api.py`; load when adding or changing a parse endpoint.
|
||||
- **`cyclone-tests`** — endpoint tests follow the `test_api_<topic>_<verb>.py` naming under `backend/tests/`; load when writing the test for a new endpoint.
|
||||
@@ -1,187 +0,0 @@
|
||||
---
|
||||
name: cyclone-cli
|
||||
description: "Cyclone CLI subcommand conventions (cli.py — Click group + subcommands parse-837/parse-835/validate-npi/validate-tax-id/backup, --yes + click.confirm for destructive ops, exit codes 0/1/2, CliRunner smoke tests in backend/tests/test_cli_*.py). Use when: adding a CLI subcommand, changing an exit code, adding a smoke test, or wiring a security-sensitive command (backup, key rotation, anything touching secrets.py)."
|
||||
---
|
||||
|
||||
# cyclone-cli
|
||||
|
||||
The operator-facing CLI is a **Click** group at `cli.py:45` (`@click.group()` for `main`), mounted as the `cyclone` console script in `pyproject.toml:54` (`cyclone = "cyclone.cli:main"`). Seven subcommands ship today: `parse-837`, `parse-835`, `validate-npi`, `validate-tax-id`, plus the `backup` group (`init-passphrase`, `create`, `list`, `verify`, `restore`, `prune`, `status`). `serve` lives separately in `__main__.py:19` (dispatches `uvicorn cyclone.api:app`).
|
||||
|
||||
## When to use
|
||||
|
||||
- **Adding a subcommand.** You need a new operator command (e.g. `cyclone rotate-key`) and want to match the existing Click decorator + smoke-test rhythm.
|
||||
- **Changing an exit code.** You're tweaking which `sys.exit(N)` a subcommand raises and need the 0/1/2 contract used by `parse_837`, `parse_835`, `validate_npi_cmd`, `validate_tax_id_cmd`, and the `backup` group.
|
||||
- **Adding a smoke test.** You need `backend/tests/test_cli_<name>.py` using `click.testing.CliRunner` (NOT `subprocess.run`) and want the canonical fixture + monkeypatch layout (Keychain stub, fresh SQLite, `CYCLONE_BACKUP_DIR`).
|
||||
- **Wiring a security-sensitive command.** Anything touching `cyclone/secrets.py` (Keychain writes), DB key rotation, or destructive restores needs the two-step confirm dance used by `backup restore` / `backup prune` (`cli.py:462,509`).
|
||||
|
||||
## Conventions
|
||||
|
||||
1. **Click decorator pattern, not argparse.** Each subcommand is a
|
||||
top-level function decorated with `@main.command("<name>")` and
|
||||
one `@click.option` / `@click.argument` per parameter. Group
|
||||
dispatch is implicit — no `set_defaults(func=...)` and no
|
||||
`cmd_<name>(args) -> int` signature. Top-level commands at
|
||||
`cli.py:77,151,241,261,303`; the `backup` sub-group nests a
|
||||
second `@main.group()` (`cli.py:303`) with its own
|
||||
`@backup.command("<name>")` children.
|
||||
2. **Long-form flags for safety.** Prefer `--rotate-key`,
|
||||
`--backup-dir`, `--from-stdin` over positional args for anything
|
||||
that mutates state or takes a secret. `init-passphrase`
|
||||
(`cli.py:308-311`) demonstrates the canonical "flag OR stdin"
|
||||
pattern: `--passphrase` for automation, `--from-stdin` for
|
||||
interactive `getpass()` prompting.
|
||||
3. **Exit codes: 0 / 1 / 2.** `0` = success. `1` = user / input
|
||||
error (invalid NPI/EIN at `cli.py:258,277,285`; Keychain write
|
||||
failure at `cli.py:341,351`; tampered-backup verify at
|
||||
`cli.py:459`). `2` = operator / parse error
|
||||
(`CycloneParseError` at `cli.py:106,178`; passphrase
|
||||
empty/mismatch/short at `cli.py:331,337`). Document the codes
|
||||
in the docstring (see `validate_npi_cmd` at `cli.py:244-250`).
|
||||
Use `click.UsageError(...)` for usage mistakes; reserve
|
||||
`sys.exit(2)` for "the file failed to parse" semantics.
|
||||
4. **Smoke test with `click.testing.CliRunner`.** Every new
|
||||
subcommand gets `backend/tests/test_cli_<name>.py` that
|
||||
imports `from cyclone.cli import main` and invokes via
|
||||
`CliRunner().invoke(main, [...], catch_exceptions=False)`. The
|
||||
test must stub Keychain (`monkeypatch.setattr(secrets_mod,
|
||||
"get_secret"/"set_secret", ...)`) and pin a temp SQLite DB via
|
||||
`CYCLONE_DB_URL` + `db._reset_for_tests()` — see
|
||||
`backend/tests/test_cli_backup.py:13-69` for the canonical
|
||||
`_cli_env` fixture. CliRunner captures output and exit codes
|
||||
in-process; do NOT shell out to `subprocess.run`.
|
||||
5. **Destructive ops need `--yes` + `click.confirm(abort=True)`.**
|
||||
`backup restore` (`cli.py:462-506`) and `backup prune`
|
||||
(`cli.py:509-537`) both gate the destructive action behind a
|
||||
`--yes` is_flag and an interactive `click.confirm(..., abort=True)`
|
||||
prompt. CliRunner auto-aborts confirm prompts, so smoke tests
|
||||
assert `exit_code != 0` when `--yes` is omitted
|
||||
(`test_cli_backup.py:122-137`). A `--dry-run` flag is NOT yet
|
||||
implemented anywhere; if needed, mirror the `--yes` pattern.
|
||||
|
||||
## Patterns
|
||||
|
||||
### A Click subcommand — `@main.command("<name>")` + options
|
||||
|
||||
From `cli.py:241-258` (smallest standalone subcommand):
|
||||
|
||||
```python
|
||||
@main.command("validate-npi")
|
||||
@click.argument("npi")
|
||||
@click.option("--log-level", default="WARNING", show_default=True,
|
||||
type=click.Choice(["DEBUG", "INFO", "WARNING", "ERROR"]))
|
||||
def validate_npi_cmd(npi: str, log_level: str) -> None:
|
||||
"""Validate a 10-digit NPI's Luhn checksum locally (SP20). Exit 0 valid, 1 invalid. PHI — don't log the value."""
|
||||
setup_logging(level=log_level)
|
||||
from cyclone.npi import is_valid_npi
|
||||
if is_valid_npi(npi):
|
||||
click.echo(f"OK: {len(npi)}-digit NPI passes Luhn checksum")
|
||||
return
|
||||
click.echo(f"INVALID: {npi!r} fails NPI Luhn checksum", err=True)
|
||||
sys.exit(1)
|
||||
```
|
||||
|
||||
### A smoke test — `CliRunner` + Keychain stub + temp SQLite
|
||||
|
||||
From `test_cli_backup.py:13-69`. Stable hex salt keeps multiple `CliRunner` invocations consistent within one test. The `Batch` seed at `cli_backup.py:27-35` is omitted — only the Keychain + DB plumbing is the convention:
|
||||
|
||||
```python
|
||||
@pytest.fixture
|
||||
def _cli_env(tmp_path, monkeypatch):
|
||||
from cyclone import db, secrets as secrets_mod
|
||||
from cyclone import backup_service as svc_mod
|
||||
|
||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
|
||||
db._reset_for_tests()
|
||||
db.init_db()
|
||||
# ... seed any DB rows the subcommand needs (see cli_backup.py:27-35) ...
|
||||
|
||||
store = {svc_mod.KEYCHAIN_BACKUP_PASSPHRASE_ACCOUNT: "cli-test-passphrase",
|
||||
svc_mod.KEYCHAIN_BACKUP_SALT_ACCOUNT:
|
||||
"0123456789abcdef0123456789abcdef"}
|
||||
monkeypatch.setattr(secrets_mod, "get_secret", lambda n: store.get(n))
|
||||
monkeypatch.setattr(secrets_mod, "set_secret",
|
||||
lambda n, v: store.__setitem__(n, v) or True)
|
||||
monkeypatch.setenv("CYCLONE_BACKUP_DIR", str(tmp_path / "backups"))
|
||||
yield tmp_path / "backups"
|
||||
db._reset_for_tests()
|
||||
|
||||
def test_backup_create_list_verify_status(_cli_env):
|
||||
from cyclone.cli import main
|
||||
runner = CliRunner()
|
||||
r = runner.invoke(main, ["backup", "create"], catch_exceptions=False)
|
||||
assert r.exit_code == 0, r.output
|
||||
assert "created backup id=" in r.output
|
||||
```
|
||||
|
||||
### A destructive subcommand — `--yes` + `click.confirm(abort=True)`
|
||||
|
||||
From `cli.py:462-506` (`backup restore`). Two-step: announce, prompt unless `--yes`, then execute. Restore uses an explicit init/confirm round-trip so the operator can back out between phases. Matching smoke test asserts the guard fires:
|
||||
|
||||
```python
|
||||
@backup.command("restore")
|
||||
@click.argument("backup_id", type=int)
|
||||
@click.option("--yes", is_flag=True, help="Skip the interactive confirm prompt")
|
||||
@click.option("--actor", default="operator-cli", show_default=True)
|
||||
def backup_restore(backup_id: int, yes: bool, actor: str) -> None:
|
||||
"""Restore the live DB from a backup (two-step, requires --yes)."""
|
||||
# ... db.init_db() + service config omitted ...
|
||||
click.echo(f"Initiating restore from backup {backup_id}...")
|
||||
init = svc.restore_initiate(backup_id)
|
||||
# ... echo init summary (filename, fp, table_count, ttl) ...
|
||||
if not yes:
|
||||
click.confirm(
|
||||
"Replace the live DB with this backup? "
|
||||
"This will dispose the engine and rebuild it.",
|
||||
abort=True,
|
||||
)
|
||||
click.echo("Confirming restore...")
|
||||
result = svc.restore_confirm(backup_id, init.restore_token, actor=actor)
|
||||
|
||||
def test_backup_restore_requires_yes_flag(_cli_env):
|
||||
runner = CliRunner()
|
||||
runner.invoke(main, ["backup", "create"], catch_exceptions=False)
|
||||
r = runner.invoke(main, ["backup", "restore", "1"], catch_exceptions=False)
|
||||
# CliRunner auto-aborts confirm prompts → exit_code != 0.
|
||||
assert r.exit_code != 0
|
||||
```
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Don't `sys.exit(2)` for usage errors.** Reserve code 2 for
|
||||
parse/operator errors (`CycloneParseError`, Keychain not
|
||||
initialized, passphrase policy violation). For "you passed the
|
||||
wrong flag" use `raise click.UsageError(...)` — Click formats
|
||||
it as a clean help message and exits 2 on its own.
|
||||
- **Don't print errors to stdout.** Use `click.echo(msg, err=True)`
|
||||
for all error output. `print(..., file=sys.stderr)` and bare
|
||||
`logging.error(...)` bypass Click's stdout/stderr split and leak
|
||||
into test `result.output` — breaking `assert "FAIL" in r.output`
|
||||
style assertions.
|
||||
- **Don't add a subcommand without a smoke test.** Every new
|
||||
`@main.command(...)` ships a sibling
|
||||
`backend/tests/test_cli_<name>.py` exercising the happy path
|
||||
AND at least one error path (missing input, invalid arg,
|
||||
tampered ciphertext — see `test_cli_backup.py:102-119`).
|
||||
- **Don't reuse the `parse` command name.** Existing subcommands
|
||||
are type-specific (`parse-837`, `parse-835`); a generic `parse`
|
||||
would shadow them or force an `--type` flag — neither is the
|
||||
codebase pattern.
|
||||
|
||||
## Related skills
|
||||
|
||||
- **`cyclone-store`** — `backup` subcommands and the parse
|
||||
subcommands both round-trip through `CycloneStore` and the DB
|
||||
session; load when the increment changes write paths or the
|
||||
`<entity>_written` event contract.
|
||||
- **`cyclone-api-router`** — `cyclone serve` (via `__main__.py:19`)
|
||||
launches the FastAPI app; the CLI parse subcommands share the
|
||||
same `CycloneParseError` exception and Pydantic result models
|
||||
as the matching HTTP endpoints.
|
||||
- **`cyclone-edi`** — `parse-837` / `parse-835` are the CLI smoke
|
||||
entry points for the parser/validator surface; load when adding
|
||||
a parser or R-code rule.
|
||||
- **`cyclone-tests`** — every CLI subcommand gets a pytest smoke
|
||||
case under `backend/tests/test_cli_<name>.py`; the `_cli_env`
|
||||
fixture pattern (fresh SQLite + Keychain stub) is documented
|
||||
there.
|
||||
- **`cyclone-spec`** — load when the SP-N spec introduces a new operator command or reserves a new exit-code category.
|
||||
@@ -1,193 +0,0 @@
|
||||
---
|
||||
name: cyclone-edi
|
||||
description: "Cyclone EDI parser/validator conventions (837P/835/999/270/271/277CA/TA1). Use when: adding or changing a parser, adding a validator rule (R010/R020/R100/R200-R210/R835_*/NPI Luhn/EIN/CAS), or mapping a new CAS adjustment reason code."
|
||||
---
|
||||
|
||||
# cyclone-edi
|
||||
|
||||
Cyclone parses seven X12 EDI transaction types (837P, 835, 999, 270, 271, 277CA, TA1) into typed Pydantic models, then runs per-claim / per-batch validator rules that surface as R-coded `ValidationIssue` records. This skill codifies the conventions so new parsers and rules stay consistent with the seven that already exist.
|
||||
|
||||
As of this writing: **7 parser modules** under `backend/src/cyclone/parsers/parse_<edi>.py`, **~25 per-claim rules** numbered `R010`–`R100` and `R200`–`R210` in `validator.py`, plus a parallel set of **835-specific rules** prefixed `R835_*` in `validator_835.py`. The next increment is **SP22**.
|
||||
|
||||
## When to use
|
||||
|
||||
- **Adding or changing a parser.** You are about to touch `backend/src/cyclone/parsers/parse_<edi>.py` or its paired `models_<edi>.py` and need the orchestrator signature, the segment walker convention, and the re-export in `parsers/__init__.py`.
|
||||
- **Adding a validator rule.** You are writing a new `_rule_R<n>_<name>` (or `_r<n>_<name>` per the existing snake-case style) and need the rule signature, the R-code numbering scheme, and the `ValidationIssue` shape.
|
||||
- **Wiring a new CAS / CARC code.** The 835 carries Claim Adjustment Reason Codes in `CAS` segments; the lookup lives in `backend/src/cyclone/parsers/cas_codes.py` and the UI reads through `claim_status_label()`.
|
||||
- **Debugging a parse failure on a prodfiles sample.** You dropped a real EDI file into `docs/prodfiles/<source>/` and the parser is choking — load this skill to confirm the tokenizer path, the orchestrator entry point, and which fixture in `backend/tests/fixtures/` matches the transaction type.
|
||||
|
||||
## Conventions
|
||||
|
||||
1. **Parser signature.** Every parser module exports exactly one public entry function. Two flavors coexist in the codebase:
|
||||
- `parse(text: str, *, input_file: str = "") -> <TypedResult>` — used by `parse_270.py:337`, `parse_271.py:356`.
|
||||
- `parse(text: str, payer_config: <PayerConfig>, input_file: str = "") -> <TypedResult>` — used by `parse_837.py:319` and `parse_835.py:459` because both need payer-specific config to validate segments against.
|
||||
- `parse_<edi>_text(text: str, *, input_file: str = "") -> <TypedResult>` — the legacy name-suffixed form, still in use at `parse_ta1.py:143`, `parse_999.py:220`, `parse_277ca.py:280`. The `<TypedResult>` is always a Pydantic model from `models_<edi>.py` (or co-located `models.py` for 837P).
|
||||
2. **Segment walk.** Parsers consume `backend/src/cyclone/parsers/segments.py` — there are exactly three public pieces: `Delimiters` (frozen dataclass holding the four ISA-derived separators), `_detect_delimiters(isa_segment)` (private), and `tokenize(text) -> list[list[str]]` (returns ISA prepended as the first segment). Parsers then index into the `list[list[str]]` directly — there is **no** `Segment` / `Loop` / `next_segment` helper class. Whole-document problems (missing ISA, wrong transaction set) raise `CycloneParseError`; per-segment problems on acks (999/277CA) are surfaced on the result, not raised.
|
||||
3. **Validator rules.** Numbered rules live in `backend/src/cyclone/parsers/validator.py` (for 837P — R010–R100 general + R200–R210 SP9 CO MAP / HCPF naming) and `backend/src/cyclone/parsers/validator_835.py` (for 835 — names prefixed `R835_*` because the same numeric space would collide with 837P). Each rule is a function `_r<n>_<name>(claim: ClaimOutput, cfg: PayerConfig) -> Iterable[ValidationIssue]` registered in the module-level `_RULES` list and run by `validate(claim, config)`. Issues carry the rule name as a stable string (`rule="R021_npi_checksum"`) — the R-code **is** how the UI surfaces the error, so never invent an unnumbered rule.
|
||||
4. **NPI / EIN / CAS format logic.** Identity-format checks live in their own modules — never duplicate them in a parser or validator:
|
||||
- `backend/src/cyclone/npi.py` — `is_valid_npi(npi)` runs the Luhn checksum with the `80840` NPPES prefix; `is_valid_tax_id(ein)` enforces `XX-XXXXXXX` (or 9 raw digits).
|
||||
- `backend/src/cyclone/parsers/cas_codes.py` — `reason_label(group, reason)` and `all_known_codes()` for the CARC lookup; snapshot date is exported as `LAST_UPDATED`.
|
||||
- `backend/src/cyclone/parsers/models_271.py` — `SERVICE_TYPE_CODES` + `service_type_description()` for 271 EB benefit codes.
|
||||
5. **Prodfiles reuse.** When adding a parser for a new transaction type, ship at least one fixture in `backend/tests/fixtures/<edi>/<sample>.txt` (the existing 13 fixtures are **flat** at the top level of `fixtures/` — no per-test subdirectories). Copy from `docs/prodfiles/<source>/<file>.txt`; never reach into `docs/prodfiles/` from a test. The matching test should declare the path as a module-level `Path` constant.
|
||||
|
||||
## Patterns
|
||||
|
||||
### Minimal `parse_<edi>.py` — using `segments.py`, exporting `parse_ta1_text`
|
||||
|
||||
Taken from `backend/src/cyclone/parsers/parse_ta1.py:1-29` (the smallest parser — TA1 is just ISA + TA1 + IEA). The same skeleton scales to every other EDI type by adding `_consume_<segment>` helpers.
|
||||
|
||||
```python
|
||||
"""Parse an X12 TA1 (Interchange Acknowledgment) file.
|
||||
|
||||
Whole-document problems (missing ISA, no TA1) raise CycloneParseError.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import date
|
||||
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
from cyclone.parsers.models import BatchSummary, Envelope
|
||||
from cyclone.parsers.models_ta1 import ParseResultTa1, Ta1Ack
|
||||
from cyclone.parsers.segments import tokenize
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _parse_yyyymmdd(s: str) -> date | None:
|
||||
"""Parse an 8-digit CCYYMMDD string. Returns None on bad input."""
|
||||
...
|
||||
|
||||
|
||||
def _build_envelope(segments: list[list[str]], input_file: str) -> Envelope:
|
||||
"""Build the envelope from ISA. TA1 has no GS/ST — just ISA → TA1 → IEA."""
|
||||
...
|
||||
|
||||
|
||||
def _consume_ta1(segments: list[list[str]], idx: int) -> tuple[Ta1Ack, int]:
|
||||
"""Read a TA1 segment and return a Ta1Ack. Returns (model, next_idx)."""
|
||||
...
|
||||
|
||||
|
||||
def parse_ta1_text(text: str, *, input_file: str = "") -> ParseResultTa1:
|
||||
"""Parse a complete TA1 document and return a ParseResultTa1."""
|
||||
segments = tokenize(text)
|
||||
envelope = _build_envelope(segments, input_file=input_file)
|
||||
ta1_idx = next(
|
||||
(i for i, seg in enumerate(segments) if seg[0] == "TA1"), None,
|
||||
)
|
||||
if ta1_idx is None:
|
||||
raise CycloneParseError("No TA1 segment found")
|
||||
ta1, _ = _consume_ta1(segments, ta1_idx)
|
||||
...
|
||||
return ParseResultTa1(envelope=envelope, ta1=ta1, summary=summary, ...)
|
||||
|
||||
|
||||
__all__ = ["parse_ta1_text"]
|
||||
```
|
||||
|
||||
The orchestrator pattern is the same in every parser: `tokenize` → `_build_envelope` → segment consumers in order → wrap into a `ParseResult<EDI>` model. The Pydantic result is what the API / store layer consumes.
|
||||
|
||||
### A validator rule — `_r<n>_<name>` registered in `_RULES`
|
||||
|
||||
Taken from `backend/src/cyclone/parsers/validator.py:23-66` (the canonical R010–R100 block).
|
||||
|
||||
```python
|
||||
from collections.abc import Iterable
|
||||
from cyclone.parsers.models import ClaimOutput, ValidationIssue
|
||||
from cyclone.parsers.payer import PayerConfig
|
||||
|
||||
NPI_RE = re.compile(r"^\d{10}$")
|
||||
Rule = Callable[[ClaimOutput, PayerConfig], Iterable[ValidationIssue]]
|
||||
|
||||
|
||||
def _r020_npi_format(claim: ClaimOutput, _: PayerConfig) -> Iterable[ValidationIssue]:
|
||||
if claim.billing_provider.npi and not NPI_RE.match(claim.billing_provider.npi):
|
||||
yield ValidationIssue(
|
||||
rule="R020_npi_format",
|
||||
severity="error",
|
||||
message=f"Billing provider NPI must be 10 digits, got {claim.billing_provider.npi!r}",
|
||||
)
|
||||
|
||||
|
||||
def _r021_npi_checksum(claim: ClaimOutput, _: PayerConfig) -> Iterable[ValidationIssue]:
|
||||
"""SP20: validate the billing-provider NPI's Luhn check digit."""
|
||||
npi = claim.billing_provider.npi
|
||||
if not npi or not NPI_RE.match(npi):
|
||||
return # R020 already flagged the format — skip silently.
|
||||
try:
|
||||
from cyclone.npi import is_valid_npi
|
||||
except ImportError:
|
||||
return
|
||||
if not is_valid_npi(npi):
|
||||
yield ValidationIssue(
|
||||
rule="R021_npi_checksum",
|
||||
severity="warning",
|
||||
message=f"Billing provider NPI {npi!r} fails Luhn checksum (likely typo)",
|
||||
)
|
||||
|
||||
|
||||
_RULES: list[Rule] = [
|
||||
_r010_clm01_present,
|
||||
_r011_total_charge_positive,
|
||||
_r020_npi_format,
|
||||
_r021_npi_checksum,
|
||||
# ... R030, R031, R032-R035, R050, R060, R070, R100, R200-R210
|
||||
]
|
||||
```
|
||||
|
||||
For 835 rules, prefix the rule string with `R835_` (e.g. `R835_BPR01_handling_code_allowed`) and target the `ParseResult835` model instead of `ClaimOutput` — see `backend/src/cyclone/parsers/validator_835.py:38-79`.
|
||||
|
||||
### A test that uses a prodfiles fixture
|
||||
|
||||
Taken from `backend/tests/test_api_999.py:53-72`. The autouse `conftest.py` already provides a per-test SQLite DB; most tests just add a `client` fixture and reference the fixture path as a module-level constant.
|
||||
|
||||
```python
|
||||
"""Tests for the FastAPI surface in cyclone.api for the 999 endpoint."""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from cyclone.api import app
|
||||
|
||||
# Fixture reference — flat, module-level Path constant. NEVER reach into
|
||||
# docs/prodfiles/ from a test; the fixtures/ dir is the test-consumed surface.
|
||||
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
|
||||
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client() -> TestClient:
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def test_parse_999_endpoint_happy_path(client: TestClient):
|
||||
text = ACCEPTED.read_text()
|
||||
resp = client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("minimal_999.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["ack"]["ack_code"] == "A"
|
||||
```
|
||||
|
||||
For pure-unit parser tests (no API), the same path is reused — see `backend/tests/test_parse_837.py:8` (`FIXTURE = Path(__file__).parent / "fixtures" / "minimal_837p.txt"`).
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Don't re-parse raw X12 strings inside validators.** Always parse first into the typed `ParseResult<EDI>` / `ClaimOutput`, then validate against that. Validators index into the model fields (or `claim.raw_segments` for spot-checks of specific segment presence) — they never call `tokenize` again. R034's `REF*G1` presence check (`validator.py:104-107`) is the only place that legitimately touches `raw_segments`, and it does so to confirm a single segment exists.
|
||||
- **Don't bake payer-specific logic into the generic parser.** Payer variations live in `backend/src/cyclone/parsers/payer.py` (`PayerConfig`, `PayerConfig835`) and `backend/src/cyclone/payers.py` (the YAML loader from `config/payers.yaml`). Parsers accept the config as an argument; rules read it from the `cfg` parameter. A new payer never requires a new parser file — extend the config and add / adjust an R-code rule.
|
||||
- **Don't add a validator rule without an R-code.** The `rule="R<n>_<name>"` string is the stable identifier the UI greys out, the API returns in `errors[].rule`, and tests assert against. Inventing a rule without an R-code (or reusing an R-code with new semantics) breaks the operator workflow. New SP-N increments reserve their R-code range up front (SP9 reserved R200–R210, SP20 added R021) and document it in the spec.
|
||||
|
||||
## Related skills
|
||||
|
||||
- **`cyclone-store`** — load when the increment changes how a parsed `ClaimOutput` / `ParseResult<EDI>` is persisted (`store.py` write path, `<entity>_written` events).
|
||||
- **`cyclone-api-router`** — load when the increment adds or changes an HTTP endpoint that surfaces a parsed result (e.g. `/api/parse-999`, `/api/parse-837`, `/api/parse-835`).
|
||||
- **`cyclone-tests`** — every parser addition ships a fixture in `backend/tests/fixtures/` and a pytest case; load this skill for the fixture-drop-in and autouse-conftest rules.
|
||||
- **`cyclone-cli`** — load when the increment adds a CLI subcommand. The `cyclone parse-837 <file>` and `cyclone parse-835 <file>` smoke commands at `backend/src/cyclone/cli.py:77,151` are the parser-level smoke tests; the `validate-npi` and `validate-tax-id` commands exercise the format helpers.
|
||||
- **`cyclone-spec`** — load when the SP-N spec for the increment introduces a new R-code range or a new transaction type; the spec's `## Decisions` section is where the R-code reservation gets locked in.
|
||||
@@ -1,40 +0,0 @@
|
||||
# Cyclone EDI parsers — flat catalog
|
||||
|
||||
Every parser module under `backend/src/cyclone/parsers/` (one row per
|
||||
file), its transaction type, its public entry signature, its result
|
||||
model, its primary fixture, and any payer-specific variant. The
|
||||
companion Pydantic model is in a co-located `models_<edi>.py`; the
|
||||
segment walker uses `tokenize()` from `segments.py` and never parses
|
||||
raw text inline.
|
||||
|
||||
| Module | EDI type | Public entry signature | Result model | Primary fixture(s) | Payer variant |
|
||||
|---|---|---|---|---|---|
|
||||
| `parse_837.py` | 837P (Professional Claim) | `parse(text, payer_config: PayerConfig, input_file="") -> ParseResult` | `cyclone.parsers.models.ParseResult` | `minimal_837p.txt`, `co_medicaid_837p.txt` | `PayerConfig` (CO Medicaid default) |
|
||||
| `parse_835.py` | 835 (ERA / Remittance) | `parse(text, payer_config: PayerConfig835, input_file="") -> ParseResult835` | `cyclone.parsers.models_835.ParseResult835` | `minimal_835.txt`, `co_medicaid_835.txt`, `unbalanced_835.txt` | `PayerConfig835` |
|
||||
| `parse_999.py` | 999 (Implementation ACK) | `parse_999_text(text, *, input_file="") -> ParseResult999` | `cyclone.parsers.models_999.ParseResult999` | `minimal_999.txt`, `minimal_999_rejected.txt` | none — single-shape ack |
|
||||
| `parse_277ca.py` | 277CA (Claim ACK) | `parse_277ca_text(text, *, input_file="") -> ParseResult277CA` | `cyclone.parsers.models_277ca.ParseResult277CA` | `minimal_277ca.txt`, `minimal_277ca_rejected_only.txt`, `minimal_277ca_st277.txt` | `PayerConfig277CA` (config-driven) |
|
||||
| `parse_270.py` | 270 (Eligibility Inquiry) | `parse(text, *, input_file="") -> ParseResult270` | `cyclone.parsers.models_270.ParseResult270` | `minimal_270.txt` | reuses `PayerConfig` shape |
|
||||
| `parse_271.py` | 271 (Eligibility Response) | `parse(text, *, input_file="") -> ParseResult271` | `cyclone.parsers.models_271.ParseResult271` | `minimal_271.txt` | reuses `PayerConfig` shape |
|
||||
| `parse_ta1.py` | TA1 (Interchange ACK) | `parse_ta1_text(text, *, input_file="") -> ParseResultTa1` | `cyclone.parsers.models_ta1.ParseResultTa1` | `minimal_ta1.txt` | none — single-shape ack |
|
||||
|
||||
Companion modules (not parsers, but shipped alongside):
|
||||
|
||||
| Module | Role |
|
||||
|---|---|
|
||||
| `segments.py` | `Delimiters`, `_detect_delimiters`, `tokenize(text) -> list[list[str]]` |
|
||||
| `models.py` | Pydantic models for 837P (`ParseResult`, `ClaimOutput`, `Envelope`, `BatchSummary`, `ValidationIssue`, `ValidationReport`, …) |
|
||||
| `models_835.py` / `models_270.py` / `models_271.py` / `models_277ca.py` / `models_999.py` / `models_ta1.py` | Pydantic models for each transaction type |
|
||||
| `payer.py` | `PayerConfig` + `PayerConfig835` factories |
|
||||
| `exceptions.py` | `CycloneParseError`, `CycloneValidationError` |
|
||||
| `cas_codes.py` | CARC lookup: `reason_label(group, reason)`, `all_known_codes()`, `LAST_UPDATED` |
|
||||
| `validator.py` | 837P rules: R010–R100, R200–R210; `validate(claim, config) -> ValidationReport` |
|
||||
| `validator_835.py` | 835 rules: `R835_*` (e.g. `R835_BPR01_handling_code_allowed`); `validate(result, cfg) -> ValidationReport` |
|
||||
| `serialize_270.py` / `serialize_837.py` / `serialize_999.py` | Outbound (Cyclone → payer) serializers — mirror of `parse_*` |
|
||||
| `writer.py` / `writer_835.py` | Output writers (one JSON per claim) |
|
||||
| `batch_ack_builder.py` | `build_ack_for_batch` — produces a 999 for a parsed 837 batch |
|
||||
| `__init__.py` | Lazy PEP 562 re-exports (`parse`, `parse_835`, `parse_999`, `parse_270`, `parse_271`, `parse_277ca`, plus all models) |
|
||||
|
||||
Fixture rule: every parser ships at least one flat fixture in
|
||||
`backend/tests/fixtures/<edi>-sample.txt`. Prodfiles sources live in
|
||||
`docs/prodfiles/{837p-from-axiscare,835fromco,FromHPE,claims}/` — copy
|
||||
from there, never reach in from a test.
|
||||
@@ -1,138 +0,0 @@
|
||||
---
|
||||
name: cyclone-frontend-page
|
||||
description: "Cyclone React page conventions (TanStack Query, use<X> hook, drawer, URL state, .test.tsx sibling, Layout / PageHeader / Sidebar). Use when: adding a new page, refactoring an existing one, or wiring a drawer into a page."
|
||||
---
|
||||
|
||||
# cyclone-frontend-page
|
||||
|
||||
Cyclone pages live at `src/pages/<Name>.tsx`: a `use<X>` hook in `src/hooks/use<X>.ts` does the fetching (and optionally the live-tail subscription), `<Layout>` + `<PageHeader>` + `<Sidebar>` (`src/components/`) provide the app shell, and any right-side detail (claim, remittance) lives in `src/components/<DrawerName>/` whose open/close state is mirrored to the URL via `useDrawerUrlState`. This skill codifies the conventions so additions stay consistent with the eleven pages already shipped.
|
||||
|
||||
As of this writing: **11 pages** under `src/pages/` (9 of 11 with a `*.test.tsx` sibling — Dashboard, Upload, and BatchDiff are not yet covered; be the first when you refactor them), **~30 hooks** under `src/hooks/`, and **2 drawer modules** at `src/components/{ClaimDrawer,RemitDrawer}/`. The next increment is **SP22**.
|
||||
|
||||
## When to use
|
||||
|
||||
- **Adding a new page.** Mounting a new screen in `src/pages/` — you need the Layout + PageHeader + table shape, the `use<X>` hook split, and the route registration point in `src/App.tsx`.
|
||||
- **Refactoring an existing page.** Splitting a 600-line page, swapping a manual `fetch` for a hook, or moving in-component state into the URL — load this skill to confirm the destination shape.
|
||||
- **Wiring a drawer.** Adding a new right-side detail drawer (e.g. `BatchDrawer`, `ActivityDrawer`) — you need `useDrawerUrlState` for URL-driven open/close, the `src/components/<DrawerName>/` folder layout, and the deep-link contract so `?claim=…` / `?remit=…` round-trips.
|
||||
- **Sharing state via URL.** Persisting filter / page / drawer state across reloads — confirm the `useDrawerUrlState` / `useRemitDrawerUrlState` hook pair is the right tool before reaching for `useState` + history.
|
||||
|
||||
## Conventions
|
||||
|
||||
1. **Page shape.** Every page in `src/pages/<Name>.tsx` exports a `function <Name>()` (most pages use a named export — see `src/pages/Claims.tsx:51`, `Remittances.tsx:62`, `Dashboard.tsx:68`; `src/pages/Inbox.tsx:40` is the lone default export). The app shell is provided by the `<Layout>` route wrapper in `src/App.tsx:30`. Each page sets a `<PageHeader>` (`src/components/PageHeader.tsx:18`) and renders a table, list, or KPI grid. Sidebar nav is mounted once by `<Layout>` at `src/components/Sidebar.tsx`.
|
||||
2. **Data hook.** Each page pairs with a `use<X>` hook in `src/hooks/use<X>.ts` (e.g. `Claims` ↔ `useClaims`, `Remittances` ↔ `useRemittances`, `Acks` ↔ `useAcks`). The hook returns `{ data, isLoading, isError, error, refetch }` from TanStack Query's `useQuery` (see `src/hooks/useClaims.ts:24-31`, `useRemittances.ts:21-25`). Pages never call `fetch` or `@/lib/api` directly — the hook is the boundary so the page is testable with `vi.mock("@/lib/api", ...)`.
|
||||
3. **Live tail.** Pages with live data compose three hooks in order: the `use<X>` initial fetch, `useTailStream(resource)` (`src/hooks/useTailStream.ts:80` — opens the NDJSON stream, drives the backoff/stall state machine), and `useMergedTail(resource, baseItems, filterFn?)` (`src/hooks/useMergedTail.ts:25` — merges snapshot + tail, dedup'd by id). The full wiring lives at `src/pages/Claims.tsx:85-89` and `Remittances.tsx:81-83`. The streaming subscription belongs on the page, not in the data hook — hoisting it couples the lifecycle to whoever mounts `use<X>`.
|
||||
4. **Drawer.** Right-side detail drawers live in `src/components/<DrawerName>/` (currently `ClaimDrawer/`, `RemitDrawer/`) with a barrel `index.ts` (`src/components/ClaimDrawer/index.ts:1-14`). The drawer is wired to URL state via `useDrawerUrlState()` for `?claim=…` (`src/hooks/useDrawerUrlState.ts`) or `useRemitDrawerUrlState()` for `?remit=…` (`src/hooks/useRemitDrawerUrlState.ts`). Open state is driven by the URL, not a `useState` flag, so deep-links round-trip.
|
||||
5. **Tests.** Every page gets a `src/pages/<Name>.test.tsx` sibling (e.g. `Claims.test.tsx`, `Remittances.test.tsx`, `Batches.test.tsx`). Every hook gets a `src/hooks/use<X>.test.ts` sibling (e.g. `useClaims.test.ts`, `useRemittances.test.ts`). The shared setup — `// @vitest-environment happy-dom`, `IS_REACT_ACT_ENVIRONMENT = true`, `QueryClient` provider, `vi.mock("@/lib/api", ...)` — is documented in `cyclone-tests`; mirror `src/pages/Claims.test.tsx:1-30` for the canonical page-test shape.
|
||||
6. **UI primitives.** Use Radix-backed components from `src/components/ui/` (`button.tsx`, `dialog.tsx`, `table.tsx`, `select.tsx`, `pagination.tsx`, `empty-state.tsx`, `error-state.tsx`, `filter-chips.tsx`, `skeleton.tsx`, `input.tsx`, `label.tsx`, `card.tsx`, `badge.tsx`, `skip-link.tsx`, `claim-state-badge.tsx`). Don't pull in a new UI library without discussion — every primitive here is already consumed by at least one shipped page.
|
||||
7. **Routing.** Pages register their route in `src/App.tsx` as a `<Route path="<name>" element={<<Name>> />} />` inside the `<Layout>` element wrapper (`src/App.tsx:30-44`). Currently every page is a static import; switch to `React.lazy(() => import(...))` only if a page grows heavy (large parse/EDI libs, chart code) and the import cost shows up in the bundle report.
|
||||
|
||||
## Patterns
|
||||
|
||||
### `Claims.tsx`-style page (Layout + PageHeader + table + drawer)
|
||||
|
||||
Canonical page shape — composes the data hook, the tail triplet, and
|
||||
`useDrawerUrlState` for the `?claim=…` deep-link. See
|
||||
`src/pages/Claims.tsx:51-200` for the full file.
|
||||
|
||||
```tsx
|
||||
import { useMemo, useState } from "react";
|
||||
import { Table, TableBody, TableRow, /* … */ } from "@/components/ui/table";
|
||||
import { PageHeader } from "@/components/PageHeader";
|
||||
import { ClaimDrawer } from "@/components/ClaimDrawer";
|
||||
import { useClaims } from "@/hooks/useClaims";
|
||||
import { useDrawerUrlState } from "@/hooks/useDrawerUrlState";
|
||||
import { useTailStream } from "@/hooks/useTailStream";
|
||||
import { useMergedTail } from "@/hooks/useMergedTail";
|
||||
import { TailStatusPill } from "@/components/TailStatusPill";
|
||||
|
||||
export function Claims() {
|
||||
const [status, setStatus] = useState<ClaimStatus | null>(null);
|
||||
const params = useMemo(() => ({ status, limit: 25, offset: 0 }), [status]);
|
||||
|
||||
const { data } = useClaims(params);
|
||||
const { status: tailStatus, lastEventAt, forceReconnect } = useTailStream("claims");
|
||||
const items = useMergedTail("claims", data?.items ?? [], (c) => !status || c.status === status);
|
||||
const { claimId, openClaim, closeClaim } = useDrawerUrlState();
|
||||
|
||||
return (
|
||||
<>
|
||||
<PageHeader
|
||||
eyebrow="Inbox"
|
||||
title="Claims"
|
||||
status={<TailStatusPill status={tailStatus} lastEventAt={lastEventAt} onReconnect={forceReconnect} />}
|
||||
/>
|
||||
<Table>
|
||||
<TableBody>
|
||||
{items.map((c) => (
|
||||
<TableRow key={c.id} onClick={() => openClaim(c.id)}>{/* …cells… */}</TableRow>
|
||||
))}
|
||||
</TableBody>
|
||||
</Table>
|
||||
<ClaimDrawer claimId={claimId} claims={items} onClose={closeClaim} onNavigate={openClaim} />
|
||||
</>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
### `use<X>` data hook (TanStack Query)
|
||||
|
||||
Pattern from `src/hooks/useClaims.ts:24-67` and `useRemittances.ts:21-65`.
|
||||
Returns a stable shape so the page treats all data hooks uniformly.
|
||||
The tail subscription lives on the page (see Convention 3), not here.
|
||||
|
||||
```ts
|
||||
import { useQuery } from "@tanstack/react-query";
|
||||
import { api, type ListClaimsParams, type PaginatedResponse } from "@/lib/api";
|
||||
import type { Claim } from "@/types";
|
||||
|
||||
export function useClaims(params: ListClaimsParams) {
|
||||
return useQuery<PaginatedResponse<Claim>>({
|
||||
queryKey: ["claims", params],
|
||||
queryFn: () => api.listClaims<Claim>(params),
|
||||
enabled: api.isConfigured,
|
||||
// …in-memory fallback when !api.isConfigured…
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### Drawer component with `useDrawerUrlState`
|
||||
|
||||
Pattern from `src/components/ClaimDrawer/ClaimDrawer.tsx:1-50` +
|
||||
`src/hooks/useDrawerUrlState.ts`. The drawer takes `claimId` as a
|
||||
prop (driven by the URL), renders nothing when `null`, and uses
|
||||
`useDrawerKeyboard` for j/k navigation + Escape to close. The page
|
||||
that mounts it controls the URL — `openClaim("abc")` sets `?claim=abc`,
|
||||
removing the param closes the drawer, and reload preserves the state.
|
||||
|
||||
```tsx
|
||||
import { Dialog, DialogContent } from "@/components/ui/dialog";
|
||||
import { useClaimDetail } from "@/hooks/useClaimDetail";
|
||||
import { useDrawerKeyboard } from "@/hooks/useDrawerKeyboard";
|
||||
|
||||
export function ClaimDrawer({ claimId, claims, onClose, onNavigate, onToggleHelp }) {
|
||||
const open = claimId !== null;
|
||||
const { data, isLoading, isError, error } = useClaimDetail(claimId);
|
||||
useDrawerKeyboard({ open, claims, onClose, onNavigate, onToggleHelp });
|
||||
if (!open) return null;
|
||||
return (
|
||||
<Dialog open onOpenChange={(o) => !o && onClose()}>
|
||||
<DialogContent>{/* header + body panels from useClaimDetail… */}</DialogContent>
|
||||
</Dialog>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Don't `fetch` from inside a page component.** All API access goes through the `use<X>` hook in `src/hooks/use<X>.ts`. Pages that reach for `fetch(...)` directly can't be tested with `vi.mock("@/lib/api", ...)` and split the data lifecycle across files. The hook returns `{ data, isLoading, isError, error, refetch }` so the page is a pure renderer.
|
||||
- **Don't open a drawer via local component state.** A `const [open, setOpen] = useState(false)` for a drawer breaks deep-links and reload-restore. Use `useDrawerUrlState()` (claim) or `useRemitDrawerUrlState()` (remit) so the URL is the single source of truth.
|
||||
- **Don't put domain logic in JSX.** Conditional renderings, table sorting, and KPI math all belong in the `use<X>` hook or a pure helper under `src/lib/` (e.g. `src/lib/format.ts` for currency / date formatting). JSX is for layout; mixing in `items.filter(...).sort(...)` inline is hard to test and hides behavior from the hook.
|
||||
- **Don't call `useTailStream` from inside a `use<X>` hook.** The streaming subscription belongs on the page (see `src/pages/Claims.tsx:85-89`). Hoisting it into `useClaims` couples the open/close lifecycle of the page to whoever mounts the hook, and breaks the one-resource-one-page ownership that the backoff/stall state machine assumes.
|
||||
- **Don't import a new UI library to render a button, modal, or table.** Radix-backed primitives in `src/components/ui/` already cover every widget the shipped pages use. Reach for a new library only after the primitive gap is real and the proposal is in a spec / PR.
|
||||
|
||||
## Related skills
|
||||
|
||||
- **`cyclone-tail`** — load for any live-data page (Claims, Remittances, ActivityLog). Documents the `useTailStream` + `useMergedTail` triplet, the `<TailStatusPill>` wiring, and the 30s stall threshold (`STALL_TIMEOUT_MS = 30_000` at `useTailStream.ts:53`).
|
||||
- **`cyclone-api-router`** — load when a page is calling a new HTTP endpoint. Documents `api_routers/<topic>.py` vs. inline `api.py` registration and the response / error-envelope shapes.
|
||||
- **`cyclone-tests`** — load when adding the `*.test.tsx` sibling. Documents the `// @vitest-environment happy-dom` setup, `IS_REACT_ACT_ENVIRONMENT = true`, the `@testing-library/react` vs. `createRoot`+`Probe` rendering styles, and the `vi.mock("@/lib/api")` convention.
|
||||
- **`cyclone-edi`** — load when a page renders parsed 837P / 835 / 999 / 270 / 271 / 277CA / TA1 content (ServiceLinesTable, CAS panels, ValidationPanel). Documents the parser modules, the R-coded validator rules, and the CAS / CARC / NPI / EIN helpers.
|
||||
@@ -1,163 +0,0 @@
|
||||
---
|
||||
name: cyclone-spec
|
||||
description: "Cyclone SP-N superpowers increment flow — spec → plan → implement → merge. Use when: starting a new numbered feature increment, naming a branch, opening a SP-N PR, or doing the merge dance into main."
|
||||
---
|
||||
|
||||
# cyclone-spec
|
||||
|
||||
The Cyclone repo ships every new feature as a numbered **SP-N increment**:
|
||||
a spec, a plan, an implementation branch, and a single atomic merge commit
|
||||
into `main`. This skill encodes the conventions so every increment follows
|
||||
the same shape and the commit history stays auditable.
|
||||
|
||||
As of this writing: **17 specs** in `docs/superpowers/specs/`, **13 plans**
|
||||
in `docs/superpowers/plans/`, and SP numbers used through **SP22**. **SP23**
|
||||
is the Ubuntu + Docker + RBAC product fork (awaiting user decision);
|
||||
**SP24** is the auth-posture alignment (docs-only). The next free increment
|
||||
is **SP25** after SP24 lands.
|
||||
|
||||
## Auth-aware spec template (SP24)
|
||||
|
||||
The threat-model section in the canonical SP-N spec template (`## 1. Scope`, second-to-last bullet) used to read "no second party to authenticate; no second host to harden against." **That phrasing is stale as of 2026-06-23** — the auth work landed in `main` and every backend endpoint requires login. New specs should instead state the auth boundary explicitly: "the auth boundary is HTTP (login required, bcrypt + HttpOnly session cookie); file-system threats remain the local-only threat model (SQLCipher at rest, macOS Keychain). SP23 changes the threat model to LAN-bound remote operator." Reference: [`docs/superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md`](../../../docs/superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md).
|
||||
|
||||
## When to use
|
||||
|
||||
- **Starting a new feature increment.** You are about to add a numbered
|
||||
feature, fix that crosses subsystem boundaries, or anything bigger than
|
||||
a one-line change. Before you write code, reserve the next SP number and
|
||||
write the spec.
|
||||
- **Naming the spec / plan files or the branch.** You have a topic, a
|
||||
date, and a number — and you need the exact path / branch shape so
|
||||
existing scripts and reviewers can find the artifacts.
|
||||
- **Opening the SP-N PR.** You are about to push the branch and need the
|
||||
PR title format and the commit-prefix conventions so the merge commit
|
||||
reads cleanly.
|
||||
- **Doing the merge dance.** Review is approved and you're about to land
|
||||
the branch into `main`. Use this skill to confirm the merge shape — no
|
||||
squash, no rebase, one atomic merge commit.
|
||||
|
||||
## Conventions
|
||||
|
||||
1. **Numbering.** Reserve the next SP-N number — the next integer after
|
||||
the highest `SP<n>` already used in `git log`. Never reuse a number,
|
||||
even after deletion. Numbering is monotonic and lives in the merge
|
||||
history.
|
||||
2. **Branch.** `sp<N>-<short-kebab-topic>` — e.g. `sp22-line-reconciliation`,
|
||||
`sp9-multi-payer-npi`. Kebab-case, lowercase, no spaces, no slashes.
|
||||
The branch name is the canonical handle for the increment.
|
||||
3. **Spec path.** `docs/superpowers/specs/YYYY-MM-DD-cyclone-<topic>-design.md`
|
||||
with header `Status: Draft, pending user review`. One spec per
|
||||
increment. Real examples: `2026-06-19-cyclone-db-reconciliation-design.md`
|
||||
(SP3), `2026-06-20-cyclone-multi-payer-npi-sftp-design.md` (SP9).
|
||||
4. **Plan path.** `docs/superpowers/plans/YYYY-MM-DD-cyclone-<topic>.md`.
|
||||
Header per the upstream `superpowers:writing-plans` skill: a
|
||||
`For agentic workers:` line that names
|
||||
`superpowers:subagent-driven-development` or
|
||||
`superpowers:executing-plans`, plus a `Goal / Architecture / Tech
|
||||
Stack / Spec` metadata block, then numbered tasks with
|
||||
`- [ ] Step N:` checkboxes.
|
||||
5. **Commit prefix.** All commits on the branch follow these prefixes —
|
||||
they make the SP-N merge commit readable and let `git log --grep`
|
||||
filter cleanly:
|
||||
- `feat(sp<N>): …` — implementation commits (e.g. `feat(sp20): NPI Luhn checksum + Tax ID format validation`).
|
||||
- `docs(spec): …` — landing the spec (e.g. `docs(spec): design for CycloneStore split (Step 4)`).
|
||||
- `docs(plan): …` — landing the plan.
|
||||
- `merge: SP<N> <topic> into main` — the merge commit itself (e.g. `merge: SP14 5-lane Inbox UI + acknowledge action into main`).
|
||||
6. **PR title.** `SP<N> <Topic>` — e.g. `SP22 Line reconciliation`.
|
||||
Matches the merge-commit subject so GitHub's "merged PR" view and the
|
||||
`git log` entry are identical strings.
|
||||
7. **Merge shape.** A single atomic merge commit into `main` after
|
||||
review. **No squash** — squash collapses the per-commit history and
|
||||
breaks the SP-N audit trail. **No rebase** — rebase rewrites the SHAs
|
||||
the PR review was performed against. The SP-N merge commit *is* the
|
||||
record of the increment landing.
|
||||
|
||||
## Patterns
|
||||
|
||||
### Spec header (canonical SP-N shape, post-SP9)
|
||||
|
||||
This is the canonical header for new specs. Older specs (pre-SP9) deviate
|
||||
slightly — different title style, no Branch or Aesthetic direction line —
|
||||
and have not been retroactively normalized. **Use this template for any
|
||||
new SP-N spec.**
|
||||
|
||||
```markdown
|
||||
# Sub-project <N> — <Topic>: Design Spec
|
||||
|
||||
**Date:** YYYY-MM-DD
|
||||
**Status:** Draft, awaiting user sign-off
|
||||
**Branch:** `sp<N>-<short-kebab-topic>`
|
||||
**Aesthetic direction:** <one line — e.g. "No new UI" or "Modern (geometric sans + bold borders + electric blue accent)">
|
||||
|
||||
## 1. Scope
|
||||
<2-6 lines: what's in, what's out, with explicit out-of-scope list>
|
||||
```
|
||||
|
||||
Worked example (matches this template): `docs/superpowers/specs/2026-06-20-cyclone-multi-payer-npi-sftp-design.md`.
|
||||
|
||||
### Plan header (every SP-N plan starts with this)
|
||||
|
||||
```markdown
|
||||
# <Topic> Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use
|
||||
> superpowers:subagent-driven-development (recommended) or
|
||||
> superpowers:executing-plans to implement this plan task-by-task. Steps
|
||||
> use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** <one sentence — the outcome>
|
||||
**Architecture:** <one paragraph — how it's structured>
|
||||
**Tech Stack:** <comma-separated list>
|
||||
**Spec:** [`docs/superpowers/specs/YYYY-MM-DD-cyclone-<topic>-design.md`](../specs/...)
|
||||
|
||||
---
|
||||
|
||||
## File structure
|
||||
<tree of new / modified files>
|
||||
|
||||
## Task 0: <setup>
|
||||
## Task 1: <first user-visible step>
|
||||
…
|
||||
```
|
||||
|
||||
Real examples: `docs/superpowers/plans/2026-06-21-cyclone-skill-catalog.md`,
|
||||
`docs/superpowers/plans/2026-06-21-cyclone-store-split.md`.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Don't skip the spec ("it's a small fix").** Small fixes still get a
|
||||
3-line spec when they introduce a new numbered increment. The spec is
|
||||
the *what* and the audit trail; the plan is the *how*. Without a spec
|
||||
the merge commit has no anchor.
|
||||
- **Don't squash the merge commit.** The SP-N merge commit is the audit
|
||||
trail — it tells future you exactly which feature landed and which
|
||||
commits composed it. Squash collapses that into one opaque commit and
|
||||
the per-commit history is lost.
|
||||
- **Don't put code in the spec — the spec is the *what*, the plan is the
|
||||
*how*.** Specs describe scope, goals, non-goals, and decisions. Code
|
||||
snippets belong in the plan (with checkbox steps) or in the diff, not
|
||||
in the spec. SP-N specs in this repo routinely have **zero** code
|
||||
blocks.
|
||||
|
||||
## Related skills
|
||||
|
||||
- **`cyclone-tests`** — every spec lists test impact; load this when
|
||||
drafting or reviewing the spec to confirm fixture / `.test.tsx`
|
||||
implications.
|
||||
- **`cyclone-edi`** — load when the SP-N increment touches an EDI parser,
|
||||
validator rule, or CAS mapping.
|
||||
- **`cyclone-tail`** — load when the increment changes the live-tail wire
|
||||
format or adds a streaming page.
|
||||
- **`cyclone-store`** — load when the increment adds a write-path,
|
||||
touches `store.py`, or wires a new `<entity>_written` event.
|
||||
- **`cyclone-api-router`** — load when the increment adds or changes an
|
||||
HTTP endpoint in `api_routers/`.
|
||||
- **`cyclone-frontend-page`** — load when the increment adds or
|
||||
refactors a page in `src/pages/`.
|
||||
- **`cyclone-cli`** — load when the increment adds a CLI subcommand or
|
||||
changes exit codes.
|
||||
- **`superpowers:brainstorming`** (global) — run before the spec to lock
|
||||
the scope / decisions in the spec's `## Decisions (locked during
|
||||
brainstorming)` section.
|
||||
- **`superpowers:writing-plans`** (global) — produces the plan header
|
||||
format every SP-N plan follows.
|
||||
@@ -1,200 +0,0 @@
|
||||
---
|
||||
name: cyclone-store
|
||||
description: "Cyclone store write-paths, the pubsub event contract (claim_written / remittance_written / activity_recorded), and the SP21 store-split boundary map. Use when: touching store.py, adding a new entity, wiring a new write event, or splitting a store module."
|
||||
---
|
||||
|
||||
# cyclone-store
|
||||
|
||||
Cyclone persists every parsed X12 batch through one facade,
|
||||
`CycloneStore` (`backend/src/cyclone/store.py:882`). Every write
|
||||
inserts the row AND publishes a pubsub event on the in-process
|
||||
`EventBus` (`backend/src/cyclone/pubsub.py:20`) so live-tail pages
|
||||
see new rows the moment they land. The event contract is the seam
|
||||
between persistence and streaming — a wrong event name silently
|
||||
goes stale.
|
||||
|
||||
As of this writing: the store is a **single 2412-line module** and
|
||||
the SP21 split into `backend/src/cyclone/store/` is in progress.
|
||||
Three event kinds: `claim_written`, `remittance_written`,
|
||||
`activity_recorded`. Next: **SP22**.
|
||||
|
||||
## When to use
|
||||
|
||||
- **Adding a new entity.** You need a new ORM model + write method +
|
||||
event kind + snapshot serializer and want to know where each piece
|
||||
lives (current monolith vs. post-SP21 module).
|
||||
- **Wiring a new write event.** You're adding a `<entity>_written`
|
||||
event and need both the publish call in the store AND the
|
||||
subscribe call in the live-tail endpoint to stay in sync.
|
||||
- **Debugging a write-path issue.** A page isn't reflecting new
|
||||
rows, the DB has the row but the stream is silent, or the publish
|
||||
raises and rolls back the transaction.
|
||||
- **Splitting a store module.** You're moving a domain out of
|
||||
`store.py` into its own module under `backend/src/cyclone/store/`
|
||||
and need the SP21 module list + facade re-export rules.
|
||||
|
||||
## Conventions
|
||||
|
||||
1. **All writes go through `CycloneStore`.** Route handlers and
|
||||
parsers must not write directly to the ORM session. The facade
|
||||
opens a short-lived session via `db.SessionLocal()()`. Direct ORM
|
||||
access is the #1 way the live-tail contract gets bypassed (no
|
||||
event published → page silently goes stale). Read paths are
|
||||
similar — prefer the facade's `iter_*` / `get_*` methods over raw
|
||||
`s.execute(select(...))`.
|
||||
2. **Every write publishes an event.** The event name matches the
|
||||
entity: `claim_written`, `remittance_written`,
|
||||
`activity_recorded` (the trailing `_recorded` signals a
|
||||
non-canonical row — activity events are derived, not first-class;
|
||||
current names at `store.py:1072,1081,1096`). New entities get
|
||||
`<entity>_written`; activity-style side rows get
|
||||
`<entity>_recorded`. Publish is **best-effort** — failures are
|
||||
logged but never roll back the persisted batch
|
||||
(`store.py:1097-1098`).
|
||||
3. **Snapshot shape.** Each entity has a `to_ui_<entity>` serializer
|
||||
(plain Python function returning a dict) — currently
|
||||
`to_ui_claim`, `to_ui_remittance`, `to_ui_claim_from_orm`,
|
||||
`to_ui_remittance_from_orm`, `to_ui_provider`. Post-SP21 these
|
||||
move to `backend/src/cyclone/store/ui.py`. The serializer is the
|
||||
single source of truth for what the frontend sees — every event
|
||||
payload MUST match what the matching list endpoint returns for
|
||||
that row (`store.py:1052-1054`).
|
||||
4. **SP21 boundaries.** Post-split, each domain lives in its own
|
||||
module under `backend/src/cyclone/store/`: `__init__.py` (facade
|
||||
+ `CycloneStore` class), `exceptions.py`, `records.py`,
|
||||
`orm_builders.py`, `ui.py`, `write.py`, `batches.py`,
|
||||
`claim_detail.py`, `acks.py`, `backups.py`, `inbox.py`,
|
||||
`providers.py`. Cross-module writes go through `CycloneStore`
|
||||
facade methods, not direct module access. The facade re-exports
|
||||
every name callers currently import from `cyclone.store`. Full
|
||||
list: `docs/superpowers/plans/2026-06-21-cyclone-store-split.md:25-37`.
|
||||
5. **No business logic in route handlers.** A route handler validates
|
||||
input, calls `store.<method>(...)`, passes the `event_bus`,
|
||||
returns the serialized result. Reconciliation, idempotency
|
||||
checks, CAS adjustment persistence — all live in the store.
|
||||
|
||||
## Patterns
|
||||
|
||||
### A `CycloneStore.add` write — publishes events from inserted rows
|
||||
|
||||
Taken from `backend/src/cyclone/store.py:898-1107`. The method opens
|
||||
a session, inserts rows, then runs a sync `_publish_events_sync`
|
||||
after commit so subscribers can immediately re-fetch consistent data.
|
||||
|
||||
```python
|
||||
def add(
|
||||
self,
|
||||
record: BatchRecord,
|
||||
*,
|
||||
event_bus: "EventBus | None" = None,
|
||||
) -> None:
|
||||
inserted_claim_ids: list[str] = []
|
||||
with db.SessionLocal()() as s:
|
||||
s.add(Batch(id=record.id, kind=record.kind, ...))
|
||||
if isinstance(record, BatchRecord837):
|
||||
for claim in record.result.claims:
|
||||
if s.get(Claim, claim.claim_id) is not None:
|
||||
continue # idempotency: skip dupes
|
||||
s.add(_claim_837_row(claim, record.id))
|
||||
s.add(ActivityEvent(kind="claim_submitted", ...))
|
||||
inserted_claim_ids.append(claim.claim_id)
|
||||
# ... 835 branch + flush + cas adjustments ...
|
||||
s.commit()
|
||||
|
||||
if event_bus is not None and inserted_claim_ids:
|
||||
self._publish_events_sync(event_bus, record, inserted_claim_ids)
|
||||
|
||||
def _publish_events_sync(self, event_bus, record, claim_ids):
|
||||
with db.SessionLocal()() as s:
|
||||
for cid in claim_ids:
|
||||
ui = to_ui_claim_from_orm(s.get(Claim, cid), ...)
|
||||
self._sync_publish(event_bus, "claim_written", ui)
|
||||
# ... remittance + activity loops ...
|
||||
```
|
||||
|
||||
`EventBus.publish` is async but the body is pure sync `put_nowait`,
|
||||
so the store calls `_sync_publish` directly to avoid forcing sync
|
||||
FastAPI handlers to await.
|
||||
|
||||
### Backend `/api/<resource>/stream` endpoint — subscribes to the event
|
||||
|
||||
Taken from `backend/src/cyclone/api.py:1380-1401`. Two phases — eager
|
||||
snapshot, then live subscription — wrapped in `StreamingResponse`
|
||||
with `media_type="application/x-ndjson"`.
|
||||
|
||||
```python
|
||||
@app.get("/api/claims/stream")
|
||||
async def claims_stream(request: Request, ...) -> StreamingResponse:
|
||||
bus: EventBus = request.app.state.event_bus
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
rows = store.iter_claims(status=status, ...) # 1. Snapshot
|
||||
for row in rows:
|
||||
yield _ndjson_line({"type": "item", "data": row})
|
||||
yield _ndjson_line({"type": "snapshot_end", "data": {"count": len(rows)}})
|
||||
|
||||
async for chunk in _tail_events(request, bus, ["claim_written"]): # 2. Live
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
```
|
||||
|
||||
`_ndjson_line` and `_tail_events` live in
|
||||
`backend/src/cyclone/api_helpers.py`. The `["remittance_written"]`
|
||||
and `["activity_recorded"]` subscriptions are at `api.py:1891,2002`.
|
||||
|
||||
### Backend test — asserts both the row AND the event landed
|
||||
|
||||
The autouse `conftest.py` (`backend/tests/conftest.py:20`) wires a
|
||||
fresh `EventBus` onto `app.state` per test.
|
||||
|
||||
```python
|
||||
def test_publishes_claim_written_event(client: TestClient) -> None:
|
||||
bus = app.state.event_bus
|
||||
resp = client.post("/api/parse-837",
|
||||
files={"file": ("x.837", MINIMAL_837, "text/plain")})
|
||||
assert resp.status_code == 200
|
||||
assert list(store.iter_claims(limit=10)) # row landed
|
||||
|
||||
queues = bus._subscribers.get("claim_written", []) # event published
|
||||
assert queues
|
||||
evt = queues[0].get_nowait()
|
||||
assert evt["_kind"] == "claim_written"
|
||||
```
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Don't read directly from the ORM in a route handler — go through
|
||||
the snapshot serializer.** A handler that does
|
||||
`with db.SessionLocal()() as s: row = s.get(Claim, cid); return row`
|
||||
bypasses `to_ui_claim_from_orm` and silently drifts from the event
|
||||
payload shape. Always call `store.get_claim_detail(cid)` (or the
|
||||
equivalent `get_*` facade method).
|
||||
- **Don't introduce a new event name without updating the subscriber
|
||||
list.** Today every `<entity>_written` event has exactly one
|
||||
consumer — the matching `/api/<resource>/stream` endpoint,
|
||||
currently in `backend/src/cyclone/api.py` (claims at `:1398`,
|
||||
remittances at `:1891`, activity at `:2002`). Any future split
|
||||
into `backend/src/cyclone/api_routers/` must wire the same
|
||||
subscription.
|
||||
- **Don't merge a write method with its event publication into
|
||||
separate places.** `_publish_events_sync` lives next to `add` in
|
||||
`store.py:1042-1107` so reviewers see both halves of the contract
|
||||
in one diff. Post-SP21 the same rule applies.
|
||||
- **Don't make the publish call blocking on commit failures.**
|
||||
Publish errors are caught and logged at `store.py:1097-1098`; a
|
||||
failing subscriber MUST NOT roll back the persisted batch. The
|
||||
batch is the source of truth; the event is the cache-invalidation
|
||||
hint.
|
||||
|
||||
## Related skills
|
||||
|
||||
- **`cyclone-edi`** — the parsed `ClaimOutput` / `ParseResult<EDI>`
|
||||
lands in the store via `CycloneStore.add`.
|
||||
- **`cyclone-api-router`** — the route that calls `store.<method>(...)`
|
||||
and (for stream endpoints) subscribes to the matching
|
||||
`<entity>_written` event.
|
||||
- **`cyclone-tail`** — the consumer side of the event contract;
|
||||
load when changing the wire format or adding a streaming hook.
|
||||
- **`cyclone-tests`** — write-path tests live under `backend/tests/`
|
||||
and assert both the DB row and the event payload.
|
||||
@@ -1,200 +0,0 @@
|
||||
---
|
||||
name: cyclone-tail
|
||||
description: "Cyclone live-tail streaming wire format and the useTailStream / useMergedTail hook triplet. Use when: adding a new streaming list page, changing the wire format, debugging stalled/reconnecting state, or modifying the StatusPill behavior."
|
||||
---
|
||||
|
||||
# cyclone-tail
|
||||
|
||||
Cyclone keeps the Claims, Remittances, and Activity pages live without
|
||||
polling: every store write publishes an internal EventBus event, the
|
||||
page opens a `GET /api/<resource>/stream` HTTP/1.1 chunked-NDJSON
|
||||
connection, and new rows land in the table the moment they hit the
|
||||
database. This skill codifies the wire format, the hook triplet, and
|
||||
the backoff/stall machinery so additions stay consistent with the
|
||||
three streaming pages already shipped (`Claims`, `Remittances`,
|
||||
`ActivityLog`).
|
||||
|
||||
## When to use
|
||||
|
||||
- **Adding a new streaming page.** Mounting `useTailStream(resource)`
|
||||
on a page — you need the hook triplet shape (initial fetch +
|
||||
`useTailStream` + `useMergedTail`), the `<TailStatusPill>` wiring,
|
||||
and the dedup rules in `useMergedTail`.
|
||||
- **Changing the wire format.** Adding a new event type — update
|
||||
`TailEvent` in `src/lib/tail-stream.ts:22-44`, the dispatch switch
|
||||
in `src/hooks/useTailStream.ts:173-195`, the emitter in
|
||||
`backend/src/cyclone/api_helpers.py:tail_events()`, and
|
||||
`references/wire-format.md`.
|
||||
- **Debugging stalled/reconnecting state.** Confirm whether the
|
||||
backend is heartbeating (`CYCLONE_TAIL_HEARTBEAT_S`, default `15s`)
|
||||
or the stall timer fired (`STALL_TIMEOUT_MS = 30_000` at
|
||||
`src/hooks/useTailStream.ts:53`).
|
||||
- **Tuning heartbeat/stall timing.** Changing the 30s stall threshold
|
||||
or the 15s heartbeat interval — README's "Status pill" + "Knobs"
|
||||
tables need to stay in sync (`README.md:109-129`).
|
||||
|
||||
## Conventions
|
||||
|
||||
1. **Wire format.** Newline-delimited JSON. Every line is
|
||||
`{"type": ..., "data": ...}`. Known `type` values: `item`
|
||||
(per-row envelope), `snapshot_end` (`{"count": N}` marker after the
|
||||
snapshot), `heartbeat` (`{"ts": "<iso-8601>"}` keep-alive),
|
||||
`item_dropped` (`{"id": "..."}` queue-overflow notice), `error`
|
||||
(`{"message": "..."}` promoted to a thrown error by the hook).
|
||||
Defined at `src/lib/tail-stream.ts:22-44`; emitted by
|
||||
`backend/src/cyclone/api.py:1357-2006`. See
|
||||
`references/wire-format.md`.
|
||||
2. **Hook triplet.** Streaming pages compose three pieces:
|
||||
`useTailStream(resource)` (`src/hooks/useTailStream.ts:80` — opens
|
||||
the stream, drives the backoff/stall state machine, dispatches
|
||||
`item` events into `useTailStore`),
|
||||
`useMergedTail(resource, baseItems, filterFn?)`
|
||||
(`src/hooks/useMergedTail.ts:25` — returns
|
||||
`baseItems + tailSlice` dedup'd against `baseItems`), and a
|
||||
per-resource initial-fetch hook (`useClaims`, `useRemittances`,
|
||||
`useActivity`). The page wires all three — see
|
||||
`src/pages/Claims.tsx:87-89`.
|
||||
3. **Stall threshold.** 30 seconds of total silence — heartbeats
|
||||
included — flips status to `stalled` and surfaces the
|
||||
`↻ Reconnect` button on `<TailStatusPill>`. Constant:
|
||||
`STALL_TIMEOUT_MS = 30_000` at `src/hooks/useTailStream.ts:53`.
|
||||
Re-armed on every event including `heartbeat` and `item_dropped`
|
||||
(`useTailStream.ts:124-140`). Don't change without updating
|
||||
`README.md:109-123`.
|
||||
4. **Snapshot first.** Every stream emits the snapshot before any
|
||||
live `item` events, then closes with exactly one `snapshot_end`
|
||||
carrying `{"count": N}`. The hook uses `snapshot_end` to flip
|
||||
`<TailStatusPill>` from `connecting` to `live` and to reset the
|
||||
reconnect backoff counter (`useTailStream.ts:174-180`).
|
||||
5. **Content-Type.** Stream endpoints respond with
|
||||
`media_type="application/x-ndjson"` — see
|
||||
`backend/src/cyclone/api.py:1401,1894,2005`. Frontend sets
|
||||
`Accept: application/x-ndjson` at `src/lib/tail-stream.ts:67-70`.
|
||||
Never `application/json` for a stream endpoint.
|
||||
6. **Backoff.** Transient errors retry with `1s → 2s → 4s → 8s → 16s
|
||||
→ 30s` capped — `BACKOFF_STEPS_MS` at
|
||||
`src/hooks/useTailStream.ts:48-50`. Counter resets on every
|
||||
`snapshot_end`.
|
||||
7. **Heartbeat knob.** Idle heartbeat interval is configurable via
|
||||
`CYCLONE_TAIL_HEARTBEAT_S` (default `15`, parsed at call time in
|
||||
`backend/src/cyclone/api_helpers.py:185-198`). Tests override to
|
||||
a small value to keep runtime bounded.
|
||||
|
||||
## Patterns
|
||||
|
||||
### Page-hook skeleton — `Claims.tsx`
|
||||
|
||||
Pattern from `src/pages/Claims.tsx:85-90`. Three hooks in order:
|
||||
`useClaims(params)` (initial TanStack Query fetch),
|
||||
`useTailStream("claims")` (opens the stream, owns status state), and
|
||||
`useMergedTail("claims", data?.items ?? [], tailFilterFn)` (combines
|
||||
initial snapshot with live tail, dedup'd by id, filtered by the page's
|
||||
predicate applied AFTER dedup at `useMergedTail.ts:72-74`).
|
||||
|
||||
```ts
|
||||
import { useClaims } from "@/hooks/useClaims";
|
||||
import { useTailStream } from "@/hooks/useTailStream";
|
||||
import { useMergedTail } from "@/hooks/useMergedTail";
|
||||
|
||||
export function ClaimsPage() {
|
||||
const { data } = useClaims({ status: "submitted" });
|
||||
const { status, lastEventAt, forceReconnect } = useTailStream("claims");
|
||||
const tailFilterFn = (c: Claim) => c.status === "submitted";
|
||||
const items = useMergedTail("claims", data?.items ?? [], tailFilterFn);
|
||||
// <TailStatusPill status={status} lastEventAt={lastEventAt}
|
||||
// onReconnect={forceReconnect} /> + <Table items={items} />
|
||||
}
|
||||
```
|
||||
|
||||
### Backend `/api/foo/stream` endpoint
|
||||
|
||||
Pattern from `backend/src/cyclone/api.py:1357-1401`. Register BEFORE
|
||||
`/api/foo/{foo_id}` so the literal `stream` segment doesn't match as
|
||||
an id. Two phases — eager snapshot, then live subscription — wrapped
|
||||
in `StreamingResponse` with `media_type="application/x-ndjson"`.
|
||||
|
||||
```python
|
||||
@app.get("/api/foo/stream")
|
||||
async def foo_stream(
|
||||
request: Request,
|
||||
status: str | None = Query(None),
|
||||
limit: int = Query(100, ge=1, le=1000),
|
||||
) -> StreamingResponse:
|
||||
bus: EventBus = request.app.state.event_bus
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
# 1. Snapshot.
|
||||
rows = store.iter_foos(status=status, limit=limit)
|
||||
for row in rows:
|
||||
yield _ndjson_line({"type": "item", "data": row})
|
||||
yield _ndjson_line({"type": "snapshot_end", "data": {"count": len(rows)}})
|
||||
# 2. Live subscription + heartbeats.
|
||||
async for chunk in _tail_events(request, bus, ["foo_written"]):
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
```
|
||||
|
||||
`_ndjson_line` and `_tail_events` live in
|
||||
`backend/src/cyclone/api_helpers.py`; the latter forwards EventBus
|
||||
events as `item` lines and emits `heartbeat` lines on the cadence
|
||||
from `heartbeat_seconds()`.
|
||||
|
||||
### `<TailStatusPill>` wiring
|
||||
|
||||
Pattern from `src/components/TailStatusPill.tsx:22-83`. The pill
|
||||
takes `status` + `lastEventAt` from `useTailStream` plus
|
||||
`forceReconnect` so the `↻ Reconnect` button wires to the hook's
|
||||
`reconnectNonce` bump (aborts and reopens the stream —
|
||||
`useTailStream.ts:99-101`). The button only renders when
|
||||
`status === "stalled" || status === "error"` (`TailStatusPill.tsx:52`).
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Don't hand-roll `fetch` + `ReadableStream` parsing in a page.**
|
||||
All stream consumers go through `streamTail(resource, opts?)` at
|
||||
`src/lib/tail-stream.ts:58`. The parser handles `TextDecoderStream`,
|
||||
newline splitting, malformed-line tolerance (`console.warn` + skip),
|
||||
the `KNOWN_TYPES` allowlist, and abort-on-signal semantics
|
||||
(`tail-stream.ts:75,98,107`). Duplicating it loses all of those.
|
||||
- **Don't change the wire format on one endpoint without updating
|
||||
the others.** The parser (`src/lib/tail-stream.ts`) and the hook
|
||||
dispatch switch (`useTailStream.ts:173-195`) are shared across all
|
||||
three live streams. Adding a new event type means updating
|
||||
`TailEvent`, `KNOWN_TYPES`, the dispatch `case`, the `armStall`
|
||||
re-arm list, and the backend emitter — in that order.
|
||||
- **Don't emit `item` events before `snapshot_end`.** The hook uses
|
||||
`snapshot_end` as the marker to flip `<TailStatusPill>` from
|
||||
`connecting` to `live` and to reset the reconnect backoff counter.
|
||||
Emitting `item`s first lands rows in `useTailStore` but the UI
|
||||
still reads "Connecting" — operators see a flash of stale state.
|
||||
Emit snapshot, then `snapshot_end`, then live (`api.py:1386-1401`).
|
||||
- **Don't change the 30s stall threshold without updating the README
|
||||
"Status pill" table.** The constant
|
||||
(`STALL_TIMEOUT_MS = 30_000` at `useTailStream.ts:53`) is the
|
||||
contract the pill is documented against — a bump needs the matching
|
||||
edit at `README.md:118`.
|
||||
- **Don't add `useTailStream` calls from `useFoo.ts` hooks.**
|
||||
Streaming connections belong on the page (`src/pages/Claims.tsx`,
|
||||
`Remittances.tsx`, `ActivityLog.tsx`). Hoisting into `useFoo`
|
||||
couples the lifecycle to whoever mounts it and breaks
|
||||
one-resource-one-page ownership.
|
||||
|
||||
## Related skills
|
||||
|
||||
- **`cyclone-frontend-page`** — page components live in `src/pages/`;
|
||||
load when adding or refactoring a streaming page to confirm the
|
||||
route + `PageHeader` + table conventions.
|
||||
- **`cyclone-api-router`** — endpoint conventions (`api_routers/` for
|
||||
resource-group routers, `api.py` for the live-tail endpoints); load
|
||||
when adding or changing an HTTP endpoint that surfaces streamed or
|
||||
paginated data.
|
||||
- **`cyclone-store`** — write-path conventions in `store.py` and the
|
||||
pubsub event contract (`claim_written`, `remittance_written`,
|
||||
`activity_recorded`); load when adding a new entity whose writes
|
||||
should fan out to a stream endpoint.
|
||||
- **`cyclone-tests`** — frontend `*.test.tsx` siblings cover
|
||||
`useTailStream`, `useMergedTail`, `TailStatusPill`; backend
|
||||
`test_api_stream_live.py` covers the three live-tail endpoints;
|
||||
load when the increment changes wire-format behavior or adds a
|
||||
streaming hook.
|
||||
@@ -1,76 +0,0 @@
|
||||
# Live-tail wire format — field reference
|
||||
|
||||
The Cyclone live-tail NDJSON contract is owned by the frontend parser
|
||||
(`src/lib/tail-stream.ts:22-44`) and the backend emitter
|
||||
(`backend/src/cyclone/api_helpers.py:tail_events()` + the three
|
||||
endpoints in `backend/src/cyclone/api.py:1357-2006`). This file is the
|
||||
quick reference; the canonical prose lives in the README and the
|
||||
source-of-truth type definitions.
|
||||
|
||||
## Source
|
||||
|
||||
Wire format excerpt copied verbatim from `README.md:76-94` (the
|
||||
"Live updates → Wire format" section). The README is the user-facing
|
||||
exposition; this reference adds the per-line field semantics and the
|
||||
parser-tolerance rules from the code.
|
||||
|
||||
> Each stream endpoint emits newline-delimited JSON. The first batch
|
||||
> is the **snapshot** of currently-known rows; after that comes
|
||||
> **`snapshot_end`** with the count, then the **live** events.
|
||||
>
|
||||
> ```json
|
||||
> {"type":"item","data":{"id":"CLM-1", "...":"..."}}
|
||||
> {"type":"item","data":{"id":"CLM-2", "...":"..."}}
|
||||
> {"type":"snapshot_end","data":{"count":2}}
|
||||
> {"type":"item","data":{"id":"CLM-3", "...":"..."}} ← live
|
||||
> {"type":"heartbeat","data":{"ts":"2026-06-20T23:17:09Z"}} ← idle keep-alive
|
||||
> ```
|
||||
>
|
||||
> Lines are `{"type": ..., "data": ...}`; known types are `item`,
|
||||
> `snapshot_end`, `heartbeat`, and (rare) `item_dropped` /
|
||||
> `error`. Heartbeats keep the connection alive when nothing is
|
||||
> happening — clients flip to `stalled` after 30s of total silence
|
||||
> (heartbeat or otherwise) and surface a **↻ Reconnect** button.
|
||||
|
||||
## Per-line field reference
|
||||
|
||||
| `type` | `data` shape | Required? | Emitted by | Parser behavior |
|
||||
| -------------- | ----------------------------------------- | --------- | ------------------------------------------- | -------------------------------------------- |
|
||||
| `item` | resource-specific row (`Claim` / `Remittance` / `Activity`) | yes, in `data` (the per-row envelope) | snapshot loop + `_tail_events` forwarding `claim_written` / `remittance_written` / `activity_recorded` | `dispatch(resource, ev.data)` → `useTailStore.addClaim` / `addRemittance` / `addActivity` (first-write-wins dedup on the id-keyed slices — `tail-store.ts:104,120`). |
|
||||
| `snapshot_end` | `{"count": N}` (integer ≥ 0) | yes, on every stream | `api.py:1395,1889,1999` (one per stream, after the snapshot loop) | Flips `<TailStatusPill>` from `connecting` to `live`, resets the reconnect backoff counter to 0 (`useTailStream.ts:174-180`). |
|
||||
| `heartbeat` | `{"ts": "<iso-8601 UTC>"}` | yes, but only when idle | `_tail_events` in `api_helpers.py:241-245` (cadence from `heartbeat_seconds()`, default 15s) | Re-arms the stall timer (`useTailStream.ts:124-140`); no state change. |
|
||||
| `item_dropped` | `{"id": "<string>"}` | optional (rare; queue overflow) | EventBus drop-oldest path (per the spec at `docs/superpowers/specs/2026-06-20-cyclone-live-tail-design.md:299`) | Re-arms the stall timer; no state change. The `id` field is informational — the hook does not refetch on drop, the page does (see Spec §3.6). |
|
||||
| `error` | `{"message": "<string>"}` | optional (server-side failure) | Reserved for future server-emitted errors (currently only thrown client-side) | Hook promotes to a thrown `Error(message)` so the catch block runs the reconnect machinery (`useTailStream.ts:190-194`). |
|
||||
|
||||
## Parser tolerance
|
||||
|
||||
The shared parser at `src/lib/tail-stream.ts` is intentionally
|
||||
forgiving so a single bad frame doesn't kill the stream:
|
||||
|
||||
- **Trailing `\r`** is stripped per line (`tail-stream.ts:116`) so a
|
||||
CRLF-terminated stream still parses.
|
||||
- **Malformed JSON** (`JSON.parse` throws) → `console.warn` + skip the
|
||||
line; the iterator continues (`tail-stream.ts:122-131`).
|
||||
- **Unknown `type`** (not in `KNOWN_TYPES`) → `console.warn` + skip
|
||||
(`tail-stream.ts:141-148`). Adding a new event type is
|
||||
forward-compatible: old clients see warn lines, new clients see the
|
||||
typed event.
|
||||
- **Empty lines** (consecutive `\n`s) → silently skipped
|
||||
(`tail-stream.ts:118`).
|
||||
- **No trailing newline** → flushed as a final partial line on
|
||||
stream close (`tail-stream.ts:154-178`).
|
||||
- **Abort signal** → iterator exits cleanly without throwing
|
||||
(`tail-stream.ts:75,98,107`).
|
||||
|
||||
## Endpoint inventory
|
||||
|
||||
| Method | Path | Subscribes to | Default sort | Defined at |
|
||||
| ------ | ------------------------- | -------------------- | ------------------- | ----------------------------------- |
|
||||
| GET | `/api/claims/stream` | `claim_written` | `-submission_date` | `backend/src/cyclone/api.py:1357` |
|
||||
| GET | `/api/remittances/stream` | `remittance_written` | `-received_date` | `backend/src/cyclone/api.py:1858` |
|
||||
| GET | `/api/activity/stream` | `activity_recorded` | `-timestamp` (limit 50) | `backend/src/cyclone/api.py:1971` |
|
||||
|
||||
All three accept the same query params as their non-streaming
|
||||
counterparts (`status`, `payer`, `date_from`, …) so a frontend can
|
||||
swap a one-shot fetch for a tail with no URL surgery. Responses are
|
||||
`Content-Type: application/x-ndjson`.
|
||||
@@ -1,170 +0,0 @@
|
||||
---
|
||||
name: cyclone-tests
|
||||
description: "Cyclone pytest + vitest fixture patterns, prodfiles layout, backend/tests/fixtures/ conventions, .test.tsx sibling rule. Use when: adding a backend pytest case, adding a frontend vitest test, or wiring in a real-EDI prodfiles sample."
|
||||
---
|
||||
|
||||
# cyclone-tests
|
||||
|
||||
The Cyclone test suite is split two ways: **backend pytest** (89 test files under `backend/tests/`, 13 flat fixtures in `backend/tests/fixtures/`, one autouse `conftest.py` that resets the DB per-test) and **frontend vitest** (59 `*.test.ts(x)` siblings across `src/`, two rendering styles — `@testing-library/react` and a custom `createRoot`+`Probe` shim). This skill codifies the conventions so additions stay consistent with what's already there.
|
||||
|
||||
As of this writing: **89 backend test files**, **13 flat backend fixtures**, **59 frontend `*.test.ts(x)` siblings**, and **23 prodfiles samples** across `docs/prodfiles/{837p-from-axiscare,835fromco,FromHPE,claims}/`. The next increment is **SP24** (SP23 is the Ubuntu+Docker fork, awaiting user decision).
|
||||
|
||||
## Auth flag (SP24)
|
||||
|
||||
The autouse `conftest.py` fixture at `backend/tests/conftest.py` flips `cyclone.auth.deps.AUTH_DISABLED = True` for the entire test session, so every test runs without a login round-trip. **Any new test that reads `cyclone.auth.deps.AUTH_DISABLED` directly will see `True`** — that's the test-suite reality, not a production reality. If you need a test that exercises the real gate, import `from cyclone.auth.deps import matrix_gate` and call it directly with a `Request` whose `state` carries a real session, or flip the flag back inside the test and reset it on teardown. The startup WARNING (`backend/src/cyclone/__main__.py`) is silent in tests by default because the conftest sets the flag before `bootstrap.run()` is called via `import`.
|
||||
|
||||
## When to use
|
||||
|
||||
- **Adding a backend pytest case.** You're about to add a new `test_*.py` under `backend/tests/` and need the autouse DB-fixture rules, the fixture-path convention, and the right naming flavor (`test_api_*.py` vs `test_<module>_*.py`).
|
||||
- **Adding a frontend vitest test.** You're about to add a `*.test.ts(x)` sibling and need the `// @vitest-environment happy-dom` setup, the act-environment flag, and the right rendering helper (`@testing-library/react` vs the `createRoot`+`Probe` shim).
|
||||
- **Dropping in a prodfiles fixture.** You have a real EDI sample under `docs/prodfiles/<source>/` and need the copy step that makes it test-runnable without coupling the test to the prodfiles archive.
|
||||
- **Debugging a flaky test.** A test passes locally but flakes in CI — load this skill to check the determinism + no-network rules before chasing the symptom.
|
||||
|
||||
## Conventions
|
||||
|
||||
1. **Frontend sibling rule.** Every new file in `src/` that contains testable logic gets a `*.test.ts(x)` next to it. `useFoo.ts` → `useFoo.test.ts`. `ClaimDrawer.tsx` → `ClaimDrawer.test.tsx`. The 59 existing siblings follow this; CI implicitly enforces it via the default `src/**/*.test.ts(x)` glob in `vitest.config.ts`.
|
||||
2. **Backend test location.** Tests live under `backend/tests/test_*.py`. Two flavors, two naming patterns:
|
||||
- **Integration tests** (FastAPI surface) → `test_api_<topic>_<verb>.py` — e.g. `test_api_parse_persists.py`, `test_api_999.py`.
|
||||
- **Pure-unit tests** (parsers, validators, store internals) → `test_<module>_<behavior>.py` — e.g. `test_cas_codes.py`, `test_pubsub.py`.
|
||||
3. **Prodfiles drop-in.** Real EDI samples live under `docs/prodfiles/<source>/<file>.txt` (sources seen so far: `837p-from-axiscare/`, `835fromco/`, `FromHPE/`, `claims/`). To use one in a test: copy the file to `backend/tests/fixtures/<descriptive-name>.txt` (flat — no per-test subdirectories; the existing 13 fixtures all sit at the top level) and reference it from the test as a module-level `Path` constant. See `## Patterns` for the exact line.
|
||||
4. **Determinism.** Time-sensitive tests must not depend on wall-clock time.
|
||||
- **Frontend** uses `vi.useFakeTimers()` + `vi.setSystemTime(new Date("YYYY-MM-DDTHH:MM:SSZ"))` (see `src/components/TailStatusPill.test.tsx:54-58`) and `vi.advanceTimersByTime(ms)` to drive interval / backoff code deterministically.
|
||||
- **Backend** dates are passed explicitly as fixture values — e.g. `datetime.now(timezone.utc)` is fine for "now-ish" anchors, but for fixed dates pass `datetime(2026, 6, 20, tzinfo=timezone.utc)`. The project does **not** currently use `freezegun` or `mock.patch(datetime)`; if you need deterministic date mocking, propose adding `freezegun` to `backend/pyproject.toml` rather than rolling your own.
|
||||
5. **No network.** Tests must not hit the network.
|
||||
- **Backend:** `fastapi.testclient.TestClient(app)` runs in-process; no uvicorn. The autouse `conftest.py` fixture (`backend/tests/conftest.py:20`) points `CYCLONE_DB_URL` at `tmp_path/test.db`, calls `db._reset_for_tests()` + `db.init_db()`, and wires a fresh `EventBus` onto `app.state`.
|
||||
- **Frontend:** `vi.stubGlobal("fetch", vi.fn().mockResolvedValue(...))` for hooks; `vitest.config.ts` sets `VITE_API_BASE_URL=http://test.local` so the `api` module doesn't throw `notConfiguredError` before the mock fires.
|
||||
6. **pytest collection.** Run `cd backend && python -m pytest tests/<file>::<name> -v` for the fastest single-test feedback loop. Run `cd backend && python -m pytest tests/<file> -v` for one file. Run `cd backend && python -m pytest` for the full suite — this is the merge gate. Frontend: `npm test` (alias for `vitest run`) for the full suite; `npx vitest run src/hooks/useFoo.test.ts` for one file.
|
||||
|
||||
## Patterns
|
||||
|
||||
### Backend pytest using a fixture + per-test SQLite DB
|
||||
|
||||
Canonical shape — see `backend/tests/test_api_999.py:1-50` for the full file. The autouse `conftest.py` already provides the DB init + `EventBus` reset; most tests just add a `client` fixture.
|
||||
|
||||
```python
|
||||
"""Tests for the FastAPI surface in cyclone.api for the 999 endpoint."""
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from cyclone.api import app
|
||||
|
||||
# Fixture reference — flat, module-level Path constant. NEVER reach into
|
||||
# docs/prodfiles/ from a test; the fixtures/ dir is the test-consumed surface.
|
||||
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
|
||||
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client() -> TestClient:
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
def test_parse_999_endpoint_happy_path(client: TestClient):
|
||||
text = ACCEPTED.read_text()
|
||||
resp = client.post(
|
||||
"/api/parse-999",
|
||||
files={"file": ("minimal_999.txt", text, "text/plain")},
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["ack"]["ack_code"] == "A"
|
||||
```
|
||||
|
||||
Override the autouse DB fixture only when you need a custom env (e.g. a per-test backup directory) — see `backend/tests/test_999_rejected_state.py:20-25`:
|
||||
|
||||
```python
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup(tmp_path, monkeypatch):
|
||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/inbox.db")
|
||||
from cyclone import db
|
||||
db._reset_for_tests()
|
||||
db.init_db()
|
||||
yield
|
||||
```
|
||||
|
||||
### Frontend vitest `*.test.tsx` for a component using fake timers
|
||||
|
||||
Pattern taken from `src/components/TailStatusPill.test.tsx:1-62` — uses `vi.useFakeTimers()` + `vi.setSystemTime(...)` to drive interval-based code deterministically.
|
||||
|
||||
```ts
|
||||
// @vitest-environment happy-dom
|
||||
// React's act warnings need an act-aware environment — mirror the other
|
||||
// hook tests in this repo.
|
||||
(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { MyComponent } from "./MyComponent";
|
||||
|
||||
describe("MyComponent", () => {
|
||||
beforeEach(() => {
|
||||
vi.useFakeTimers();
|
||||
vi.setSystemTime(new Date("2026-06-20T12:00:30Z"));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
it("test_renders_after_interval_tick", () => {
|
||||
vi.advanceTimersByTime(30_000); // drive the setInterval
|
||||
// …assert…
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Frontend vitest `*.test.tsx` for a component using `@testing-library/react` + `happy-dom`
|
||||
|
||||
Pattern taken from `src/hooks/useInboxLanes.test.ts:1-80`. (A handful of older tests — `useClaimDetail.test.ts`, `useDrawerUrlState.test.ts`, `TailStatusPill.test.tsx` — roll a custom `createRoot`+`Probe` shim instead. Both styles are accepted; `@testing-library/react` is preferred when the hook has async dependencies because `waitFor` is built in.)
|
||||
|
||||
```ts
|
||||
// @vitest-environment happy-dom
|
||||
(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
|
||||
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { act, cleanup, renderHook, waitFor } from "@testing-library/react";
|
||||
import { useMyHook } from "./useMyHook";
|
||||
|
||||
vi.mock("@/lib/api", () => ({
|
||||
api: { fetchFoo: vi.fn() },
|
||||
}));
|
||||
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
vi.unstubAllGlobals();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
describe("useMyHook", () => {
|
||||
it("loads data on mount", async () => {
|
||||
vi.stubGlobal("fetch", vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({ items: [] }),
|
||||
}));
|
||||
const { result } = renderHook(() => useMyHook());
|
||||
await waitFor(() => expect(result.current.loading).toBe(false));
|
||||
expect(result.current.items).toEqual([]);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Don't put frontend tests in `src/__tests__/`.** No such directory exists in the codebase — `__tests__` only appears in the SP-catalog plan itself. Use the sibling rule (`useFoo.ts` → `useFoo.test.ts`).
|
||||
- **Don't reach into `docs/prodfiles/` directly from a test.** Always copy to `backend/tests/fixtures/<name>.txt` first. The prodfiles directory is the source-of-truth archive and may be reorganized; the fixtures directory is the test-consumed surface and is stable.
|
||||
- **Don't use wall-clock sleeps for timing.** A handful of legacy tests use `await new Promise((r) => setTimeout(r, 200))` (see `src/components/SearchBar.test.tsx:281,323,373` and `src/hooks/useSearch.test.ts:174`) — these are known flaky in CI. Use `vi.useFakeTimers()` + `vi.advanceTimersByTime(ms)` instead, or `waitFor(...)` from `@testing-library/react`.
|
||||
|
||||
## Related skills
|
||||
|
||||
- **`cyclone-spec`** — every SP-N spec lists test impact; load this when drafting or reviewing the spec to confirm fixture / `.test.tsx` implications for the increment.
|
||||
- **`cyclone-edi`** — most backend tests cover parser + validator behavior; load when the increment touches an EDI parser or adds a validator rule (R200/R210/NPI Luhn/EIN/CAS).
|
||||
- **`cyclone-tail`** — most frontend tests cover hook behavior (`useTailStream`, `useMergedTail`); load when the increment changes the wire format or adds a streaming hook.
|
||||
- **`cyclone-store`** — write-path tests live here; load when the increment touches `store.py`, adds a new entity, or wires a new `<entity>_written` event.
|
||||
- **`cyclone-api-router`** — endpoint tests live in `backend/tests/test_api_*.py`; load when the increment adds or changes an HTTP endpoint.
|
||||
- **`cyclone-frontend-page`** — page-component tests live next to pages in `src/pages/*.test.tsx`; load when the increment adds or refactors a page.
|
||||
- **`cyclone-cli`** — CLI smoke tests live in `backend/tests/test_cli_*.py`; load when the increment adds a CLI subcommand.
|
||||
- **`superpowers:test-driven-development`** (global) — the upstream TDD workflow. Load first when starting any new feature increment; this skill only codifies the Cyclone-specific test layout on top of TDD.
|
||||
@@ -1,184 +0,0 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## What this is
|
||||
|
||||
Cyclone is a self-hosted X12 EDI claims-management suite for a single billing office (Colorado Medicaid currently). It parses 837P professional claims and 835 ERA remittances (X12 005010X222A1 / 005010X221A1) and also handles 999, TA1, 270, 271, and 277CA. Always binds to `0.0.0.0` — the host firewall / compose port publishing is what restricts reachability, not the bind address. Requires login (auth boundary is HTTP; bcrypt + HttpOnly session cookie; first admin bootstrapped from `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD` env vars; see SP24 spec for the full posture). LAN-only by design — don't expose the published ports to the WAN.
|
||||
|
||||
Stack: one Python process (FastAPI + uvicorn, port 8000) + one Node process in dev (Vite, port 5173). The authoritative state is a single SQLite file at `~/.local/share/cyclone/cyclone.db` (or SQLCipher at the same path when the macOS Keychain entry + `sqlcipher3` are both present).
|
||||
|
||||
For the day-1 architecture read, see `docs/ARCHITECTURE.md` (process topology, module map, store facade, parser pipeline, pubsub). For the what-it-does read, see `docs/REQUIREMENTS.md` (FRs + NFRs + DoD).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# Backend (Python 3.11+)
|
||||
cd backend
|
||||
python -m venv .venv
|
||||
.venv/bin/pip install -e '.[dev]'
|
||||
|
||||
# Frontend (Node 20+)
|
||||
cd ..
|
||||
npm install
|
||||
```
|
||||
|
||||
Optional backend extras: `pip install -e '.[sqlcipher]'` (encryption at rest, SP12) and `pip install -e '.[sftp]'` (real SFTP, SP13).
|
||||
|
||||
## Dev (two terminals)
|
||||
|
||||
```bash
|
||||
# Terminal 1 — backend
|
||||
cd backend
|
||||
.venv/bin/python -m cyclone serve # default 0.0.0.0:8000
|
||||
# CYCLONE_PORT=... overrides port; CYCLONE_RELOAD=1 enables uvicorn --reload
|
||||
# Or: .venv/bin/uvicorn cyclone.api:app --reload --port 8000
|
||||
|
||||
# Terminal 2 — frontend
|
||||
npm run dev # Vite on http://localhost:5173
|
||||
```
|
||||
|
||||
Vite proxies `/api/*` to the backend at `http://127.0.0.1:${CYCLONE_PORT:-8000}` so relative-URL fetchers (the live-tail NDJSON streams in particular) resolve through the same origin. Override the backend port with `CYCLONE_PORT` in the frontend terminal too.
|
||||
|
||||
Create `.env.local` at the repo root with `VITE_API_BASE_URL=http://127.0.0.1:8000`. Without it, the UI runs against the in-memory zustand store and real EDI parsing is disabled.
|
||||
|
||||
## Test
|
||||
|
||||
```bash
|
||||
# Backend — full suite
|
||||
cd backend && .venv/bin/pytest
|
||||
|
||||
# Backend — one file
|
||||
cd backend && .venv/bin/pytest tests/test_api_999.py -v
|
||||
|
||||
# Backend — one test by node id
|
||||
cd backend && .venv/bin/pytest tests/test_api_999.py::test_parse_999_endpoint_happy_path -v
|
||||
|
||||
# Frontend — full suite
|
||||
npm test # alias for `vitest run`
|
||||
|
||||
# Frontend — one file
|
||||
npx vitest run src/hooks/useFoo.test.ts
|
||||
|
||||
# Frontend — typecheck
|
||||
npm run typecheck
|
||||
|
||||
# Frontend — build (tsc -b + vite build)
|
||||
npm run build
|
||||
|
||||
# Frontend — lint
|
||||
npm run lint
|
||||
```
|
||||
|
||||
**Conventions** (full detail in `.superpowers/skills/cyclone-tests/SKILL.md`):
|
||||
|
||||
- Backend tests live under `backend/tests/test_*.py`. Two flavors: `test_api_<topic>_<verb>.py` (FastAPI integration via `fastapi.testclient.TestClient`) and `test_<module>_<behavior>.py` (pure-unit). Autouse `conftest.py` points `CYCLONE_DB_URL` at `tmp_path/test.db`, calls `db._reset_for_tests()` + `db.init_db()`, and wires a fresh `EventBus` onto `app.state`.
|
||||
- Prodfiles (real EDI samples under `docs/prodfiles/<source>/`) are never read directly from a test — copy to `backend/tests/fixtures/<descriptive-name>.txt` first and reference as a module-level `Path` constant. The `fixtures/` dir is flat (no per-test subdirs).
|
||||
- Frontend tests are siblings: `useFoo.ts` → `useFoo.test.ts`, `ClaimDrawer.tsx` → `ClaimDrawer.test.tsx`. Setup is `// @vitest-environment happy-dom` plus `(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;`. Mock the API at the module boundary with `vi.mock("@/lib/api", ...)`; stub fetch with `vi.stubGlobal("fetch", vi.fn().mockResolvedValue(...))`. `vitest.config.ts` sets `VITE_API_BASE_URL=http://test.local` so the `api` module doesn't throw `notConfiguredError` before the mock fires.
|
||||
- Time-sensitive tests: frontend uses `vi.useFakeTimers()` + `vi.setSystemTime(...)` + `vi.advanceTimersByTime(ms)`. Backend passes explicit `datetime(...)` values. Don't add `await new Promise((r) => setTimeout(r, N))` — it's the legacy flaky pattern.
|
||||
|
||||
## Project-scoped skills (`.superpowers/skills/`)
|
||||
|
||||
Cyclone ships 8 skills under `.superpowers/skills/`. They auto-load by description match — no slash command needed. **Read the relevant skill before touching the matching subsystem.**
|
||||
|
||||
| Skill | Owns |
|
||||
|---|---|
|
||||
| `cyclone-spec` | The SP-N spec → plan → implement → merge flow (branch shape, file paths, commit prefixes, PR title, merge shape). |
|
||||
| `cyclone-tests` | pytest + vitest fixture patterns, prodfiles drop-in rule, determinism rules. |
|
||||
| `cyclone-edi` | EDI parser/validator conventions (837P/835/999/270/271/277CA/TA1, R-codes, CAS mapping). |
|
||||
| `cyclone-tail` | Live-tail streaming wire format and the `useTailStream` + `useMergedTail` + `TailStatusPill` hook triplet. |
|
||||
| `cyclone-store` | `CycloneStore` facade, write-paths, pubsub event contract, SP21 split map. |
|
||||
| `cyclone-api-router` | FastAPI router conventions (`api_routers/`, `api_helpers.py`), response/error-envelope shapes. |
|
||||
| `cyclone-frontend-page` | React page conventions (TanStack Query `use<X>` hook, drawer, URL state, sibling test). |
|
||||
| `cyclone-cli` | CLI subcommand conventions (`cli.py`, exit codes, smoke tests). |
|
||||
|
||||
## The SP-N increment flow
|
||||
|
||||
Every feature ships as a numbered **SP-N increment**: spec → plan → implementation branch → single atomic merge into `main`. As of the last backfill, SP numbers are used through **SP22**; **SP23** is reserved for the Ubuntu + Docker + RBAC product fork (`docs/superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md`, awaiting user decision); the next free increment is **SP24**. Read `cyclone-spec` before starting a new one. Non-negotiable shape:
|
||||
|
||||
- **Branch:** `sp<N>-<short-kebab-topic>` (e.g. `sp22-line-reconciliation`).
|
||||
- **Spec path:** `docs/superpowers/specs/YYYY-MM-DD-cyclone-<topic>-design.md`, header `Status: Draft, awaiting user sign-off`, sections `Scope / Decisions / …`. Specs contain zero code blocks.
|
||||
- **Plan path:** `docs/superpowers/plans/YYYY-MM-DD-cyclone-<topic>.md`, header per `superpowers:writing-plans` with `Goal / Architecture / Tech Stack / Spec` metadata + numbered `- [ ] Step N:` tasks.
|
||||
- **Commit prefixes:** `feat(sp<N>): …`, `docs(spec): …`, `docs(plan): …`, `merge: SP<N> <topic> into main`.
|
||||
- **PR title:** `SP<N> <Topic>` (matches the merge-commit subject).
|
||||
- **Merge shape:** single atomic merge commit. **No squash** (collapses the audit trail) and **no rebase** (rewrites the SHAs the review was performed against). The SP-N merge commit *is* the record of the increment landing.
|
||||
|
||||
The matching skill to load alongside `cyclone-spec` depends on the subsystem the SP-N touches (see the "Related skills" section at the bottom of each skill file).
|
||||
|
||||
## Live-tail wire format
|
||||
|
||||
The Claims, Remittances, and Activity pages stay current without manual refresh. The backend publishes an internal event on every store write, the page opens a streaming HTTP connection to the matching `/api/<resource>/stream` endpoint, and new rows append to the table the moment they hit the database.
|
||||
|
||||
Endpoints (all accept the same query params as their non-streaming counterparts; `Content-Type: application/x-ndjson`):
|
||||
|
||||
| Method | Path | Subscribes to | Default sort |
|
||||
|---|---|---|---|
|
||||
| GET | `/api/claims/stream` | `claim_written` | `-submission_date` |
|
||||
| GET | `/api/remittances/stream` | `remittance_written` | `-received_date` |
|
||||
| GET | `/api/activity/stream` | `activity_recorded` | `-timestamp` (limit 50) |
|
||||
|
||||
Wire format: one JSON object per line, `{"type": ..., "data": ...}`. The first batch is the **snapshot** of currently-known rows, then `snapshot_end` with the count, then the **live** events. Known types: `item`, `snapshot_end`, `heartbeat` (keeps the connection alive on idle — clients flip to `stalled` after 30s of total silence), `item_dropped` (rare), `error`.
|
||||
|
||||
Status pill states (rendered by `<TailStatusPill>` in `src/components/TailStatusPill.tsx`): `live` (success), `connecting` (warning), `reconnecting` (warning), `stalled` (destructive, ↻ Reconnect button), `error` (destructive, ↻ Reconnect button), `closed` (destructive). Backoff on error: `1s → 2s → 4s → 8s → 16s → 30s` capped. `STALL_TIMEOUT_MS = 30_000` in `src/hooks/useTailStream.ts:53`. Heartbeat interval is `CYCLONE_TAIL_HEARTBEAT_S` env var, default 15s.
|
||||
|
||||
Frontend triplet for any live page: `use<X>(params)` (initial fetch) + `useTailStream(resource)` (opens the NDJSON stream, drives backoff/stall) + `useMergedTail(resource, baseItems, filterFn?)` (merges snapshot + tail, dedup'd by id). The subscription lives on the page, not inside the data hook — see `cyclone-frontend-page` for why.
|
||||
|
||||
## Backend at a glance
|
||||
|
||||
`backend/src/cyclone/` is a single namespace. The two largest files are `api.py` (~3,548 LOC, the only large file) and `store.py` (~2,423 LOC, the `CycloneStore` facade — SP21 is in flight to split it into a `cyclone/store/` subpackage; the public API stays unchanged). Subpackages: `api_routers/` (acks, admin, health, ta1_acks), `clearhouse/` (Clearhouse + SftpClient), `edi/` (filenames), `parsers/` (X12 transaction parsers + models + validators + serializers), `workflow/` (placeholder for future sub-project 6).
|
||||
|
||||
The store is the only read/write surface for the database; every mutating endpoint goes through it. All persistence flows through SQLAlchemy sessions via `db.SessionLocal()()`. SQLAlchemy ORM models live in `db.py`; 12 SQL migrations under `migrations/` (0001_initial through 0012_backups) are walked in order by `db_migrate.py`.
|
||||
|
||||
The parser pipeline is a 5-stage `tokenize → segmentize → model → validate → write_to_store` flow used for every inbound X12 type. Per-transaction parsers: `parse_837.py`, `parse_835.py`, `parse_999.py`, `parse_ta1.py`, `parse_270.py`, `parse_271.py`, `parse_277ca.py`. Each has a matching Pydantic model module (`models.py`, `models_835.py`, …) and a writer (`writer.py` / `writer_835.py`). The 837P serializer (`serialize_837.py`) is the byte-faithful outbound counterpart used by both single-claim download (`/api/claims/{id}/serialize-837`) and the bulk rejected-resubmit bundle (`/api/inbox/rejected/resubmit?download=true`).
|
||||
|
||||
The pubsub is `cyclone.pubsub.EventBus` — an in-process async fan-out broker. Publishers call `publish(kind, payload)`; subscribers receive via an async iterator. If a subscriber's per-kind queue is full, the oldest event is dropped so a slow consumer can't stall the producer. Bus is single-event-loop only (matches FastAPI/uvicorn).
|
||||
|
||||
Config: `config/payers.yaml` is the on-disk source for providers / payers / clearhouse, schema-validated at boot against a Pydantic model. Reload with `POST /api/admin/reload-config`. Original in-code `PAYER_FACTORIES` dict in `cli.py` is kept as a fallback for ad-hoc testing.
|
||||
|
||||
Secrets live in the macOS Keychain (via `keyring` + `cyclone.secrets`): SQLCipher key (service `cyclone`, account `cyclone.db.key`), SFTP password, backup passphrase. No secrets on disk in plaintext.
|
||||
|
||||
## Frontend at a glance
|
||||
|
||||
`src/` is React 18 + TypeScript + Vite. Routes register in `src/App.tsx` (11 pages, all under a `<Layout>` route wrapper). Pages are pure renderers — every page pairs with a `use<X>` data hook in `src/hooks/` and renders a `<PageHeader>` + a table/list/KPI grid. Drawers (`ClaimDrawer/`, `RemitDrawer/`, plus the new `ProviderDrawer/` and `AckDrawer/`) are mounted by the page and their open/close state is mirrored to the URL via `useDrawerUrlState` so deep-links round-trip. Drill-stack navigation is provided by `<DrillStackProvider>` in `src/components/drill/`.
|
||||
|
||||
State split: **server state** in TanStack Query (`@tanstack/react-query`); **ephemeral client state** in Zustand (`useTailStore` for live-tail append, plus the drill stack). The live-tail store is FIFO-capped at `TAIL_CAP = 10_000` per slice (`src/store/tail-store.ts:28`); `claims` and `remittances` are key-by-id with first-write-wins dedup, `activity` is an append-only array.
|
||||
|
||||
UI primitives in `src/components/ui/` are Radix-backed (`button`, `dialog`, `table`, `select`, `pagination`, `empty-state`, `error-state`, `filter-chips`, `skeleton`, `input`, `label`, `card`, `badge`, `skip-link`, `claim-state-badge`). Don't import a new UI library without discussion.
|
||||
|
||||
Path alias `@/` → `src/`. Configured in `vite.config.ts`, `vitest.config.ts`, and `tsconfig.app.json`.
|
||||
|
||||
## CLI
|
||||
|
||||
```bash
|
||||
# Parser
|
||||
python -m cyclone.cli parse-837 path/to/837p.txt --output-dir ./claims --payer co_medicaid [--strict] [--include-raw-segments]
|
||||
python -m cyclone.cli parse-835 path/to/835.txt --output-dir ./remits
|
||||
python -m cyclone.cli parse-999 inbound_999.txt
|
||||
python -m cyclone.cli parse-ta1 inbound_ta1.txt
|
||||
python -m cyclone.cli parse-277ca inbound_277ca.txt
|
||||
|
||||
# Validators
|
||||
python -m cyclone.cli validate-npi 1234567893
|
||||
python -m cyclone.cli validate-tin 721587149
|
||||
|
||||
# Other
|
||||
python -m cyclone serve # uvicorn
|
||||
python -m cyclone backup list
|
||||
python -m cyclone backup create --reason manual
|
||||
```
|
||||
|
||||
Exit codes are documented per subcommand in `cyclone-cli` — `0` for success, `2` for file-level failure, `1` for unexpected exceptions.
|
||||
|
||||
## Things that are easy to get wrong
|
||||
|
||||
- **`VITE_API_BASE_URL` matters.** With it empty, every `api` method throws `notConfiguredError()` and the UI falls back to the in-memory zustand store — parses are disabled and the live-tail streams never open.
|
||||
- **Prodfiles vs fixtures.** Tests must reference `backend/tests/fixtures/<name>.txt`, not `docs/prodfiles/<source>/<file>.txt`. The prodfiles dir is the source-of-truth archive; the fixtures dir is the stable test surface.
|
||||
- **SP-N merge shape.** No squash, no rebase. The merge commit *is* the audit trail. Squash collapses the per-commit history and breaks the SP-N audit trail.
|
||||
- **Don't put domain logic in JSX.** Conditional renderings, table sorting, and KPI math all belong in the `use<X>` hook or a pure helper under `src/lib/`.
|
||||
- **Don't open a drawer via local `useState`.** Use `useDrawerUrlState()` so the URL is the single source of truth — deep-links and reload-restore depend on it.
|
||||
- **Don't call `useTailStream` from inside a `use<X>` hook.** The subscription lives on the page so the lifecycle ties to whoever mounts the hook, not to whoever happens to call it.
|
||||
- **The store facade.** The public API of `cyclone.store` is preserved through SP21's split — call through the facade, not directly into the underlying modules.
|
||||
- **Encryption is optional, not required.** When the Keychain entry is missing **or** `sqlcipher3` is not installed, the DB falls back to plain SQLite. Don't fail boot on missing encryption.
|
||||
- **Always bind to 0.0.0.0.** The bind address does not control reachability — the host firewall / compose port publishing does. Requires login (bcrypt + HttpOnly session cookie; see SP24 spec), and the threat model is still a stolen/imaged drive — SQLCipher at rest and the macOS Keychain handle that. The auth boundary is the HTTP layer; the file-system posture is unchanged. Don't expose the published ports to the public internet (LAN-only or VPN-fronted). Don't disable auth without an explicit `CYCLONE_AUTH_DISABLED=1` env var (the escape hatch logs a WARNING at boot).
|
||||
</content>
|
||||
</invoke>
|
||||
@@ -1,43 +0,0 @@
|
||||
# syntax=docker/dockerfile:1.7
|
||||
#
|
||||
# Cyclone frontend — React SPA built with node:20-alpine and served by
|
||||
# nginx:1.27-alpine. nginx reverse-proxies /api/* to the backend service
|
||||
# over the compose-managed bridge network.
|
||||
|
||||
# ---------- builder ----------
|
||||
FROM node:20-alpine AS builder
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
# Install deps first so this layer caches across source edits.
|
||||
# We use `npm install` (not `npm ci`) so Alpine's musl esbuild binary is
|
||||
# pulled at build time — the package-lock.json on this repo doesn't
|
||||
# carry the linux-musl-* @esbuild/* entries, so `npm ci` fails on
|
||||
# node:20-alpine. `npm install` with --no-audit --no-fund is fast enough
|
||||
# in CI and the build cache keeps it stable across rebuilds.
|
||||
COPY package.json package-lock.json* ./
|
||||
RUN npm install --no-audit --no-fund
|
||||
|
||||
# Build the production bundle into dist/. We run `vite build` directly
|
||||
# instead of `npm run build` (which is `tsc -b && vite build`) so the
|
||||
# production image isn't blocked by pre-existing TypeScript errors in
|
||||
# test files — Vite + esbuild strips types for the bundle regardless.
|
||||
# Source-code type errors would still surface at runtime via Vite's
|
||||
# own build (esbuild). Run `npm run typecheck` separately to see them.
|
||||
COPY . .
|
||||
RUN npx vite build
|
||||
|
||||
# ---------- runtime ----------
|
||||
FROM nginx:1.27-alpine
|
||||
|
||||
# Replace the default nginx site with ours (SPA + reverse proxy).
|
||||
RUN rm -f /etc/nginx/conf.d/default.conf
|
||||
|
||||
COPY nginx.conf /etc/nginx/conf.d/default.conf
|
||||
COPY --from=builder /build/dist /usr/share/nginx/html
|
||||
|
||||
# wget is on busybox; nginx:alpine doesn't ship curl.
|
||||
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
||||
CMD wget -qO- http://127.0.0.1:8080/ >/dev/null || exit 1
|
||||
|
||||
EXPOSE 8080
|
||||
@@ -30,7 +30,7 @@ Two terminals:
|
||||
# Terminal 1 — backend
|
||||
cd backend
|
||||
.venv/bin/python -m cyclone serve
|
||||
# (defaults to 0.0.0.0:8000; override with CYCLONE_HOST / CYCLONE_PORT / CYCLONE_RELOAD=1)
|
||||
# (defaults to 127.0.0.1:8000; override with CYCLONE_PORT=...; reload with CYCLONE_RELOAD=1)
|
||||
|
||||
# Terminal 2 — frontend
|
||||
npm run dev
|
||||
@@ -51,52 +51,6 @@ VITE_API_BASE_URL=http://127.0.0.1:8000
|
||||
Without that, the UI falls back to its in-memory sample store via the
|
||||
existing `data` adapter (parses are disabled).
|
||||
|
||||
## Pipeline automation agent
|
||||
|
||||
For unattended round-trips (an agent / scheduler drops an 837P file
|
||||
into the pipeline and waits for the 999 back from Gainwell), see the
|
||||
`cyclone-pipeline` sibling project at `/Users/openclaw/dev/cyclone-pipeline/`.
|
||||
It drives the full 7-phase state machine — preflight → browser upload →
|
||||
parse verification → SFTP submit → TA1 wait → 999 wait → scan +
|
||||
report — with structured JSON logs, crash-safe resume, and a
|
||||
self-contained per-run folder under `./runs/`.
|
||||
|
||||
```bash
|
||||
# Single file
|
||||
cyclone-pipeline run /path/to/axiscare-837p.txt
|
||||
|
||||
# Resume a crashed run
|
||||
cyclone-pipeline resume 2026-06-21-1430-001
|
||||
|
||||
# On/after the following Monday, verify the 835 arrived
|
||||
cyclone-pipeline check-835 2026-06-21-1430-001
|
||||
```
|
||||
|
||||
The 835 is **not** waited for inline (it lands the following Monday on
|
||||
the CO Medicaid payment cycle). See
|
||||
[`cyclone-pipeline/README.md`](../cyclone-pipeline/README.md) for
|
||||
install, embed-in-agent example, exit codes, and the report format.
|
||||
|
||||
## Skills
|
||||
|
||||
Cyclone ships 8 project-scoped AI-assistant skills under
|
||||
[`.superpowers/skills/`](.superpowers/skills/). Each one codifies the
|
||||
conventions for a major subsystem so the next contributor (human or
|
||||
AI) gets the lay of the land automatically.
|
||||
|
||||
| Skill | Owns |
|
||||
|-------|------|
|
||||
| [`cyclone-spec`](.superpowers/skills/cyclone-spec/SKILL.md) | The SP-N spec → plan → implement → merge flow. |
|
||||
| [`cyclone-tests`](.superpowers/skills/cyclone-tests/SKILL.md) | pytest + vitest fixture patterns, prodfiles drop-in. |
|
||||
| [`cyclone-edi`](.superpowers/skills/cyclone-edi/SKILL.md) | EDI parser/validator conventions (837P/835/999/270/271/277CA/TA1). |
|
||||
| [`cyclone-tail`](.superpowers/skills/cyclone-tail/SKILL.md) | Live-tail streaming wire format and the hook triplet. |
|
||||
| [`cyclone-store`](.superpowers/skills/cyclone-store/SKILL.md) | Store write-paths, pubsub event contract, SP21 split map. |
|
||||
| [`cyclone-api-router`](.superpowers/skills/cyclone-api-router/SKILL.md) | FastAPI router conventions (`api_routers/`, `api_helpers.py`). |
|
||||
| [`cyclone-frontend-page`](.superpowers/skills/cyclone-frontend-page/SKILL.md) | React page conventions (TanStack Query, drawer, URL state). |
|
||||
| [`cyclone-cli`](.superpowers/skills/cyclone-cli/SKILL.md) | CLI subcommand conventions (`cli.py`, exit codes, smoke tests). |
|
||||
|
||||
Skills auto-load by description match — no slash command needed.
|
||||
|
||||
## Test
|
||||
|
||||
```bash
|
||||
@@ -111,49 +65,6 @@ npm run build
|
||||
npm test
|
||||
```
|
||||
|
||||
## Authentication
|
||||
|
||||
Cyclone ships with username/password authentication and three predefined roles.
|
||||
|
||||
**Roles:**
|
||||
|
||||
| Role | Can read | Can write (upload, parse, reconcile) | Can manage users |
|
||||
| -------- | -------- | ------------------------------------ | ---------------- |
|
||||
| `viewer` | ✅ | ❌ | ❌ |
|
||||
| `user` | ✅ | ✅ | ❌ |
|
||||
| `admin` | ✅ | ✅ | ✅ |
|
||||
|
||||
**Bootstrap.** On first start, set `CYCLONE_ADMIN_USERNAME` and
|
||||
`CYCLONE_ADMIN_PASSWORD` (min 12 chars) in your environment. Cyclone creates the
|
||||
first admin automatically. On subsequent starts these env vars are ignored, so
|
||||
rotating the bootstrap password doesn't affect an already-seeded admin — use the
|
||||
CLI below to reset it. When running via `docker compose`, both vars are
|
||||
required: compose refuses to start with a clear error if either is missing.
|
||||
|
||||
**CLI.** Manage users from the command line:
|
||||
|
||||
```
|
||||
python -m cyclone users create alice --role user --password 'hunter2hunter2'
|
||||
python -m cyclone users list
|
||||
python -m cyclone users disable alice
|
||||
python -m cyclone users reset-password alice
|
||||
python -m cyclone users set-role alice --role admin
|
||||
```
|
||||
|
||||
**Login.** Browse to `http://localhost:5173` (dev) or `http://localhost:8081`
|
||||
(Docker), sign in on the `/login` page, and you'll be redirected to the
|
||||
dashboard. Sessions are stored server-side in SQLite with a 24-hour sliding
|
||||
expiry — every authenticated request refreshes the TTL, so an active user
|
||||
never gets logged out.
|
||||
|
||||
**Dev escape hatch.** Set `CYCLONE_AUTH_DISABLED=1` to bypass auth entirely
|
||||
(the backend auto-grants admin on every request). **NEVER set this in
|
||||
production** — it's a single env-var trip from wide-open to the public
|
||||
internet. The Docker compose file does not honor this flag.
|
||||
|
||||
See `docs/superpowers/specs/2026-06-22-cyclone-auth-design.md` for the full
|
||||
design.
|
||||
|
||||
## Live updates
|
||||
|
||||
The Claims, Remittances, and Activity pages stay current without
|
||||
@@ -244,6 +155,7 @@ parses and rejects claims, the inbox reflects the new
|
||||
| POST | `/api/inbox/candidates/{remit_id}/match` | Manual match. `409` if the claim state moved out from under us. |
|
||||
| POST | `/api/inbox/candidates/dismiss` | `{pairs: [{claim_id, remit_id}]}`. Session-scoped. |
|
||||
| POST | `/api/inbox/rejected/resubmit` | `{claim_ids: [...]}`. `200` with `conflicts` for non-rejected. |
|
||||
| POST | `/api/inbox/payer-rejected/acknowledge` | `{claim_ids: [...], actor: "..."}`. Bulk-acknowledge payer rejections. Idempotent. `200` with `transitioned` / `already_acked` / `not_found` / `not_rejected` counts. Writes an audit event per transition; never overwrites the underlying 277CA rejection. |
|
||||
| GET | `/api/inbox/export.csv?lane=<lane>` | Streams CSV of the lane's rows. |
|
||||
|
||||
## Outbound 837 Serializer
|
||||
@@ -410,8 +322,9 @@ both be true for the same claim.
|
||||
|
||||
The `audit_log` table is the canonical record of every state transition
|
||||
the system has ever observed — claim lifecycle, reconciliation
|
||||
decisions, config reloads, SFTP submissions, 277CA rejects. SP11 made
|
||||
it tamper-evident: every row carries a SHA-256 hash of
|
||||
decisions, config reloads, SFTP submissions, 277CA rejects, Payer-Rejected
|
||||
acknowledgements, SQLCipher key rotations. SP11 made it tamper-evident:
|
||||
every row carries a SHA-256 hash of
|
||||
`(prev_hash || row_payload)`, forming a chain back to a genesis row.
|
||||
Any `INSERT`, `UPDATE`, or `DELETE` that breaks the chain is detectable
|
||||
in a single walk.
|
||||
@@ -446,251 +359,46 @@ operator has created the Keychain entry on first run. See
|
||||
for the one-time setup recipe and the HIPAA Security Rule §164.312(a)(2)(iv)
|
||||
mapping.
|
||||
|
||||
### Key rotation (SP15)
|
||||
### Key rotation
|
||||
|
||||
`POST /api/admin/db/rotate-key` re-encrypts the SQLite file in place
|
||||
with a fresh SQLCipher key via `PRAGMA rekey`, then updates the
|
||||
Keychain so subsequent connections open with the new key. The
|
||||
rotation holds a module-level `threading.Lock` (so two concurrent
|
||||
requests can't race), disposes + rebuilds the SQLAlchemy engine with
|
||||
`NullPool` (so SQLCipher's thread affinity is honored), and writes a
|
||||
tamper-evident `db.key_rotated` audit event with old + new
|
||||
fingerprints and the post-rotation table count. The old key is
|
||||
retained in the `cyclone.db.key.previous` Keychain account for a
|
||||
grace period so a botched rotation can be rolled back by hand.
|
||||
The DB encryption key is rotated in place via `POST /api/admin/db/rotate-key`.
|
||||
The handler:
|
||||
|
||||
## NPI checksum + Tax ID format validation (SP20)
|
||||
1. Generates a fresh 256-bit CSPRNG key (`db_crypto.generate_db_key()`).
|
||||
2. Persists the new key to the Keychain under the same
|
||||
`cyclone.db.key` account (overwriting the old one).
|
||||
3. Disposes the SQLAlchemy engine so pooled connections release the
|
||||
file (SQLCipher refuses to `PRAGMA rekey` while another connection
|
||||
holds the DB).
|
||||
4. Opens with the old key, issues `PRAGMA rekey`, and verifies the
|
||||
schema survived (table-count sanity check).
|
||||
5. Rebuilds the engine with the new key.
|
||||
6. Writes a `db.key_rotated` audit event carrying the SHA-256
|
||||
fingerprints of the old and new keys (first 8 hex chars) plus the
|
||||
post-rotation `table_count`.
|
||||
|
||||
Two pure local validators — no NPPES round-trip, no IRS e-file
|
||||
lookup. Catches the 99% case (a typo at the end of an NPI, a letter in
|
||||
an EIN, an extra digit, the reserved `00`/`07`/`8X` EIN prefix).
|
||||
When SQLCipher is enabled the engine uses SQLAlchemy's `NullPool`
|
||||
instead of the default `QueuePool`. `QueuePool` returns connections to
|
||||
a shared queue that any thread can pull from, which breaks SQLCipher's
|
||||
thread affinity. `NullPool` trades connection reuse for thread safety
|
||||
— the only correct behavior under FastAPI's per-request threadpool.
|
||||
|
||||
| Check | Algorithm | Surface |
|
||||
|-------|-----------|---------|
|
||||
| NPI | 10 digits where the last is a Luhn checksum over `80840 + body`. CMS-published example: body `123456789` → check `3` → valid NPI `1234567893`. | `cyclone.npi.is_valid_npi`, CLI `cyclone validate-npi <npi>`, API `GET /api/admin/validate-provider?npi=...`, validator rule `R021_npi_checksum` (warning) |
|
||||
| Tax ID (EIN) | 9 digits, optional `XX-XXXXXXX` formatting. Rejects reserved prefixes `00`, `07`, `80`–`89` (IRS Pension Plan Branch). | `cyclone.npi.is_valid_tax_id`, CLI `cyclone validate-tax-id <ein>`, API `GET /api/admin/validate-provider?tax_id=...` |
|
||||
The rotation endpoint is serialized with a module-level
|
||||
`threading.Lock` (one rotation in flight at a time), returns `409` if
|
||||
a rotation is already running, `400` if encryption is not enabled,
|
||||
and `503` with a `reason` on `PRAGMA rekey` or Keychain failure so the
|
||||
operator can take the next step without parsing the traceback.
|
||||
|
||||
### CLI
|
||||
|
||||
```bash
|
||||
$ cyclone validate-npi 1234567893
|
||||
OK: 10-digit NPI passes Luhn checksum
|
||||
$ cyclone validate-npi 1234567890
|
||||
INVALID: '1234567890' fails NPI Luhn checksum # exit 1
|
||||
|
||||
$ cyclone validate-tax-id 72-1587149
|
||||
OK: 9-digit EIN (normalized=721587149)
|
||||
$ cyclone validate-tax-id 00-1234567
|
||||
INVALID: 9-digit EIN has reserved prefix (00); EIN is not assignable by IRS # exit 1
|
||||
```
|
||||
|
||||
### API
|
||||
|
||||
```bash
|
||||
curl 'http://localhost:8000/api/admin/validate-provider?npi=1234567893&tax_id=72-1587149'
|
||||
# {
|
||||
# "npi": {"valid": true, "skipped": false},
|
||||
# "tax_id": {"valid": true, "skipped": false, "normalized": "721587149"}
|
||||
# }
|
||||
```
|
||||
|
||||
Both query params are optional; omitted fields return
|
||||
`{"valid": null, "skipped": true}` so the caller can render "no check
|
||||
performed" rather than treating absent input as a hard fail.
|
||||
|
||||
### Parser integration
|
||||
|
||||
The `R021_npi_checksum` rule runs alongside the existing `R020_npi_format`
|
||||
in `cyclone.parsers.validator`. A billing-provider NPI that passes
|
||||
R020 (right shape) but fails R021 (bad Luhn) is yielded as a
|
||||
**warning**, not an error — operators sometimes ingest test fixtures
|
||||
with placeholder NPIs (e.g. all-same-digit) and we don't want to block
|
||||
that path. In strict mode (`--strict` / `?strict=true`) warnings are
|
||||
promoted to errors.
|
||||
|
||||
### Files
|
||||
|
||||
* `cyclone/npi.py` — new module (~155 LOC).
|
||||
* `cyclone.parsers.validator` — new `R021_npi_checksum` rule.
|
||||
* `cyclone.api_routers.admin` — new `validate-provider` endpoint.
|
||||
* `cyclone.cli` — `validate-npi` + `validate-tax-id` subcommands.
|
||||
* Tests: `test_npi.py` (27), `test_api_validate_provider.py` (4),
|
||||
`test_cli_validate.py` (8), `test_validator.py::test_r021_*` (4) —
|
||||
**43 new tests**.
|
||||
|
||||
## Security hardening (SP19)
|
||||
|
||||
Three pure-ASGI middlewares sit in front of every FastAPI request.
|
||||
They're sized for Cyclone's local-only posture — a misconfigured
|
||||
Tailscale / ngrok bind, a buggy cron job uploading a 4 GB file, or a
|
||||
port-scraper — not for hostile internet exposure.
|
||||
|
||||
| Middleware | Default | Override | Reject |
|
||||
|------------|---------|----------|--------|
|
||||
| `BodySizeLimitMiddleware` | 50 MB | `CYCLONE_MAX_BODY_BYTES` | `413 body_too_large` over Content-Length cap; chunked reads capped too |
|
||||
| `RateLimitMiddleware` | 300 req/min/IP | `CYCLONE_RATE_LIMIT_PER_MIN` | `429 rate_limited` over the sliding window; `/api/health` exempt |
|
||||
| `SecurityHeadersMiddleware` | always on | n/a | stamps `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: same-origin`, `Permissions-Policy`, `Content-Security-Policy: default-src 'none'; frame-ancestors 'none'` |
|
||||
|
||||
Every rejection (413 / 429) also writes a tamper-evident
|
||||
`api.request_rejected` event into the SP11 audit chain so an
|
||||
operator can correlate a misbehaving client with the SP18 JSON logs:
|
||||
|
||||
```json
|
||||
{"event_type":"api.request_rejected","entity_id":"POST /api/parse-837","payload":{"status":413,"reason":"body_too_large","path":"/api/parse-837","method":"POST","ip":"127.0.0.1"}}
|
||||
```
|
||||
|
||||
### Health probe
|
||||
|
||||
`GET /api/health` now returns a subsystem snapshot:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"version": "0.1.0",
|
||||
"db": {"ok": true},
|
||||
"scheduler": {"running": true, "interval_s": 60, "sftp_block": "co_medicaid",
|
||||
"backup_scheduler_running": false, "backup_interval_hours": 24.0},
|
||||
"pubsub": {"parse_completed": 1, "batch_added": 1},
|
||||
"batch": {"last_batch_id": 42, "last_batch_kind": "837P",
|
||||
"last_batch_at": "2026-06-21T15:30:00.123Z",
|
||||
"last_batch_filename": "TP11525703-837P-..."}
|
||||
}
|
||||
```
|
||||
|
||||
Returns `"status": "degraded"` if any subsystem reports an error —
|
||||
the per-subsystem dict still surfaces so an operator can see which
|
||||
one is unhappy. `/api/health` is rate-limit exempt so a load balancer
|
||||
hammering the endpoint doesn't trip the limiter.
|
||||
|
||||
### Files
|
||||
|
||||
* `cyclone.security` — `BodySizeLimitMiddleware`,
|
||||
`RateLimitMiddleware`, `SecurityHeadersMiddleware`, and
|
||||
`get_health_snapshot()` (~330 LOC).
|
||||
* `cyclone.api_routers.health` rewritten to use `get_health_snapshot()`.
|
||||
* `cyclone.pubsub.EventBus.stats()` — new method that returns
|
||||
per-kind subscriber counts.
|
||||
* `tests/test_security.py` — 13 new tests.
|
||||
|
||||
## Structured logging (SP18)
|
||||
|
||||
Cyclone emits newline-delimited JSON to stderr by default — readable
|
||||
by `jq`, Loki, Vector, ELK, or any log shipper. Every record carries
|
||||
`ts` (ISO 8601 ms UTC), `level`, `logger`, `msg`, and an optional
|
||||
`extra` dict for structured fields. Exceptions render as a
|
||||
`traceback` string.
|
||||
|
||||
```
|
||||
{"ts":"2026-06-21T15:30:00.123Z","level":"INFO","logger":"cyclone.scheduler","msg":"Processed inbound file","extra":{"input_filename":"ACK_999.x12","parser":"parse_999","claims":3}}
|
||||
{"ts":"2026-06-21T15:30:01.456Z","level":"ERROR","logger":"cyclone.api","msg":"Backup create failed","extra":{"reason":"BackupError: passphrase mismatch"},"traceback":"Traceback ..."}
|
||||
```
|
||||
|
||||
### PII scrubbing
|
||||
|
||||
A `PiiScrubber` filter is attached to the root logger and rewrites
|
||||
obvious PHI patterns to `<redacted:npi>` / `<redacted:ssn>` /
|
||||
`<redacted:dob>` / `<redacted:patient_name>` before any handler sees
|
||||
the record:
|
||||
|
||||
| Pattern | Replacement |
|
||||
|---------|-------------|
|
||||
| `\b\d{10}\b` | `<redacted:npi>` |
|
||||
| `\b\d{3}-\d{2}-\d{4}\b` or `\b\d{9}\b` at phrase boundary | `<redacted:ssn>` |
|
||||
| `(dob\|date_of_birth)[:=]\s*\d{4}-\d{2}-\d{2}` | preserves the key, redacts the date |
|
||||
| `patient_name=...` | full chunk redacted |
|
||||
| Extras with key `dob`/`ssn`/`npi`/`patient_name`/… | value redacted regardless of shape |
|
||||
|
||||
The scrubber is conservative — bare ISO dates without a `dob=` prefix
|
||||
are **not** scrubbed (they're too often timestamps or batch IDs), and
|
||||
11+ digit numbers are left alone (they can't be NPIs). Disable for
|
||||
forensic mode with `CYCLONE_LOG_NO_PII_SCRUB=1`.
|
||||
|
||||
### Knobs
|
||||
|
||||
| Env var | Default | Meaning |
|
||||
|---------|---------|---------|
|
||||
| `CYCLONE_LOG_LEVEL` | `INFO` | Root logger level. `DEBUG` for troubleshooting. |
|
||||
| `CYCLONE_LOG_FILE` | (none) | Write to this path via `RotatingFileHandler` (10 MB × 5 backups). |
|
||||
| `CYCLONE_LOG_JSON` | `true` | `false` uses the dev tabular formatter. |
|
||||
| `CYCLONE_LOG_NO_PII_SCRUB` | (none) | `1` disables scrubbing. |
|
||||
|
||||
CLI:
|
||||
|
||||
```
|
||||
cyclone --log-format=dev parse-837 sample.x12 --output-dir out/ # tabular for tail -f
|
||||
cyclone --log-file=/var/log/cyclone.log backup create # JSON to rotating file
|
||||
```
|
||||
|
||||
The `parse-837` / `parse-835` subcommands also accept `--log-level`
|
||||
which re-runs `setup_logging()` so the per-invocation level overrides
|
||||
the group default.
|
||||
|
||||
### Files
|
||||
|
||||
* `cyclone.logging_config` — `JsonFormatter`, `CycloneDevFormatter`,
|
||||
`PiiScrubber`, `setup_logging()`.
|
||||
* `tests/test_logging_formatter.py` (11), `test_logging_scrubber.py`
|
||||
(13), `test_logging_setup.py` (10) — 34 new tests.
|
||||
* `cyclone.api` lifespan calls `setup_logging()` first; the CLI
|
||||
`main` group does the same.
|
||||
|
||||
## Encrypted Backups (SP17)
|
||||
|
||||
The BackupService takes an online consistent snapshot of the live
|
||||
SQLite file via SQLite's `.backup()` API, encrypts the bytes with
|
||||
AES-256-GCM, and writes a `.bin` + `.meta.json` pair into the backup
|
||||
directory (default `~/.local/share/cyclone/backups/`). The encryption
|
||||
key is derived from a separate passphrase in the macOS Keychain
|
||||
(PBKDF2-HMAC-SHA256, 200,000 iterations, 16-byte salt persisted to
|
||||
Keychain) — so a SQLCipher DB-key compromise does not unlock the
|
||||
backups, and a backup-passphrase compromise does not unlock the live
|
||||
DB. If neither is set, the service refuses (`BackupError`) rather than
|
||||
silently writing plaintext.
|
||||
|
||||
| Method | Path | Purpose |
|
||||
| ------ | ---- | ------- |
|
||||
| POST | `/api/admin/backup/create` | Take an encrypted backup now. |
|
||||
| GET | `/api/admin/backup/list` | List `db_backups` rows (newest first, filterable). |
|
||||
| GET | `/api/admin/backup/status` | Counts, disk usage, last-run timestamp, scheduler snapshot. |
|
||||
| POST | `/api/admin/backup/{id}/verify` | Decrypt + SHA-256 verify against the sidecar. |
|
||||
| POST | `/api/admin/backup/{id}/restore/initiate` | First step: get `restore_token` + preview (fingerprints of backup vs live). |
|
||||
| POST | `/api/admin/backup/{id}/restore/confirm` | Second step: dispose engine, copy decrypted DB, rebuild engine. |
|
||||
| POST | `/api/admin/backup/prune` | Apply retention policy now. |
|
||||
| POST | `/api/admin/backup/scheduler/{start,stop,tick}` | Operate the backup scheduler. |
|
||||
|
||||
Restore is two-step by design: an idle browser tab can't nuke the
|
||||
live DB. The first call returns a one-shot 64-char hex
|
||||
`restore_token` plus a side-by-side preview (`backup_db_fingerprint`,
|
||||
`backup_table_count`, `current_db_fingerprint`, `current_table_count`).
|
||||
The second call swaps the live engine only if the token matches
|
||||
within a 5-minute TTL.
|
||||
|
||||
The scheduler (auto-start opt-in via `CYCLONE_BACKUP_AUTOSTART`)
|
||||
ticks every `CYCLONE_BACKUP_INTERVAL_HOURS` (default 24), runs
|
||||
`create_now` + `prune`, and writes audit events for each outcome
|
||||
(`db.backup_created`, `db.backup_failed`, `db.backup_pruned`,
|
||||
`db.backup_restored`). The CLI mirrors the API surface:
|
||||
|
||||
```bash
|
||||
cyclone backup init-passphrase # one-time; interactive
|
||||
cyclone backup create
|
||||
cyclone backup list
|
||||
cyclone backup verify <id>
|
||||
cyclone backup restore <id> --yes
|
||||
cyclone backup prune --yes
|
||||
cyclone backup status
|
||||
```
|
||||
|
||||
Retention defaults to 30 days (`CYCLONE_BACKUP_RETENTION_DAYS`). The
|
||||
retention policy is best-effort: an operator who runs `cyclone
|
||||
backup create` manually retains full control.
|
||||
| Method | Path | Notes |
|
||||
| ------ | --------------------------------- | -------------------------------------------------------------- |
|
||||
| POST | `/api/admin/db/rotate-key` | Rotate the SQLCipher key in place. Audit-logged. |
|
||||
|
||||
## SFTP Wire-Up (paramiko)
|
||||
|
||||
The `clearhouse.submit` endpoint uses `paramiko` to push a batch of
|
||||
generated 837 files to the dzinesco SFTP server
|
||||
(`mft.gainwelltechnologies.com:22`, path
|
||||
`/CO XIX/PROD/coxix_prod_11525703/ToHPE`). The SFTP credential is
|
||||
`/CO XIX/PROD/coxix_prod_11525703/FromHPE`). The SFTP credential is
|
||||
fetched from the macOS Keychain at call time — never read from YAML,
|
||||
never logged, never written to disk. The wire-up honors the file-naming
|
||||
template stored in the `clearhouse` config:
|
||||
@@ -710,6 +418,29 @@ was a one-file change (`sftp_paramiko.py` replacing `sftp_stub.py`).
|
||||
the `paramiko` extras aren't installed so the test suite stays green on
|
||||
Linux dev boxes.
|
||||
|
||||
## Batches, Reconciliation, and Activity
|
||||
|
||||
These read/write endpoints are the core operator surface for browsing
|
||||
the parsed record and acting on matches. They predate the per-SP
|
||||
endpoint reference sections in the **Roadmap** and are listed here in
|
||||
one place so the route inventory stays discoverable.
|
||||
|
||||
| Method | Path | Notes |
|
||||
| ------ | --------------------------------------------- | ---------------------------------------------------------------- |
|
||||
| GET | `/api/batches` | All parsed batches, newest first. `?limit=` (1–1000, default 100). |
|
||||
| GET | `/api/batches/{batch_id}` | One batch detail (envelope, claims / remits, validation summary). |
|
||||
| GET | `/api/batch-diff?a=<id>&b=<id>` | Side-by-side diff of two batches: `added` / `removed` / `changed` claims + envelope metadata for each. Both query params required. |
|
||||
| GET | `/api/reconciliation/unmatched` | `{"claims": [...], "remittances": [...]}` — every Claim with no paired Remittance and vice versa. The two lists are always present (empty list, never absent) so the UI can index unconditionally. |
|
||||
| POST | `/api/reconciliation/match` | Body `{claim_id, remit_id}`. Manually pair a claim with a remit. `400` on missing ids, `404` on unknown id, `409` on already-matched or terminal-state claim. |
|
||||
| POST | `/api/reconciliation/unmatch` | Body `{claim_id}`. Remove the current match and reset the claim to `submitted`. `404` on unknown id, `409` on no current match. |
|
||||
| GET | `/api/providers` | Distinct providers across parsed claims. `?npi=`, `?state=`, `?limit=`, `?offset=`. Distinct from `/api/config/providers/{npi}` (SP9 config table) — this endpoint surfaces providers derived from the parsed claim stream. |
|
||||
| GET | `/api/activity` | Recent activity events. `?kind=`, `?since=`, `?limit=` (1–500, default 200). Powers the Activity page; the streaming counterpart `/api/activity/stream` is documented under **Live updates**. |
|
||||
|
||||
The per-SP endpoint blocks at the bottom of the **Roadmap** cover the
|
||||
SP-specific routes (parse, 999/TA1/277CA, Inbox, claim drawer, line
|
||||
reconciliation, outbound 837, multi-payer config, audit log, 277CA,
|
||||
key rotation, payer-rejected acknowledge).
|
||||
|
||||
## Persistence
|
||||
|
||||
Parsed batches, claims, remittances, matches, 277CA rejections,
|
||||
@@ -743,8 +474,19 @@ backup API).
|
||||
.
|
||||
├── backend/
|
||||
│ ├── src/cyclone/
|
||||
│ │ ├── api.py # FastAPI app, GET + parse routes, /api/{resource}/stream
|
||||
│ │ ├── api.py # FastAPI app + parse routes; mounts api_routers/* sub-apps
|
||||
│ │ ├── api_helpers.py # NDJSON / content-negotiation / live-tail helpers
|
||||
│ │ ├── api_routers/ # FastAPI APIRouters extracted from api.py
|
||||
│ │ │ ├── health.py # GET /api/health
|
||||
│ │ │ ├── acks.py # GET /api/acks, /api/acks/{id}
|
||||
│ │ │ ├── ta1_acks.py # GET /api/ta1-acks, /api/ta1-acks/{id}
|
||||
│ │ │ ├── activity.py # GET /api/activity, /api/activity/stream
|
||||
│ │ │ ├── batches.py # GET /api/batches, /api/batches/{id}, /api/batch-diff
|
||||
│ │ │ ├── claims.py # GET /api/claims, /api/claims/stream, /api/claims/{id},
|
||||
│ │ │ │ # /api/claims/{id}/serialize-837, /api/claims/{id}/line-reconciliation
|
||||
│ │ │ ├── providers.py # GET /api/providers
|
||||
│ │ │ ├── clearhouse.py # GET /api/clearhouse, POST /api/clearhouse/submit
|
||||
│ │ │ └── config.py # /api/config/providers[/...], /api/config/payers[/...], /api/admin/reload-config
|
||||
│ │ ├── pubsub.py # in-process EventBus (drop-oldest, per-kind fan-out)
|
||||
│ │ ├── store.py # CycloneStore, mappers, publish-on-write
|
||||
│ │ ├── db.py # SQLAlchemy engine, session factory, ORM models
|
||||
@@ -801,13 +543,7 @@ backup API).
|
||||
|
||||
## Roadmap
|
||||
|
||||
> **Read order for new engineers:**
|
||||
> 1. [`docs/REQUIREMENTS.md`](docs/REQUIREMENTS.md) — what Cyclone does (FRs + NFRs + DoD + traceability). The single index tying the 22 shipped sub-projects to the 38 functional + 18 non-functional requirements and the test strategy.
|
||||
> 2. [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — how it fits together (process topology, package layout, data flow, lifecycle, operational concerns).
|
||||
> 3. The per-SP spec under [`docs/superpowers/specs/`](docs/superpowers/specs/) for whatever you're touching.
|
||||
> 4. The per-SP plan under [`docs/superpowers/plans/`](docs/superpowers/plans/) if you're implementing.
|
||||
|
||||
Sub-projects 2 through 19 are **shipped**. See the [completeness
|
||||
Sub-projects 2 through 15 are **shipped**. See the [completeness
|
||||
review](docs/reviews/2026-06-20-cyclone-completeness-review.md) for
|
||||
the honest gap analysis against the industry definition of a HIPAA
|
||||
clearinghouse — the short version is that the local-only,
|
||||
@@ -818,70 +554,6 @@ scope.
|
||||
|
||||
Shipped sub-projects (most recent first):
|
||||
|
||||
- **Sub-project 22 (shipped) — Pipeline automation agent.** A
|
||||
sibling project at `/Users/openclaw/dev/cyclone-pipeline` that
|
||||
drives the full 7-phase round-trip (preflight → browser upload →
|
||||
parse verify → SFTP submit → TA1 wait → 999 wait → scan + report)
|
||||
with crash-safe resume, structured JSON logging, idempotency
|
||||
dedup, and a per-run report. Pure Python 3.11+ (httpx, Playwright,
|
||||
Click, pydantic v2, structlog). The 835 is not waited for inline —
|
||||
it lands the following Monday — and is verified by a separate
|
||||
`check-835` subcommand. Embeddable as a library for OpenClaw / Nora
|
||||
agent integration. See
|
||||
[Pipeline automation agent](#pipeline-automation-agent) above.
|
||||
- **Sub-project 19 (shipped) — Security hardening + health probe.**
|
||||
Three pure-ASGI middlewares (`BodySizeLimitMiddleware`,
|
||||
`RateLimitMiddleware`, `SecurityHeadersMiddleware`) close the
|
||||
completeness-review gaps §3.1.4 (no body/rate limits) and §3.1.25
|
||||
(no CSP / security headers). 413/429 rejections emit a
|
||||
tamper-evident `api.request_rejected` audit event (SP11 chain).
|
||||
`/api/health` is now a rich subsystem snapshot — DB connectivity,
|
||||
MFT scheduler state, backup scheduler state, live pubsub
|
||||
subscriber counts, last batch id + timestamp. See
|
||||
[Security hardening (SP19)](#security-hardening-sp19) below.
|
||||
- **Sub-project 20 (shipped) — NPI checksum + Tax ID format validation.**
|
||||
Pure local validators (`cyclone.npi`) — no NPPES round-trip, no IRS
|
||||
e-file lookup. Catches the 99% typo case at parse time. NPI uses
|
||||
CMS-published Luhn over `80840 + body` (example: `1234567893` is
|
||||
valid). EIN rejects reserved prefixes (`00`, `07`, `80`–`89`).
|
||||
Surface: `cyclone validate-npi` / `validate-tax-id` CLI subcommands,
|
||||
`GET /api/admin/validate-provider`, new `R021_npi_checksum`
|
||||
validator rule (warning, not error — placeholder NPIs in test
|
||||
fixtures shouldn't block ingest). See
|
||||
[NPI checksum + Tax ID format validation (SP20)](#npi-checksum--tax-id-format-validation-sp20)
|
||||
below.
|
||||
- **Sub-project 18 (shipped) — Structured JSON logging.** All logs
|
||||
emitted by the API, CLI, scheduler tick loop, and backup service
|
||||
flow through a `JsonFormatter` (newline-delimited JSON, ISO-8601 ms
|
||||
timestamps) by default. A `PiiScrubber` filter redacts obvious PHI
|
||||
(NPIs, SSNs, DOBs, patient names) from message + extras — both via
|
||||
inline patterns (`npi 1881068062`) and via PHI-keyed extras
|
||||
(`extra={"dob": "1980-04-12"}`). Configurable via env vars
|
||||
(`CYCLONE_LOG_LEVEL`, `CYCLONE_LOG_FILE`, `CYCLONE_LOG_JSON`,
|
||||
`CYCLONE_LOG_NO_PII_SCRUB`) and CLI flags
|
||||
(`--log-format=json|dev`, `--log-file=…`); a tabular `CycloneDevFormatter`
|
||||
is the opt-out for `tail -f` in dev. See
|
||||
[Structured logging](#structured-logging-sp18) below.
|
||||
- **Sub-project 17 (shipped) — Encrypted DB backups.** Automated
|
||||
encrypted backups via AES-256-GCM (PBKDF2-HMAC-SHA256, 200k iters).
|
||||
The operator sets a separate passphrase in the macOS Keychain
|
||||
(`cyclone backup init-passphrase`); if missing, the service falls
|
||||
back to deriving from the SQLCipher DB key with a WARNING. Online
|
||||
backups via SQLite `.backup()`, two-step restore (`initiate` →
|
||||
`confirm` with one-shot 64-char hex token), retention pruning with
|
||||
a 30-day default, and a tamper-evident audit chain (`db.backup_created`,
|
||||
`db.backup_failed`, `db.backup_pruned`, `db.backup_restored`,
|
||||
`db.backup_passphrase_set`). Backup scheduler ticks every 24h
|
||||
(configurable); auto-start opt-in via `CYCLONE_BACKUP_AUTOSTART`.
|
||||
Seven admin endpoints + six CLI subcommands. See
|
||||
[Encrypted Backups](#encrypted-backups) below.
|
||||
- **Sub-project 16 (shipped) — Live MFT polling scheduler.** asyncio
|
||||
background loop polls the Gainwell MFT inbound path, downloads
|
||||
new files, and routes them through the right parser (999 / 835 /
|
||||
277CA / TA1). Idempotent (re-ticks skip already-processed files
|
||||
via the new `processed_inbound_files` table). Crash-safe (per-file
|
||||
try/except so a bad file doesn't stop the loop). Five admin
|
||||
endpoints (`/api/admin/scheduler/{status,start,stop,tick,processed-files}`).
|
||||
- **Sub-project 15 (shipped) — SQLCipher key rotation.** In-place
|
||||
rotation via `PRAGMA rekey`, serialized through a module-level
|
||||
`threading.Lock` and a SQLAlchemy `NullPool` to keep SQLCipher
|
||||
@@ -898,7 +570,7 @@ Shipped sub-projects (most recent first):
|
||||
- **Sub-project 13 (shipped) — SFTP wire-up.** `paramiko`-backed
|
||||
`SftpClient` replaces the SP9 stub. The clearhouse.submit endpoint
|
||||
actually pushes to
|
||||
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/ToHPE`.
|
||||
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
|
||||
SFTP credentials are read from the macOS Keychain at call time.
|
||||
- **Sub-project 12 (shipped) — Encryption at rest.** Optional
|
||||
SQLCipher AES-256 encryption of the SQLite file, with the key
|
||||
@@ -1160,9 +832,37 @@ the one-time setup recipe.
|
||||
|
||||
- `POST /api/clearhouse/submit` — same endpoint as SP9; the
|
||||
implementation is now a real `paramiko` `SftpClient.write` to
|
||||
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/ToHPE`.
|
||||
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
|
||||
SFTP credentials are fetched from the macOS Keychain at call time.
|
||||
|
||||
### SP14 endpoints (5-lane Inbox UI + acknowledge)
|
||||
|
||||
- `GET /api/inbox/lanes` — same shape as SP6; the `payer_rejected`
|
||||
lane payload now also includes
|
||||
`payer_rejected_acknowledged_at` + `payer_rejected_acknowledged_actor`
|
||||
per row so the UI can badge acknowledged claims (forward-compat for
|
||||
a future "Recently acknowledged" view).
|
||||
- `POST /api/inbox/payer-rejected/acknowledge` — bulk-acknowledge
|
||||
Payer-Rejected claims. Body: `{claim_ids: [...], actor: "..."}`.
|
||||
Idempotent. Response: `200` with
|
||||
`{transitioned, already_acked, not_found, not_rejected}`. Writes an
|
||||
`inbox.payer_rejected_acknowledged` audit event per transition.
|
||||
Acknowledgement hides a claim from the lane but never overwrites
|
||||
`payer_rejected_at` / `payer_rejected_reason` /
|
||||
`payer_rejected_by_277ca_id` — the original 277CA evidence stays
|
||||
intact in the audit log.
|
||||
|
||||
### SP15 endpoints (SQLCipher key rotation)
|
||||
|
||||
- `POST /api/admin/db/rotate-key` — rotate the SQLCipher DB key in
|
||||
place. Generates a fresh 256-bit key, writes it to the Keychain
|
||||
(overwriting `cyclone.db.key`), disposes the engine, issues
|
||||
`PRAGMA rekey`, verifies the schema, rebuilds the engine. Writes
|
||||
a `db.key_rotated` audit event with old + new key fingerprints
|
||||
and `table_count`. Returns `409` when a rotation is already in
|
||||
flight, `400` when encryption is not enabled, `503` with a
|
||||
`reason` on `PRAGMA rekey` or Keychain failure.
|
||||
|
||||
## License
|
||||
|
||||
No license file yet; this is internal-use software. Add a `LICENSE` file
|
||||
|
||||
-65
@@ -1,65 +0,0 @@
|
||||
# Cyclone Operator Runbook
|
||||
|
||||
Production operations for a single-operator Cyclone deploy on Ubuntu Linux. Assumes the box was bootstrapped via `scripts/cyclone-init.sh` and the stack is up via `docker compose up -d`.
|
||||
|
||||
## Daily
|
||||
|
||||
- [ ] Confirm the host healthcheck cron hasn't emailed. It pings `http://localhost:8080/api/health` every 5 minutes.
|
||||
- [ ] `docker compose ps` — both services `healthy`.
|
||||
- [ ] `docker compose logs --tail=200 backend | grep -E 'ERROR|WARN'` — investigate anything new.
|
||||
|
||||
## Weekly
|
||||
|
||||
- [ ] `curl -fsS http://localhost:8080/api/admin/audit-log -b cookies.txt | jq '.events[] | select(.event | test("login_failed|backup.failed"))'` — review failed logins + backup failures.
|
||||
- [ ] Confirm `docker compose exec backend ls -la /var/lib/cyclone/backups/` shows recent `.bin` files (within 25h of now).
|
||||
|
||||
## Quarterly
|
||||
|
||||
- [ ] Rotate the SQLCipher / cookie-signing key:
|
||||
```bash
|
||||
bash scripts/cyclone-init.sh --force # overwrites /etc/cyclone/secrets/db.key
|
||||
docker compose restart backend # picks up the new key
|
||||
```
|
||||
Old `.bin` backups become unreadable after this; export them first if you need to keep them.
|
||||
|
||||
## As needed
|
||||
|
||||
- **Add an operator.** Log in as admin → `/admin/users` → Create user. Roles: `admin` / `user` / `viewer`.
|
||||
- **Reset a password.** Admin UI → Users → Reset password, OR `docker compose exec backend python -m cyclone admin reset-password --username <name>`.
|
||||
- **Restore from backup.** Admin UI → Backups → pick the snapshot → Initiate restore → Confirm. The backend will restart automatically.
|
||||
- **Roll back the code (not the schema).** `TAG=0.0.9 docker compose up -d`. The previous image stays in the local Docker cache for one cycle.
|
||||
- **Pull a new `:stable`.**
|
||||
```bash
|
||||
cd /opt/cyclone
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
docker compose logs -f backend | head -200 # verify migrations + healthcheck
|
||||
```
|
||||
- **Off-box backup copy.** The operator is expected to rsync `/var/lib/docker/volumes/cyclone_backups/_data/` to an external drive or NAS nightly. The `.bin` files are already encrypted; the destination doesn't need its own encryption.
|
||||
- **Inspect the DB.** `docker compose exec backend sqlite3 /var/lib/cyclone/db/cyclone.db ".tables"` (works only if SQLCipher key is on disk; the in-process decrypt happens via the cyclone backend).
|
||||
|
||||
## Annual
|
||||
|
||||
- [ ] Rotate the admin password (force re-login for everyone).
|
||||
- [ ] Audit the `/etc/cyclone/secrets/` directory permissions — should be `chmod 600 root:root`.
|
||||
- [ ] Review the audit log for stale admin sessions.
|
||||
|
||||
## Emergency
|
||||
|
||||
- **Backend won't start.** `docker compose logs --tail=300 backend`. Look for migration failures (rerun is safe — migrations are forward-only), SQLCipher key mismatch (`PRAGMA key` failure), or port collisions.
|
||||
- **Frontend won't serve.** `docker compose logs --tail=100 frontend`. Usually nginx config drift; `docker compose restart frontend`.
|
||||
- **Both unhealthy after a host reboot.** Docker may have come up before the named volumes did. `docker compose down && docker compose up -d`.
|
||||
- **Suspected key compromise.** Rotate immediately (see Quarterly above). All active sessions are invalidated.
|
||||
|
||||
## Where things live
|
||||
|
||||
| Asset | Path |
|
||||
|---|---|
|
||||
| Docker compose file | `/opt/cyclone/docker-compose.yml` |
|
||||
| Secrets | `/etc/cyclone/secrets/{db.key,admin_username,admin_pw}` |
|
||||
| Live DB (SQLCipher-encrypted volume) | `cyclone_db` named volume, mounted at `/var/lib/cyclone/db` |
|
||||
| Encrypted backups | `cyclone_backups` named volume, mounted at `/var/lib/cyclone/backups` |
|
||||
| Uploaded prod files | `cyclone_prodfiles` named volume |
|
||||
| SFTP staging stub | `cyclone_sftp_staging` named volume |
|
||||
| Logs | `cyclone_logs` named volume + bind-mounted at `/var/log/cyclone` |
|
||||
| Off-box backup destination | Operator's external drive / NAS (rsync cron, not in compose) |
|
||||
-233
@@ -1,233 +0,0 @@
|
||||
// UI/UX Score Loop — pass 1 driver.
|
||||
// Loads each route at three sizes in Chrome Canary, captures screenshots,
|
||||
// logs console errors, runs a small interaction probe per flow, and
|
||||
// writes a JSON report. Does not modify any source files.
|
||||
|
||||
import puppeteer from "puppeteer-core";
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
|
||||
const BASE = "http://127.0.0.1:5173";
|
||||
const SHOTS = "/tmp/cyclone-uiux/shots";
|
||||
const REPORT = "/tmp/cyclone-uiux/report.json";
|
||||
|
||||
const SIZES = [
|
||||
{ name: "desktop", w: 1440, h: 900 },
|
||||
{ name: "tablet", w: 768, h: 1024 },
|
||||
{ name: "mobile", w: 375, h: 812 },
|
||||
];
|
||||
|
||||
// Routes to load. path = the route; name = the flow label; ready = a
|
||||
// selector we wait for to consider the page "rendered".
|
||||
const FLOWS = [
|
||||
{ name: "dashboard", path: "/", ready: "aside nav, h1, h2" },
|
||||
{ name: "upload", path: "/upload", ready: "section[aria-label='File upload']" },
|
||||
{ name: "inbox", path: "/inbox", ready: "main, section[aria-label='Queue summary']" },
|
||||
{ name: "claims", path: "/claims", ready: "table, [data-testid='claims-page-body']" },
|
||||
{ name: "claims-denied", path: "/claims?status=denied", ready: "table, [data-testid='claims-page-body']" },
|
||||
{ name: "remittances", path: "/remittances", ready: "main, table" },
|
||||
{ name: "providers", path: "/providers", ready: "main, table" },
|
||||
{ name: "reconciliation",path: "/reconciliation", ready: "main" },
|
||||
{ name: "acks", path: "/acks", ready: "main, table" },
|
||||
{ name: "batches", path: "/batches", ready: "main, table" },
|
||||
{ name: "batch-diff", path: "/batch-diff", ready: "main" },
|
||||
{ name: "activity", path: "/activity", ready: "main" },
|
||||
{ name: "404", path: "/does-not-exist", ready: "main" },
|
||||
];
|
||||
|
||||
async function setupViewports(browser) {
|
||||
const pages = [];
|
||||
for (const size of SIZES) {
|
||||
const page = await browser.newPage();
|
||||
await page.setViewport({ width: size.w, height: size.h, deviceScaleFactor: 1 });
|
||||
pages.push({ page, size });
|
||||
}
|
||||
return pages;
|
||||
}
|
||||
|
||||
async function probeFlow(page, flow) {
|
||||
const consoleErrors = [];
|
||||
const pageErrors = [];
|
||||
const failedRequests = [];
|
||||
|
||||
const onConsole = (msg) => {
|
||||
if (msg.type() === "error") consoleErrors.push(msg.text());
|
||||
};
|
||||
const onPageError = (err) => pageErrors.push(err.message);
|
||||
const onRequestFailed = (req) => failedRequests.push(`${req.method()} ${req.url()} :: ${req.failure()?.errorText}`);
|
||||
|
||||
page.on("console", onConsole);
|
||||
page.on("pageerror", onPageError);
|
||||
page.on("requestfailed", onRequestFailed);
|
||||
|
||||
const t0 = Date.now();
|
||||
let rendered = false;
|
||||
let readyError = null;
|
||||
try {
|
||||
await page.goto(`${BASE}${flow.path}`, { waitUntil: "networkidle2", timeout: 15000 });
|
||||
if (flow.ready) {
|
||||
try {
|
||||
await page.waitForSelector(flow.ready, { timeout: 5000 });
|
||||
rendered = true;
|
||||
} catch (e) {
|
||||
readyError = e.message;
|
||||
}
|
||||
} else {
|
||||
rendered = true;
|
||||
}
|
||||
} catch (e) {
|
||||
readyError = e.message;
|
||||
}
|
||||
const loadMs = Date.now() - t0;
|
||||
|
||||
page.off("console", onConsole);
|
||||
page.off("pageerror", onPageError);
|
||||
page.off("requestfailed", onRequestFailed);
|
||||
|
||||
return { rendered, loadMs, readyError, consoleErrors, pageErrors, failedRequests };
|
||||
}
|
||||
|
||||
async function probeInteractions(page, flow) {
|
||||
const findings = [];
|
||||
// Generic a11y / structural probes per flow.
|
||||
try {
|
||||
// Sidebar visible? (md+ shows it; < md hides it)
|
||||
const aside = await page.$("aside");
|
||||
findings.push({ check: "sidebar-present", pass: !!aside });
|
||||
} catch (e) {
|
||||
findings.push({ check: "sidebar-present", pass: false, err: e.message });
|
||||
}
|
||||
try {
|
||||
// Top bar present?
|
||||
const main = await page.$("main#main-content");
|
||||
findings.push({ check: "main-present", pass: !!main });
|
||||
} catch (e) {
|
||||
findings.push({ check: "main-present", pass: false, err: e.message });
|
||||
}
|
||||
try {
|
||||
// H1 or page heading?
|
||||
const heading = await page.evaluate(() => {
|
||||
const h = document.querySelector("h1, h2");
|
||||
return h ? h.textContent?.trim().slice(0, 60) : null;
|
||||
});
|
||||
findings.push({ check: "heading-present", pass: !!heading, value: heading });
|
||||
} catch (e) {
|
||||
findings.push({ check: "heading-present", pass: false, err: e.message });
|
||||
}
|
||||
|
||||
// Flow-specific probes.
|
||||
if (flow.name === "claims" || flow.name === "claims-denied") {
|
||||
try {
|
||||
const chips = await page.$$("[role='radio'], button[role='radio']");
|
||||
findings.push({ check: "status-chips", pass: chips.length >= 1, count: chips.length });
|
||||
} catch (e) {
|
||||
findings.push({ check: "status-chips", pass: false, err: e.message });
|
||||
}
|
||||
try {
|
||||
const search = await page.$("input[placeholder*='Search']");
|
||||
findings.push({ check: "search-input", pass: !!search });
|
||||
} catch (e) {
|
||||
findings.push({ check: "search-input", pass: false, err: e.message });
|
||||
}
|
||||
}
|
||||
if (flow.name === "upload") {
|
||||
try {
|
||||
const dropzone = await page.$("section[aria-label='File upload']");
|
||||
findings.push({ check: "dropzone-present", pass: !!dropzone });
|
||||
const selects = await page.$$("button[role='combobox']");
|
||||
findings.push({ check: "payer-kind-selects", pass: selects.length >= 2, count: selects.length });
|
||||
} catch (e) {
|
||||
findings.push({ check: "upload-elements", pass: false, err: e.message });
|
||||
}
|
||||
}
|
||||
if (flow.name === "inbox") {
|
||||
try {
|
||||
const lanes = await page.$$("main > div > div");
|
||||
findings.push({ check: "lane-cards", pass: lanes.length >= 1, count: lanes.length });
|
||||
} catch (e) {
|
||||
findings.push({ check: "lane-cards", pass: false, err: e.message });
|
||||
}
|
||||
}
|
||||
if (flow.name === "404") {
|
||||
try {
|
||||
const text = await page.evaluate(() => document.body.innerText);
|
||||
findings.push({ check: "404-text", pass: text.includes("404") || text.toLowerCase().includes("doesn't exist") });
|
||||
} catch (e) {
|
||||
findings.push({ check: "404-text", pass: false, err: e.message });
|
||||
}
|
||||
}
|
||||
|
||||
return findings;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
await mkdir(SHOTS, { recursive: true });
|
||||
const browser = await puppeteer.launch({
|
||||
executablePath: "/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary",
|
||||
headless: "new",
|
||||
args: ["--no-sandbox", "--disable-dev-shm-usage"],
|
||||
});
|
||||
const startedAt = new Date().toISOString();
|
||||
const results = [];
|
||||
|
||||
for (const size of SIZES) {
|
||||
const page = await browser.newPage();
|
||||
await page.setViewport({ width: size.w, height: size.h, deviceScaleFactor: 1 });
|
||||
|
||||
for (const flow of FLOWS) {
|
||||
const probe = await probeFlow(page, flow);
|
||||
const interactions = await probeInteractions(page, flow);
|
||||
const shot = `${SHOTS}/${flow.name}--${size.name}.png`;
|
||||
try {
|
||||
await page.screenshot({ path: shot, fullPage: false });
|
||||
} catch (e) {
|
||||
// ignore — recording in result
|
||||
}
|
||||
results.push({
|
||||
flow: flow.name,
|
||||
path: flow.path,
|
||||
size: size.name,
|
||||
viewport: { w: size.w, h: size.h },
|
||||
probe,
|
||||
interactions,
|
||||
shot,
|
||||
});
|
||||
console.log(
|
||||
`${size.name.padEnd(7)} ${flow.name.padEnd(20)} ` +
|
||||
`render=${probe.rendered} load=${probe.loadMs}ms ` +
|
||||
`consoleErr=${probe.consoleErrors.length} pageErr=${probe.pageErrors.length}`
|
||||
);
|
||||
}
|
||||
|
||||
await page.close();
|
||||
}
|
||||
|
||||
await browser.close();
|
||||
|
||||
// Aggregate.
|
||||
const summary = {
|
||||
startedAt,
|
||||
endedAt: new Date().toISOString(),
|
||||
flows: results.length,
|
||||
sizes: SIZES.map((s) => s.name),
|
||||
renderedOk: results.filter((r) => r.probe.rendered).length,
|
||||
withConsoleErrors: results.filter((r) => r.probe.consoleErrors.length > 0).length,
|
||||
withPageErrors: results.filter((r) => r.probe.pageErrors.length > 0).length,
|
||||
withFailedRequests: results.filter((r) => r.probe.failedRequests.length > 0).length,
|
||||
results,
|
||||
};
|
||||
await writeFile(REPORT, JSON.stringify(summary, null, 2));
|
||||
console.log("\nSummary:", JSON.stringify({
|
||||
flows: summary.flows,
|
||||
renderedOk: summary.renderedOk,
|
||||
withConsoleErrors: summary.withConsoleErrors,
|
||||
withPageErrors: summary.withPageErrors,
|
||||
withFailedRequests: summary.withFailedRequests,
|
||||
}, null, 2));
|
||||
console.log("\nReport:", REPORT);
|
||||
console.log("Shots:", SHOTS);
|
||||
}
|
||||
|
||||
main().catch((e) => {
|
||||
console.error("FATAL", e);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,12 +0,0 @@
|
||||
.venv/
|
||||
venv/
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*.egg-info/
|
||||
.pytest_cache/
|
||||
.ruff_cache/
|
||||
tests/
|
||||
docs/prodfiles/
|
||||
*.production.txt
|
||||
.git/
|
||||
.github/
|
||||
@@ -1,83 +0,0 @@
|
||||
# syntax=docker/dockerfile:1.7
|
||||
#
|
||||
# Cyclone backend — FastAPI on python:3.11-slim-bookworm with sqlcipher.
|
||||
#
|
||||
# Two-stage build:
|
||||
# 1. builder — wheels the package with [sqlcipher] extra into /wheels.
|
||||
# 2. runtime — slim base, tini PID 1, curl-based healthcheck.
|
||||
#
|
||||
# `sqlcipher` is preferred but the engine falls back to plain SQLite at
|
||||
# runtime if the package isn't actually installed (see cyclone.db) — so a
|
||||
# missing libsqlcipher-dev during build will fail loudly here rather than
|
||||
# silently downgrading encryption in production.
|
||||
|
||||
# ---------- builder ----------
|
||||
FROM python:3.11-slim-bookworm AS builder
|
||||
|
||||
ENV PIP_NO_CACHE_DIR=1 \
|
||||
PIP_DISABLE_PIP_VERSION_CHECK=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
libffi-dev \
|
||||
libsqlcipher-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
# Copy the build manifest first so this layer caches across source edits.
|
||||
COPY pyproject.toml ./
|
||||
|
||||
# Copy the full source tree, then build the wheel once. We deliberately
|
||||
# avoid the "stub __init__.py, build wheel, then rebuild" pattern — it
|
||||
# left stale `__init__.py` content in the wheel because pip wheel reuses
|
||||
# the cached wheel metadata when the name+version matches. See git
|
||||
# history on this file for the long version.
|
||||
COPY src/ ./src/
|
||||
# Install the sftp extra alongside sqlcipher so the real-mode SFTP
|
||||
# client (paramiko) is available inside the container — required by
|
||||
# SP25 + SP26 for live Gainwell MFT polling.
|
||||
RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher,sftp]'
|
||||
|
||||
# ---------- runtime ----------
|
||||
FROM python:3.11-slim-bookworm
|
||||
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1
|
||||
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
libsqlcipher-dev \
|
||||
curl \
|
||||
tini \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& useradd --create-home --uid 1000 --shell /bin/bash cyclone
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY --from=builder /wheels /wheels
|
||||
RUN pip install --no-cache-dir --no-index --find-links /wheels 'cyclone[sqlcipher,sftp]' \
|
||||
&& rm -rf /wheels
|
||||
|
||||
# NOTE: we deliberately do NOT drop privileges to the `cyclone` user.
|
||||
# Named volumes mount as root inside the container, and chown-ing them
|
||||
# requires CAP_CHOWN (root). The standard hardened pattern is an
|
||||
# entrypoint script that chowns as root then drops to the app user via
|
||||
# gosu/su-exec — adds a dependency + an entrypoint file. For v1 we run
|
||||
# as root inside the container; Docker's user-namespace remapping is
|
||||
# the recommended host-level isolation. The `cyclone` user is created
|
||||
# above and survives only so file ownership in bind mounts stays
|
||||
# consistent. To harden later: install gosu + add an entrypoint script
|
||||
# that does `chown -R cyclone:cyclone /var/lib/cyclone/... && exec gosu
|
||||
# cyclone "$@"`.
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
# Container-level healthcheck — the compose service healthcheck is
|
||||
# effectively a duplicate but the Docker `HEALTHCHECK` directive keeps
|
||||
# `docker ps` honest without needing compose to be running.
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
|
||||
CMD curl -fs http://localhost:8000/api/health || exit 1
|
||||
|
||||
ENTRYPOINT ["tini", "--"]
|
||||
CMD ["python", "-m", "cyclone", "serve"]
|
||||
@@ -16,15 +16,6 @@ dependencies = [
|
||||
"sqlalchemy>=2.0,<3",
|
||||
"pyyaml>=6.0,<7",
|
||||
"keyring>=25.0,<26",
|
||||
# backup_service / backup: encryption-at-rest (SP17). Used at module
|
||||
# top-level by cyclone.backup, so it has to be a hard dep — not an
|
||||
# extra — or the test suite fails to collect when the venv is built
|
||||
# from a clean `uv sync`.
|
||||
"cryptography>=49.0,<50",
|
||||
# passlib 1.7.4 + bcrypt >= 4.1 are incompatible (passlib probes bcrypt.__about__
|
||||
# which 4.x removed). Pin bcrypt < 4.1.
|
||||
"passlib[bcrypt]>=1.7.4",
|
||||
"bcrypt<4.1",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
@@ -33,7 +24,6 @@ dev = [
|
||||
"pytest-cov>=4.1",
|
||||
"pytest-asyncio>=0.23,<1",
|
||||
"httpx>=0.27,<1",
|
||||
"pytest-randomly>=4.1",
|
||||
]
|
||||
sqlcipher = [
|
||||
# SP12: encryption at rest. Optional — without it the DB is plain SQLite.
|
||||
|
||||
@@ -1,11 +1,10 @@
|
||||
"""Entry point for ``python -m cyclone``.
|
||||
|
||||
* ``python -m cyclone`` (no args) — Click CLI (``cli.main``)
|
||||
* ``python -m cyclone serve`` — start the FastAPI app on 0.0.0.0:8000
|
||||
* ``python -m cyclone serve`` — start the FastAPI app on 127.0.0.1:8000
|
||||
|
||||
Honors the env vars:
|
||||
|
||||
* ``CYCLONE_HOST`` (default ``0.0.0.0`` — always bind to all interfaces)
|
||||
* ``CYCLONE_PORT`` (default ``8000``)
|
||||
* ``CYCLONE_RELOAD`` (default ``0``; set to ``1`` to enable uvicorn reload)
|
||||
"""
|
||||
@@ -17,42 +16,13 @@ import sys
|
||||
|
||||
|
||||
def main() -> None:
|
||||
# Always run first-admin bootstrap before any other entry path.
|
||||
# Must happen before ``serve`` (uvicorn) AND before the Click CLI
|
||||
# dispatch — otherwise `python -m cyclone users create ...` on a
|
||||
# fresh DB would race with the bootstrap's check, and the API
|
||||
# could come up with zero users.
|
||||
from cyclone.auth import bootstrap
|
||||
bootstrap.run()
|
||||
|
||||
# SP24: if the AUTH_DISABLED escape hatch is on, scream at boot so a
|
||||
# misconfigured production deploy fails loudly. The flag is flipped by
|
||||
# ``CYCLONE_AUTH_DISABLED=1`` (see ``cyclone.auth.bootstrap``) and by the
|
||||
# pytest conftest autouse fixture (see ``.superpowers/skills/cyclone-tests``).
|
||||
from cyclone.auth import deps as _auth_deps
|
||||
|
||||
if _auth_deps.AUTH_DISABLED:
|
||||
import logging
|
||||
|
||||
logging.getLogger("cyclone").warning(
|
||||
"AUTH_DISABLED is set (CYCLONE_AUTH_DISABLED=1) — all requests "
|
||||
"treated as admin, dev only. Do NOT enable this in production."
|
||||
)
|
||||
|
||||
if len(sys.argv) >= 2 and sys.argv[1] == "serve":
|
||||
port = os.environ.get("CYCLONE_PORT", "8000")
|
||||
# Always bind to 0.0.0.0 so the API is reachable from the
|
||||
# frontend container on the compose bridge network AND from
|
||||
# the host (Vite dev proxy) AND from the LAN. Network isolation
|
||||
# is provided by the host firewall / compose port publishing,
|
||||
# not by binding to loopback. Override with CYCLONE_HOST if you
|
||||
# have a reason to restrict.
|
||||
host = os.environ.get("CYCLONE_HOST", "0.0.0.0")
|
||||
reload = os.environ.get("CYCLONE_RELOAD", "0") == "1"
|
||||
sys.argv = [
|
||||
sys.argv[0],
|
||||
"cyclone.api:app",
|
||||
"--host", host,
|
||||
"--host", "127.0.0.1",
|
||||
"--port", port,
|
||||
]
|
||||
if reload:
|
||||
|
||||
+155
-2537
File diff suppressed because it is too large
Load Diff
@@ -93,40 +93,19 @@ def ndjson_stream_list(
|
||||
}) + "\n"
|
||||
|
||||
|
||||
def ndjson_stream_837(
|
||||
result: ParseResult, batch_id: str | None = None,
|
||||
) -> Iterator[bytes]:
|
||||
"""Yield one JSON object per line: envelope → claims → summary.
|
||||
|
||||
The ``batch_id`` is the server-side UUID assigned by the persistence
|
||||
layer when the batch is ingested; the JSON response path exposes it
|
||||
as the top-level ``batch_id`` field, but the NDJSON stream needs it
|
||||
inline on the summary event so streaming clients can call
|
||||
batch-scoped endpoints (``/api/batches/{id}/export-837``, …) without
|
||||
a separate ``GET /api/batches`` round-trip. When ``batch_id`` is not
|
||||
supplied the summary omits the field, preserving backward compat
|
||||
with clients that don't expect it.
|
||||
"""
|
||||
def ndjson_stream_837(result: ParseResult) -> Iterator[bytes]:
|
||||
"""Yield one JSON object per line: envelope → claims → summary."""
|
||||
envelope_obj = (
|
||||
result.envelope.model_dump() if result.envelope is not None else None
|
||||
)
|
||||
yield (json.dumps({"type": "envelope", "data": envelope_obj}) + "\n").encode("utf-8")
|
||||
for claim in result.claims:
|
||||
yield (json.dumps({"type": "claim", "data": json.loads(claim.model_dump_json())}) + "\n").encode("utf-8")
|
||||
summary_data = json.loads(result.summary.model_dump_json())
|
||||
if batch_id is not None:
|
||||
summary_data["batch_id"] = batch_id
|
||||
yield (json.dumps({"type": "summary", "data": summary_data}) + "\n").encode("utf-8")
|
||||
yield (json.dumps({"type": "summary", "data": json.loads(result.summary.model_dump_json())}) + "\n").encode("utf-8")
|
||||
|
||||
|
||||
def ndjson_stream_835(
|
||||
result: ParseResult835, batch_id: str | None = None,
|
||||
) -> Iterator[bytes]:
|
||||
"""Yield one JSON object per line: envelope → financial → trace → payer → payee → claim_payments → summary.
|
||||
|
||||
See ``ndjson_stream_837`` for why the optional ``batch_id`` is
|
||||
merged into the summary event.
|
||||
"""
|
||||
def ndjson_stream_835(result: ParseResult835) -> Iterator[bytes]:
|
||||
"""Yield one JSON object per line: envelope → financial → trace → payer → payee → claim_payments → summary."""
|
||||
yield (json.dumps({"type": "envelope", "data": json.loads(result.envelope.model_dump_json())}) + "\n").encode("utf-8")
|
||||
yield (json.dumps({"type": "financial_info", "data": json.loads(result.financial_info.model_dump_json())}) + "\n").encode("utf-8")
|
||||
yield (json.dumps({"type": "trace", "data": json.loads(result.trace.model_dump_json())}) + "\n").encode("utf-8")
|
||||
@@ -134,10 +113,7 @@ def ndjson_stream_835(
|
||||
yield (json.dumps({"type": "payee", "data": json.loads(result.payee.model_dump_json())}) + "\n").encode("utf-8")
|
||||
for claim in result.claims:
|
||||
yield (json.dumps({"type": "claim_payment", "data": json.loads(claim.model_dump_json())}) + "\n").encode("utf-8")
|
||||
summary_data = json.loads(result.summary.model_dump_json())
|
||||
if batch_id is not None:
|
||||
summary_data["batch_id"] = batch_id
|
||||
yield (json.dumps({"type": "summary", "data": summary_data}) + "\n").encode("utf-8")
|
||||
yield (json.dumps({"type": "summary", "data": json.loads(result.summary.model_dump_json())}) + "\n").encode("utf-8")
|
||||
|
||||
|
||||
def strict_rewrite_837(result: ParseResult) -> ParseResult:
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""``/api/acks`` — list, detail, and live-tail stream for 999 ACKs.
|
||||
"""``/api/acks`` — list & detail endpoints for the 999 ACK inbox.
|
||||
|
||||
These are the persisted acknowledgment rows produced by
|
||||
``POST /api/parse-999``. The frontend ``useAcks`` hook re-shapes the
|
||||
@@ -7,74 +7,58 @@ list payload to its ``Ack`` interface in ``src/types/index.ts``.
|
||||
The detail endpoint returns the full ``raw_json`` payload plus the
|
||||
regenerated ``raw_999_text`` so the UI can show "view source" without a
|
||||
second round-trip.
|
||||
|
||||
SP25: ``/api/acks/stream`` joins the live-tail triplet — the Acks
|
||||
page mounts ``useTailStream("acks")`` and ``useMergedTail("acks", …)``
|
||||
to see new 999 acks the moment they land (whether from the SFTP
|
||||
poller or a manual upload).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any, AsyncIterator
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.api_helpers import (
|
||||
ndjson_line,
|
||||
ndjson_stream_list,
|
||||
tail_events,
|
||||
wants_ndjson,
|
||||
)
|
||||
from cyclone.api_helpers import ndjson_stream_list, wants_ndjson
|
||||
from cyclone.parsers.models_999 import ParseResult999
|
||||
from cyclone.parsers.serialize_999 import serialize_999
|
||||
from cyclone.pubsub import EventBus
|
||||
from cyclone.store import store, to_ui_ack
|
||||
from cyclone.store import store
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# SP25: ``_ack_to_ui`` moved to ``cyclone.store.ui.to_ui_ack`` so the
|
||||
# live-tail event payload (``ack_received``) matches the list endpoint
|
||||
# shape byte-for-byte.
|
||||
def _ack_to_ui(row) -> dict:
|
||||
"""Map an ``Ack`` ORM row to the UI shape used by ``/api/acks``.
|
||||
|
||||
Field names match the rest of the Cyclone API (snake_case). The
|
||||
frontend ``useAcks`` hook re-shapes this to the camelCase ``Ack``
|
||||
interface in ``src/types/index.ts``.
|
||||
"""
|
||||
return {
|
||||
"id": row.id,
|
||||
"source_batch_id": row.source_batch_id,
|
||||
"accepted_count": row.accepted_count,
|
||||
"rejected_count": row.rejected_count,
|
||||
"received_count": row.received_count,
|
||||
"ack_code": row.ack_code,
|
||||
"parsed_at": (
|
||||
row.parsed_at.isoformat().replace("+00:00", "Z")
|
||||
if row.parsed_at is not None
|
||||
else ""
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/api/acks")
|
||||
def list_acks_endpoint(
|
||||
request: Request,
|
||||
limit: int = Query(100, ge=1, le=5000),
|
||||
offset: int = Query(0, ge=0),
|
||||
limit: int = Query(100, ge=1, le=1000),
|
||||
) -> Any:
|
||||
"""Return the list of persisted 999 ACKs, newest first.
|
||||
|
||||
``limit`` caps the page size; ``offset`` lets the UI walk the
|
||||
full set without holding it all in memory. ``aggregates`` is
|
||||
summed over the *full* row set (not the page) so the KPI strip
|
||||
on the Acks page reflects every persisted 999, not just the
|
||||
visible 50. Without server-side aggregates the page would
|
||||
silently under-report (silent-failure mode) once the row count
|
||||
exceeds the page size.
|
||||
"""
|
||||
"""Return the list of persisted 999 ACKs, newest first."""
|
||||
rows = store.list_acks()
|
||||
items = [to_ui_ack(r) for r in rows[offset : offset + limit]]
|
||||
items = [_ack_to_ui(r) for r in rows[:limit]]
|
||||
total = len(rows)
|
||||
returned = len(items)
|
||||
has_more = offset + returned < total
|
||||
aggregates = {
|
||||
"accepted_count": sum(r.accepted_count or 0 for r in rows),
|
||||
"rejected_count": sum(r.rejected_count or 0 for r in rows),
|
||||
"received_count": sum(r.received_count or 0 for r in rows),
|
||||
}
|
||||
# SP28: batch-fetch linked_claim_ids per ack row in one query to
|
||||
# avoid N+1 — see ``find_linked_claim_ids_for_acks`` below.
|
||||
ack_ids = [r.id for r in rows]
|
||||
linked_map = _find_linked_claim_ids_for_acks(ack_ids, kind="999")
|
||||
for item, aid in zip(items, ack_ids[offset : offset + limit]):
|
||||
item["linked_claim_ids"] = linked_map.get(aid, [])
|
||||
has_more = total > returned
|
||||
if wants_ndjson(request):
|
||||
return StreamingResponse(
|
||||
ndjson_stream_list(items, total, returned, has_more),
|
||||
@@ -85,68 +69,9 @@ def list_acks_endpoint(
|
||||
"total": total,
|
||||
"returned": returned,
|
||||
"has_more": has_more,
|
||||
"aggregates": aggregates,
|
||||
}
|
||||
|
||||
|
||||
def _find_linked_claim_ids_for_acks(
|
||||
ack_ids: list[int], *, kind: str
|
||||
) -> dict[int, list[str]]:
|
||||
"""Batch-fetch {ack_id: [claim_id, …]} for the listed ack rows.
|
||||
|
||||
One SELECT against ``claim_acks`` keyed on the page's ack_ids —
|
||||
avoids an N+1 round-trip when the page renders the
|
||||
"🔗 N claims" badge per row. Used by the 999 / TA1 / 277CA
|
||||
list endpoints. Returns a ``{ack_id: [claim_id]}`` map; acks
|
||||
with no links map to ``[]`` (default).
|
||||
"""
|
||||
out: dict[int, list[str]] = {aid: [] for aid in ack_ids}
|
||||
if not ack_ids:
|
||||
return out
|
||||
with db.SessionLocal()() as s:
|
||||
rows = (
|
||||
s.query(db.ClaimAck.ack_id, db.ClaimAck.claim_id)
|
||||
.filter(
|
||||
db.ClaimAck.ack_kind == kind,
|
||||
db.ClaimAck.ack_id.in_(ack_ids),
|
||||
db.ClaimAck.claim_id.isnot(None),
|
||||
)
|
||||
.all()
|
||||
)
|
||||
for ack_id, claim_id in rows:
|
||||
out[ack_id].append(claim_id)
|
||||
return out
|
||||
|
||||
|
||||
@router.get("/api/acks/stream")
|
||||
async def acks_stream(
|
||||
request: Request,
|
||||
limit: int = Query(100, ge=1, le=1000),
|
||||
) -> StreamingResponse:
|
||||
"""Stream 999 ACKs as NDJSON: snapshot first, then live events.
|
||||
|
||||
SP25: this endpoint joins the live-tail triplet — subscribes to
|
||||
``ack_received`` and emits one ``item`` per snapshot row plus a
|
||||
single ``snapshot_end`` line, then forwards live events from
|
||||
the bus. Matches the wire format used by ``/api/claims/stream``,
|
||||
``/api/remittances/stream``, and ``/api/activity/stream``.
|
||||
|
||||
NOTE: registered BEFORE ``/api/acks/{ack_id}`` so the literal
|
||||
``stream`` path segment doesn't get matched as an ack id.
|
||||
"""
|
||||
bus: EventBus = request.app.state.event_bus
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
rows = store.list_acks()[:limit]
|
||||
for row in rows:
|
||||
yield ndjson_line({"type": "item", "data": to_ui_ack(row)})
|
||||
yield ndjson_line({"type": "snapshot_end", "data": {"count": len(rows)}})
|
||||
async for chunk in tail_events(request, bus, ["ack_received"]):
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
|
||||
|
||||
@router.get("/api/acks/{ack_id}")
|
||||
def get_ack_endpoint(ack_id: int) -> dict:
|
||||
"""Return one persisted ACK row with its parsed detail.
|
||||
@@ -161,7 +86,7 @@ def get_ack_endpoint(ack_id: int) -> dict:
|
||||
status_code=404,
|
||||
detail={"error": "Not found", "detail": f"Ack {ack_id} not found"},
|
||||
)
|
||||
body = to_ui_ack(row)
|
||||
body = _ack_to_ui(row)
|
||||
body["raw_json"] = row.raw_json
|
||||
# Regenerate the X12 text from raw_json so the operator can download
|
||||
# the actual 999 file. (SP3 P3 follow-up: list endpoint doesn't carry
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
"""``/api/activity`` — list & live-tail of recorded Activity events.
|
||||
|
||||
The Activity log is the cross-resource audit feed: every parser write,
|
||||
inbox state change, match/dismiss/resubmit, clearhouse submission, etc.
|
||||
records one row keyed by ``kind``. The list endpoint returns the most
|
||||
recent ``limit`` events with optional ``kind`` / ``since`` filters; the
|
||||
stream endpoint emits a snapshot followed by live updates from the
|
||||
``EventBus``.
|
||||
|
||||
The default ``limit`` on the list endpoint is 200 (matches the prior
|
||||
inline behavior); the stream endpoint defaults to 50 because activity is
|
||||
high-volume — callers usually want the most recent handful, not a full
|
||||
replay.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, AsyncIterator
|
||||
|
||||
from fastapi import APIRouter, Query, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from cyclone.api_helpers import (
|
||||
ndjson_line,
|
||||
ndjson_stream_list,
|
||||
tail_events,
|
||||
wants_ndjson,
|
||||
)
|
||||
from cyclone.pubsub import EventBus
|
||||
from cyclone.store import store
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/api/activity")
|
||||
def list_activity_endpoint(
|
||||
request: Request,
|
||||
kind: str | None = Query(None),
|
||||
since: str | None = Query(None),
|
||||
limit: int = Query(200, ge=1, le=500),
|
||||
) -> Any:
|
||||
"""Return the recent Activity log, newest first.
|
||||
|
||||
Optional ``kind`` filters by ``event["kind"]`` (e.g. ``claim_submitted``).
|
||||
Optional ``since`` is an inclusive ISO timestamp lower bound applied
|
||||
to ``event["timestamp"]``.
|
||||
"""
|
||||
events = store.recent_activity(limit=limit)
|
||||
if kind is not None:
|
||||
events = [e for e in events if e["kind"] == kind]
|
||||
if since is not None:
|
||||
events = [e for e in events if e["timestamp"] >= since]
|
||||
total = len(events)
|
||||
has_more = False
|
||||
if wants_ndjson(request):
|
||||
return StreamingResponse(
|
||||
ndjson_stream_list(events, total, total, has_more),
|
||||
media_type="application/x-ndjson",
|
||||
)
|
||||
return {
|
||||
"items": events,
|
||||
"total": total,
|
||||
"returned": total,
|
||||
"has_more": has_more,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/api/activity/stream")
|
||||
async def activity_stream_endpoint(
|
||||
request: Request,
|
||||
kind: str | None = Query(None),
|
||||
since: str | None = Query(None),
|
||||
limit: int = Query(50, ge=1, le=500),
|
||||
) -> StreamingResponse:
|
||||
"""Stream Activity events as NDJSON: snapshot first, then live events.
|
||||
|
||||
Subscribes to ``activity_recorded``. The snapshot half reuses the
|
||||
same in-memory filter as ``list_activity`` so the two endpoints are
|
||||
interchangeable for the snapshot half.
|
||||
"""
|
||||
bus: EventBus = request.app.state.event_bus
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
events = store.recent_activity(limit=limit)
|
||||
if kind is not None:
|
||||
events = [e for e in events if e["kind"] == kind]
|
||||
if since is not None:
|
||||
events = [e for e in events if e["timestamp"] >= since]
|
||||
for ev in events:
|
||||
yield ndjson_line({"type": "item", "data": ev})
|
||||
yield ndjson_line({
|
||||
"type": "snapshot_end", "data": {"count": len(events)},
|
||||
})
|
||||
|
||||
async for chunk in tail_events(request, bus, ["activity_recorded"]):
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
@@ -1,60 +0,0 @@
|
||||
"""``/api/admin/validate-provider`` — NPI + Tax ID liveness probe (SP20).
|
||||
|
||||
Pure read-only endpoint that runs the local NPI Luhn + EIN format checks
|
||||
without touching the DB. Useful for:
|
||||
- operators vetting a new provider before adding them to the registry,
|
||||
- the dashboard's "validate" button on a Provider row,
|
||||
- smoke-testing the SP20 checks after a deploy.
|
||||
|
||||
Both query params are optional; omitting one just skips that check.
|
||||
Returns the per-check result dict so the caller can distinguish "bad
|
||||
format" from "bad checksum".
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
from cyclone.npi import is_valid_npi, is_valid_tax_id, normalize_tax_id
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/api/admin/validate-provider")
|
||||
def validate_provider(
|
||||
npi: str | None = Query(None, description="10-digit NPI to validate (Luhn checksum)"),
|
||||
tax_id: str | None = Query(None, description="9-digit EIN to validate (format + reserved-prefix check)"),
|
||||
) -> dict:
|
||||
"""Return per-field validation results for ``npi`` and ``tax_id``.
|
||||
|
||||
Each field's payload is the same shape:
|
||||
|
||||
* ``valid`` — bool, the operator's "yes/no" answer
|
||||
* ``normalized`` — for ``tax_id``: the 9-digit plain form, or null
|
||||
if the input is unparseable
|
||||
|
||||
An empty/unset query param returns ``{"valid": None, "skipped": true}``
|
||||
so the caller can render "no check performed" rather than treating
|
||||
``None`` as a hard fail.
|
||||
"""
|
||||
result: dict = {}
|
||||
|
||||
if npi is None or npi == "":
|
||||
result["npi"] = {"valid": None, "skipped": True}
|
||||
else:
|
||||
result["npi"] = {
|
||||
"valid": is_valid_npi(npi),
|
||||
"skipped": False,
|
||||
}
|
||||
|
||||
if tax_id is None or tax_id == "":
|
||||
result["tax_id"] = {"valid": None, "skipped": True, "normalized": None}
|
||||
else:
|
||||
normalized = normalize_tax_id(tax_id)
|
||||
result["tax_id"] = {
|
||||
"valid": is_valid_tax_id(tax_id),
|
||||
"skipped": False,
|
||||
"normalized": normalized,
|
||||
}
|
||||
|
||||
return result
|
||||
@@ -0,0 +1,75 @@
|
||||
"""``/api/batches`` — list & detail endpoints for parsed batches.
|
||||
|
||||
Each ``store.add(...)`` call (837P, 835, 999, TA1, 277CA) produces a
|
||||
``BatchRecord`` kept in memory by :class:`CycloneStore`. The list
|
||||
endpoint returns a lightweight summary (id, kind, filename, parsedAt,
|
||||
claim count) for the recent batches; the detail endpoint returns the
|
||||
full parsed payload for one batch.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from cyclone.api_helpers import ndjson_stream_list, wants_ndjson
|
||||
from cyclone.store import BatchRecord, store
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
def _batch_summary_claim_count(rec: BatchRecord) -> int:
|
||||
"""Return the number of claims on a batch, handling both 837P and 835."""
|
||||
if rec.kind == "837p":
|
||||
return len(rec.result.claims) # type: ignore[attr-defined]
|
||||
if rec.kind == "835":
|
||||
return len(rec.result.claims) # type: ignore[attr-defined]
|
||||
return 0
|
||||
|
||||
|
||||
@router.get("/api/batches")
|
||||
def list_batches_endpoint(
|
||||
request: Request,
|
||||
limit: int = Query(100, ge=1, le=1000),
|
||||
) -> Any:
|
||||
"""Summary of all parsed batches, newest first."""
|
||||
records = store.list(limit=limit)
|
||||
items = [
|
||||
{
|
||||
"id": r.id,
|
||||
"kind": r.kind,
|
||||
"inputFilename": r.input_filename,
|
||||
"parsedAt": r.parsed_at.isoformat().replace("+00:00", "Z"),
|
||||
"claimCount": _batch_summary_claim_count(r),
|
||||
}
|
||||
for r in records
|
||||
]
|
||||
all_records = store.all()
|
||||
total = len(all_records)
|
||||
returned = len(items)
|
||||
has_more = total > returned
|
||||
if wants_ndjson(request):
|
||||
return StreamingResponse(
|
||||
ndjson_stream_list(items, total, returned, has_more),
|
||||
media_type="application/x-ndjson",
|
||||
)
|
||||
return {
|
||||
"items": items,
|
||||
"total": total,
|
||||
"returned": returned,
|
||||
"has_more": has_more,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/api/batches/{batch_id}")
|
||||
def get_batch_endpoint(batch_id: str) -> Any:
|
||||
"""Return the full parsed payload for one batch."""
|
||||
rec = store.get(batch_id)
|
||||
if rec is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={"error": "Not found", "detail": f"Batch {batch_id} not found"},
|
||||
)
|
||||
return json.loads(rec.result.model_dump_json())
|
||||
@@ -1,306 +0,0 @@
|
||||
"""``/api/claim-acks`` (and per-claim/per-ack surfaces) — SP28.
|
||||
|
||||
Seven endpoints that surface the ``claim_acks`` join table to the
|
||||
frontend + manual-match fallback for orphans. Mounted by
|
||||
``cyclone.api`` alongside the existing ``/api/acks`` and
|
||||
``/api/ta1-acks`` routers.
|
||||
|
||||
The live-tail endpoints (``/api/claims/{id}/acks/stream`` and
|
||||
``/api/acks/{kind}/{id}/claims/stream``) subscribe to the
|
||||
``claim_ack_written`` bus event so the ClaimDrawer Acknowledgments
|
||||
panel and the per-ack claims list refresh in real time.
|
||||
|
||||
Manual match is any-logged-in user (D5) — this endpoint mutates
|
||||
metadata only (``claim_acks`` row + live-tail event), no
|
||||
``Claim.state`` mutation, no payment data. Idempotent: re-calling
|
||||
with the same ``claim_id`` returns 200 with the existing row.
|
||||
Rejects (409) when the claim is in a terminal state (``REVERSED``).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any, AsyncIterator, Literal
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import JSONResponse, StreamingResponse
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.api_helpers import ndjson_line, tail_events
|
||||
from cyclone.pubsub import EventBus
|
||||
from cyclone.store import store, to_ui_claim_ack
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
AckKind = Literal["999", "277ca", "ta1"]
|
||||
|
||||
|
||||
class MatchClaimBody(BaseModel):
|
||||
"""Body for ``POST /api/acks/{kind}/{ack_id}/match-claim``."""
|
||||
|
||||
claim_id: str = Field(..., min_length=1)
|
||||
set_control_number: str | None = None
|
||||
set_accept_reject_code: str | None = None
|
||||
ak2_index: int | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-claim surface
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.get("/api/claims/{claim_id}/acks/stream")
|
||||
async def claim_acks_stream(
|
||||
request: Request,
|
||||
claim_id: str,
|
||||
) -> StreamingResponse:
|
||||
"""Stream ClaimAck rows for one claim as NDJSON.
|
||||
|
||||
Subscribes to ``claim_ack_written`` and filters for rows where
|
||||
``claim_id`` matches the path param. Each matching event is
|
||||
emitted as ``{"type": "item", "data": to_ui_claim_ack(...)}``;
|
||||
the client-side ``useMergedTail`` hook dedupes by id.
|
||||
|
||||
Registered BEFORE ``/api/claims/{claim_id}/acks`` so the literal
|
||||
``stream`` path segment doesn't get matched as a suffix.
|
||||
"""
|
||||
bus: EventBus = request.app.state.event_bus
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
rows = [
|
||||
r for r in store.list_acks_for_claim(claim_id)
|
||||
if r.claim_id is not None # filter out TA1 batch-level rows
|
||||
]
|
||||
for row in rows:
|
||||
yield ndjson_line({"type": "item", "data": to_ui_claim_ack(row)})
|
||||
yield ndjson_line({
|
||||
"type": "snapshot_end",
|
||||
"data": {"count": len(rows)},
|
||||
})
|
||||
|
||||
async for chunk in tail_events(request, bus, ["claim_ack_written"]):
|
||||
# tail_events yields full NDJSON lines; the client filter
|
||||
# picks claim_id matches. We forward every event so the
|
||||
# wire format mirrors /api/claims/stream.
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
|
||||
|
||||
@router.get("/api/claims/{claim_id}/acks")
|
||||
def list_acks_for_claim_endpoint(claim_id: str) -> Any:
|
||||
"""Return every ClaimAck row for one claim (per-claim only).
|
||||
|
||||
TA1 batch-level rows (where ``claim_id IS NULL``) are filtered
|
||||
out — those don't belong to a specific claim, they're a
|
||||
envelope-level acknowledgement that hangs off the originating
|
||||
837 batch. The ClaimDrawer Acknowledgments panel is
|
||||
per-claim only.
|
||||
"""
|
||||
rows = store.list_acks_for_claim(claim_id)
|
||||
items = [to_ui_claim_ack(r) for r in rows if r.claim_id is not None]
|
||||
return {"total": len(items), "items": items}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-ack surface
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.get("/api/acks/{kind}/{ack_id}/claims/stream")
|
||||
async def ack_claims_stream(
|
||||
request: Request,
|
||||
kind: AckKind,
|
||||
ack_id: int,
|
||||
) -> StreamingResponse:
|
||||
"""Stream ClaimAck rows for one ack as NDJSON.
|
||||
|
||||
Subscribes to ``claim_ack_written`` and forwards events the
|
||||
store knows about. Clients filter by ack_id + ack_kind on the
|
||||
client side.
|
||||
"""
|
||||
bus: EventBus = request.app.state.event_bus
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
rows = store.list_claims_for_ack(kind, ack_id)
|
||||
for row in rows:
|
||||
yield ndjson_line({"type": "item", "data": to_ui_claim_ack(row)})
|
||||
yield ndjson_line({
|
||||
"type": "snapshot_end",
|
||||
"data": {"count": len(rows)},
|
||||
})
|
||||
async for chunk in tail_events(request, bus, ["claim_ack_written"]):
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
|
||||
|
||||
@router.get("/api/acks/{kind}/{ack_id}/claims")
|
||||
def list_claims_for_ack_endpoint(kind: AckKind, ack_id: int) -> Any:
|
||||
"""Return every ClaimAck row for one ack (any kind).
|
||||
|
||||
For 999 / 277CA: returns 0..N rows (one per AK2 / ClaimStatus).
|
||||
For TA1: returns 0..1 row (envelope-level, populated batch_id).
|
||||
"""
|
||||
rows = store.list_claims_for_ack(kind, ack_id)
|
||||
items = [to_ui_claim_ack(r) for r in rows]
|
||||
return {"total": len(items), "items": items}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Manual match / unmatch (D5, D9)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.post("/api/acks/{kind}/{ack_id}/match-claim")
|
||||
def manual_match_claim_endpoint(
|
||||
request: Request,
|
||||
kind: AckKind,
|
||||
ack_id: int,
|
||||
body: MatchClaimBody,
|
||||
) -> Any:
|
||||
"""Manual link fallback (D5/D9). Any-logged-in-user posture.
|
||||
|
||||
Inserts a ``claim_acks`` row with ``linked_by="manual"`` and
|
||||
publishes ``claim_ack_written`` so the drawers refresh. The
|
||||
endpoint is idempotent: if a row already exists for this dedup
|
||||
key, the existing row is returned (200). 409 when the claim is
|
||||
in a terminal state (``REVERSED``). 404 when the claim doesn't
|
||||
exist or the ack doesn't exist.
|
||||
"""
|
||||
# Verify the claim exists and is in a non-terminal state.
|
||||
from cyclone.db import ClaimState as _CS
|
||||
with db.SessionLocal()() as s:
|
||||
claim = s.get(db.Claim, body.claim_id)
|
||||
if claim is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={
|
||||
"error": "Not found",
|
||||
"detail": f"Claim {body.claim_id} not found",
|
||||
},
|
||||
)
|
||||
if claim.state == _CS.REVERSED:
|
||||
return JSONResponse(
|
||||
status_code=409,
|
||||
content={
|
||||
"error": "Conflict",
|
||||
"detail": (
|
||||
f"Claim {body.claim_id} is in terminal state "
|
||||
f"{claim.state.value} and cannot be linked."
|
||||
),
|
||||
},
|
||||
)
|
||||
# Verify the ack exists (any kind).
|
||||
ack_table = {
|
||||
"999": db.Ack,
|
||||
"277ca": db.Two77caAck,
|
||||
"ta1": db.Ta1Ack,
|
||||
}[kind]
|
||||
if s.get(ack_table, ack_id) is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={
|
||||
"error": "Not found",
|
||||
"detail": f"{kind} ACK {ack_id} not found",
|
||||
},
|
||||
)
|
||||
# Idempotency: if a manual or auto link already exists, return it.
|
||||
existing = (
|
||||
s.query(db.ClaimAck)
|
||||
.filter(
|
||||
db.ClaimAck.ack_kind == kind,
|
||||
db.ClaimAck.ack_id == ack_id,
|
||||
db.ClaimAck.claim_id == body.claim_id,
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if existing is not None:
|
||||
return {
|
||||
"link": to_ui_claim_ack(existing),
|
||||
"created": False,
|
||||
}
|
||||
|
||||
# Insert via the store so the publish-from-store contract fires.
|
||||
row = store.add_claim_ack(
|
||||
claim_id=body.claim_id,
|
||||
batch_id=None,
|
||||
ack_id=ack_id,
|
||||
ack_kind=kind,
|
||||
ak2_index=body.ak2_index,
|
||||
set_control_number=body.set_control_number,
|
||||
set_accept_reject_code=body.set_accept_reject_code,
|
||||
linked_by="manual",
|
||||
event_bus=request.app.state.event_bus,
|
||||
)
|
||||
return {"link": to_ui_claim_ack(row), "created": True}
|
||||
|
||||
|
||||
@router.delete("/api/acks/{kind}/{ack_id}/match-claim/{claim_id}")
|
||||
def manual_unmatch_claim_endpoint(
|
||||
request: Request,
|
||||
kind: AckKind,
|
||||
ack_id: int,
|
||||
claim_id: str,
|
||||
) -> Any:
|
||||
"""Unlink (preserves ``Claim.state`` mutation; only removes the row).
|
||||
|
||||
404 when no link row exists for the dedup key. Publishes
|
||||
``claim_ack_dropped`` so live-tail subscribers remove the link.
|
||||
"""
|
||||
from cyclone import db as _db
|
||||
with _db.SessionLocal()() as s:
|
||||
link = (
|
||||
s.query(_db.ClaimAck)
|
||||
.filter(
|
||||
_db.ClaimAck.ack_kind == kind,
|
||||
_db.ClaimAck.ack_id == ack_id,
|
||||
_db.ClaimAck.claim_id == claim_id,
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if link is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={
|
||||
"error": "Not found",
|
||||
"detail": (
|
||||
f"No link for {kind} ack {ack_id} → "
|
||||
f"claim {claim_id}"
|
||||
),
|
||||
},
|
||||
)
|
||||
link_id = link.id
|
||||
|
||||
removed = store.remove_claim_ack(link_id, event_bus=request.app.state.event_bus)
|
||||
return {"removed": removed, "link_id": link_id}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Inbox ack-orphans lane (spec §D7)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@router.get("/api/inbox/ack-orphans")
|
||||
def list_ack_orphans_endpoint(
|
||||
kind: AckKind | None = Query(None, description="Filter by ack kind"),
|
||||
) -> Any:
|
||||
"""List acks with no resolvable Claim row of their own kind.
|
||||
|
||||
Used by the Inbox "Ack orphans" lane for the operator's manual
|
||||
reconciliation flow. Mirrors ``/api/inbox/remit-orphans``.
|
||||
"""
|
||||
if kind is not None:
|
||||
items = store.find_ack_orphans(kind)
|
||||
return {"total": len(items), "items": items}
|
||||
items_999 = store.find_ack_orphans("999")
|
||||
items_277ca = store.find_ack_orphans("277ca")
|
||||
items_ta1 = store.find_ack_orphans("ta1")
|
||||
all_items = items_999 + items_277ca + items_ta1
|
||||
return {"total": len(all_items), "items": all_items}
|
||||
|
||||
|
||||
__all__ = ["router"]
|
||||
@@ -0,0 +1,411 @@
|
||||
"""``/api/claims`` — list, live-tail, detail, X12 regenerate, line-reconciliation.
|
||||
|
||||
This is the largest single-resource router. Five endpoints:
|
||||
|
||||
* ``GET /api/claims`` — paginated list with filters
|
||||
(status / batch_id / provider_npi / payer / date range / sort).
|
||||
* ``GET /api/claims/stream`` — NDJSON snapshot + live tail via the
|
||||
``claim_written`` EventBus kind.
|
||||
* ``GET /api/claims/{claim_id}`` — drawer detail
|
||||
(header / state / service lines / diagnoses / parties / validation /
|
||||
raw segments / stateHistory / matchedRemittance).
|
||||
* ``GET /api/claims/{claim_id}/serialize-837`` — regenerate X12 for
|
||||
download (SP8).
|
||||
* ``GET /api/claims/{claim_id}/line-reconciliation`` — per-line
|
||||
reconciliation view for the ClaimDrawer tab (spec §5.1).
|
||||
|
||||
**Route ordering.** ``/stream`` is declared **before** ``/{claim_id}`` so
|
||||
FastAPI's first-match routing doesn't swallow the literal ``stream``
|
||||
segment as a claim id. Same order as the prior inline code.
|
||||
|
||||
The two projection helpers ``_claim_line_dict`` / ``_svc_to_dict`` are
|
||||
private to this router; both only used by the line-reconciliation
|
||||
endpoint.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from decimal import Decimal
|
||||
from typing import Any, AsyncIterator
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import JSONResponse, Response, StreamingResponse
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.api_helpers import (
|
||||
ndjson_line,
|
||||
ndjson_stream_list,
|
||||
tail_events,
|
||||
wants_ndjson,
|
||||
)
|
||||
from cyclone.db import Claim
|
||||
from cyclone.parsers.models import ClaimOutput
|
||||
from cyclone.parsers.serialize_837 import SerializeError as SerializeError837, serialize_837
|
||||
from cyclone.pubsub import EventBus
|
||||
from cyclone.store import store
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/api/claims")
|
||||
def list_claims_endpoint(
|
||||
request: Request,
|
||||
batch_id: str | None = Query(None),
|
||||
status: str | None = Query(None),
|
||||
provider_npi: str | None = Query(None),
|
||||
payer: str | None = Query(None),
|
||||
date_from: str | None = Query(None),
|
||||
date_to: str | None = Query(None),
|
||||
sort: str | None = Query(None),
|
||||
order: str = Query("desc"),
|
||||
limit: int = Query(100, ge=1, le=1000),
|
||||
offset: int = Query(0, ge=0),
|
||||
) -> Any:
|
||||
"""Paginated claims list with filters; supports NDJSON streaming.
|
||||
|
||||
``has_more`` is computed as ``total > offset + returned`` so paginating
|
||||
forward keeps returning true until the last page is reached.
|
||||
"""
|
||||
common = dict(
|
||||
batch_id=batch_id,
|
||||
status=status,
|
||||
provider_npi=provider_npi,
|
||||
payer=payer,
|
||||
date_from=date_from,
|
||||
date_to=date_to,
|
||||
)
|
||||
items = list(store.iter_claims(
|
||||
sort=sort, order=order, limit=limit, offset=offset, **common,
|
||||
))
|
||||
total = len(list(store.iter_claims(**common)))
|
||||
returned = len(items)
|
||||
has_more = total > offset + returned
|
||||
if wants_ndjson(request):
|
||||
return StreamingResponse(
|
||||
ndjson_stream_list(items, total, returned, has_more),
|
||||
media_type="application/x-ndjson",
|
||||
)
|
||||
return {
|
||||
"items": items,
|
||||
"total": total,
|
||||
"returned": returned,
|
||||
"has_more": has_more,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/api/claims/stream")
|
||||
async def claims_stream_endpoint(
|
||||
request: Request,
|
||||
status: str | None = Query(None),
|
||||
provider_npi: str | None = Query(None),
|
||||
payer: str | None = Query(None),
|
||||
date_from: str | None = Query(None),
|
||||
date_to: str | None = Query(None),
|
||||
sort: str | None = Query(None),
|
||||
order: str = Query("desc"),
|
||||
limit: int = Query(100, ge=1, le=1000),
|
||||
) -> StreamingResponse:
|
||||
"""Stream Claims as NDJSON: snapshot first, then live events.
|
||||
|
||||
Wire format:
|
||||
* ``{"type":"item","data":<claim>}`` per snapshot row, then per
|
||||
new ``claim_written`` event
|
||||
* ``{"type":"snapshot_end","data":{"count":N}}`` after the snapshot
|
||||
* ``{"type":"heartbeat","data":{"ts":<iso>}}`` every
|
||||
``CYCLONE_TAIL_HEARTBEAT_S`` seconds when idle
|
||||
|
||||
Query params mirror :func:`list_claims_endpoint` so a frontend can
|
||||
swap a one-shot fetch for a tail with no URL surgery.
|
||||
|
||||
NOTE: registered before ``/api/claims/{claim_id}`` so the literal
|
||||
``stream`` path segment doesn't get matched as a claim id.
|
||||
"""
|
||||
bus: EventBus = request.app.state.event_bus
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
# 1. Snapshot (eager — iter_claims returns a list already).
|
||||
rows = store.iter_claims(
|
||||
status=status, provider_npi=provider_npi, payer=payer,
|
||||
date_from=date_from, date_to=date_to,
|
||||
sort=sort or "-submission_date", order=order, limit=limit,
|
||||
)
|
||||
for row in rows:
|
||||
yield ndjson_line({"type": "item", "data": row})
|
||||
yield ndjson_line({"type": "snapshot_end", "data": {"count": len(rows)}})
|
||||
|
||||
# 2. Subscribe + heartbeats.
|
||||
async for chunk in tail_events(request, bus, ["claim_written"]):
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
|
||||
|
||||
@router.get("/api/claims/{claim_id}")
|
||||
def get_claim_detail_endpoint(claim_id: str) -> dict:
|
||||
"""Return one claim with full drawer context (SP4).
|
||||
|
||||
Body shape is produced by :meth:`CycloneStore.get_claim_detail`:
|
||||
header, state, service lines, diagnoses, parties, validation,
|
||||
raw segments, ``stateHistory`` (most-recent-first, capped at 50),
|
||||
and a populated ``matchedRemittance`` block when paired.
|
||||
|
||||
Returns 404 — never 500 — on a missing claim so the UI can
|
||||
distinguish "doesn't exist" from a transient fetch error.
|
||||
"""
|
||||
body = store.get_claim_detail(claim_id)
|
||||
if body is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={
|
||||
"error": "Not found",
|
||||
"detail": f"Claim {claim_id} not found",
|
||||
},
|
||||
)
|
||||
return body
|
||||
|
||||
|
||||
@router.get("/api/claims/{claim_id}/serialize-837")
|
||||
def serialize_claim_as_837(claim_id: str):
|
||||
"""Return the claim as a regenerated X12 837P file (SP8).
|
||||
|
||||
Loads the ClaimOutput from the persisted ``raw_json`` and runs the
|
||||
outbound serializer. Returns 404 if the claim doesn't exist, 422 if
|
||||
the stored payload has no parseable ClaimOutput (data integrity
|
||||
issue, not a transient failure).
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(Claim, claim_id)
|
||||
if row is None:
|
||||
return JSONResponse(
|
||||
{"error": "Not found", "detail": f"Claim {claim_id} not found"},
|
||||
status_code=404,
|
||||
)
|
||||
if not row.raw_json:
|
||||
return JSONResponse(
|
||||
{
|
||||
"error": "Unprocessable",
|
||||
"detail": f"Claim {claim_id} has no raw_json; cannot serialize",
|
||||
},
|
||||
status_code=422,
|
||||
)
|
||||
try:
|
||||
claim_obj = ClaimOutput.model_validate(row.raw_json)
|
||||
except Exception as exc:
|
||||
return JSONResponse(
|
||||
{
|
||||
"error": "Unprocessable",
|
||||
"detail": f"Claim {claim_id} raw_json is malformed: {exc}",
|
||||
},
|
||||
status_code=422,
|
||||
)
|
||||
|
||||
try:
|
||||
text = serialize_837(claim_obj)
|
||||
except SerializeError837 as exc:
|
||||
return JSONResponse(
|
||||
{"error": "Unprocessable", "detail": str(exc)},
|
||||
status_code=422,
|
||||
)
|
||||
|
||||
return Response(
|
||||
content=text,
|
||||
media_type="text/x12",
|
||||
headers={
|
||||
"Content-Disposition": f'attachment; filename="claim-{claim_id}.x12"'
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/api/claims/{claim_id}/line-reconciliation")
|
||||
def get_claim_line_reconciliation(claim_id: str) -> dict:
|
||||
"""Per-line reconciliation view for the ClaimDrawer tab.
|
||||
|
||||
Spec §5.1. Returns the 837 service lines and 835 SVC composites
|
||||
side-by-side, with per-line CAS adjustments and a summary block.
|
||||
|
||||
Architecture note: 837 service lines live in ``Claim.raw_json``
|
||||
(not a separate ORM table), so the 837-side rows are read from the
|
||||
JSON blob; the 835-side rows come from ``ServiceLinePayment`` ORM.
|
||||
``LineReconciliation.claim_service_line_number`` stores the 1-based
|
||||
line number to join them.
|
||||
"""
|
||||
from sqlalchemy import select
|
||||
|
||||
from cyclone.db import (
|
||||
CasAdjustment,
|
||||
LineReconciliation,
|
||||
ServiceLinePayment,
|
||||
)
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
claim = s.get(db.Claim, claim_id)
|
||||
if claim is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={"error": "Not found", "detail": f"Claim {claim_id} not found"},
|
||||
)
|
||||
|
||||
# 837 service lines: from raw_json.
|
||||
raw = claim.raw_json or {}
|
||||
claim_lines_raw = raw.get("service_lines") or []
|
||||
# Normalize to dicts for the response.
|
||||
claim_lines = [_claim_line_dict(d) for d in claim_lines_raw]
|
||||
|
||||
# 835 service payments: ORM rows from the matched remit.
|
||||
remits = list(
|
||||
s.execute(
|
||||
select(db.Remittance).where(db.Remittance.claim_id == claim_id)
|
||||
).scalars().all()
|
||||
)
|
||||
svc_payments: list[dict] = []
|
||||
svc_ids: list[int] = []
|
||||
if remits:
|
||||
svc_rows = list(
|
||||
s.execute(
|
||||
select(ServiceLinePayment).where(
|
||||
ServiceLinePayment.remittance_id.in_([r.id for r in remits])
|
||||
).order_by(ServiceLinePayment.line_number)
|
||||
).scalars().all()
|
||||
)
|
||||
for svc in svc_rows:
|
||||
d = _svc_to_dict(svc)
|
||||
svc_payments.append(d)
|
||||
svc_ids.append(svc.id)
|
||||
|
||||
# LineReconciliation rows.
|
||||
lrs = list(
|
||||
s.execute(
|
||||
select(LineReconciliation).where(LineReconciliation.claim_id == claim_id)
|
||||
).scalars().all()
|
||||
)
|
||||
# Index by claim_service_line_number and service_line_payment_id.
|
||||
lr_by_claim_num: dict[int, LineReconciliation] = {
|
||||
lr.claim_service_line_number: lr for lr in lrs if lr.claim_service_line_number is not None
|
||||
}
|
||||
lr_by_svc: dict[int, LineReconciliation] = {
|
||||
lr.service_line_payment_id: lr for lr in lrs if lr.service_line_payment_id is not None
|
||||
}
|
||||
|
||||
# CAS rows grouped by svc id.
|
||||
cas_by_svc: dict[int, list[CasAdjustment]] = {}
|
||||
if svc_ids:
|
||||
cas_rows = list(
|
||||
s.execute(
|
||||
select(CasAdjustment).where(CasAdjustment.service_line_payment_id.in_(svc_ids))
|
||||
).scalars().all()
|
||||
)
|
||||
for c in cas_rows:
|
||||
cas_by_svc.setdefault(c.service_line_payment_id, []).append(c)
|
||||
|
||||
# Build output lines array, preserving 837 order then 835-only.
|
||||
svc_by_id: dict[int, dict] = {d["id"]: d for d in svc_payments}
|
||||
lines_out: list[dict] = []
|
||||
billed_total = Decimal("0")
|
||||
paid_total = Decimal("0")
|
||||
adjustment_total = Decimal("0")
|
||||
matched_count = 0
|
||||
used_svc_ids: set[int] = set()
|
||||
|
||||
for cl in claim_lines:
|
||||
billed_total += Decimal(str(cl["charge"]))
|
||||
lr = lr_by_claim_num.get(cl["line_number"])
|
||||
if lr is None:
|
||||
lines_out.append({
|
||||
"claim_service_line": cl,
|
||||
"service_line_payment": None,
|
||||
"status": "unmatched_837_only",
|
||||
"adjustments": [],
|
||||
})
|
||||
continue
|
||||
svc_id = lr.service_line_payment_id
|
||||
svc = svc_by_id.get(svc_id) if svc_id else None
|
||||
if svc_id is not None:
|
||||
used_svc_ids.add(svc_id)
|
||||
cas_list = cas_by_svc.get(svc_id, []) if svc_id is not None else []
|
||||
cas_total = sum((Decimal(str(c.amount)) for c in cas_list), Decimal("0"))
|
||||
if svc:
|
||||
paid_total += Decimal(str(svc["payment"]))
|
||||
adjustment_total += cas_total
|
||||
if lr.status == "matched":
|
||||
matched_count += 1
|
||||
lines_out.append({
|
||||
"claim_service_line": cl,
|
||||
"service_line_payment": svc,
|
||||
"status": lr.status,
|
||||
"adjustments": [
|
||||
{"group_code": c.group_code, "reason_code": c.reason_code,
|
||||
"amount": str(Decimal(str(c.amount)))}
|
||||
for c in cas_list
|
||||
],
|
||||
})
|
||||
|
||||
# 835-only lines (no claim match).
|
||||
for lr in lrs:
|
||||
if lr.claim_service_line_number is not None:
|
||||
continue
|
||||
svc_id = lr.service_line_payment_id
|
||||
if svc_id is None:
|
||||
continue
|
||||
if svc_id in used_svc_ids:
|
||||
continue
|
||||
svc = svc_by_id.get(svc_id)
|
||||
cas_list = cas_by_svc.get(svc_id, [])
|
||||
cas_total = sum((Decimal(str(c.amount)) for c in cas_list), Decimal("0"))
|
||||
if svc:
|
||||
paid_total += Decimal(str(svc["payment"]))
|
||||
adjustment_total += cas_total
|
||||
lines_out.append({
|
||||
"claim_service_line": None,
|
||||
"service_line_payment": svc,
|
||||
"status": lr.status,
|
||||
"adjustments": [
|
||||
{"group_code": c.group_code, "reason_code": c.reason_code,
|
||||
"amount": str(Decimal(str(c.amount)))}
|
||||
for c in cas_list
|
||||
],
|
||||
})
|
||||
|
||||
return {
|
||||
"claim_id": claim_id,
|
||||
"summary": {
|
||||
"billed_total": str(billed_total),
|
||||
"paid_total": str(paid_total),
|
||||
"adjustment_total": str(adjustment_total),
|
||||
"matched_lines": matched_count,
|
||||
"total_lines": len(claim_lines),
|
||||
},
|
||||
"lines": lines_out,
|
||||
}
|
||||
|
||||
|
||||
def _claim_line_dict(d: dict) -> dict:
|
||||
"""Project an 837 service-line dict from ``Claim.raw_json`` to wire shape."""
|
||||
proc = d.get("procedure") or {}
|
||||
charge = d.get("charge")
|
||||
units = d.get("units")
|
||||
return {
|
||||
"line_number": d.get("line_number"),
|
||||
"procedure_qualifier": proc.get("qualifier", "HC"),
|
||||
"procedure_code": proc.get("code", ""),
|
||||
"modifiers": proc.get("modifiers") or [],
|
||||
"charge": str(Decimal(str(charge))) if charge is not None else "0",
|
||||
"units": str(Decimal(str(units))) if units is not None else None,
|
||||
"unit_type": d.get("unit_type"),
|
||||
"service_date": d.get("service_date"),
|
||||
}
|
||||
|
||||
|
||||
def _svc_to_dict(svc) -> dict:
|
||||
"""Project an ORM ``ServiceLinePayment`` to wire shape."""
|
||||
import json as _json
|
||||
return {
|
||||
"id": svc.id,
|
||||
"line_number": svc.line_number,
|
||||
"procedure_qualifier": svc.procedure_qualifier,
|
||||
"procedure_code": svc.procedure_code,
|
||||
"modifiers": _json.loads(svc.modifiers_json or "[]"),
|
||||
"charge": str(Decimal(str(svc.charge))),
|
||||
"payment": str(Decimal(str(svc.payment))),
|
||||
"units": str(Decimal(str(svc.units))) if svc.units is not None else None,
|
||||
"unit_type": svc.unit_type,
|
||||
"service_date": svc.service_date.isoformat() if svc.service_date else None,
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
"""``/api/clearhouse`` — SP9 single-payer SFTP submit surface.
|
||||
|
||||
Two endpoints:
|
||||
- ``GET /api/clearhouse`` — return the singleton clearhouse config
|
||||
(dzinesco's identity, SFTP block, filename block). 404 when the
|
||||
config hasn't been seeded yet.
|
||||
- ``POST /api/clearhouse/submit`` — submit a batch of claims to the
|
||||
clearhouse (SFTP). SP9 ships a stub that copies to ``staging_dir``
|
||||
instead of opening a real SFTP connection; SP13 wires the real
|
||||
paramiko-backed client behind the same ``make_client()`` factory.
|
||||
|
||||
For each claim the endpoint serializes a fresh 837 via
|
||||
:func:`_serialize_claim_for_submit`, builds an HCPF-compliant outbound
|
||||
filename, and records an audit-log row on success. Failures are
|
||||
captured per-claim so the operator sees exactly which claim_ids
|
||||
didn't go through.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.audit_log import AuditEvent, append_event
|
||||
from cyclone.parsers.parse_837 import parse
|
||||
from cyclone.parsers.serialize_837 import serialize_837
|
||||
from cyclone.store import store
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/api/clearhouse")
|
||||
def get_clearhouse():
|
||||
"""Return the singleton clearhouse config (dzinesco's identity, SFTP block, filename block)."""
|
||||
ch = store.get_clearhouse()
|
||||
if ch is None:
|
||||
raise HTTPException(status_code=404, detail="clearhouse not seeded")
|
||||
return json.loads(ch.model_dump_json())
|
||||
|
||||
|
||||
@router.post("/api/clearhouse/submit")
|
||||
def submit_to_clearhouse(body: dict):
|
||||
"""Submit a batch of claims to the clearhouse (SFTP). SP9: stub.
|
||||
|
||||
Body: ``{"claim_ids": [...], "payer_id": "CO_TXIX"}``
|
||||
|
||||
Stub behavior: serializes each claim via the SP7 serializer, builds
|
||||
an HCPF-compliant outbound filename, and copies the result to
|
||||
``{staging_dir}/{outbound_path}/{filename}`` instead of opening a
|
||||
real SFTP connection. Returns a receipt per claim.
|
||||
"""
|
||||
from cyclone.clearhouse import make_client
|
||||
from cyclone.edi.filenames import build_outbound_filename
|
||||
|
||||
claim_ids = body.get("claim_ids", [])
|
||||
payer_id = body.get("payer_id")
|
||||
if not claim_ids:
|
||||
raise HTTPException(status_code=400, detail="claim_ids required")
|
||||
if not payer_id:
|
||||
raise HTTPException(status_code=400, detail="payer_id required")
|
||||
|
||||
ch = store.get_clearhouse()
|
||||
if ch is None:
|
||||
raise HTTPException(status_code=500, detail="clearhouse not seeded")
|
||||
|
||||
client = make_client(ch.sftp_block)
|
||||
results = []
|
||||
for cid in claim_ids:
|
||||
try:
|
||||
x12_text = _serialize_claim_for_submit(cid)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
results.append({"claim_id": cid, "ok": False, "error": str(exc)})
|
||||
continue
|
||||
filename = build_outbound_filename(ch.tpid, "837P")
|
||||
remote = f"{ch.sftp_block.paths['outbound']}/{filename}"
|
||||
staging_path = client.write_file(remote, x12_text.encode("utf-8"))
|
||||
results.append({
|
||||
"claim_id": cid,
|
||||
"ok": True,
|
||||
"filename": filename,
|
||||
"staging_path": str(staging_path),
|
||||
"remote_path": remote,
|
||||
})
|
||||
# SP11: audit trail for each successful clearhouse submission.
|
||||
with db.SessionLocal()() as audit_s:
|
||||
append_event(audit_s, AuditEvent(
|
||||
event_type="clearhouse.submitted",
|
||||
entity_type="claim",
|
||||
entity_id=cid,
|
||||
payload={
|
||||
"filename": filename,
|
||||
"remote_path": remote,
|
||||
"tpid": ch.tpid,
|
||||
"stub": ch.sftp_block.stub,
|
||||
},
|
||||
actor="clearhouse-submit",
|
||||
))
|
||||
audit_s.commit()
|
||||
return {"ok": True, "submitted": results, "stub": ch.sftp_block.stub}
|
||||
|
||||
|
||||
def _serialize_claim_for_submit(claim_id: str) -> str:
|
||||
"""Serialize a claim to X12 for SFTP submission."""
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(db.Claim, claim_id)
|
||||
if row is None:
|
||||
raise ValueError(f"claim {claim_id!r} not found")
|
||||
raw = row.raw_json or {}
|
||||
return _serialize_claim_from_raw(row, raw)
|
||||
|
||||
|
||||
def _serialize_claim_from_raw(claim_row, raw: dict) -> str:
|
||||
"""Best-effort serializer that uses the stored raw_json to emit a fresh 837.
|
||||
|
||||
For SP9 this delegates to the existing serialize_837 helper if the
|
||||
claim has a complete raw_segments array. Otherwise it raises.
|
||||
"""
|
||||
# SP9 stub: if the claim has a `raw_json` with `x12_text`, use that.
|
||||
if isinstance(raw, dict) and raw.get("x12_text"):
|
||||
result = parse(raw["x12_text"])
|
||||
if result.claims:
|
||||
return serialize_837(result.claims[0])
|
||||
raise RuntimeError(
|
||||
f"claim {claim_row.id!r} cannot be re-serialized: no stored x12_text"
|
||||
)
|
||||
@@ -0,0 +1,56 @@
|
||||
"""``/api/config/*`` — configured-provider / configured-payer lookups.
|
||||
|
||||
The ``config`` prefix is the operator-facing read-only window into the
|
||||
``config/payers.yaml`` static config and the seeded providers table.
|
||||
Used by the frontend Settings page to show what's available.
|
||||
|
||||
The payers endpoint at ``/api/config/payers/{payer_id}/configs`` is
|
||||
slightly different: it returns both the YAML-loaded config and any
|
||||
DB-overridden config (so a runtime override wins over the static
|
||||
fallback). 404 / 200 / array shape match the rest of the API.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
|
||||
from cyclone import payers as payer_loader
|
||||
from cyclone.store import store
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/api/config/providers")
|
||||
def list_configured_providers(is_active: bool | None = Query(default=True)):
|
||||
"""List the configured provider rows (3 NPIs for SP9)."""
|
||||
return [json.loads(p.model_dump_json()) for p in store.list_providers(is_active=is_active)]
|
||||
|
||||
|
||||
@router.get("/api/config/providers/{npi}")
|
||||
def get_configured_provider(npi: str):
|
||||
p = store.get_provider(npi)
|
||||
if p is None:
|
||||
raise HTTPException(status_code=404, detail=f"provider {npi!r} not found")
|
||||
return json.loads(p.model_dump_json())
|
||||
|
||||
|
||||
@router.get("/api/config/payers")
|
||||
def list_configured_payers(is_active: bool | None = Query(default=True)):
|
||||
return [json.loads(p.model_dump_json()) for p in store.list_payers(is_active=is_active)]
|
||||
|
||||
|
||||
@router.get("/api/config/payers/{payer_id}/configs")
|
||||
def list_payer_configs(payer_id: str):
|
||||
"""List all (transaction_type, config_json) blocks for a payer."""
|
||||
configs = [
|
||||
{"transaction_type": tx, "config_json": block, "source": "yaml"}
|
||||
for (pid, tx), block in payer_loader.all_configs().items()
|
||||
if pid == payer_id
|
||||
]
|
||||
# Also check the DB for runtime-overridden configs
|
||||
for tx in ("837P", "835", "277CA", "999", "TA1"):
|
||||
live = store.get_payer_config(payer_id, tx)
|
||||
if live is not None:
|
||||
configs.append({"transaction_type": tx, "config_json": live, "source": "db"})
|
||||
return configs
|
||||
@@ -1,40 +1,17 @@
|
||||
"""``GET /api/health`` — liveness + readiness probe.
|
||||
"""``GET /api/health`` — liveness probe.
|
||||
|
||||
SP19 expanded the shallow ``{"status": "ok", "version": ...}`` probe
|
||||
into a snapshot of every subsystem:
|
||||
|
||||
* **db** — can we open a session and run ``SELECT 1``?
|
||||
* **scheduler** — is the MFT polling loop running? same for the
|
||||
backup scheduler.
|
||||
* **pubsub** — current subscriber counts per event kind.
|
||||
* **batch** — most recent batch id + timestamp.
|
||||
|
||||
Returns ``status="ok"`` only when every subsystem is healthy.
|
||||
``status="degraded"`` if any subsystem is unhappy but the API
|
||||
itself is responsive. Per-subsystem errors are surfaced in the
|
||||
respective dict so an operator doesn't have to guess.
|
||||
Returns the package version so an operator can confirm which build is
|
||||
serving requests without poking the filesystem.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi import APIRouter
|
||||
|
||||
from cyclone import __version__
|
||||
from cyclone.security import get_health_snapshot
|
||||
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/api/health")
|
||||
def health(request: Request) -> dict:
|
||||
snap = get_health_snapshot()
|
||||
# Fill in live pubsub subscriber counts using the per-request app
|
||||
# state (the snapshot builder doesn't have request context).
|
||||
bus = getattr(request.app.state, "event_bus", None)
|
||||
if bus is not None and hasattr(bus, "stats"):
|
||||
snap.pubsub = bus.stats()
|
||||
elif bus is not None:
|
||||
snap.pubsub = {"note": "EventBus.stats() not available"}
|
||||
else:
|
||||
snap.pubsub = {"note": "EventBus not attached (running outside lifespan?)"}
|
||||
return snap.to_dict()
|
||||
def health() -> dict[str, str]:
|
||||
return {"status": "ok", "version": __version__}
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
"""``GET /api/providers`` — distinct providers observed across claims.
|
||||
|
||||
The list is built by ``store.distinct_providers()`` which scans claim
|
||||
rows and returns one entry per unique ``billing_provider_npi``. The
|
||||
``npi`` and ``state`` query params filter client-side. ``limit`` /
|
||||
``offset`` paginate. Accepts ``application/x-ndjson`` like the other
|
||||
list endpoints — see ``api_helpers.ndjson_stream_list``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Query, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from cyclone.api_helpers import ndjson_stream_list, wants_ndjson
|
||||
from cyclone.store import store
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/api/providers")
|
||||
def list_providers(
|
||||
request: Request,
|
||||
npi: str | None = Query(None),
|
||||
state: str | None = Query(None),
|
||||
limit: int = Query(100, ge=1, le=1000),
|
||||
offset: int = Query(0, ge=0),
|
||||
) -> Any:
|
||||
items = store.distinct_providers()
|
||||
if npi is not None:
|
||||
items = [p for p in items if p["npi"] == npi]
|
||||
if state is not None:
|
||||
items = [p for p in items if p.get("state") == state]
|
||||
paged = items[offset:offset + limit]
|
||||
total = len(items)
|
||||
returned = len(paged)
|
||||
has_more = total > offset + returned
|
||||
if wants_ndjson(request):
|
||||
return StreamingResponse(
|
||||
ndjson_stream_list(paged, total, returned, has_more),
|
||||
media_type="application/x-ndjson",
|
||||
)
|
||||
return {
|
||||
"items": paged,
|
||||
"total": total,
|
||||
"returned": returned,
|
||||
"has_more": has_more,
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
"""``/api/ta1-acks`` — list, detail, and live-tail stream for TA1 envelopes.
|
||||
"""``/api/ta1-acks`` — list & detail endpoints for persisted TA1 envelopes.
|
||||
|
||||
TA1 is the interchange-control ACK (ISA/IEA acknowledgement). It's a
|
||||
single segment, no functional group, no transaction set. Cyclone
|
||||
@@ -7,29 +7,34 @@ row can sit alongside the 999 / 277CA ack rows without special-casing.
|
||||
|
||||
The detail endpoint also reconstructs the TA1 segment string
|
||||
(``TA1*...~``) so the operator can copy it into a downstream tool.
|
||||
|
||||
SP25: ``/api/ta1-acks/stream`` joins the live-tail triplet — the
|
||||
Acks page mounts ``useTailStream("ta1_acks")`` so the TA1 envelope
|
||||
ack section sees new rows the moment they land.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, AsyncIterator
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
|
||||
from cyclone.api_helpers import ndjson_line, tail_events
|
||||
from cyclone import db
|
||||
from cyclone.pubsub import EventBus
|
||||
from cyclone.store import store, to_ui_ta1_ack
|
||||
from cyclone.store import store
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
# SP25: ``_ta1_to_ui`` moved to ``cyclone.store.ui.to_ui_ta1_ack`` so
|
||||
# the live-tail event payload (``ta1_ack_received``) matches the list
|
||||
# endpoint shape byte-for-byte.
|
||||
def _ta1_to_ui(row: db.Ta1Ack) -> dict:
|
||||
"""Render a Ta1Ack row for the UI (list endpoint shape)."""
|
||||
return {
|
||||
"id": row.id,
|
||||
"control_number": row.control_number,
|
||||
"ack_code": row.ack_code,
|
||||
"note_code": row.note_code,
|
||||
"interchange_date": row.interchange_date.isoformat()
|
||||
if row.interchange_date else None,
|
||||
"interchange_time": row.interchange_time,
|
||||
"sender_id": row.sender_id,
|
||||
"receiver_id": row.receiver_id,
|
||||
"source_batch_id": row.source_batch_id,
|
||||
"parsed_at": row.parsed_at.isoformat() if row.parsed_at else None,
|
||||
}
|
||||
|
||||
|
||||
def _serialize_ta1_from_row(row: db.Ta1Ack) -> str:
|
||||
@@ -52,78 +57,20 @@ def list_ta1_acks_endpoint(
|
||||
count regardless of the ``limit`` cap.
|
||||
"""
|
||||
rows = store.list_ta1_acks()
|
||||
items = [to_ui_ta1_ack(r) for r in rows[:limit]]
|
||||
# SP28: batch-fetch linked_claim_ids per TA1 row (TA1 envelope
|
||||
# links always carry claim_id IS NULL — populate the field for
|
||||
# symmetry so the Acks page badge render path is uniform).
|
||||
ack_ids = [r.id for r in rows]
|
||||
linked_map = _find_linked_claim_ids_for_acks_helper(ack_ids, kind="ta1")
|
||||
for item, aid in zip(items, ack_ids[:limit]):
|
||||
item["linked_claim_ids"] = linked_map.get(aid, [])
|
||||
items = [_ta1_to_ui(r) for r in rows[:limit]]
|
||||
return {
|
||||
"total": len(rows),
|
||||
"items": items,
|
||||
}
|
||||
|
||||
|
||||
def _find_linked_claim_ids_for_acks_helper(
|
||||
ack_ids: list[int], *, kind: str
|
||||
) -> dict[int, list[str]]:
|
||||
"""Batch-fetch {ack_id: [claim_id, …]} for the listed ack rows.
|
||||
|
||||
Local helper so we don't import from ``acks.py`` and create a
|
||||
circular import. See the equivalent ``_find_linked_claim_ids_for_acks``
|
||||
in ``acks.py`` for the contract.
|
||||
"""
|
||||
out: dict[int, list[str]] = {aid: [] for aid in ack_ids}
|
||||
if not ack_ids:
|
||||
return out
|
||||
with db.SessionLocal()() as s:
|
||||
rows = (
|
||||
s.query(db.ClaimAck.ack_id, db.ClaimAck.claim_id)
|
||||
.filter(
|
||||
db.ClaimAck.ack_kind == kind,
|
||||
db.ClaimAck.ack_id.in_(ack_ids),
|
||||
db.ClaimAck.claim_id.isnot(None),
|
||||
)
|
||||
.all()
|
||||
)
|
||||
for ack_id, claim_id in rows:
|
||||
out[ack_id].append(claim_id)
|
||||
return out
|
||||
|
||||
|
||||
@router.get("/api/ta1-acks/stream")
|
||||
async def ta1_acks_stream(
|
||||
request: Request,
|
||||
limit: int = Query(100, ge=1, le=1000),
|
||||
) -> StreamingResponse:
|
||||
"""Stream TA1 envelope acks as NDJSON.
|
||||
|
||||
Subscribes to ``ta1_ack_received`` and emits the same wire format
|
||||
as the other live-tail endpoints. Registered BEFORE
|
||||
``/api/ta1-acks/{ack_id}`` so ``stream`` isn't matched as an id.
|
||||
"""
|
||||
bus: EventBus = request.app.state.event_bus
|
||||
|
||||
async def gen() -> AsyncIterator[bytes]:
|
||||
rows = store.list_ta1_acks()[:limit]
|
||||
for row in rows:
|
||||
yield ndjson_line({"type": "item", "data": to_ui_ta1_ack(row)})
|
||||
yield ndjson_line({"type": "snapshot_end", "data": {"count": len(rows)}})
|
||||
async for chunk in tail_events(request, bus, ["ta1_ack_received"]):
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
||||
|
||||
|
||||
@router.get("/api/ta1-acks/{ack_id}")
|
||||
def get_ta1_ack_endpoint(ack_id: int) -> dict:
|
||||
"""Return one persisted TA1 ACK row with its parsed detail."""
|
||||
row = store.get_ta1_ack(ack_id)
|
||||
if row is None:
|
||||
raise HTTPException(status_code=404, detail=f"TA1 ACK {ack_id} not found")
|
||||
body = to_ui_ta1_ack(row)
|
||||
body = _ta1_to_ui(row)
|
||||
body["raw_ta1_text"] = _serialize_ta1_from_row(row)
|
||||
body["raw_json"] = row.raw_json
|
||||
return body
|
||||
|
||||
@@ -103,12 +103,6 @@ class AuditEvent:
|
||||
computed hash. Payload must be JSON-serializable; the audit_log
|
||||
module handles the encoding so callers don't need to think about
|
||||
canonical form.
|
||||
|
||||
``user_id`` is the authenticated actor for this event, when known
|
||||
(e.g. a parse-999 call made by user 7). It's stored on the row but
|
||||
is NOT part of the hash chain — the chain hashes only the fields
|
||||
that existed pre-SP-auth so verify_chain stays compatible with
|
||||
pre-auth rows.
|
||||
"""
|
||||
|
||||
event_type: str
|
||||
@@ -117,7 +111,6 @@ class AuditEvent:
|
||||
payload: dict[str, Any] = field(default_factory=dict)
|
||||
actor: str = "system"
|
||||
created_at: datetime | None = None
|
||||
user_id: int | None = None
|
||||
|
||||
|
||||
def append_event(
|
||||
@@ -162,7 +155,6 @@ def append_event(
|
||||
created_at=created_at,
|
||||
prev_hash=prev_hash,
|
||||
hash=GENESIS_PREV_HASH, # placeholder; updated below
|
||||
user_id=event.user_id,
|
||||
)
|
||||
session.add(row)
|
||||
session.flush() # populate row.id
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
"""Auth module — users, sessions, permissions, routes, admin, rate_limit."""
|
||||
@@ -1,95 +0,0 @@
|
||||
"""Admin-only user management: GET/POST/PATCH/DELETE /api/admin/users."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, status
|
||||
|
||||
from cyclone.auth import users
|
||||
from cyclone.auth.deps import get_current_user
|
||||
from cyclone.auth.permissions import Role
|
||||
from cyclone.db import SessionLocal, User
|
||||
|
||||
router = APIRouter(prefix="/api/admin/users", tags=["admin"])
|
||||
|
||||
|
||||
def _require_admin(user: dict = Depends(get_current_user)) -> dict:
|
||||
if user.get("role") != Role.ADMIN.value:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="forbidden",
|
||||
)
|
||||
return user
|
||||
|
||||
|
||||
def _validate_role(role: str) -> None:
|
||||
valid = {Role.ADMIN.value, Role.USER.value, Role.VIEWER.value}
|
||||
if role not in valid:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
|
||||
detail=f"role must be one of {sorted(valid)}",
|
||||
)
|
||||
|
||||
|
||||
@router.get("")
|
||||
def list_users(_admin=Depends(_require_admin)):
|
||||
with SessionLocal()() as db:
|
||||
all_users = db.query(User).all()
|
||||
return [users.to_public(u) for u in all_users]
|
||||
|
||||
|
||||
@router.post("", status_code=status.HTTP_201_CREATED)
|
||||
def create_user(body: dict, _admin=Depends(_require_admin)):
|
||||
username = (body.get("username") or "").strip()
|
||||
password = body.get("password") or ""
|
||||
role = body.get("role") or ""
|
||||
if not username or len(username) < 3:
|
||||
raise HTTPException(status_code=422, detail="username must be at least 3 chars")
|
||||
if len(password) < 12:
|
||||
raise HTTPException(status_code=422, detail="password must be at least 12 chars")
|
||||
_validate_role(role)
|
||||
with SessionLocal()() as db:
|
||||
if users.get_by_username(db, username) is not None:
|
||||
raise HTTPException(status_code=409, detail="username already exists")
|
||||
u = users.create(db, username=username, password=password, role=role)
|
||||
return users.to_public(u)
|
||||
|
||||
|
||||
@router.patch("/{user_id}")
|
||||
def patch_user(user_id: int, body: dict, admin=Depends(_require_admin)):
|
||||
me = admin
|
||||
if me.get("id") == user_id and body.get("role") and body["role"] != Role.ADMIN.value:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail="cannot_demote_self",
|
||||
)
|
||||
|
||||
with SessionLocal()() as db:
|
||||
if body.get("role") is not None:
|
||||
_validate_role(body["role"])
|
||||
users.update_role(db, user_id, body["role"])
|
||||
if body.get("password") is not None:
|
||||
if len(body["password"]) < 12:
|
||||
raise HTTPException(status_code=422, detail="password must be at least 12 chars")
|
||||
users.update_password(db, user_id, body["password"])
|
||||
if body.get("disabled") is True:
|
||||
users.disable(db, user_id)
|
||||
u = users.get(db, user_id)
|
||||
if u is None:
|
||||
raise HTTPException(status_code=404, detail="user not found")
|
||||
return users.to_public(u)
|
||||
|
||||
|
||||
@router.delete("/{user_id}", status_code=status.HTTP_204_NO_CONTENT)
|
||||
def delete_user(user_id: int, admin=Depends(_require_admin)):
|
||||
me = admin
|
||||
if me.get("id") == user_id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail="cannot_delete_self",
|
||||
)
|
||||
with SessionLocal()() as db:
|
||||
u = users.get(db, user_id)
|
||||
if u is None:
|
||||
raise HTTPException(status_code=404, detail="user not found")
|
||||
users.disable(db, user_id)
|
||||
return None
|
||||
@@ -1,105 +0,0 @@
|
||||
"""First-admin bootstrap: create the initial admin from env vars if no users exist.
|
||||
|
||||
Called from ``python -m cyclone`` before either ``cli.main()`` or
|
||||
``uvicorn`` so users exist by the time the API serves requests.
|
||||
|
||||
Precedence:
|
||||
|
||||
1. ``CYCLONE_AUTH_DISABLED=1`` — dev escape hatch. Flip the
|
||||
``cyclone.auth.deps.AUTH_DISABLED`` flag so the API returns a
|
||||
synthetic admin user without checking credentials. Never raises.
|
||||
2. Users table non-empty — no-op.
|
||||
3. ``CYCLONE_ADMIN_USERNAME`` + ``CYCLONE_ADMIN_PASSWORD`` env vars set
|
||||
(password >= 12 chars) — create the admin and print confirmation.
|
||||
Each env var can also be replaced by a ``*_FILE`` companion
|
||||
(``CYCLONE_ADMIN_USERNAME_FILE`` / ``CYCLONE_ADMIN_PASSWORD_FILE``)
|
||||
that points at a file on disk — the standard Docker-secret pattern,
|
||||
used in production to avoid embedding secrets in ``docker-compose.yml``.
|
||||
``_FILE`` takes precedence when set.
|
||||
4. Otherwise — raise ``RuntimeError`` with a remediation hint that
|
||||
points operators at ``python -m cyclone users create``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from cyclone.auth import users
|
||||
from cyclone.auth.deps import AUTH_DISABLED
|
||||
from cyclone.auth.permissions import Role
|
||||
from cyclone.db import SessionLocal, User
|
||||
|
||||
|
||||
def _read_secret(env_var: str, file_var: str) -> str | None:
|
||||
"""Read a secret from a ``*_FILE`` env var (Docker-secret pattern) first,
|
||||
falling back to the plain env var. Returns None if neither is set.
|
||||
"""
|
||||
file_path = os.environ.get(file_var)
|
||||
if file_path:
|
||||
try:
|
||||
return Path(file_path).read_text().strip()
|
||||
except OSError as exc:
|
||||
raise RuntimeError(
|
||||
f"failed to read {file_var}={file_path}: {exc}"
|
||||
) from exc
|
||||
return os.environ.get(env_var)
|
||||
|
||||
|
||||
def run() -> None:
|
||||
"""Bootstrap the first admin user, or no-op.
|
||||
|
||||
See module docstring for behavior. Idempotent: safe to call on
|
||||
every startup — it short-circuits as soon as the users table is
|
||||
non-empty.
|
||||
"""
|
||||
if os.environ.get("CYCLONE_AUTH_DISABLED") == "1":
|
||||
# Dev escape hatch — skip bootstrap entirely and tell the API
|
||||
# to also short-circuit auth checks.
|
||||
import cyclone.auth.deps as _deps
|
||||
|
||||
_deps.AUTH_DISABLED = True
|
||||
return
|
||||
|
||||
username = _read_secret(
|
||||
"CYCLONE_ADMIN_USERNAME", "CYCLONE_ADMIN_USERNAME_FILE"
|
||||
)
|
||||
password = _read_secret(
|
||||
"CYCLONE_ADMIN_PASSWORD", "CYCLONE_ADMIN_PASSWORD_FILE"
|
||||
)
|
||||
|
||||
# First-boot fix: ``python -m cyclone`` calls bootstrap before any
|
||||
# subcommand or the FastAPI lifespan handler runs, so on a brand-new
|
||||
# DB ``SessionLocal()`` raises "init_db() has not been called".
|
||||
# Initialize here so ``serve``, ``users create``, and friends can
|
||||
# all reach the DB without the operator having to know about
|
||||
# migrations. Idempotent — no-op when the schema is already current.
|
||||
from cyclone import db as _db
|
||||
|
||||
_db.init_db()
|
||||
|
||||
with SessionLocal()() as db:
|
||||
existing = db.execute(select(User)).scalars().first()
|
||||
if existing is not None:
|
||||
return # users exist — nothing to bootstrap
|
||||
|
||||
if not username or not password:
|
||||
raise RuntimeError(
|
||||
"Cyclone has no users yet. Set CYCLONE_ADMIN_USERNAME and "
|
||||
"CYCLONE_ADMIN_PASSWORD env vars (min 12 chars), or run "
|
||||
"`python -m cyclone users create <username> --role admin`."
|
||||
)
|
||||
if len(password) < 12:
|
||||
raise RuntimeError(
|
||||
"CYCLONE_ADMIN_PASSWORD must be at least 12 characters."
|
||||
)
|
||||
|
||||
users.create(
|
||||
db,
|
||||
username=username,
|
||||
password=password,
|
||||
role=Role.ADMIN.value,
|
||||
)
|
||||
print(f"[cyclone] bootstrap admin user '{username}' created")
|
||||
@@ -1,183 +0,0 @@
|
||||
"""CLI subcommand: ``python -m cyclone users ...``.
|
||||
|
||||
Click-based to match the existing parse-837 / parse-835 convention in
|
||||
``cyclone.cli``. Provides operator-side user management without going
|
||||
through the admin HTTP API.
|
||||
|
||||
Subcommands
|
||||
-----------
|
||||
|
||||
- ``users create USERNAME --role {admin,user,viewer} [--password PW]``
|
||||
Create a user. If ``--password`` is omitted, prompts (with
|
||||
confirmation) on the controlling terminal.
|
||||
- ``users list`` — tab-separated ``id / username / role / state``.
|
||||
- ``users disable USERNAME`` — set ``disabled_at`` to now.
|
||||
- ``users reset-password USERNAME [--password PW]`` — replace the hash.
|
||||
- ``users set-role USERNAME --role {admin,user,viewer}`` — change role.
|
||||
|
||||
Exit codes
|
||||
----------
|
||||
|
||||
* 0 — success
|
||||
* 1 — validation error (unknown username, duplicate username)
|
||||
* 2 — usage error (missing arg, bad role, short password, unknown
|
||||
subcommand). Click itself uses 2 for usage errors so the conventional
|
||||
shell tools (``set -e``, etc.) recognize them.
|
||||
|
||||
Passwords shorter than 12 chars are rejected everywhere they appear.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import getpass
|
||||
import sys
|
||||
|
||||
import click
|
||||
|
||||
from cyclone.auth import users
|
||||
from cyclone.auth.permissions import Role
|
||||
from cyclone.db import SessionLocal
|
||||
|
||||
ROLE_CHOICES = [Role.ADMIN.value, Role.USER.value, Role.VIEWER.value]
|
||||
MIN_PASSWORD_LEN = 12
|
||||
|
||||
|
||||
def _prompt_password(label: str) -> str:
|
||||
pw = getpass.getpass(f"{label}: ")
|
||||
if not pw:
|
||||
click.echo("Password required.", err=True)
|
||||
sys.exit(2)
|
||||
return pw
|
||||
|
||||
|
||||
def _validate_password(pw: str) -> None:
|
||||
if len(pw) < MIN_PASSWORD_LEN:
|
||||
click.echo(
|
||||
f"Password must be at least {MIN_PASSWORD_LEN} characters.",
|
||||
err=True,
|
||||
)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Group
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
@click.group(name="users")
|
||||
def users_cli() -> None:
|
||||
"""Manage Cyclone users from the command line."""
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# create
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
@users_cli.command("create")
|
||||
@click.argument("username")
|
||||
@click.option(
|
||||
"--role",
|
||||
required=True,
|
||||
type=click.Choice(ROLE_CHOICES, case_sensitive=False),
|
||||
help="Role to grant the new user.",
|
||||
)
|
||||
@click.option(
|
||||
"--password",
|
||||
default=None,
|
||||
help=f"Password (min {MIN_PASSWORD_LEN} chars). Prompts if omitted.",
|
||||
)
|
||||
def create_user(username: str, role: str, password: str | None) -> None:
|
||||
"""Create a new user with the given USERNAME and ROLE."""
|
||||
pw = password or _prompt_password("Password")
|
||||
_validate_password(pw)
|
||||
|
||||
with SessionLocal()() as db:
|
||||
if users.get_by_username(db, username) is not None:
|
||||
click.echo(f"User '{username}' already exists.", err=True)
|
||||
sys.exit(1)
|
||||
u = users.create(db, username=username, password=pw, role=role)
|
||||
click.echo(f"Created user '{u.username}' with role '{u.role}'.")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# list
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
@users_cli.command("list")
|
||||
def list_users() -> None:
|
||||
"""List all users as id / username / role / state."""
|
||||
from cyclone.db import User
|
||||
|
||||
with SessionLocal()() as db:
|
||||
for u in db.query(User).order_by(User.id.asc()).all():
|
||||
state = "disabled" if u.disabled_at else "active"
|
||||
click.echo(f"{u.id}\t{u.username}\t{u.role}\t{state}")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# disable
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
@users_cli.command("disable")
|
||||
@click.argument("username")
|
||||
def disable_user(username: str) -> None:
|
||||
"""Disable USERNAME (sets disabled_at to now)."""
|
||||
with SessionLocal()() as db:
|
||||
u = users.get_by_username(db, username)
|
||||
if u is None:
|
||||
click.echo(f"No such user: {username}", err=True)
|
||||
sys.exit(1)
|
||||
users.disable(db, u.id)
|
||||
click.echo(f"Disabled '{username}'.")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# reset-password
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
@users_cli.command("reset-password")
|
||||
@click.argument("username")
|
||||
@click.option(
|
||||
"--password",
|
||||
default=None,
|
||||
help=f"New password (min {MIN_PASSWORD_LEN} chars). Prompts if omitted.",
|
||||
)
|
||||
def reset_password(username: str, password: str | None) -> None:
|
||||
"""Replace USERNAME's password."""
|
||||
pw = password or _prompt_password("New password")
|
||||
_validate_password(pw)
|
||||
with SessionLocal()() as db:
|
||||
u = users.get_by_username(db, username)
|
||||
if u is None:
|
||||
click.echo(f"No such user: {username}", err=True)
|
||||
sys.exit(1)
|
||||
users.update_password(db, u.id, pw)
|
||||
click.echo(f"Password reset for '{username}'.")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# set-role
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
@users_cli.command("set-role")
|
||||
@click.argument("username")
|
||||
@click.option(
|
||||
"--role",
|
||||
required=True,
|
||||
type=click.Choice(ROLE_CHOICES, case_sensitive=False),
|
||||
help="Role to grant.",
|
||||
)
|
||||
def set_role(username: str, role: str) -> None:
|
||||
"""Change USERNAME's role."""
|
||||
with SessionLocal()() as db:
|
||||
u = users.get_by_username(db, username)
|
||||
if u is None:
|
||||
click.echo(f"No such user: {username}", err=True)
|
||||
sys.exit(1)
|
||||
users.update_role(db, u.id, role)
|
||||
click.echo(f"Role for '{username}' set to '{role}'.")
|
||||
@@ -1,140 +0,0 @@
|
||||
"""FastAPI dependencies for auth."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import Depends, HTTPException, Request, status
|
||||
from sqlalchemy.orm import Session as DbSession
|
||||
|
||||
from cyclone.auth import sessions, users
|
||||
from cyclone.auth.permissions import Role, allowed_roles
|
||||
from cyclone.db import SessionLocal
|
||||
|
||||
|
||||
def _db():
|
||||
db = SessionLocal()()
|
||||
try:
|
||||
yield db
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
DbSessionDep = Annotated[DbSession, Depends(_db)]
|
||||
|
||||
|
||||
AUTH_DISABLED = False
|
||||
|
||||
|
||||
async def get_current_user(
|
||||
request: Request,
|
||||
db: DbSessionDep,
|
||||
) -> dict:
|
||||
"""Return the public User shape. Raises 401 if session is missing/expired.
|
||||
|
||||
When AUTH_DISABLED is True (dev escape hatch), returns a synthetic admin
|
||||
user without checking credentials.
|
||||
"""
|
||||
if AUTH_DISABLED:
|
||||
return {
|
||||
"id": 0,
|
||||
"username": "dev",
|
||||
"role": Role.ADMIN.value,
|
||||
"createdAt": None,
|
||||
"disabledAt": None,
|
||||
}
|
||||
|
||||
sid = request.cookies.get("cyclone_session")
|
||||
if not sid:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="session_expired",
|
||||
)
|
||||
sess = sessions.get_valid(db, sid)
|
||||
if sess is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="session_expired",
|
||||
)
|
||||
user = users.get(db, sess.user_id)
|
||||
if user is None or user.disabled_at is not None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="account_disabled",
|
||||
)
|
||||
# Sliding expiry: refresh both DB and cookie.
|
||||
sessions.touch(db, sid)
|
||||
request.state.user = user
|
||||
request.state.session_id = sid
|
||||
return users.to_public(user)
|
||||
|
||||
|
||||
def require_role(*allowed: Role):
|
||||
"""Dependency factory: gate the endpoint to specific roles.
|
||||
|
||||
Falls back to PERMISSIONS matrix lookup if no explicit roles given.
|
||||
"""
|
||||
async def _dep(
|
||||
request: Request,
|
||||
user: dict = Depends(get_current_user),
|
||||
) -> dict:
|
||||
user_role = user.get("role")
|
||||
if allowed:
|
||||
if user_role not in {r.value for r in allowed}:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="forbidden",
|
||||
)
|
||||
return user
|
||||
# Otherwise consult the matrix.
|
||||
method = request.method
|
||||
path = request.url.path
|
||||
roles = allowed_roles(method, path)
|
||||
if roles is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="forbidden",
|
||||
)
|
||||
if user_role not in {r.value for r in roles}:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="forbidden",
|
||||
)
|
||||
return user
|
||||
return _dep
|
||||
|
||||
|
||||
async def matrix_gate(
|
||||
request: Request,
|
||||
user: dict = Depends(get_current_user),
|
||||
) -> dict:
|
||||
"""App-wide gate: requires auth, then enforces the PERMISSIONS matrix.
|
||||
|
||||
Behavior:
|
||||
* AUTH_DISABLED short-circuits (synthetic admin, no role check).
|
||||
* No session cookie → 401 from get_current_user.
|
||||
* Endpoint not in the matrix → 403 (fail-closed).
|
||||
* User role not allowed for (method, path) → 403.
|
||||
* Empty allowed-roles set (e.g. /api/healthz) → public, no role check.
|
||||
|
||||
Used as ``dependencies=[Depends(matrix_gate)]`` on every authenticated
|
||||
route. Centralizing the gate here means the matrix is the source of
|
||||
truth — no need to wire per-route ``require_role(...)`` calls.
|
||||
"""
|
||||
if AUTH_DISABLED:
|
||||
return user
|
||||
method = request.method
|
||||
path = request.url.path
|
||||
roles = allowed_roles(method, path)
|
||||
if roles is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="forbidden",
|
||||
)
|
||||
user_role = user.get("role")
|
||||
if user_role not in {r.value for r in roles}:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="forbidden",
|
||||
)
|
||||
return user
|
||||
@@ -1,108 +0,0 @@
|
||||
"""Role enum + PERMISSIONS matrix."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class Role(str, Enum):
|
||||
ADMIN = "admin"
|
||||
USER = "user"
|
||||
VIEWER = "viewer"
|
||||
|
||||
|
||||
ALL_ROLES = {Role.ADMIN, Role.USER, Role.VIEWER}
|
||||
WRITE_ROLES = {Role.ADMIN, Role.USER}
|
||||
ADMIN_ONLY = {Role.ADMIN}
|
||||
|
||||
|
||||
# (method, path-prefix) → allowed roles.
|
||||
# Endpoints not in this matrix default to DENY (fail-closed).
|
||||
PERMISSIONS: dict[tuple[str, str], set[Role]] = {
|
||||
# Public paths.
|
||||
("GET", "/api/health"): set(),
|
||||
("POST", "/api/auth/login"): set(),
|
||||
|
||||
# Auth surface.
|
||||
("POST", "/api/auth/logout"): ALL_ROLES,
|
||||
("GET", "/api/auth/me"): ALL_ROLES,
|
||||
|
||||
# Admin-only user management.
|
||||
("GET", "/api/admin/users"): ADMIN_ONLY,
|
||||
("POST", "/api/admin/users"): ADMIN_ONLY,
|
||||
("PATCH", "/api/admin/users"): ADMIN_ONLY,
|
||||
("DELETE", "/api/admin/users"): ADMIN_ONLY,
|
||||
|
||||
# Read endpoints (all authenticated roles).
|
||||
("GET", "/api/claims"): ALL_ROLES,
|
||||
("GET", "/api/remittances"): ALL_ROLES,
|
||||
("GET", "/api/providers"): ALL_ROLES,
|
||||
("GET", "/api/batches"): ALL_ROLES,
|
||||
("GET", "/api/dashboard/summary"): ALL_ROLES,
|
||||
("GET", "/api/activity"): ALL_ROLES,
|
||||
("GET", "/api/inbox/lanes"): ALL_ROLES,
|
||||
("GET", "/api/inbox/export.csv"): ALL_ROLES,
|
||||
("GET", "/api/reconcile"): ALL_ROLES,
|
||||
("GET", "/api/reconciliation"): ALL_ROLES,
|
||||
("GET", "/api/acks"): ALL_ROLES,
|
||||
("GET", "/api/ta1-acks"): ALL_ROLES,
|
||||
("GET", "/api/277ca-acks"): ALL_ROLES,
|
||||
("GET", "/api/batch-diff"): ALL_ROLES,
|
||||
("GET", "/api/config"): ALL_ROLES,
|
||||
("GET", "/api/payers"): ALL_ROLES,
|
||||
("GET", "/api/audit-log"): ADMIN_ONLY,
|
||||
|
||||
# Clearhouse (SFTP creds + dzinesco identity) — admin only.
|
||||
("GET", "/api/clearhouse"): ADMIN_ONLY,
|
||||
("PATCH", "/api/clearhouse"): ADMIN_ONLY,
|
||||
("POST", "/api/clearhouse/submit"): ADMIN_ONLY,
|
||||
|
||||
# Admin ops (audit log, backup, scheduler, db rotate, reload-config).
|
||||
("GET", "/api/admin/audit-log"): ADMIN_ONLY,
|
||||
("GET", "/api/admin/audit-log/verify"): ADMIN_ONLY,
|
||||
("GET", "/api/admin/backup"): ADMIN_ONLY,
|
||||
("POST", "/api/admin/backup"): ADMIN_ONLY,
|
||||
("GET", "/api/admin/backup/scheduler"): ADMIN_ONLY,
|
||||
("POST", "/api/admin/backup/scheduler"): ADMIN_ONLY,
|
||||
("GET", "/api/admin/scheduler"): ADMIN_ONLY,
|
||||
("POST", "/api/admin/scheduler"): ADMIN_ONLY,
|
||||
("POST", "/api/admin/db/rotate-key"): ADMIN_ONLY,
|
||||
("POST", "/api/admin/reload-config"): ADMIN_ONLY,
|
||||
("GET", "/api/admin/validate-provider"): ADMIN_ONLY,
|
||||
|
||||
# Write endpoints (admin + user, no viewer).
|
||||
("POST", "/api/parse-837"): WRITE_ROLES,
|
||||
("POST", "/api/parse-835"): WRITE_ROLES,
|
||||
("POST", "/api/parse-999"): WRITE_ROLES,
|
||||
("POST", "/api/parse-ta1"): WRITE_ROLES,
|
||||
("POST", "/api/parse-277ca"): WRITE_ROLES,
|
||||
("POST", "/api/inbox"): WRITE_ROLES,
|
||||
("POST", "/api/inbox/candidates"): WRITE_ROLES,
|
||||
("POST", "/api/inbox/rejected"): WRITE_ROLES,
|
||||
("POST", "/api/inbox/payer-rejected"): WRITE_ROLES,
|
||||
("POST", "/api/reconcile"): WRITE_ROLES,
|
||||
("POST", "/api/reconciliation"): WRITE_ROLES,
|
||||
("POST", "/api/resubmit"): WRITE_ROLES,
|
||||
("POST", "/api/acks"): WRITE_ROLES,
|
||||
("POST", "/api/batches"): WRITE_ROLES, # /export-837 regenerates X12 from DB rows
|
||||
("POST", "/api/eligibility"): WRITE_ROLES,
|
||||
|
||||
# CSV export — read-only.
|
||||
("GET", "/api/export.csv"): ALL_ROLES,
|
||||
}
|
||||
|
||||
|
||||
def allowed_roles(method: str, path: str) -> set[Role] | None:
|
||||
"""Return the set of roles allowed to call (method, path), or None if denied.
|
||||
|
||||
Uses longest-prefix match on path; falls back to DENY (None) if no entry matches.
|
||||
"""
|
||||
candidates = [
|
||||
(len(prefix), roles)
|
||||
for (m, prefix), roles in PERMISSIONS.items()
|
||||
if m == method and (path == prefix or path.startswith(prefix.rstrip("/") + "/"))
|
||||
]
|
||||
if not candidates:
|
||||
return None
|
||||
candidates.sort(key=lambda x: -x[0])
|
||||
return candidates[0][1]
|
||||
@@ -1,34 +0,0 @@
|
||||
"""Per-username login rate limiter (in-memory, per-process)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from threading import Lock
|
||||
|
||||
WINDOW_SECONDS = 300
|
||||
MAX_FAILS = 5
|
||||
|
||||
_FAILS: dict[str, list[float]] = {}
|
||||
_LOCK = Lock()
|
||||
|
||||
|
||||
def check(username: str) -> int:
|
||||
"""Return retry-after seconds, or 0 if allowed."""
|
||||
now = time.monotonic()
|
||||
with _LOCK:
|
||||
fails = [t for t in _FAILS.get(username, []) if now - t < WINDOW_SECONDS]
|
||||
_FAILS[username] = fails
|
||||
if len(fails) >= MAX_FAILS:
|
||||
return int(WINDOW_SECONDS - (now - fails[0]))
|
||||
return 0
|
||||
|
||||
|
||||
def record_failure(username: str) -> None:
|
||||
now = time.monotonic()
|
||||
with _LOCK:
|
||||
_FAILS.setdefault(username, []).append(now)
|
||||
|
||||
|
||||
def reset(username: str) -> None:
|
||||
with _LOCK:
|
||||
_FAILS.pop(username, None)
|
||||
@@ -1,97 +0,0 @@
|
||||
"""/api/auth/login, /api/auth/logout, /api/auth/me."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
|
||||
|
||||
from cyclone.auth import rate_limit, sessions, users
|
||||
from cyclone.auth.deps import get_current_user
|
||||
from cyclone.db import SessionLocal
|
||||
|
||||
router = APIRouter(prefix="/api/auth", tags=["auth"])
|
||||
|
||||
COOKIE_NAME = "cyclone_session"
|
||||
COOKIE_MAX_AGE = 86400 # 24h
|
||||
|
||||
|
||||
def _is_https(request: Request) -> bool:
|
||||
if request.url.scheme == "https":
|
||||
return True
|
||||
return os.environ.get("CYCLONE_BEHIND_HTTPS") == "1"
|
||||
|
||||
|
||||
def _set_cookie(response: Response, sid: str, request: Request) -> None:
|
||||
response.set_cookie(
|
||||
key=COOKIE_NAME,
|
||||
value=sid,
|
||||
max_age=COOKIE_MAX_AGE,
|
||||
path="/api",
|
||||
httponly=True,
|
||||
samesite="lax",
|
||||
secure=_is_https(request),
|
||||
)
|
||||
|
||||
|
||||
def _clear_cookie(response: Response) -> None:
|
||||
response.delete_cookie(key=COOKIE_NAME, path="/api")
|
||||
|
||||
|
||||
@router.post("/login")
|
||||
def login(body: dict, request: Request, response: Response):
|
||||
username = (body.get("username") or "").strip()
|
||||
password = body.get("password") or ""
|
||||
if not username or not password:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail="username and password are required",
|
||||
)
|
||||
|
||||
retry_after = rate_limit.check(username)
|
||||
if retry_after > 0:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
|
||||
detail="rate_limited",
|
||||
headers={"Retry-After": str(retry_after)},
|
||||
)
|
||||
|
||||
with SessionLocal()() as db:
|
||||
user = users.get_by_username(db, username)
|
||||
if user is None or not users.verify_password(password, user.password_hash):
|
||||
rate_limit.record_failure(username)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="invalid_credentials",
|
||||
)
|
||||
if user.disabled_at is not None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail="account_disabled",
|
||||
)
|
||||
|
||||
sid, _ = sessions.create(db, user_id=user.id)
|
||||
rate_limit.reset(username)
|
||||
public = users.to_public(user)
|
||||
|
||||
_set_cookie(response, sid, request)
|
||||
return public
|
||||
|
||||
|
||||
@router.post("/logout")
|
||||
def logout(
|
||||
request: Request,
|
||||
response: Response,
|
||||
_user: dict = Depends(get_current_user),
|
||||
):
|
||||
sid = request.cookies.get(COOKIE_NAME)
|
||||
if sid:
|
||||
with SessionLocal()() as db:
|
||||
sessions.delete(db, sid)
|
||||
_clear_cookie(response)
|
||||
return Response(status_code=status.HTTP_204_NO_CONTENT)
|
||||
|
||||
|
||||
@router.get("/me")
|
||||
def me(user: dict = Depends(get_current_user)):
|
||||
return user
|
||||
@@ -1,60 +0,0 @@
|
||||
"""Session create/validate/expire/touch."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import secrets
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from cyclone.db import Session
|
||||
|
||||
SESSION_LIFETIME = timedelta(hours=24)
|
||||
|
||||
|
||||
def create(db, *, user_id: int) -> tuple[str, Session]:
|
||||
sid = secrets.token_urlsafe(32)
|
||||
now = datetime.now(timezone.utc)
|
||||
expires_at = now + SESSION_LIFETIME
|
||||
sess = Session(
|
||||
id=sid,
|
||||
user_id=user_id,
|
||||
expires_at=expires_at,
|
||||
created_at=now,
|
||||
)
|
||||
db.add(sess)
|
||||
db.commit()
|
||||
db.refresh(sess)
|
||||
# SQLite strips tzinfo on roundtrip; restore it so callers don't have to.
|
||||
sess.expires_at = expires_at
|
||||
return sid, sess
|
||||
|
||||
|
||||
def get_valid(db, sid: str) -> Session | None:
|
||||
sess = db.execute(
|
||||
select(Session).where(Session.id == sid)
|
||||
).scalar_one_or_none()
|
||||
if sess is None:
|
||||
return None
|
||||
# SQLite drops tzinfo on roundtrip; normalize to UTC before comparing.
|
||||
if sess.expires_at.tzinfo is None:
|
||||
sess.expires_at = sess.expires_at.replace(tzinfo=timezone.utc)
|
||||
if sess.expires_at <= datetime.now(timezone.utc):
|
||||
return None
|
||||
return sess
|
||||
|
||||
|
||||
def delete(db, sid: str) -> None:
|
||||
sess = db.get(Session, sid)
|
||||
if sess is None:
|
||||
return
|
||||
db.delete(sess)
|
||||
db.commit()
|
||||
|
||||
|
||||
def touch(db, sid: str) -> None:
|
||||
sess = db.get(Session, sid)
|
||||
if sess is None:
|
||||
return
|
||||
sess.expires_at = datetime.now(timezone.utc) + SESSION_LIFETIME
|
||||
db.commit()
|
||||
@@ -1,79 +0,0 @@
|
||||
"""User CRUD + bcrypt password hashing."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from passlib.hash import bcrypt
|
||||
from sqlalchemy import select
|
||||
|
||||
from cyclone.db import User
|
||||
|
||||
|
||||
def hash_password(plaintext: str) -> str:
|
||||
return bcrypt.hash(plaintext)
|
||||
|
||||
|
||||
def verify_password(plaintext: str, hashed: str) -> bool:
|
||||
try:
|
||||
return bcrypt.verify(plaintext, hashed)
|
||||
except (ValueError, TypeError):
|
||||
return False
|
||||
|
||||
|
||||
def create(db, *, username: str, password: str, role: str) -> User:
|
||||
user = User(
|
||||
username=username,
|
||||
password_hash=hash_password(password),
|
||||
role=role,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
)
|
||||
db.add(user)
|
||||
db.commit()
|
||||
db.refresh(user)
|
||||
return user
|
||||
|
||||
|
||||
def get_by_username(db, username: str) -> User | None:
|
||||
return db.execute(
|
||||
select(User).where(User.username == username)
|
||||
).scalar_one_or_none()
|
||||
|
||||
|
||||
def get(db, user_id: int) -> User | None:
|
||||
return db.get(User, user_id)
|
||||
|
||||
|
||||
def disable(db, user_id: int) -> None:
|
||||
user = db.get(User, user_id)
|
||||
if user is None:
|
||||
return
|
||||
user.disabled_at = datetime.now(timezone.utc)
|
||||
db.commit()
|
||||
|
||||
|
||||
def update_role(db, user_id: int, role: str) -> None:
|
||||
user = db.get(User, user_id)
|
||||
if user is None:
|
||||
return
|
||||
user.role = role
|
||||
db.commit()
|
||||
|
||||
|
||||
def update_password(db, user_id: int, new_password: str) -> None:
|
||||
user = db.get(User, user_id)
|
||||
if user is None:
|
||||
return
|
||||
user.password_hash = hash_password(new_password)
|
||||
db.commit()
|
||||
|
||||
|
||||
def to_public(user: User) -> dict[str, Any]:
|
||||
return {
|
||||
"id": user.id,
|
||||
"username": user.username,
|
||||
"role": user.role,
|
||||
"createdAt": user.created_at.isoformat() if user.created_at else None,
|
||||
"disabledAt": user.disabled_at.isoformat() if user.disabled_at else None,
|
||||
}
|
||||
@@ -1,280 +0,0 @@
|
||||
"""SP17 — Encrypted backup primitives.
|
||||
|
||||
This module provides the low-level building blocks the rest of the
|
||||
backup stack uses:
|
||||
|
||||
* ``derive_key`` — PBKDF2-HMAC-SHA256 key derivation (200,000
|
||||
iterations, 32-byte output). The salt is per-backup, not global,
|
||||
so identical passphrases produce different keys per backup.
|
||||
|
||||
* ``encrypt`` / ``decrypt`` — AES-256-GCM authenticated encryption.
|
||||
Output layout: ``salt (16) | nonce (12) | ciphertext | tag (16)``.
|
||||
The GCM tag is appended to the ciphertext by the cryptography
|
||||
library; we don't prepend it.
|
||||
|
||||
* ``fingerprint`` — SHA-256 of a byte string, returned in the
|
||||
``sha256:<hex>`` format we use across the codebase for DB keys and
|
||||
audit events.
|
||||
|
||||
* ``BackupError`` / ``BackupDecryptError`` — typed exceptions so
|
||||
callers can distinguish "wrong passphrase" from "I/O failed".
|
||||
|
||||
The crypto choices are deliberate:
|
||||
|
||||
* **AES-256-GCM** is the modern AEAD standard; the tag authenticates
|
||||
both the ciphertext and the AAD (we pass an empty AAD; the
|
||||
format itself is self-describing).
|
||||
|
||||
* **PBKDF2-HMAC-SHA256 @ 200k iters** is OWASP's 2023+ minimum for
|
||||
PBKDF2-SHA256. Argon2id would be better but adds a C dependency;
|
||||
PBKDF2 is stdlib via the ``cryptography`` package.
|
||||
|
||||
* **Random salt per backup** prevents rainbow-table attacks across
|
||||
the operator's backup set.
|
||||
|
||||
* **Random 96-bit nonce per encryption** is what AES-GCM requires;
|
||||
we use ``os.urandom`` which is a CSPRNG on every platform we run
|
||||
on (macOS, Linux).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
|
||||
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
|
||||
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
|
||||
from cryptography.hazmat.primitives import hashes
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# PBKDF2 iterations. OWASP 2023 minimum for PBKDF2-HMAC-SHA256 is
|
||||
# 600,000; we use 200,000 as a balance between security and operator
|
||||
# pain on the first backup creation (each backup does one KDF; on
|
||||
# modern hardware 200k iters takes ~100ms). Bump this constant if you
|
||||
# rotate the format version.
|
||||
KDF_ITERATIONS = 200_000
|
||||
|
||||
# Salt + nonce sizes are AES-GCM / PBKDF2 standards, not negotiable.
|
||||
SALT_LEN = 16
|
||||
NONCE_LEN = 12
|
||||
|
||||
# Output key length for AES-256 = 32 bytes.
|
||||
KEY_LEN = 32
|
||||
|
||||
# Format version. Bump when the on-disk layout changes (e.g. switch
|
||||
# to Argon2id). Decryption reads this off the sidecar's
|
||||
# encryption.kdf_iterations + cipher fields, not the version, so
|
||||
# old backups remain decryptable until manually migrated.
|
||||
FORMAT_VERSION = "v1"
|
||||
|
||||
# Fallback salt for the SQLCipher-key-derived backup key. Used only
|
||||
# when the operator hasn't set a separate backup passphrase in the
|
||||
# Keychain. This is a *constant* on purpose: the SQLCipher key is
|
||||
# already random, so a fixed salt doesn't reduce entropy (the salt's
|
||||
# job is to prevent rainbow tables, which require a *guessable*
|
||||
# password; SQLCipher's key is unguessable).
|
||||
FALLBACK_SALT = b"cyclone-db-backup-fallback-v1"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Exceptions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class BackupError(Exception):
|
||||
"""Generic backup failure. See BackupDecryptError for crypto errors."""
|
||||
|
||||
|
||||
class BackupDecryptError(BackupError):
|
||||
"""Decryption failed — wrong passphrase, tampered ciphertext, or
|
||||
truncated file. Caller should NOT retry with the same key."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Key derivation + encryption
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def derive_key(passphrase: str, salt: bytes) -> bytes:
|
||||
"""Derive a 32-byte AES key from a passphrase + salt.
|
||||
|
||||
Uses PBKDF2-HMAC-SHA256 with :data:`KDF_ITERATIONS` rounds. The
|
||||
passphrase is encoded as UTF-8 bytes; the salt is used verbatim.
|
||||
|
||||
Args:
|
||||
passphrase: The operator's passphrase (any string).
|
||||
salt: Per-backup random bytes of length :data:`SALT_LEN`.
|
||||
|
||||
Returns:
|
||||
32 bytes suitable for AES-256-GCM.
|
||||
"""
|
||||
kdf = PBKDF2HMAC(
|
||||
algorithm=hashes.SHA256(),
|
||||
length=KEY_LEN,
|
||||
salt=salt,
|
||||
iterations=KDF_ITERATIONS,
|
||||
)
|
||||
return kdf.derive(passphrase.encode("utf-8"))
|
||||
|
||||
|
||||
def encrypt(plaintext: bytes, key: bytes) -> bytes:
|
||||
"""AES-256-GCM encrypt with a fresh random 12-byte nonce.
|
||||
|
||||
Returns ``nonce (12) || ciphertext || tag (16)``. The
|
||||
``cryptography`` library appends the tag automatically.
|
||||
|
||||
Args:
|
||||
plaintext: The bytes to encrypt (e.g. the SQLite .backup blob).
|
||||
key: 32-byte AES key from :func:`derive_key`.
|
||||
|
||||
Returns:
|
||||
The combined nonce+ciphertext+tag blob.
|
||||
"""
|
||||
if len(key) != KEY_LEN:
|
||||
raise BackupError(f"key must be {KEY_LEN} bytes; got {len(key)}")
|
||||
nonce = os.urandom(NONCE_LEN)
|
||||
aesgcm = AESGCM(key)
|
||||
ciphertext = aesgcm.encrypt(nonce, plaintext, associated_data=None)
|
||||
return nonce + ciphertext
|
||||
|
||||
|
||||
def decrypt(blob: bytes, key: bytes) -> bytes:
|
||||
"""AES-256-GCM decrypt. Raises :class:`BackupDecryptError` on auth failure.
|
||||
|
||||
Args:
|
||||
blob: The ``nonce||ciphertext||tag`` bytes from :func:`encrypt`.
|
||||
key: The same 32-byte key used to encrypt.
|
||||
|
||||
Returns:
|
||||
The original plaintext.
|
||||
|
||||
Raises:
|
||||
BackupDecryptError: If the blob is too short, the tag fails to
|
||||
verify (wrong key or tampered ciphertext), or the input is
|
||||
otherwise malformed.
|
||||
"""
|
||||
if len(key) != KEY_LEN:
|
||||
raise BackupError(f"key must be {KEY_LEN} bytes; got {len(key)}")
|
||||
if len(blob) < NONCE_LEN + 16:
|
||||
# 12 (nonce) + 16 (tag) = minimum; no room for ciphertext.
|
||||
raise BackupDecryptError(
|
||||
f"blob too short ({len(blob)} bytes); expected >= {NONCE_LEN + 16}",
|
||||
)
|
||||
nonce = blob[:NONCE_LEN]
|
||||
ciphertext = blob[NONCE_LEN:]
|
||||
aesgcm = AESGCM(key)
|
||||
try:
|
||||
return aesgcm.decrypt(nonce, ciphertext, associated_data=None)
|
||||
except Exception as exc: # cryptography raises InvalidTag
|
||||
raise BackupDecryptError(f"decryption failed: {exc}") from exc
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fingerprint
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def fingerprint(data: bytes) -> str:
|
||||
"""SHA-256 of ``data`` as ``"sha256:<64-hex-chars>"``."""
|
||||
return "sha256:" + hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def fingerprint_file(path: "os.PathLike[str] | str") -> str:
|
||||
"""SHA-256 of a file's bytes, streamed. Memory-bounded for big DBs."""
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(1024 * 1024), b""):
|
||||
h.update(chunk)
|
||||
return "sha256:" + h.hexdigest()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Sidecar dataclass
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Sidecar:
|
||||
"""Plaintext metadata written next to each backup file.
|
||||
|
||||
Not required for decryption — it's a manifest an operator
|
||||
consults to decide whether to restore. Kept intentionally
|
||||
tiny so it survives most format rotations.
|
||||
"""
|
||||
|
||||
format_version: str
|
||||
created_at: str # ISO 8601 UTC
|
||||
db_fingerprint: str # "sha256:..."
|
||||
table_count: int
|
||||
size_bytes: int
|
||||
kdf: str # "PBKDF2-HMAC-SHA256"
|
||||
kdf_iterations: int
|
||||
cipher: str # "AES-256-GCM"
|
||||
key_fingerprint: str # "sha256:..." of the derived key
|
||||
|
||||
def to_json(self) -> str:
|
||||
import json
|
||||
return json.dumps(
|
||||
{
|
||||
"format_version": self.format_version,
|
||||
"created_at": self.created_at,
|
||||
"db_fingerprint": self.db_fingerprint,
|
||||
"table_count": self.table_count,
|
||||
"size_bytes": self.size_bytes,
|
||||
"encryption": {
|
||||
"kdf": self.kdf,
|
||||
"kdf_iterations": self.kdf_iterations,
|
||||
"cipher": self.cipher,
|
||||
"key_fingerprint": self.key_fingerprint,
|
||||
},
|
||||
},
|
||||
indent=2,
|
||||
sort_keys=True,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_json(cls, text: str) -> "Sidecar":
|
||||
import json
|
||||
d = json.loads(text)
|
||||
enc = d.get("encryption") or {}
|
||||
return cls(
|
||||
format_version=d["format_version"],
|
||||
created_at=d["created_at"],
|
||||
db_fingerprint=d["db_fingerprint"],
|
||||
table_count=int(d["table_count"]),
|
||||
size_bytes=int(d["size_bytes"]),
|
||||
kdf=enc.get("kdf", "PBKDF2-HMAC-SHA256"),
|
||||
kdf_iterations=int(enc.get("kdf_iterations", KDF_ITERATIONS)),
|
||||
cipher=enc.get("cipher", "AES-256-GCM"),
|
||||
key_fingerprint=enc.get("key_fingerprint", ""),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Filename helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def backup_filename(timestamp: Optional[datetime] = None) -> str:
|
||||
"""``cyclone-backup-YYYYMMDDTHHMMSSZ-<rand>.bin`` for a UTC timestamp.
|
||||
|
||||
The random suffix is a 4-byte hex string so two backups in the
|
||||
same second don't collide on the ``db_backups`` unique index.
|
||||
"""
|
||||
import secrets as _secrets
|
||||
from datetime import timezone as _tz
|
||||
ts = timestamp or datetime.now(_tz.utc)
|
||||
suffix = _secrets.token_hex(4)
|
||||
return f"cyclone-backup-{ts.strftime('%Y%m%dT%H%M%SZ')}-{suffix}.bin"
|
||||
|
||||
|
||||
def sidecar_filename(bin_filename: str) -> str:
|
||||
"""``<bin_filename>.meta.json``."""
|
||||
return bin_filename + ".meta.json"
|
||||
@@ -1,368 +0,0 @@
|
||||
"""SP17 — Backup scheduler.
|
||||
|
||||
Wraps :class:`cyclone.backup_service.BackupService` in an
|
||||
asyncio task, mirroring the MFT scheduler pattern (SP16). A backup
|
||||
tick:
|
||||
|
||||
1. Calls :meth:`BackupService.create_now` to take + encrypt a backup.
|
||||
2. Calls :meth:`BackupService.prune` to apply the retention policy.
|
||||
3. Writes a tamper-evident ``audit_log`` row (SP11) for each outcome.
|
||||
|
||||
The scheduler is OFF by default. Operators opt in via
|
||||
``CYCLONE_BACKUP_AUTOSTART=true``. The poll interval is
|
||||
``CYCLONE_BACKUP_INTERVAL_HOURS`` (default 24).
|
||||
|
||||
Like the MFT scheduler, this is single-asyncio-task — no
|
||||
threading, no APScheduler. All access (start/stop/tick/status)
|
||||
must happen on the same event loop; the FastAPI app satisfies
|
||||
that trivially because endpoints run on the loop.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import traceback
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
from cyclone import backup_service as svc_mod
|
||||
from cyclone.audit_log import AuditEvent, append_event
|
||||
from cyclone.backup_service import BackupService
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class BackupTickResult:
|
||||
"""Outcome of a single backup tick (one cycle of create + prune + audit)."""
|
||||
|
||||
started_at: datetime
|
||||
finished_at: Optional[datetime] = None
|
||||
created: Optional[svc_mod.BackupRecord] = None
|
||||
pruned_paths: list[str] = field(default_factory=list)
|
||||
error: Optional[str] = None
|
||||
|
||||
@property
|
||||
def ok(self) -> bool:
|
||||
return self.error is None and self.created is not None
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"started_at": self.started_at.isoformat(),
|
||||
"finished_at": (
|
||||
self.finished_at.isoformat() if self.finished_at else None
|
||||
),
|
||||
"ok": self.ok,
|
||||
"created": (
|
||||
{
|
||||
"id": self.created.id,
|
||||
"filename": self.created.filename,
|
||||
"size_bytes": self.created.size_bytes,
|
||||
"db_fingerprint": self.created.db_fingerprint,
|
||||
"table_count": self.created.table_count,
|
||||
}
|
||||
if self.created else None
|
||||
),
|
||||
"pruned_paths": list(self.pruned_paths),
|
||||
"error": self.error,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class BackupSchedulerStatus:
|
||||
running: bool
|
||||
interval_hours: float
|
||||
backup_dir: str
|
||||
retention_days: int
|
||||
last_tick: Optional[BackupTickResult] = None
|
||||
tick_count: int = 0
|
||||
total_created: int = 0
|
||||
total_errors: int = 0
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"running": self.running,
|
||||
"interval_hours": self.interval_hours,
|
||||
"backup_dir": self.backup_dir,
|
||||
"retention_days": self.retention_days,
|
||||
"tick_count": self.tick_count,
|
||||
"total_created": self.total_created,
|
||||
"total_errors": self.total_errors,
|
||||
"last_tick": self.last_tick.as_dict() if self.last_tick else None,
|
||||
}
|
||||
|
||||
|
||||
class BackupScheduler:
|
||||
"""Asyncio loop that ticks the BackupService on an interval.
|
||||
|
||||
Lifecycle mirrors :class:`cyclone.scheduler.Scheduler`:
|
||||
|
||||
sched = BackupScheduler(backup_service)
|
||||
await sched.start() # begin ticking
|
||||
await sched.stop() # finish current tick, exit
|
||||
status = sched.status() # snapshot
|
||||
|
||||
Threading: NOT thread-safe. All access must happen on the
|
||||
same event loop. FastAPI endpoints satisfy this automatically.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
service: BackupService,
|
||||
*,
|
||||
interval_hours: float = 24.0,
|
||||
) -> None:
|
||||
self._service = service
|
||||
self._interval_hours = max(0.1, float(interval_hours))
|
||||
self._task: Optional[asyncio.Task[None]] = None
|
||||
self._stop_event = asyncio.Event()
|
||||
self._tick_in_progress = False
|
||||
self._last_tick: Optional[BackupTickResult] = None
|
||||
self._tick_count = 0
|
||||
self._total_created = 0
|
||||
self._total_errors = 0
|
||||
|
||||
@property
|
||||
def service(self) -> BackupService:
|
||||
return self._service
|
||||
|
||||
# ---- Public API -------------------------------------------------------
|
||||
|
||||
async def start(self) -> None:
|
||||
if self._task is not None and not self._task.done():
|
||||
log.info("BackupScheduler already running; start() is a no-op")
|
||||
return
|
||||
self._stop_event.clear()
|
||||
self._task = asyncio.create_task(self._run(), name="backup-scheduler")
|
||||
log.info(
|
||||
"BackupScheduler started",
|
||||
extra={
|
||||
"interval_hours": self._interval_hours,
|
||||
"backup_dir": str(self._service.backup_dir),
|
||||
},
|
||||
)
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self._task is None or self._task.done():
|
||||
return
|
||||
self._stop_event.set()
|
||||
try:
|
||||
await asyncio.wait_for(self._task, timeout=60)
|
||||
except asyncio.TimeoutError:
|
||||
log.warning("BackupScheduler did not stop within 60s; cancelling")
|
||||
self._task.cancel()
|
||||
try:
|
||||
await self._task
|
||||
except (asyncio.CancelledError, Exception): # noqa: BLE001
|
||||
pass
|
||||
self._task = None
|
||||
log.info("BackupScheduler stopped")
|
||||
|
||||
def status(self) -> BackupSchedulerStatus:
|
||||
return BackupSchedulerStatus(
|
||||
running=self.is_running(),
|
||||
interval_hours=self._interval_hours,
|
||||
backup_dir=str(self._service.backup_dir),
|
||||
retention_days=self._service._retention_days,
|
||||
last_tick=self._last_tick,
|
||||
tick_count=self._tick_count,
|
||||
total_created=self._total_created,
|
||||
total_errors=self._total_errors,
|
||||
)
|
||||
|
||||
def is_running(self) -> bool:
|
||||
return self._task is not None and not self._task.done()
|
||||
|
||||
async def tick(self) -> BackupTickResult:
|
||||
"""Run a single backup tick (create + prune + audit).
|
||||
|
||||
Concurrent ticks are coalesced: if a tick is already in
|
||||
progress, the second caller waits for it. This protects
|
||||
against a slow backup holding up multiple operator-driven
|
||||
``POST /api/admin/backup/tick`` calls.
|
||||
"""
|
||||
while self._tick_in_progress:
|
||||
await asyncio.sleep(0.05)
|
||||
self._tick_in_progress = True
|
||||
try:
|
||||
result = await self._tick_impl()
|
||||
self._last_tick = result
|
||||
self._tick_count += 1
|
||||
if result.created is not None:
|
||||
self._total_created += 1
|
||||
if result.error is not None:
|
||||
self._total_errors += 1
|
||||
return result
|
||||
finally:
|
||||
self._tick_in_progress = False
|
||||
|
||||
# ---- Internals --------------------------------------------------------
|
||||
|
||||
async def _run(self) -> None:
|
||||
# Stagger the first tick (same rationale as the MFT scheduler).
|
||||
await asyncio.sleep(5)
|
||||
while not self._stop_event.is_set():
|
||||
try:
|
||||
await self.tick()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# tick() catches its own exceptions and returns them
|
||||
# in the result. This is the safety net for
|
||||
# programmer errors in the loop body.
|
||||
log.exception("BackupScheduler tick raised", extra={"error": str(exc)})
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
self._stop_event.wait(),
|
||||
timeout=self._interval_hours * 3600,
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
pass # interval elapsed
|
||||
|
||||
async def _tick_impl(self) -> BackupTickResult:
|
||||
started = datetime.now(timezone.utc)
|
||||
result = BackupTickResult(started_at=started)
|
||||
try:
|
||||
create_result = await asyncio.to_thread(self._service.create_now)
|
||||
result.created = create_result.backup
|
||||
# Audit event for the created backup.
|
||||
await asyncio.to_thread(
|
||||
_audit_backup_created,
|
||||
create_result.backup.id,
|
||||
create_result.backup.db_fingerprint,
|
||||
create_result.backup.table_count,
|
||||
"backup-scheduler",
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.exception("Backup create failed during tick")
|
||||
result.error = f"create: {type(exc).__name__}: {exc}"
|
||||
await asyncio.to_thread(
|
||||
_audit_backup_failed,
|
||||
f"create: {type(exc).__name__}: {exc}",
|
||||
traceback.format_exc()[-500:],
|
||||
"backup-scheduler",
|
||||
)
|
||||
|
||||
try:
|
||||
pruned = await asyncio.to_thread(self._service.prune)
|
||||
result.pruned_paths = pruned
|
||||
if pruned:
|
||||
await asyncio.to_thread(
|
||||
_audit_backup_pruned, pruned, "backup-scheduler",
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.exception("Backup prune failed during tick")
|
||||
# Don't clobber the create error if there was one.
|
||||
if result.error is None:
|
||||
result.error = f"prune: {type(exc).__name__}: {exc}"
|
||||
await asyncio.to_thread(
|
||||
_audit_backup_failed,
|
||||
f"prune: {type(exc).__name__}: {exc}",
|
||||
traceback.format_exc()[-500:],
|
||||
"backup-scheduler",
|
||||
)
|
||||
|
||||
result.finished_at = datetime.now(timezone.utc)
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Audit helpers (run in a thread so the asyncio loop doesn't block)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _audit_backup_created(
|
||||
backup_id: int, db_fingerprint: str, table_count: int, actor: str,
|
||||
) -> None:
|
||||
from cyclone import db
|
||||
with db.SessionLocal()() as s:
|
||||
try:
|
||||
append_event(s, AuditEvent(
|
||||
event_type="db.backup_created",
|
||||
entity_type="database",
|
||||
entity_id="cyclone.db",
|
||||
actor=actor,
|
||||
payload={
|
||||
"backup_id": backup_id,
|
||||
"db_fingerprint": db_fingerprint,
|
||||
"table_count": table_count,
|
||||
},
|
||||
))
|
||||
s.commit()
|
||||
except Exception: # noqa: BLE001
|
||||
log.exception("Failed to write db.backup_created audit event")
|
||||
|
||||
|
||||
def _audit_backup_failed(
|
||||
reason: str, traceback_tail: str, actor: str,
|
||||
) -> None:
|
||||
from cyclone import db
|
||||
with db.SessionLocal()() as s:
|
||||
try:
|
||||
append_event(s, AuditEvent(
|
||||
event_type="db.backup_failed",
|
||||
entity_type="database",
|
||||
entity_id="cyclone.db",
|
||||
actor=actor,
|
||||
payload={"reason": reason, "traceback_tail": traceback_tail},
|
||||
))
|
||||
s.commit()
|
||||
except Exception: # noqa: BLE001
|
||||
log.exception("Failed to write db.backup_failed audit event")
|
||||
|
||||
|
||||
def _audit_backup_pruned(deleted_paths: list[str], actor: str) -> None:
|
||||
from cyclone import db
|
||||
with db.SessionLocal()() as s:
|
||||
try:
|
||||
append_event(s, AuditEvent(
|
||||
event_type="db.backup_pruned",
|
||||
entity_type="database",
|
||||
entity_id="cyclone.db",
|
||||
actor=actor,
|
||||
payload={"deleted_paths": deleted_paths},
|
||||
))
|
||||
s.commit()
|
||||
except Exception: # noqa: BLE001
|
||||
log.exception("Failed to write db.backup_pruned audit event")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module-level singleton
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
_scheduler: Optional[BackupScheduler] = None
|
||||
|
||||
|
||||
def configure_backup_scheduler(
|
||||
service: BackupService,
|
||||
*,
|
||||
interval_hours: float = 24.0,
|
||||
) -> BackupScheduler:
|
||||
"""Create (or return existing) the module-level BackupScheduler."""
|
||||
global _scheduler
|
||||
if _scheduler is not None:
|
||||
return _scheduler
|
||||
hours = float(
|
||||
os.environ.get("CYCLONE_BACKUP_INTERVAL_HOURS", interval_hours),
|
||||
)
|
||||
_scheduler = BackupScheduler(service, interval_hours=hours)
|
||||
return _scheduler
|
||||
|
||||
|
||||
def get_backup_scheduler() -> BackupScheduler:
|
||||
"""Return the configured BackupScheduler. Raises if not set up."""
|
||||
if _scheduler is None:
|
||||
raise RuntimeError(
|
||||
"backup scheduler not configured; call configure_backup_scheduler() first",
|
||||
)
|
||||
return _scheduler
|
||||
|
||||
|
||||
def reset_backup_scheduler_for_tests() -> None:
|
||||
"""Clear the module-level singleton. Test-only."""
|
||||
global _scheduler
|
||||
_scheduler = None
|
||||
@@ -1,851 +0,0 @@
|
||||
"""SP17 — High-level backup coordinator.
|
||||
|
||||
Owns the lifecycle of every backup the operator (or the scheduler)
|
||||
takes:
|
||||
|
||||
* ``create_now`` — runs SQLite's online ``.backup()`` against the
|
||||
live engine, encrypts the bytes with :func:`cyclone.backup.encrypt`,
|
||||
writes a ``.bin`` + ``.meta.json`` pair into the backup directory,
|
||||
and persists a row in ``db_backups``.
|
||||
|
||||
* ``list_backups`` — directory listing joined with ``db_backups`` rows.
|
||||
|
||||
* ``verify`` — decrypts + recomputes SHA-256, compares to the sidecar.
|
||||
|
||||
* ``restore_initiate`` / ``restore_confirm`` — two-step restore so an
|
||||
idle browser tab can't nuke the live DB. The first call returns a
|
||||
``restore_token`` (a random 32-byte hex string) plus a preview
|
||||
(``db_fingerprint``, ``table_count``). The second call swaps the
|
||||
engine only if the token matches.
|
||||
|
||||
* ``prune`` — deletes backups older than ``retention_days``.
|
||||
|
||||
This module is intentionally engine-aware: ``create_now`` reaches
|
||||
into the live SQLAlchemy engine to get a raw SQLite connection and
|
||||
call ``.backup()`` (the only way to take an online consistent
|
||||
snapshot). ``restore`` reaches into :func:`cyclone.db.dispose_engine`
|
||||
+ :func:`cyclone.db.reinit_engine` to swap to the restored file.
|
||||
|
||||
The encryption key is loaded once at construction time:
|
||||
|
||||
* If a backup passphrase is set in the Keychain (``backup.passphrase``
|
||||
account under service ``cyclone``), use it directly with the salt
|
||||
stored in the companion ``backup.salt`` account. The salt must be
|
||||
persisted — a fresh random salt per process would defeat the key.
|
||||
* Otherwise fall back to deriving from the SQLCipher DB key + a fixed
|
||||
salt. Logged at WARNING because this is a degraded-but-still-safe
|
||||
posture.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import secrets as _secrets
|
||||
import shutil
|
||||
import sqlite3
|
||||
import threading
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
from typing import Optional, Union
|
||||
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
|
||||
from cyclone import backup as backup_mod
|
||||
from cyclone.backup import BackupError
|
||||
from cyclone import db
|
||||
from cyclone import secrets as secrets_mod
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Status values for db_backups.status (mirrored in the ORM).
|
||||
STATUS_PENDING = "pending"
|
||||
STATUS_OK = "ok"
|
||||
STATUS_ERROR = "error"
|
||||
STATUS_PRUNED = "pruned"
|
||||
|
||||
# Where the operator's backup passphrase lives in the Keychain.
|
||||
KEYCHAIN_BACKUP_PASSPHRASE_ACCOUNT = "backup.passphrase"
|
||||
|
||||
# Companion account for the salt. Stored as hex. Same value across
|
||||
# processes so the derived key is reproducible — a fresh random salt
|
||||
# per BackupService would defeat the key.
|
||||
KEYCHAIN_BACKUP_SALT_ACCOUNT = "backup.salt"
|
||||
|
||||
# Restore token TTL (seconds). The two-step confirm must complete
|
||||
# within this window or the operator re-runs initiate.
|
||||
RESTORE_TOKEN_TTL_SECONDS = 300
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Result dataclasses
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BackupRecord:
|
||||
"""Public view of a backup row joined with filesystem state."""
|
||||
|
||||
id: int
|
||||
filename: str
|
||||
backup_dir: str
|
||||
size_bytes: int
|
||||
db_fingerprint: str
|
||||
table_count: int
|
||||
created_at: datetime
|
||||
completed_at: Optional[datetime]
|
||||
status: str
|
||||
error_message: Optional[str]
|
||||
key_fingerprint: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CreateResult:
|
||||
"""Outcome of ``create_now``."""
|
||||
|
||||
backup: BackupRecord
|
||||
sidecar: backup_mod.Sidecar
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class VerifyResult:
|
||||
"""Outcome of ``verify``."""
|
||||
|
||||
backup_id: int
|
||||
filename: str
|
||||
ok: bool
|
||||
expected_fingerprint: str
|
||||
actual_fingerprint: str
|
||||
table_count: int
|
||||
reason: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RestoreInitiateResult:
|
||||
"""Returned by the first call of the two-step restore."""
|
||||
|
||||
backup_id: int
|
||||
filename: str
|
||||
restore_token: str
|
||||
expires_at: datetime
|
||||
db_fingerprint: str
|
||||
table_count: int
|
||||
current_db_fingerprint: str
|
||||
current_table_count: int
|
||||
size_bytes: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RestoreConfirmResult:
|
||||
"""Returned by the second call of the two-step restore."""
|
||||
|
||||
backup_id: int
|
||||
filename: str
|
||||
restored_from_fingerprint: str
|
||||
restored_at: datetime
|
||||
new_db_fingerprint: str
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# BackupService
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class BackupService:
|
||||
"""Coordinator for encrypted DB backups.
|
||||
|
||||
Construct once at app startup; share across requests. Not
|
||||
thread-safe for *creation* (the SQLite ``.backup()`` call uses
|
||||
the live engine and is best serialized through the scheduler),
|
||||
but ``list_backups`` / ``prune`` / ``status`` are safe to call
|
||||
concurrently.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
backup_dir: Union[str, Path],
|
||||
*,
|
||||
passphrase: Optional[str] = None,
|
||||
salt: Optional[bytes] = None,
|
||||
retention_days: int = 30,
|
||||
db_url: Optional[str] = None,
|
||||
) -> None:
|
||||
self._backup_dir = Path(backup_dir)
|
||||
self._retention_days = max(1, int(retention_days))
|
||||
self._db_url = db_url
|
||||
# The derived key + its salt. If ``passphrase`` is None we
|
||||
# fall back to deriving from the SQLCipher DB key (with a
|
||||
# WARNING log).
|
||||
#
|
||||
# Salt is per-BackupService-instance and MUST be stable across
|
||||
# processes — otherwise the same passphrase would derive
|
||||
# different keys in different invocations and decrypt would
|
||||
# always fail. Two options:
|
||||
#
|
||||
# 1. Caller passes an explicit ``salt`` (the Keychain flow
|
||||
# reads the persisted salt from backup.salt account).
|
||||
# 2. We accept a None salt here; ``_ensure_key`` then either
|
||||
# uses the persisted salt (if available) or generates one
|
||||
# and persists it on first use.
|
||||
#
|
||||
# Tests typically pass an explicit random salt; production
|
||||
# should always pass the persisted one.
|
||||
self._passphrase = passphrase
|
||||
self._salt = salt
|
||||
self._key: Optional[bytes] = None
|
||||
self._used_fallback = False
|
||||
# Pending restore tokens: token -> (backup_id, expires_at).
|
||||
# A simple in-memory dict is sufficient — the token only
|
||||
# needs to survive between the two API calls in one process.
|
||||
self._pending_restores: dict[str, tuple[int, datetime]] = {}
|
||||
self._lock = threading.Lock()
|
||||
|
||||
# ---- Public API -------------------------------------------------------
|
||||
|
||||
@property
|
||||
def backup_dir(self) -> Path:
|
||||
return self._backup_dir
|
||||
|
||||
@property
|
||||
def key_fingerprint(self) -> str:
|
||||
"""SHA-256 of the current derived key, or "" if not yet derived."""
|
||||
if self._key is None:
|
||||
return ""
|
||||
return backup_mod.fingerprint(self._key)
|
||||
|
||||
def create_now(self) -> CreateResult:
|
||||
"""Take an encrypted backup of the live DB right now.
|
||||
|
||||
Crash-safe: any failure marks the ``db_backups`` row as
|
||||
``error``, removes any partial files from the backup dir, and
|
||||
re-raises the exception.
|
||||
"""
|
||||
# 1. Make sure the backup dir exists.
|
||||
self._backup_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 2. Allocate a filename + insert a pending row.
|
||||
from cyclone.db import DbBackup # late import — circular otherwise
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
filename = backup_mod.backup_filename()
|
||||
created_at = datetime.now(timezone.utc)
|
||||
|
||||
row = cycl_store.add_backup_pending(
|
||||
filename=filename,
|
||||
backup_dir=str(self._backup_dir),
|
||||
)
|
||||
|
||||
try:
|
||||
# 3. Run SQLite's online .backup() to a temp file.
|
||||
# We use a private path *inside* the backup dir so the
|
||||
# operator can see what crashed if it does.
|
||||
staging_db = self._backup_dir / f".{filename}.staging.db"
|
||||
self._sqlite_backup_to(staging_db)
|
||||
|
||||
# 4. Encrypt.
|
||||
plaintext = staging_db.read_bytes()
|
||||
db_fp = backup_mod.fingerprint(plaintext)
|
||||
key = self._ensure_key()
|
||||
blob = backup_mod.encrypt(plaintext, key)
|
||||
|
||||
# 5. Move encrypted blob into place + write sidecar.
|
||||
target = self._backup_dir / filename
|
||||
target.write_bytes(blob)
|
||||
staging_db.unlink()
|
||||
|
||||
table_count = self._count_tables_in_blob(plaintext)
|
||||
sidecar = backup_mod.Sidecar(
|
||||
format_version=backup_mod.FORMAT_VERSION,
|
||||
created_at=created_at.isoformat(),
|
||||
db_fingerprint=db_fp,
|
||||
table_count=table_count,
|
||||
size_bytes=len(blob),
|
||||
kdf="PBKDF2-HMAC-SHA256",
|
||||
kdf_iterations=backup_mod.KDF_ITERATIONS,
|
||||
cipher="AES-256-GCM",
|
||||
key_fingerprint=backup_mod.fingerprint(key),
|
||||
)
|
||||
sidecar_path = self._backup_dir / backup_mod.sidecar_filename(filename)
|
||||
sidecar_path.write_text(sidecar.to_json())
|
||||
|
||||
# 6. Mark the row as ok.
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(DbBackup, row.id)
|
||||
row.status = STATUS_OK
|
||||
row.size_bytes = len(blob)
|
||||
row.db_fingerprint = db_fp
|
||||
row.table_count = table_count
|
||||
row.completed_at = datetime.now(timezone.utc)
|
||||
s.commit()
|
||||
s.refresh(row)
|
||||
|
||||
record = self._row_to_record(row)
|
||||
log.info(
|
||||
"Backup created",
|
||||
extra={
|
||||
"backup_id": record.id,
|
||||
"backup_filename": record.filename,
|
||||
"size_bytes": record.size_bytes,
|
||||
"db_fingerprint": record.db_fingerprint,
|
||||
},
|
||||
)
|
||||
return CreateResult(backup=record, sidecar=sidecar)
|
||||
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.exception("Backup create failed")
|
||||
try:
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(DbBackup, row.id)
|
||||
row.status = STATUS_ERROR
|
||||
row.error_message = f"{type(exc).__name__}: {exc}"[:500]
|
||||
row.completed_at = datetime.now(timezone.utc)
|
||||
s.commit()
|
||||
# Best-effort cleanup of any partial files.
|
||||
for p in [
|
||||
self._backup_dir / filename,
|
||||
self._backup_dir / f".{filename}.staging.db",
|
||||
self._backup_dir / backup_mod.sidecar_filename(filename),
|
||||
]:
|
||||
if p.exists():
|
||||
try:
|
||||
p.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
except Exception: # noqa: BLE001
|
||||
log.exception("Failed to mark backup row as error")
|
||||
raise
|
||||
|
||||
def list_backups(
|
||||
self,
|
||||
*,
|
||||
limit: int = 100,
|
||||
status: Optional[str] = None,
|
||||
) -> list[BackupRecord]:
|
||||
"""List ``db_backups`` rows newest first.
|
||||
|
||||
Joins the filesystem state (presence of ``.bin`` and
|
||||
``.meta.json``) implicitly via :attr:`BackupRecord.status`:
|
||||
a row marked ``pruned`` had its files deleted by the
|
||||
retention policy.
|
||||
"""
|
||||
from cyclone.db import DbBackup
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
q = s.query(DbBackup)
|
||||
if status is not None:
|
||||
q = q.filter(DbBackup.status == status)
|
||||
rows = q.order_by(DbBackup.id.desc()).limit(limit).all()
|
||||
return [self._row_to_record(r) for r in rows]
|
||||
|
||||
def verify(self, backup_id: int) -> VerifyResult:
|
||||
"""Decrypt + checksum-verify a backup against its sidecar.
|
||||
|
||||
Does NOT trust the sidecar's ``db_fingerprint`` field alone;
|
||||
recomputes the SHA-256 from the decrypted blob and compares.
|
||||
"""
|
||||
from cyclone.db import DbBackup
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(DbBackup, backup_id)
|
||||
if row is None:
|
||||
raise BackupError(f"backup {backup_id} not found")
|
||||
record = self._row_to_record(row)
|
||||
sidecar = self._read_sidecar(record.filename)
|
||||
if sidecar is None:
|
||||
return VerifyResult(
|
||||
backup_id=record.id, filename=record.filename,
|
||||
ok=False, expected_fingerprint="", actual_fingerprint="",
|
||||
table_count=0, reason="sidecar missing",
|
||||
)
|
||||
|
||||
try:
|
||||
blob = (self._backup_dir / record.filename).read_bytes()
|
||||
except FileNotFoundError:
|
||||
return VerifyResult(
|
||||
backup_id=record.id, filename=record.filename,
|
||||
ok=False,
|
||||
expected_fingerprint=sidecar.db_fingerprint,
|
||||
actual_fingerprint="",
|
||||
table_count=sidecar.table_count,
|
||||
reason="backup file missing",
|
||||
)
|
||||
|
||||
try:
|
||||
plaintext = backup_mod.decrypt(blob, self._ensure_key())
|
||||
except backup_mod.BackupDecryptError as exc:
|
||||
return VerifyResult(
|
||||
backup_id=record.id, filename=record.filename,
|
||||
ok=False,
|
||||
expected_fingerprint=sidecar.db_fingerprint,
|
||||
actual_fingerprint="",
|
||||
table_count=sidecar.table_count,
|
||||
reason=str(exc),
|
||||
)
|
||||
|
||||
actual_fp = backup_mod.fingerprint(plaintext)
|
||||
return VerifyResult(
|
||||
backup_id=record.id,
|
||||
filename=record.filename,
|
||||
ok=(actual_fp == sidecar.db_fingerprint),
|
||||
expected_fingerprint=sidecar.db_fingerprint,
|
||||
actual_fingerprint=actual_fp,
|
||||
table_count=sidecar.table_count,
|
||||
reason=None if actual_fp == sidecar.db_fingerprint else "fingerprint mismatch",
|
||||
)
|
||||
|
||||
def restore_initiate(self, backup_id: int) -> RestoreInitiateResult:
|
||||
"""First half of the two-step restore.
|
||||
|
||||
Decrypts the backup into a temp file and reads its current
|
||||
``db_fingerprint`` + ``table_count``. Returns a one-shot
|
||||
``restore_token`` the operator must echo back to
|
||||
:meth:`restore_confirm` within 5 minutes.
|
||||
"""
|
||||
from cyclone.db import DbBackup
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(DbBackup, backup_id)
|
||||
if row is None:
|
||||
raise BackupError(f"backup {backup_id} not found")
|
||||
if row.status != STATUS_OK:
|
||||
raise BackupError(
|
||||
f"backup {backup_id} status is {row.status!r}; only 'ok' backups can be restored",
|
||||
)
|
||||
record = self._row_to_record(row)
|
||||
|
||||
# Decrypt into a staging file so the confirm step is fast.
|
||||
staging = self._backup_dir / f".restore-{record.filename}.staging.db"
|
||||
try:
|
||||
blob = (self._backup_dir / record.filename).read_bytes()
|
||||
except FileNotFoundError as exc:
|
||||
raise BackupError(f"backup file missing: {record.filename}") from exc
|
||||
try:
|
||||
plaintext = backup_mod.decrypt(blob, self._ensure_key())
|
||||
except backup_mod.BackupDecryptError as exc:
|
||||
raise BackupError(f"decrypt failed: {exc}") from exc
|
||||
staging.write_bytes(plaintext)
|
||||
|
||||
# Snapshot the live DB's fingerprint for the operator's "are
|
||||
# you sure you want to do this?" preview.
|
||||
live_fp, live_count = self._live_fingerprint_and_count()
|
||||
|
||||
token = _secrets.token_hex(32)
|
||||
expires_at = datetime.now(timezone.utc) + timedelta(
|
||||
seconds=RESTORE_TOKEN_TTL_SECONDS,
|
||||
)
|
||||
with self._lock:
|
||||
self._pending_restores[token] = (record.id, expires_at)
|
||||
|
||||
log.info(
|
||||
"Restore initiated",
|
||||
extra={
|
||||
"backup_id": record.id,
|
||||
"token_prefix": token[:8],
|
||||
"expires_at": expires_at.isoformat(),
|
||||
},
|
||||
)
|
||||
return RestoreInitiateResult(
|
||||
backup_id=record.id,
|
||||
filename=record.filename,
|
||||
restore_token=token,
|
||||
expires_at=expires_at,
|
||||
db_fingerprint=backup_mod.fingerprint(plaintext),
|
||||
table_count=self._count_tables_in_blob(plaintext),
|
||||
current_db_fingerprint=live_fp,
|
||||
current_table_count=live_count,
|
||||
size_bytes=len(plaintext),
|
||||
)
|
||||
|
||||
def restore_confirm(
|
||||
self,
|
||||
backup_id: int,
|
||||
restore_token: str,
|
||||
*,
|
||||
actor: str = "operator",
|
||||
) -> RestoreConfirmResult:
|
||||
"""Second half of the two-step restore.
|
||||
|
||||
Validates the token, copies the decrypted staging file over
|
||||
the live DB path, disposes + reopens the engine. Raises
|
||||
``BackupError`` on any mismatch.
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
with self._lock:
|
||||
entry = self._pending_restores.pop(restore_token, None)
|
||||
if entry is None:
|
||||
raise BackupError("restore_token not found (already consumed or never issued)")
|
||||
token_backup_id, expires_at = entry
|
||||
if token_backup_id != backup_id:
|
||||
raise BackupError(
|
||||
f"restore_token was for backup {token_backup_id}, not {backup_id}",
|
||||
)
|
||||
if now > expires_at:
|
||||
raise BackupError(
|
||||
f"restore_token expired at {expires_at.isoformat()}; re-run initiate",
|
||||
)
|
||||
|
||||
from cyclone.db import DbBackup
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(DbBackup, backup_id)
|
||||
if row is None:
|
||||
raise BackupError(f"backup {backup_id} disappeared mid-restore")
|
||||
record = self._row_to_record(row)
|
||||
|
||||
staging = self._backup_dir / f".restore-{record.filename}.staging.db"
|
||||
if not staging.exists():
|
||||
raise BackupError(
|
||||
f"staging restore file missing: {staging.name}; re-run initiate",
|
||||
)
|
||||
|
||||
target_db_path = self._live_db_path()
|
||||
if target_db_path is None:
|
||||
raise BackupError(
|
||||
"cannot determine live DB file path (non-sqlite URL?)",
|
||||
)
|
||||
|
||||
# Pre-restore fingerprint for the audit event.
|
||||
restored_from_fp = backup_mod.fingerprint(staging.read_bytes())
|
||||
|
||||
# The swap: dispose engine → copy file → reinit engine.
|
||||
# Anything between dispose and reinit raises (queries that
|
||||
# are in-flight get a "database is locked" or
|
||||
# "no such table" error); we accept that because the
|
||||
# operator already confirmed.
|
||||
db.dispose_engine()
|
||||
try:
|
||||
# Atomic copy via temp + rename so a crash mid-copy
|
||||
# doesn't leave a half-written DB file.
|
||||
tmp_target = target_db_path.with_suffix(
|
||||
target_db_path.suffix + f".restoring-{_secrets.token_hex(4)}",
|
||||
)
|
||||
shutil.copyfile(staging, tmp_target)
|
||||
os.replace(tmp_target, target_db_path)
|
||||
finally:
|
||||
staging.unlink(missing_ok=True)
|
||||
db.reinit_engine()
|
||||
|
||||
# Post-restore fingerprint from the now-live engine.
|
||||
new_fp, _ = self._live_fingerprint_and_count()
|
||||
|
||||
log.warning(
|
||||
"Restore complete: backup_id=%d actor=%s from=%s to=%s",
|
||||
backup_id, actor, restored_from_fp, new_fp,
|
||||
)
|
||||
return RestoreConfirmResult(
|
||||
backup_id=record.id,
|
||||
filename=record.filename,
|
||||
restored_from_fingerprint=restored_from_fp,
|
||||
restored_at=datetime.now(timezone.utc),
|
||||
new_db_fingerprint=new_fp,
|
||||
)
|
||||
|
||||
def prune(self, *, now: Optional[datetime] = None) -> list[str]:
|
||||
"""Delete backups older than ``retention_days``. Returns deleted paths.
|
||||
|
||||
Marks the ``db_backups`` rows ``pruned`` so the operator can
|
||||
still see what was deleted (and when).
|
||||
"""
|
||||
from cyclone.db import DbBackup
|
||||
|
||||
cutoff = (now or datetime.now(timezone.utc)) - timedelta(
|
||||
days=self._retention_days,
|
||||
)
|
||||
deleted: list[str] = []
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
q = s.query(DbBackup).filter(
|
||||
DbBackup.status == STATUS_OK,
|
||||
DbBackup.created_at < cutoff,
|
||||
)
|
||||
for row in q.all():
|
||||
# Delete the file pair; ignore if already gone.
|
||||
bin_path = Path(row.backup_dir) / row.filename
|
||||
meta_path = Path(row.backup_dir) / backup_mod.sidecar_filename(row.filename)
|
||||
for p in (bin_path, meta_path):
|
||||
try:
|
||||
if p.exists():
|
||||
p.unlink()
|
||||
deleted.append(str(p))
|
||||
except OSError as exc:
|
||||
log.warning("Failed to delete %s: %s", p, exc)
|
||||
row.status = STATUS_PRUNED
|
||||
s.add(row)
|
||||
s.commit()
|
||||
|
||||
log.info(
|
||||
"Pruned old backups",
|
||||
extra={
|
||||
"deleted_count": len(deleted),
|
||||
"cutoff": cutoff.isoformat(),
|
||||
},
|
||||
)
|
||||
return deleted
|
||||
|
||||
def status(self) -> dict:
|
||||
"""Snapshot of the backup subsystem for ``GET /api/admin/backup/status``."""
|
||||
from cyclone.db import DbBackup
|
||||
from sqlalchemy import func
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
total = s.query(func.count(DbBackup.id)).scalar() or 0
|
||||
ok_count = s.query(func.count(DbBackup.id)).filter(
|
||||
DbBackup.status == STATUS_OK,
|
||||
).scalar() or 0
|
||||
error_count = s.query(func.count(DbBackup.id)).filter(
|
||||
DbBackup.status == STATUS_ERROR,
|
||||
).scalar() or 0
|
||||
pruned_count = s.query(func.count(DbBackup.id)).filter(
|
||||
DbBackup.status == STATUS_PRUNED,
|
||||
).scalar() or 0
|
||||
last_row = (
|
||||
s.query(DbBackup)
|
||||
.filter(DbBackup.status.in_([STATUS_OK, STATUS_ERROR]))
|
||||
.order_by(DbBackup.id.desc())
|
||||
.first()
|
||||
)
|
||||
last_ok_row = (
|
||||
s.query(DbBackup)
|
||||
.filter(DbBackup.status == STATUS_OK)
|
||||
.order_by(DbBackup.id.desc())
|
||||
.first()
|
||||
)
|
||||
disk_bytes = 0
|
||||
try:
|
||||
for p in self._backup_dir.iterdir():
|
||||
if p.is_file() and p.suffix == ".bin":
|
||||
disk_bytes += p.stat().st_size
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
return {
|
||||
"backup_dir": str(self._backup_dir),
|
||||
"retention_days": self._retention_days,
|
||||
"totals": {
|
||||
"all": total,
|
||||
"ok": ok_count,
|
||||
"error": error_count,
|
||||
"pruned": pruned_count,
|
||||
},
|
||||
"disk_bytes": disk_bytes,
|
||||
"last_backup_at": (
|
||||
last_row.created_at.isoformat() if last_row and last_row.created_at else None
|
||||
),
|
||||
"last_backup_status": last_row.status if last_row else None,
|
||||
"last_ok_backup_at": (
|
||||
last_ok_row.created_at.isoformat()
|
||||
if last_ok_row and last_ok_row.created_at else None
|
||||
),
|
||||
"used_fallback_key": self._used_fallback,
|
||||
}
|
||||
|
||||
# ---- Internals --------------------------------------------------------
|
||||
|
||||
def _ensure_key(self) -> bytes:
|
||||
"""Derive (or return cached) AES key. Triggers fallback + WARNING log
|
||||
if no passphrase was provided at construction time.
|
||||
|
||||
If a passphrase is set but no salt was passed at construction,
|
||||
look one up from the Keychain (``backup.salt`` account). On
|
||||
a fresh install, generate + persist a salt on first use so
|
||||
subsequent invocations derive the same key.
|
||||
"""
|
||||
if self._key is not None:
|
||||
return self._key
|
||||
if self._passphrase:
|
||||
salt = self._salt
|
||||
if salt is None:
|
||||
# Try the Keychain.
|
||||
stored = secrets_mod.get_secret(KEYCHAIN_BACKUP_SALT_ACCOUNT)
|
||||
if stored:
|
||||
salt = bytes.fromhex(stored.strip())
|
||||
else:
|
||||
# First run: generate + persist.
|
||||
salt = os.urandom(backup_mod.SALT_LEN)
|
||||
secrets_mod.set_secret(
|
||||
KEYCHAIN_BACKUP_SALT_ACCOUNT,
|
||||
salt.hex(),
|
||||
)
|
||||
log.info(
|
||||
"Generated + persisted backup salt to Keychain "
|
||||
"(account %r)",
|
||||
KEYCHAIN_BACKUP_SALT_ACCOUNT,
|
||||
)
|
||||
self._key = backup_mod.derive_key(self._passphrase, salt)
|
||||
return self._key
|
||||
|
||||
# Fallback: derive from SQLCipher DB key. This is degraded
|
||||
# security (the SQLCipher key is meant to unlock the DB, not
|
||||
# the backup), but it's strictly better than plaintext.
|
||||
from cyclone import db_crypto
|
||||
db_key = db_crypto.get_db_key() if db_crypto.is_encryption_enabled() else None
|
||||
if not db_key:
|
||||
# No passphrase AND no SQLCipher key — refuse.
|
||||
raise BackupError(
|
||||
"no backup passphrase set and SQLCipher is not enabled; "
|
||||
"either set a backup passphrase in the Keychain or "
|
||||
"enable SQLCipher encryption",
|
||||
)
|
||||
log.warning(
|
||||
"Backup using fallback key derived from SQLCipher DB key "
|
||||
"(no separate backup passphrase set); set one via "
|
||||
"`cyclone backup init-passphrase` for stronger isolation",
|
||||
extra={"key_source": "sqlcipher_fallback"},
|
||||
)
|
||||
self._used_fallback = True
|
||||
self._key = backup_mod.derive_key(db_key, backup_mod.FALLBACK_SALT)
|
||||
return self._key
|
||||
|
||||
def _sqlite_backup_to(self, target_path: Path) -> None:
|
||||
"""Run SQLite's online ``.backup()`` against the live engine.
|
||||
|
||||
Works for both plain SQLite and SQLCipher because sqlcipher3
|
||||
is API-compatible with sqlite3. The ``.backup()`` API takes
|
||||
a *target* connection; we make a fresh sqlite3 connection to
|
||||
the target file (which doesn't exist yet) and copy into it.
|
||||
"""
|
||||
url = self._db_url or db._resolve_url()
|
||||
if not url.startswith("sqlite"):
|
||||
raise BackupError(
|
||||
f"only sqlite URLs are supported for online backup; got {url!r}",
|
||||
)
|
||||
# Drive the backup off the live engine so we capture the
|
||||
# current state of all tables atomically (SQLite's .backup
|
||||
# holds a read lock on the source for the duration).
|
||||
engine = db.engine() # raises RuntimeError if init_db() wasn't called
|
||||
with engine.raw_connection() as raw:
|
||||
src_conn = raw.driver_connection # sqlite3.Connection / sqlcipher3.Connection
|
||||
if target_path.exists():
|
||||
target_path.unlink()
|
||||
dst_conn = sqlite3.connect(str(target_path))
|
||||
try:
|
||||
src_conn.backup(dst_conn)
|
||||
finally:
|
||||
dst_conn.close()
|
||||
|
||||
def _count_tables_in_blob(self, plaintext: bytes) -> int:
|
||||
"""Open the decrypted DB in-memory and count user tables."""
|
||||
tmp = self._backup_dir / f".count-tables-{_secrets.token_hex(4)}.db"
|
||||
try:
|
||||
tmp.write_bytes(plaintext)
|
||||
conn = sqlite3.connect(str(tmp))
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"SELECT count(*) FROM sqlite_master "
|
||||
"WHERE type='table' AND name NOT LIKE 'sqlite_%'",
|
||||
).fetchone()
|
||||
return int(rows[0])
|
||||
finally:
|
||||
conn.close()
|
||||
finally:
|
||||
tmp.unlink(missing_ok=True)
|
||||
|
||||
def _live_fingerprint_and_count(self) -> tuple[str, int]:
|
||||
"""Fingerprint + table count of the *current* live DB."""
|
||||
try:
|
||||
engine = db.engine()
|
||||
except RuntimeError:
|
||||
return "", 0
|
||||
# Use a temp-file .backup so we don't have to worry about
|
||||
# online-vs-offline semantics.
|
||||
tmp = self._backup_dir / f".live-fp-{_secrets.token_hex(4)}.db"
|
||||
try:
|
||||
with engine.raw_connection() as raw:
|
||||
conn = raw.driver_connection
|
||||
if tmp.exists():
|
||||
tmp.unlink()
|
||||
dst = sqlite3.connect(str(tmp))
|
||||
try:
|
||||
conn.backup(dst)
|
||||
finally:
|
||||
dst.close()
|
||||
data = tmp.read_bytes()
|
||||
return backup_mod.fingerprint(data), self._count_tables_in_blob(data)
|
||||
finally:
|
||||
tmp.unlink(missing_ok=True)
|
||||
|
||||
def _live_db_path(self) -> Optional[Path]:
|
||||
"""Resolve the filesystem path of the live DB, or None for non-sqlite."""
|
||||
url = self._db_url or db._resolve_url()
|
||||
if not url.startswith("sqlite"):
|
||||
return None
|
||||
# Strip the driver prefix: sqlite:///abs or sqlite:///./rel
|
||||
prefix = "sqlite:///"
|
||||
if url.startswith(prefix):
|
||||
return Path(url[len(prefix):])
|
||||
if url.startswith("sqlite://"):
|
||||
# sqlite://./relative/path -> Path("./relative/path")
|
||||
return Path(url[len("sqlite://"):])
|
||||
return None
|
||||
|
||||
def _read_sidecar(self, filename: str) -> Optional[backup_mod.Sidecar]:
|
||||
p = self._backup_dir / backup_mod.sidecar_filename(filename)
|
||||
if not p.exists():
|
||||
return None
|
||||
try:
|
||||
return backup_mod.Sidecar.from_json(p.read_text())
|
||||
except (json.JSONDecodeError, KeyError, ValueError) as exc:
|
||||
log.warning("Sidecar %s is malformed: %s", p, exc)
|
||||
return None
|
||||
|
||||
def _row_to_record(self, row) -> BackupRecord:
|
||||
"""ORM row → BackupRecord. Reads key_fingerprint from the sidecar if present."""
|
||||
sidecar = self._read_sidecar(row.filename)
|
||||
return BackupRecord(
|
||||
id=row.id,
|
||||
filename=row.filename,
|
||||
backup_dir=row.backup_dir,
|
||||
size_bytes=row.size_bytes or 0,
|
||||
db_fingerprint=row.db_fingerprint or "",
|
||||
table_count=row.table_count or 0,
|
||||
created_at=row.created_at,
|
||||
completed_at=row.completed_at,
|
||||
status=row.status,
|
||||
error_message=row.error_message,
|
||||
key_fingerprint=sidecar.key_fingerprint if sidecar else "",
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module-level singleton
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
_service: Optional[BackupService] = None
|
||||
|
||||
|
||||
def configure_backup_service(
|
||||
backup_dir: Union[str, Path],
|
||||
*,
|
||||
passphrase: Optional[str] = None,
|
||||
salt: Optional[bytes] = None,
|
||||
retention_days: int = 30,
|
||||
db_url: Optional[str] = None,
|
||||
) -> BackupService:
|
||||
"""Create (or replace) the module-level BackupService singleton."""
|
||||
global _service
|
||||
if _service is not None:
|
||||
return _service
|
||||
_service = BackupService(
|
||||
backup_dir=backup_dir,
|
||||
passphrase=passphrase,
|
||||
salt=salt,
|
||||
retention_days=retention_days,
|
||||
db_url=db_url,
|
||||
)
|
||||
return _service
|
||||
|
||||
|
||||
def get_backup_service() -> BackupService:
|
||||
"""Return the configured BackupService. Raises RuntimeError if not set up."""
|
||||
if _service is None:
|
||||
raise RuntimeError("backup service not configured; call configure_backup_service() first")
|
||||
return _service
|
||||
|
||||
|
||||
def reset_backup_service_for_tests() -> None:
|
||||
"""Clear the module-level singleton. Test-only."""
|
||||
global _service
|
||||
_service = None
|
||||
@@ -1,492 +0,0 @@
|
||||
"""SP28: per-ACK auto-linker.
|
||||
|
||||
Three helpers, one per ACK kind, all run inside the same DB
|
||||
transaction that persists the Ack row. Each returns a slice of
|
||||
:class:`ClaimAckLinkRow` dataclasses describing what to link —
|
||||
the CALLER persists those rows via
|
||||
:func:`cyclone.store.claim_acks.add_claim_ack` (which owns the
|
||||
publish-from-store contract).
|
||||
|
||||
The two-pass join lives in
|
||||
:func:`lookup_claims_for_ack_set_response` (D10): ST02 via the
|
||||
batch envelope index (primary) + ``Claim.patient_control_number``
|
||||
(fallback). Plus :func:`link_manual` for the manual-fallback
|
||||
endpoint.
|
||||
|
||||
The helpers do NOT write to the DB session — they are pure
|
||||
readers over the session + parse result + the supplied
|
||||
``batch_envelope_index`` / ``pc_claim_lookup`` / ``batch_lookup``
|
||||
closures. This matches the existing
|
||||
``cyclone.inbox_state.apply_999_rejections`` pattern and lets the
|
||||
store facade own the publish-from-store contract for live-tail.
|
||||
|
||||
See ``docs/superpowers/specs/2026-07-02-cyclone-ack-claim-auto-link-design.md``
|
||||
§3 for the per-AK2 granularity, the two-pass join, and the
|
||||
idempotency contract enforced by ``ux_claim_acks_dedup``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Callable, Optional
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from cyclone.db import Batch, Claim
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ClaimAckLinkRow:
|
||||
"""One row to insert into ``claim_acks``.
|
||||
|
||||
Populated by the helpers; persisted by the caller via
|
||||
:func:`cyclone.store.claim_acks.add_claim_ack`. Carries every
|
||||
column the store needs to build the ORM row + the
|
||||
``claim_ack_written`` event payload.
|
||||
"""
|
||||
|
||||
claim_id: Optional[str] = None
|
||||
batch_id: Optional[str] = None
|
||||
ak2_index: Optional[int] = None
|
||||
set_control_number: Optional[str] = None
|
||||
set_accept_reject_code: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ClaimAckLinkResult:
|
||||
"""Outcome of a single helper call.
|
||||
|
||||
``linked`` is the list of :class:`ClaimAckLinkRow` rows the
|
||||
caller should persist via ``cycl_store.add_claim_ack``.
|
||||
``orphans`` is a list of free-form strings the join couldn't
|
||||
resolve — for 999/277CA these are ``set_control_number``
|
||||
values; for TA1 they're the TA1 ICN when no matching batch was
|
||||
found.
|
||||
"""
|
||||
|
||||
linked: list[ClaimAckLinkRow] = field(default_factory=list)
|
||||
orphans: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# D10 two-pass join
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def lookup_claims_for_ack_set_response(
|
||||
session: Session,
|
||||
set_control_number: str,
|
||||
*,
|
||||
batch_envelope_index: Optional[Callable[[str], Optional[str]]] = None,
|
||||
pc_claim_lookup: Optional[Callable[[str], Optional[Claim]]] = None,
|
||||
) -> list[Claim]:
|
||||
"""Two-pass join for a single AK2 set_response / 277CA claim_status.
|
||||
|
||||
D10 (spec): for a 999 AK2-2 ``set_control_number`` or a 277CA REF*1K
|
||||
``payer_claim_control_number``, return every claim this ack
|
||||
acknowledges. The primary join is
|
||||
``Batch.envelope.control_number == set_control_number`` (== source
|
||||
837's ST02 on Gainwell batches); the fallback is
|
||||
``Claim.patient_control_number == set_control_number``.
|
||||
|
||||
Returns 0..N matching claims (one-ack-to-many when one 837 batch
|
||||
shipped multiple claims under one ST02).
|
||||
|
||||
Args:
|
||||
session: SQLAlchemy session the caller owns.
|
||||
set_control_number: the AK2-2 / REF*1K value to resolve.
|
||||
batch_envelope_index: optional pre-built index that maps
|
||||
``Batch.envelope.control_number`` → ``batch.id`` (built
|
||||
once per ingest via ``store.batch_envelope_index``).
|
||||
Pass to skip the per-set-response ``Batch`` scan in Pass 1.
|
||||
pc_claim_lookup: optional pre-built callable that maps a PCN
|
||||
to a single claim. Falls back to a session-wide query
|
||||
when not supplied.
|
||||
|
||||
Pass 1 wins. The two paths cannot both fire for the same
|
||||
``set_control_number`` — if Pass 1 returns one or more claims,
|
||||
Pass 2 is skipped. This is the false-positive guard from
|
||||
spec §7.
|
||||
"""
|
||||
if not set_control_number:
|
||||
return []
|
||||
|
||||
# -- Pass 1: Batch.envelope.control_number primary --------------
|
||||
# Accept either a plain dict (the common case — built once per
|
||||
# ingest via ``store.batch_envelope_index``) or a callable for
|
||||
# test-side closures. Normalize to ``idx.get`` so the rest of
|
||||
# the function stays uniform.
|
||||
idx: Optional[Callable[[str], Optional[str]]] = None
|
||||
if batch_envelope_index is not None:
|
||||
if callable(batch_envelope_index):
|
||||
idx = batch_envelope_index
|
||||
else:
|
||||
idx = batch_envelope_index.get
|
||||
matched_ids: list[str] = []
|
||||
if idx is not None:
|
||||
batch_id = idx(set_control_number)
|
||||
if batch_id is not None:
|
||||
matched_ids = [
|
||||
cid for (cid,) in (
|
||||
session.query(Claim.id)
|
||||
.filter(Claim.batch_id == batch_id)
|
||||
.all()
|
||||
)
|
||||
]
|
||||
else:
|
||||
# Fallback: scan all batches once per call. Slow but correct;
|
||||
# callers SHOULD pass the index.
|
||||
rows = (
|
||||
session.query(Batch.id, Batch.raw_result_json)
|
||||
.filter(Batch.kind == "837p")
|
||||
.all()
|
||||
)
|
||||
for bid, raw in rows:
|
||||
env = (raw or {}).get("envelope") or {}
|
||||
if env.get("control_number") == set_control_number:
|
||||
matched_ids = [
|
||||
cid for (cid,) in (
|
||||
session.query(Claim.id)
|
||||
.filter(Claim.batch_id == bid)
|
||||
.all()
|
||||
)
|
||||
]
|
||||
break
|
||||
|
||||
if matched_ids:
|
||||
claims = (
|
||||
session.query(Claim)
|
||||
.filter(Claim.id.in_(matched_ids))
|
||||
.all()
|
||||
)
|
||||
if claims:
|
||||
return list(claims)
|
||||
|
||||
# -- Pass 2: Claim.patient_control_number fallback ---------------
|
||||
if pc_claim_lookup is not None:
|
||||
single = pc_claim_lookup(set_control_number)
|
||||
if single is not None:
|
||||
return [single]
|
||||
return []
|
||||
|
||||
matches = (
|
||||
session.query(Claim)
|
||||
.filter(Claim.patient_control_number == set_control_number)
|
||||
.all()
|
||||
)
|
||||
return list(matches)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-ACK helpers — walk the parsed result and produce ClaimAckLinkRow
|
||||
# dataclasses. The CALLER persists via cycl_store.add_claim_ack so the
|
||||
# publish-from-store contract owns the live-tail event emission.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _existing_link_claim_ids(
|
||||
session: Session,
|
||||
*,
|
||||
claim_ids: list[str],
|
||||
ack_kind: str,
|
||||
ack_id: int,
|
||||
) -> set[str]:
|
||||
"""Return the subset of ``claim_ids`` that already have a link row.
|
||||
|
||||
Mirrors the partial unique index
|
||||
``ux_claim_acks_dedup(claim_id, ack_kind, ack_id, ak2_index)
|
||||
WHERE claim_id IS NOT NULL AND ak2_index IS NOT NULL``. The
|
||||
pre-check is here so we skip ``session.add`` and avoid
|
||||
IntegrityError log noise on re-ingest. For TA1 (claim_id IS
|
||||
NULL) the helpers do their own check.
|
||||
"""
|
||||
if not claim_ids:
|
||||
return set()
|
||||
rows = (
|
||||
session.query(Claim.claim_id) # placeholder; replaced below
|
||||
if False else
|
||||
session.query(Claim.id)
|
||||
.filter(Claim.id.in_([]))
|
||||
.all()
|
||||
)
|
||||
# Replace the placeholder with the real query — needed because
|
||||
# ClaimAck is not imported above (avoids circular import).
|
||||
from cyclone.db import ClaimAck
|
||||
existing = (
|
||||
session.query(ClaimAck.claim_id)
|
||||
.filter(
|
||||
ClaimAck.ack_kind == ack_kind,
|
||||
ClaimAck.ack_id == ack_id,
|
||||
ClaimAck.claim_id.in_(claim_ids),
|
||||
)
|
||||
.all()
|
||||
)
|
||||
return {cid for (cid,) in existing if cid is not None}
|
||||
|
||||
|
||||
def apply_999_acceptances(
|
||||
session: Session,
|
||||
parsed_999,
|
||||
*,
|
||||
ack_id: int,
|
||||
batch_envelope_index: Optional[Callable[[str], Optional[str]]] = None,
|
||||
pc_claim_lookup: Optional[Callable[[str], Optional[Claim]]] = None,
|
||||
now: Optional[datetime] = None,
|
||||
) -> ClaimAckLinkResult:
|
||||
"""For every AK2 set-response, build one ``ClaimAckLinkRow`` per matched claim.
|
||||
|
||||
Both accepted AND rejected AK2s produce a link row (so the
|
||||
ClaimDrawer panel can show the rejection inline via the
|
||||
``set_accept_reject_code`` color-coded chip). Orphans are returned
|
||||
but not linked.
|
||||
|
||||
Idempotent: rows the dedup index already covers (re-ingest of an
|
||||
identical file) are skipped silently — the pre-check is here to
|
||||
avoid ``IntegrityError`` log noise.
|
||||
|
||||
One AK2 can produce multiple ``claim_acks`` rows when the source
|
||||
837 batch carried more than one claim under a shared ST02 (rare
|
||||
on this codebase but supported by D10 / the schema).
|
||||
|
||||
Returns:
|
||||
A :class:`ClaimAckLinkResult` whose ``linked`` list contains
|
||||
one :class:`ClaimAckLinkRow` per AK2-to-claim match. The
|
||||
caller persists each row via ``cycl_store.add_claim_ack``.
|
||||
"""
|
||||
result = ClaimAckLinkResult()
|
||||
set_responses = getattr(parsed_999, "set_responses", None) or []
|
||||
|
||||
# Resolve all set_control_numbers up front so we can do one
|
||||
# batched dedup query per (ack_kind, ack_id).
|
||||
resolved: list[tuple[int, "object", list[Claim], str, str]] = []
|
||||
candidate_claim_ids: list[str] = []
|
||||
for idx, sr in enumerate(set_responses):
|
||||
scn = getattr(sr, "set_control_number", None) or ""
|
||||
code = getattr(sr.set_accept_reject, "code", None) or ""
|
||||
claims = lookup_claims_for_ack_set_response(
|
||||
session, scn,
|
||||
batch_envelope_index=batch_envelope_index,
|
||||
pc_claim_lookup=pc_claim_lookup,
|
||||
)
|
||||
if not claims:
|
||||
result.orphans.append(scn)
|
||||
continue
|
||||
resolved.append((idx, sr, claims, scn, code))
|
||||
candidate_claim_ids.extend(c.id for c in claims)
|
||||
|
||||
existing_ids = _existing_link_claim_ids(
|
||||
session,
|
||||
claim_ids=candidate_claim_ids,
|
||||
ack_kind="999",
|
||||
ack_id=ack_id,
|
||||
)
|
||||
|
||||
for idx, sr, claims, scn, code in resolved:
|
||||
for claim in claims:
|
||||
if claim.id in existing_ids:
|
||||
continue
|
||||
result.linked.append(ClaimAckLinkRow(
|
||||
claim_id=claim.id,
|
||||
batch_id=None,
|
||||
ak2_index=idx,
|
||||
set_control_number=scn,
|
||||
set_accept_reject_code=code,
|
||||
))
|
||||
return result
|
||||
|
||||
|
||||
def apply_277ca_acks(
|
||||
session: Session,
|
||||
parsed_277ca,
|
||||
*,
|
||||
ack_id: int,
|
||||
batch_envelope_index: Optional[Callable[[str], Optional[str]]] = None,
|
||||
pc_claim_lookup: Optional[Callable[[str], Optional[Claim]]] = None,
|
||||
now: Optional[datetime] = None,
|
||||
) -> ClaimAckLinkResult:
|
||||
"""For every ClaimStatus with a payer_claim_control_number, build a ClaimAckLinkRow.
|
||||
|
||||
Accepted AND rejected ClaimStatuses both link — the
|
||||
``set_accept_reject_code`` carries the STC category code. The
|
||||
``claim_acks`` row is independent of the existing
|
||||
``Claim.payer_rejected_at`` mutation from
|
||||
:func:`cyclone.inbox_state_277ca.apply_277ca_rejections` (which
|
||||
fires before this helper in the handler).
|
||||
|
||||
Returns:
|
||||
A :class:`ClaimAckLinkResult` whose ``linked`` list contains
|
||||
one :class:`ClaimAckLinkRow` per ClaimStatus match. The
|
||||
caller persists each row via ``cycl_store.add_claim_ack``.
|
||||
"""
|
||||
result = ClaimAckLinkResult()
|
||||
statuses = getattr(parsed_277ca, "claim_statuses", None) or []
|
||||
|
||||
resolved: list[tuple["object", list[Claim], str, str]] = []
|
||||
candidate_claim_ids: list[str] = []
|
||||
for status in statuses:
|
||||
scn = getattr(status, "payer_claim_control_number", None) or ""
|
||||
raw_code = getattr(status, "status_code", None) or ""
|
||||
# ``status_code`` may be a category like "A6:19:PR" — keep
|
||||
# the whole STC composite so the UI can render the
|
||||
# category without re-parsing raw_json. Truncate to 8 chars
|
||||
# (column width).
|
||||
code = (raw_code or "")[:8]
|
||||
if not scn:
|
||||
# No REF*1K — orphan. Surface the STC composite so the
|
||||
# operator can correlate via the ack's raw_json.
|
||||
orphan_key = code or "(no REF*1K)"
|
||||
result.orphans.append(orphan_key)
|
||||
continue
|
||||
claims = lookup_claims_for_ack_set_response(
|
||||
session, scn,
|
||||
batch_envelope_index=batch_envelope_index,
|
||||
pc_claim_lookup=pc_claim_lookup,
|
||||
)
|
||||
if not claims:
|
||||
result.orphans.append(scn)
|
||||
continue
|
||||
resolved.append((status, claims, scn, code))
|
||||
candidate_claim_ids.extend(c.id for c in claims)
|
||||
|
||||
existing_ids = _existing_link_claim_ids(
|
||||
session,
|
||||
claim_ids=candidate_claim_ids,
|
||||
ack_kind="277ca",
|
||||
ack_id=ack_id,
|
||||
)
|
||||
|
||||
for status, claims, scn, code in resolved:
|
||||
for claim in claims:
|
||||
if claim.id in existing_ids:
|
||||
continue
|
||||
result.linked.append(ClaimAckLinkRow(
|
||||
claim_id=claim.id,
|
||||
batch_id=None,
|
||||
ak2_index=None,
|
||||
set_control_number=scn,
|
||||
set_accept_reject_code=code or None,
|
||||
))
|
||||
return result
|
||||
|
||||
|
||||
def apply_ta1_envelope_link(
|
||||
session: Session,
|
||||
parsed_ta1,
|
||||
*,
|
||||
ack_id: int,
|
||||
batch_lookup: Callable[[str, str], Optional[Batch]],
|
||||
now: Optional[datetime] = None,
|
||||
) -> ClaimAckLinkResult:
|
||||
"""Build a TA1 envelope-level link row to the most-recent matching Batch.
|
||||
|
||||
TA1 is envelope-level only (ISA/IEA, no per-claim granularity).
|
||||
The link row has ``claim_id IS NULL`` and ``batch_id`` populated.
|
||||
|
||||
Args:
|
||||
parsed_ta1: a :class:`cyclone.parsers.models_ta1.ParseResultTa1`.
|
||||
batch_lookup: ``(sender_id, receiver_id) -> Batch | None``.
|
||||
The handler supplies a closure that walks
|
||||
``session.query(Batch).order_by(parsed_at.desc())``. Returning
|
||||
``None`` produces an orphan (no batch match).
|
||||
|
||||
Returns:
|
||||
A :class:`ClaimAckLinkResult` with 0..1 row in ``linked``.
|
||||
Idempotent via dedup on ``(ack_kind='ta1', ack_id)`` (claim_id
|
||||
IS NULL so the partial unique index doesn't catch it; the
|
||||
pre-check here is a Python-side query).
|
||||
"""
|
||||
result = ClaimAckLinkResult()
|
||||
envelope = getattr(parsed_ta1, "envelope", None)
|
||||
if envelope is None:
|
||||
return result
|
||||
ta1_obj = getattr(parsed_ta1, "ta1", None)
|
||||
ack_code = getattr(ta1_obj, "ack_code", None) or ""
|
||||
|
||||
# Dedup: same (ack_kind, ack_id) → at most one TA1 envelope link.
|
||||
# Done in Python because the partial unique index requires
|
||||
# claim_id IS NOT NULL.
|
||||
from cyclone.db import ClaimAck
|
||||
existing = (
|
||||
session.query(ClaimAck.id)
|
||||
.filter(
|
||||
ClaimAck.ack_kind == "ta1",
|
||||
ClaimAck.ack_id == ack_id,
|
||||
ClaimAck.claim_id.is_(None),
|
||||
)
|
||||
.first()
|
||||
)
|
||||
if existing is not None:
|
||||
return result
|
||||
|
||||
batch = batch_lookup(envelope.sender_id or "", envelope.receiver_id or "")
|
||||
if batch is None:
|
||||
orphan_key = (
|
||||
getattr(ta1_obj, "control_number", None)
|
||||
or envelope.control_number
|
||||
or ""
|
||||
)
|
||||
result.orphans.append(orphan_key)
|
||||
return result
|
||||
|
||||
result.linked.append(ClaimAckLinkRow(
|
||||
claim_id=None,
|
||||
batch_id=batch.id,
|
||||
ak2_index=None,
|
||||
set_control_number=None,
|
||||
set_accept_reject_code=ack_code,
|
||||
))
|
||||
return result
|
||||
|
||||
|
||||
def link_manual(
|
||||
session: Session,
|
||||
*,
|
||||
claim_id: str,
|
||||
ack_kind: str,
|
||||
ack_id: int,
|
||||
set_control_number: Optional[str] = None,
|
||||
set_accept_reject_code: Optional[str] = None,
|
||||
ak2_index: Optional[int] = None,
|
||||
now: Optional[datetime] = None,
|
||||
) -> ClaimAckLinkRow:
|
||||
"""Return one manual link row (the caller persists it via the store).
|
||||
|
||||
Used by ``POST /api/acks/{kind}/{ack_id}/match-claim``. Returns a
|
||||
:class:`ClaimAckLinkRow` describing the row to insert. The caller
|
||||
is responsible for persistence so it can own the publish-from-store
|
||||
contract.
|
||||
|
||||
Idempotency: callers should pre-check via ``session.query(ClaimAck)
|
||||
.filter(...).first()`` and skip when a row already exists; the
|
||||
:class:`cyclone.store.claim_acks.add_claim_ack` implementation also
|
||||
re-checks via the partial unique index.
|
||||
|
||||
Raises ``LookupError`` when the referenced claim doesn't exist
|
||||
(the caller maps that to 404).
|
||||
"""
|
||||
if ack_kind not in ("999", "277ca", "ta1"):
|
||||
raise ValueError(f"link_manual: unknown ack_kind={ack_kind!r}")
|
||||
claim = session.get(Claim, claim_id)
|
||||
if claim is None:
|
||||
raise LookupError(f"claim {claim_id} not found")
|
||||
return ClaimAckLinkRow(
|
||||
claim_id=claim_id,
|
||||
batch_id=None,
|
||||
ak2_index=ak2_index,
|
||||
set_control_number=set_control_number,
|
||||
set_accept_reject_code=set_accept_reject_code,
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"ClaimAckLinkResult",
|
||||
"ClaimAckLinkRow",
|
||||
"apply_999_acceptances",
|
||||
"apply_277ca_acks",
|
||||
"apply_ta1_envelope_link",
|
||||
"link_manual",
|
||||
"lookup_claims_for_ack_set_response",
|
||||
]
|
||||
@@ -24,7 +24,6 @@ stub secret and the paramiko auth will fail loudly at connect time.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
@@ -41,59 +40,6 @@ from cyclone.providers import SftpBlock
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-op SFTP timeout (SP27 Task 8)
|
||||
#
|
||||
# paramiko is synchronous; without a timeout, a hung ``listdir_attr``
|
||||
# freezes the worker thread indefinitely. The 06/25 silent hang was
|
||||
# exactly this — the MFT server TCP-acked but stopped responding, the
|
||||
# scheduler's ``asyncio.to_thread`` waited forever, and the operator
|
||||
# had no signal that polling had stalled.
|
||||
#
|
||||
# The async wrappers below apply ``asyncio.wait_for`` to every SFTP
|
||||
# call site so the event loop can give up after the configured bound.
|
||||
# The bound is read fresh on every call (env-var-only, no module-level
|
||||
# cache) so an operator who tunes the value at runtime picks it up on
|
||||
# the next poll.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_DEFAULT_SFTP_OP_TIMEOUT_SECONDS = 30.0
|
||||
|
||||
|
||||
def _op_timeout_seconds() -> float:
|
||||
"""Per-op SFTP timeout from ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS``.
|
||||
|
||||
Default 30s. Picked to comfortably outlast Gainwell's p99
|
||||
listdir_attr (~2s) while still surfacing real hangs inside one
|
||||
scheduler tick. Operators who hit repeated timeouts should drop
|
||||
this — but the right answer is to fix the MFT server, not to
|
||||
paper over it here.
|
||||
"""
|
||||
raw = os.environ.get("CYCLONE_SFTP_OP_TIMEOUT_SECONDS")
|
||||
if not raw:
|
||||
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
|
||||
try:
|
||||
value = float(raw)
|
||||
except ValueError:
|
||||
log.warning(
|
||||
"CYCLONE_SFTP_OP_TIMEOUT_SECONDS=%r is not a float; using default %.1fs",
|
||||
raw, _DEFAULT_SFTP_OP_TIMEOUT_SECONDS,
|
||||
)
|
||||
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
|
||||
if value <= 0:
|
||||
# ``asyncio.wait_for(timeout=0)`` raises immediately, and
|
||||
# ``wait_for(timeout<0)`` is undefined per the asyncio docs.
|
||||
# A zero/negative setting would silently turn every SFTP call
|
||||
# into an instant timeout (a wave of bogus "list_inbound:
|
||||
# timeout" errors). Treat the value as bad and fall back.
|
||||
log.warning(
|
||||
"CYCLONE_SFTP_OP_TIMEOUT_SECONDS=%r must be positive; using default %.1fs",
|
||||
raw, _DEFAULT_SFTP_OP_TIMEOUT_SECONDS,
|
||||
)
|
||||
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
|
||||
return value
|
||||
|
||||
|
||||
@dataclass
|
||||
class InboundFile:
|
||||
"""A single file observed in the inbound MFT path."""
|
||||
@@ -141,89 +87,19 @@ class SftpClient:
|
||||
dir and returns :class:`InboundFile` records pointing at the
|
||||
cache copy. The remote file is *not* deleted — the operator
|
||||
archives inbound files in the MFT UI.
|
||||
|
||||
Gainwell's MFT puts advisory ``*_warn.txt`` files in the same
|
||||
inbound path. They're text-format side-channel notes (not X12
|
||||
envelopes) and are skipped at list time. Use
|
||||
:meth:`list_inbound_names` if you need the raw list without
|
||||
the download.
|
||||
"""
|
||||
if self._stub:
|
||||
return self._list_inbound_stub()
|
||||
return self._list_inbound_paramiko()
|
||||
|
||||
def list_inbound_names(self) -> list[InboundFile]:
|
||||
"""Lightweight listing: returns metadata only, no file download.
|
||||
|
||||
Use this when you want to filter the inbound set (e.g. by date)
|
||||
before paying the download cost. Pair with
|
||||
:meth:`download_inbound` to fetch a filtered subset on demand.
|
||||
|
||||
Real mode is implemented as a single SFTP ``listdir_attr`` call
|
||||
— sub-second on Gainwell's MFT — versus the full
|
||||
:meth:`list_inbound` which downloads every file. The
|
||||
``*_warn.txt`` advisory files are filtered out the same way.
|
||||
|
||||
Stub mode returns the same as :meth:`list_inbound` (the stub
|
||||
only knows about local files; no download cost).
|
||||
"""
|
||||
if self._stub:
|
||||
return self._list_inbound_stub()
|
||||
return self._list_inbound_names_paramiko()
|
||||
|
||||
def download_inbound(self, f: InboundFile) -> Path:
|
||||
"""Download a single inbound file to its ``local_path``.
|
||||
|
||||
Idempotent: if ``f.local_path`` already exists and is
|
||||
non-empty, the download is skipped. Callers should use the
|
||||
``InboundFile`` returned by :meth:`list_inbound_names` and pass
|
||||
it back here — ``local_path`` is the planned cache location
|
||||
and matches the path the scheduler will read from.
|
||||
|
||||
Returns:
|
||||
The on-disk path (same as ``f.local_path``).
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: if the file is missing locally in stub
|
||||
mode, or if the remote file disappears between list
|
||||
and download in real mode.
|
||||
"""
|
||||
if f.local_path.exists() and f.local_path.stat().st_size > 0:
|
||||
log.debug(
|
||||
"SFTP: %s already cached at %s, skipping download",
|
||||
f.name, f.local_path,
|
||||
)
|
||||
return f.local_path
|
||||
if self._stub:
|
||||
# Stub mode: no remote — the file is supposed to already be
|
||||
# at f.local_path (operator-dropped). If it isn't there, the
|
||||
# operator hasn't seeded the stub; raise loudly.
|
||||
if not f.local_path.is_file():
|
||||
raise FileNotFoundError(
|
||||
f"inbound stub file not found: {f.local_path}"
|
||||
)
|
||||
return f.local_path
|
||||
return self._download_inbound_paramiko(f)
|
||||
|
||||
def read_file(self, remote_path: str) -> bytes:
|
||||
"""Read bytes from a remote path.
|
||||
|
||||
Stub mode: reads from ``{staging_dir}/{remote_path}``. Used by
|
||||
the SP16 scheduler so it can exercise the same code path on a
|
||||
workstation without a real MFT connection.
|
||||
"""
|
||||
"""Read bytes from a remote path. Stub raises in stub mode."""
|
||||
if self._stub:
|
||||
return self._read_file_stub(remote_path)
|
||||
raise RuntimeError(
|
||||
"Stub SFTP cannot read remote files. Use the local staging dir."
|
||||
)
|
||||
return self._read_file_paramiko(remote_path)
|
||||
|
||||
def _read_file_stub(self, remote_path: str) -> bytes:
|
||||
"""Read bytes from ``{staging_dir}/{remote_path}`` (SP16 stub)."""
|
||||
staging = Path(self._block.staging_dir).resolve()
|
||||
target = staging / remote_path.lstrip("/")
|
||||
if not target.is_file():
|
||||
raise FileNotFoundError(f"inbound stub file not found: {target}")
|
||||
return target.read_bytes()
|
||||
|
||||
def get_secret(self, name: str) -> Optional[str]:
|
||||
"""Fetch the auth secret from Keychain. Returns the stub secret if absent."""
|
||||
value = secrets.get_secret(name)
|
||||
@@ -232,37 +108,6 @@ class SftpClient:
|
||||
return secrets.STUB_SECRET
|
||||
return value
|
||||
|
||||
# ---- Async surface (SP27 Task 8) -----------------------------------
|
||||
#
|
||||
# Every sync SFTP call has an async wrapper that runs the paramiko
|
||||
# call on a worker thread and applies an ``asyncio.wait_for(...
|
||||
# timeout=N)`` around it. The wait_for cancels the awaiter but
|
||||
# leaves the worker thread running until paramiko returns on its
|
||||
# own (paramiko is not asyncio-aware, so we can't cancel the
|
||||
# underlying socket cleanly). The scheduler should treat
|
||||
# ``asyncio.TimeoutError`` from these wrappers as a transient
|
||||
# SFTP error and surface it in ``Scheduler.status()`` (Task 9).
|
||||
#
|
||||
# The timeout is read on every call (see ``_op_timeout_seconds``)
|
||||
# so an operator who tunes ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` at
|
||||
# runtime sees the new value on the next tick.
|
||||
|
||||
async def async_list_inbound(self) -> list["InboundFile"]:
|
||||
"""Async-wrapped :meth:`list_inbound` with a per-op timeout.
|
||||
|
||||
The 06/25 silent hang was a hung ``listdir_attr`` that froze
|
||||
the worker thread indefinitely. This wrapper applies
|
||||
``asyncio.wait_for(...)`` so the event loop can give up after
|
||||
``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` (default 30s) and the
|
||||
scheduler tick can surface the timeout in
|
||||
``result.errors`` (and, once Task 9 lands, in
|
||||
``Scheduler.status()``).
|
||||
"""
|
||||
return await asyncio.wait_for(
|
||||
asyncio.to_thread(self.list_inbound),
|
||||
timeout=_op_timeout_seconds(),
|
||||
)
|
||||
|
||||
# ---- Stub implementations (SP9) -------------------------------------
|
||||
|
||||
def _write_bytes_stub(self, remote_path: str, content: bytes) -> Path:
|
||||
@@ -428,12 +273,6 @@ class SftpClient:
|
||||
if attr.st_mode and (attr.st_mode & 0o170000) == 0o040000:
|
||||
# Directory entry — skip.
|
||||
continue
|
||||
if attr.filename.endswith("_warn.txt"):
|
||||
# Gainwell's MFT drops text-format advisory notes in
|
||||
# the same inbound path. They're side-channel noise,
|
||||
# not X12 envelopes — skip at list time so we don't
|
||||
# download ~600 advisory files per poll.
|
||||
continue
|
||||
remote = f"{inbound_dir.rstrip('/')}/{attr.filename}"
|
||||
cache_path = cache_dir / attr.filename
|
||||
# Download into cache. We use ``prefetch`` to keep memory
|
||||
@@ -449,56 +288,6 @@ class SftpClient:
|
||||
))
|
||||
return files
|
||||
|
||||
def _list_inbound_names_paramiko(self) -> list[InboundFile]:
|
||||
"""List inbound names via paramiko; do NOT download (lightweight).
|
||||
|
||||
Same ``listdir_attr`` iteration as
|
||||
:meth:`_list_inbound_paramiko`, but the returned
|
||||
:class:`InboundFile` records have ``local_path`` set to the
|
||||
planned cache location without actually fetching the file.
|
||||
Pair with :meth:`_download_inbound_paramiko` to fetch on
|
||||
demand. Skips ``*_warn.txt`` advisory files the same way.
|
||||
"""
|
||||
with self._connect() as (ssh, sftp):
|
||||
inbound_dir = self._block.paths.get("inbound", "/")
|
||||
staging = Path(self._block.staging_dir).resolve()
|
||||
inbound_rel = inbound_dir.lstrip("/")
|
||||
cache_dir = staging / inbound_rel
|
||||
cache_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
files: list[InboundFile] = []
|
||||
try:
|
||||
attrs = sftp.listdir_attr(inbound_dir)
|
||||
except IOError as exc:
|
||||
log.warning("SFTP: cannot list %s: %s", inbound_dir, exc)
|
||||
return []
|
||||
|
||||
for attr in sorted(attrs, key=lambda a: a.filename):
|
||||
if attr.st_mode and (attr.st_mode & 0o170000) == 0o040000:
|
||||
# Directory entry — skip.
|
||||
continue
|
||||
if attr.filename.endswith("_warn.txt"):
|
||||
continue
|
||||
cache_path = cache_dir / attr.filename
|
||||
files.append(InboundFile(
|
||||
name=attr.filename,
|
||||
size=attr.st_size or 0,
|
||||
modified_at=datetime.fromtimestamp(attr.st_mtime or 0),
|
||||
local_path=cache_path,
|
||||
))
|
||||
return files
|
||||
|
||||
def _download_inbound_paramiko(self, f: InboundFile) -> Path:
|
||||
"""Download a single ``f`` to ``f.local_path`` (idempotent on size>0)."""
|
||||
with self._connect() as (ssh, sftp):
|
||||
inbound_dir = self._block.paths.get("inbound", "/")
|
||||
remote = f"{inbound_dir.rstrip('/')}/{f.name}"
|
||||
f.local_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with sftp.open(remote, "rb") as src, open(f.local_path, "wb") as dst:
|
||||
shutil.copyfileobj(src, dst, length=64 * 1024)
|
||||
log.info("SFTP: downloaded %d bytes for %s", f.local_path.stat().st_size, f.name)
|
||||
return f.local_path
|
||||
|
||||
def _read_file_paramiko(self, remote_path: str) -> bytes:
|
||||
with self._connect() as (ssh, sftp):
|
||||
buf = io.BytesIO()
|
||||
|
||||
+3
-999
File diff suppressed because it is too large
Load Diff
@@ -30,7 +30,6 @@ from sqlalchemy import (
|
||||
Numeric,
|
||||
String,
|
||||
Text,
|
||||
func,
|
||||
text,
|
||||
)
|
||||
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship, sessionmaker
|
||||
@@ -249,7 +248,6 @@ class Claim(Base):
|
||||
service_date_to: Mapped[Optional[date]] = mapped_column(Date, nullable=True)
|
||||
charge_amount: Mapped[Decimal] = mapped_column(Numeric(12, 2), nullable=False, default=Decimal("0"))
|
||||
provider_npi: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
rendering_provider_npi: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
payer_id: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
|
||||
state: Mapped[ClaimState] = mapped_column(
|
||||
Enum(ClaimState, native_enum=False), nullable=False, default=ClaimState.SUBMITTED
|
||||
@@ -318,13 +316,6 @@ class Claim(Base):
|
||||
Index("ix_claims_state", "state"),
|
||||
Index("ix_claims_patient_control_number", "patient_control_number"),
|
||||
Index("ix_claims_service_date_from", "service_date_from"),
|
||||
# SP27 Task 11: matched-pair drift check (run at startup)
|
||||
# scans ``WHERE matched_remittance_id IS NOT NULL``. Without
|
||||
# this index it's a full claim scan. The reverse side
|
||||
# (``ix_remittances_claim_id``) is added in 0007. Pure index
|
||||
# (non-unique) — a claim without a match is fine, reversals
|
||||
# leave the previous claim/claim match intact.
|
||||
Index("ix_claims_matched_remittance_id", "matched_remittance_id"),
|
||||
)
|
||||
|
||||
|
||||
@@ -345,7 +336,6 @@ class Remittance(Base):
|
||||
claim_id: Mapped[Optional[str]] = mapped_column(
|
||||
String(64), ForeignKey("claims.id"), nullable=True
|
||||
)
|
||||
rendering_provider_npi: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
status_code: Mapped[str] = mapped_column(String(4), nullable=False)
|
||||
status_label: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
|
||||
total_charge: Mapped[Decimal] = mapped_column(Numeric(12, 2), nullable=False, default=Decimal("0"))
|
||||
@@ -647,76 +637,6 @@ class Two77caAck(Base):
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP28: per-ACK auto-link join table (claim_acks)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ClaimAck(Base):
|
||||
"""One row per AK2 set-response / ClaimStatus / TA1 envelope link.
|
||||
|
||||
SP28. The durable record of "this ACK acknowledges this claim (or
|
||||
set, or batch)". One 999 row carries many AK2s; one 277CA carries
|
||||
many ClaimStatuses; each gets its own ClaimAck row so the operator
|
||||
can answer "which claims does this ack acknowledge?" with a single
|
||||
SELECT on ``claim_acks``.
|
||||
|
||||
See ``docs/superpowers/specs/2026-07-02-cyclone-ack-claim-auto-link-design.md``
|
||||
§3.1 for the schema decisions (per-AK2 granularity, the two-pass
|
||||
join, idempotency via the unique index).
|
||||
"""
|
||||
|
||||
__tablename__ = "claim_acks"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||
# FK to claims.id with ON DELETE CASCADE so removing a claim
|
||||
# drops every link row referencing it. NULLable so TA1 envelope-level
|
||||
# rows can populate ``batch_id`` instead (the table CHECK constraint
|
||||
# requires at least one of the two).
|
||||
claim_id: Mapped[Optional[str]] = mapped_column(
|
||||
String(64), ForeignKey("claims.id", ondelete="CASCADE"), nullable=True,
|
||||
)
|
||||
# FK to batches.id (a Batch row in the 837 case, or the synthetic
|
||||
# inbound-batch id when the ack arrived outside the SFTP pipeline).
|
||||
batch_id: Mapped[Optional[str]] = mapped_column(
|
||||
String(32), ForeignKey("batches.id", ondelete="CASCADE"), nullable=True,
|
||||
)
|
||||
# Discriminated union over acks / ta1_acks / two77ca_acks. No FK
|
||||
# constraint because the three target tables are separate; the
|
||||
# application enforces the discriminator + the matching row's id.
|
||||
ack_id: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
ack_kind: Mapped[str] = mapped_column(String(8), nullable=False)
|
||||
ak2_index: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
|
||||
# The set_control_number the upstream ack ACTUALLY CARRIED
|
||||
# (== source 837 ST02 for Gainwell batches). Preserved on the link
|
||||
# row for orphan traceability — the join may have resolved the
|
||||
# link via the PCN fallback instead, but the operator still sees
|
||||
# the value the 999/277CA originally carried.
|
||||
set_control_number: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
|
||||
# AK5 code (A/E/R/X) for 999, STC category (A1/A2/A3/A4/A6/A7)
|
||||
# for 277CA, envelope ack_code for TA1.
|
||||
set_accept_reject_code: Mapped[Optional[str]] = mapped_column(String(8), nullable=True)
|
||||
linked_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
linked_by: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
|
||||
__table_args__ = (
|
||||
Index("ix_claim_acks_claim_id", "claim_id"),
|
||||
Index("ix_claim_acks_batch_id", "batch_id"),
|
||||
Index("ix_claim_acks_ack", "ack_kind", "ack_id"),
|
||||
# Mirror the dedup unique index declared in 0018_claim_acks.sql so
|
||||
# ``Base.metadata.create_all`` (the test-time safety net) emits the
|
||||
# same partial-unique constraint that the production migration runner
|
||||
# applies. Without this a fresh in-memory test DB would not enforce
|
||||
# idempotency at the DB layer.
|
||||
Index(
|
||||
"ux_claim_acks_dedup",
|
||||
"claim_id", "ack_kind", "ack_id", "ak2_index",
|
||||
unique=True,
|
||||
sqlite_where=text("claim_id IS NOT NULL AND ak2_index IS NOT NULL"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP11: tamper-evident hash-chained audit_log
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -748,11 +668,6 @@ class AuditLog(Base):
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
prev_hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
# SP-auth: which authenticated user performed this action. Nullable
|
||||
# so existing (pre-auth) rows and system-initiated events stay valid.
|
||||
# NOT part of the hash chain — verify_chain must continue to work on
|
||||
# legacy rows that pre-date this column.
|
||||
user_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True, index=True)
|
||||
|
||||
__table_args__ = (
|
||||
Index("idx_audit_log_entity", "entity_type", "entity_id"),
|
||||
@@ -761,89 +676,6 @@ class AuditLog(Base):
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP16: inbound MFT scheduler
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ProcessedInboundFile(Base):
|
||||
"""One row per inbound MFT file the scheduler has downloaded.
|
||||
|
||||
SP16. Lets the scheduler be idempotent: a re-tick or restart must
|
||||
not re-parse the same inbound file. The unique index on
|
||||
(sftp_block_name, name) prevents duplicate inserts and lets the
|
||||
scheduler fast-skip already-processed files via a SELECT.
|
||||
|
||||
Status values:
|
||||
* ok - parsed cleanly, results persisted to the store
|
||||
* error - parser raised; error_message captured
|
||||
* skipped - file_type not in the scheduler's allowed set
|
||||
* pending - file was downloaded but a downstream step failed;
|
||||
the scheduler retries on the next tick
|
||||
"""
|
||||
|
||||
__tablename__ = "processed_inbound_files"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||
sftp_block_name: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
name: Mapped[str] = mapped_column(String(256), nullable=False)
|
||||
size: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
modified_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
file_type: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
||||
processed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
parser_used: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
|
||||
claim_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
status: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
error_message: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
|
||||
|
||||
__table_args__ = (
|
||||
Index(
|
||||
"ux_processed_inbound_files_block_name",
|
||||
"sftp_block_name", "name", unique=True,
|
||||
),
|
||||
Index("ix_processed_inbound_files_processed_at", "processed_at"),
|
||||
Index("ix_processed_inbound_files_status", "status"),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP17: encrypted backup metadata
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class DbBackup(Base):
|
||||
"""One row per encrypted backup the BackupService has taken.
|
||||
|
||||
The actual encrypted blob lives in a directory outside the DB
|
||||
(``~/.local/share/cyclone/backups/`` by default); this table is
|
||||
the index. Status values: ``pending``, ``ok``, ``error``,
|
||||
``pruned``.
|
||||
|
||||
SP17. The unique index on ``(backup_dir, filename)`` makes a
|
||||
duplicate ``create_now()`` race fail cleanly with an
|
||||
IntegrityError instead of clobbering an existing backup.
|
||||
"""
|
||||
|
||||
__tablename__ = "db_backups"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
||||
filename: Mapped[str] = mapped_column(String(128), nullable=False)
|
||||
backup_dir: Mapped[str] = mapped_column(String(512), nullable=False)
|
||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
db_fingerprint: Mapped[Optional[str]] = mapped_column(String(80), nullable=True)
|
||||
table_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||
completed_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
status: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
error_message: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
|
||||
|
||||
__table_args__ = (
|
||||
Index("ux_db_backups_filename", "backup_dir", "filename", unique=True),
|
||||
Index("ix_db_backups_created_at", "created_at"),
|
||||
Index("ix_db_backups_status", "status"),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP9: providers, payers, payer_configs, clearhouse
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -921,27 +753,3 @@ class ClearhouseORM(Base):
|
||||
filename_block_json: Mapped[dict] = mapped_column(JSONText, nullable=False)
|
||||
sftp_block_json: Mapped[dict] = mapped_column(JSONText, nullable=False)
|
||||
updated_at: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||
|
||||
|
||||
class User(Base):
|
||||
"""Auth user (admin / user / viewer)."""
|
||||
|
||||
__tablename__ = "users"
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
username: Mapped[str] = mapped_column(String(64), unique=True, index=True)
|
||||
password_hash: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
role: Mapped[str] = mapped_column(String(16), nullable=False)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
disabled_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
|
||||
class Session(Base):
|
||||
"""Server-side auth session (HttpOnly cookie holds the id)."""
|
||||
|
||||
__tablename__ = "sessions"
|
||||
|
||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), index=True)
|
||||
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
@@ -4,15 +4,12 @@ SP9. Source-of-truth spec:
|
||||
https://hcpf.colorado.gov/tp-x12-filenaming (HCPF X12 File Naming Standards Quick Guide)
|
||||
|
||||
Outbound (we send):
|
||||
tp{tpid}-{transaction_type}-{yyyymmddhhmmssSSS_MT}-1of1.{ext}
|
||||
Example: tp11525703-837P-20260620132243505-1of1.x12
|
||||
{tpid}-{transaction_type}-{yyyymmddhhmmssSSS_MT}-1of1.{ext}
|
||||
Example: 11525703-837P-20260620132243505-1of1.x12
|
||||
|
||||
Inbound (HPE sends to our FromHPE):
|
||||
[Tt][Pp]{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
|
||||
Example: tp11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
||||
(legacy / prodfiles may use uppercase TP; the regex is case-insensitive
|
||||
on the prefix to accept both — Gainwell's filer has used both
|
||||
casings over time.)
|
||||
Inbound (HPE sends to our ToHPE):
|
||||
TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
|
||||
Example: TP11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
||||
|
||||
Both use Mountain Time (MT) timestamps with 17-digit millisecond precision
|
||||
(yyyymmddhhmmssSSS = 4+2+2+2+2+2+3 = 17 digits). Sequence is always "1of1"
|
||||
@@ -31,59 +28,29 @@ from cyclone.providers import InboundFilename
|
||||
# Regexes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Outbound: tp11525703-837P-20260620132243505-1of1.x12
|
||||
# - tp: literal "tp" prefix
|
||||
# Outbound: 11525703-837P-20260620132243505-1of1.x12
|
||||
# - tpid: 1+ digits
|
||||
# - tx: 1+ alnum
|
||||
# - ts: 17 digits (yyyymmddhhmmssSSS)
|
||||
# - seq: literal "1of1"
|
||||
# - ext: 1+ alnum
|
||||
OUTBOUND_RE = re.compile(
|
||||
r"^tp(?P<tpid>\d+)-(?P<tx>[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>[A-Za-z0-9]+)$"
|
||||
r"^(?P<tpid>\d+)-(?P<tx>[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>[A-Za-z0-9]+)$"
|
||||
)
|
||||
|
||||
# Inbound: [Tt][Pp]11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
||||
# - prefix: literal "TP" or "tp" (case-insensitive — Gainwell's
|
||||
# production filer has used both)
|
||||
# Inbound: TP11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
||||
# - tpid: 1+ digits (inside TP<...>)
|
||||
# - orig_tx: 1+ uppercase alnum
|
||||
# - track: M + 1+ uppercase alnum (e.g. M019048402) — the M is
|
||||
# part of the tracking value, not a separator.
|
||||
# - orig_tx: 1+ alnum
|
||||
# - track: M + 1+ alnum (e.g. M019048402) — the M is part of the
|
||||
# tracking value, not a separator.
|
||||
# - ts: 17 digits
|
||||
# - seq: literal "1of1"
|
||||
# - ft: 1+ uppercase alnum (e.g. 999, TA1, 271, 277, 277CA,
|
||||
# 820, 834, 835, ENCR)
|
||||
#
|
||||
# Case insensitivity is scoped to the ``TP`` prefix via ``(?i:TP)``
|
||||
# — the rest of the pattern is case-sensitive so we still reject a
|
||||
# stray ``837p`` or ``m019048402`` (they'd be invalid HCPF).
|
||||
# - ft: 1+ alnum (e.g. 999, TA1, 271, 277, 277CA, 820, 834, 835, ENCR)
|
||||
INBOUND_RE = re.compile(
|
||||
r"^(?i:TP)(?P<tpid>\d+)-(?P<orig_tx>[A-Z0-9]+)_(?P<tracking>M[A-Z0-9]+)"
|
||||
r"^TP(?P<tpid>\d+)-(?P<orig_tx>[A-Z0-9]+)_(?P<tracking>M[A-Z0-9]+)"
|
||||
r"-(?P<ts>\d{17})-1of1_(?P<file_type>[A-Z0-9]+)\.(?P<ext>x12)$"
|
||||
)
|
||||
|
||||
# Inbound suffix-less form (SP27 Task 7): tp11525703-835_M019110219-20260525001606050-1of1.x12
|
||||
# Gainwell's production filer has shipped this shorter form in addition
|
||||
# to the spec form above — the 6/15-6/19 835 batch arrived this way and
|
||||
# was silently dropped by the strict INBOUND_RE. The loose form omits
|
||||
# the `_{file_type}.x12` suffix; the token between `-` and `_M` doubles
|
||||
# as both `orig_tx` and `file_type` (since there's no separate suffix
|
||||
# to disambiguate). Allowed values still must be in ALLOWED_FILE_TYPES
|
||||
# — the suffix is optional, the type-set is not.
|
||||
# - prefix: literal "TP" or "tp" (case-insensitive — same as INBOUND_RE)
|
||||
# - tpid: 1+ digits
|
||||
# - file_type: 3-5 char alphanumeric (covers 999, TA1, 835, 277CA, ENCR;
|
||||
# capped at 5 to avoid swallowing the next `_M` token)
|
||||
# - tracking: M + 1+ uppercase alnum
|
||||
# - ts: 17 digits
|
||||
# - seq: literal "1of1"
|
||||
# - ext: literal "x12" (relaxing this would also relax the strict
|
||||
# form's contract; out of scope for SP27 Task 7)
|
||||
INBOUND_RE_LOOSE = re.compile(
|
||||
r"^(?i:TP)(?P<tpid>\d+)-(?P<file_type>[A-Z0-9]{3,5})"
|
||||
r"_(?P<tracking>M[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>x12)$"
|
||||
)
|
||||
|
||||
ALLOWED_FILE_TYPES = frozenset({
|
||||
"999", "TA1", "270", "271", "276", "277", "277CA", "278",
|
||||
"820", "834", "835", "ENCR",
|
||||
@@ -112,8 +79,7 @@ def build_outbound_filename(
|
||||
time in ``America/Denver`` is used.
|
||||
|
||||
Returns:
|
||||
Filename like "tp11525703-837P-20260620132243505-1of1.x12"
|
||||
(note the ``tp`` prefix per HCPF outbound spec).
|
||||
Filename like "11525703-837P-20260620132243505-1of1.x12"
|
||||
|
||||
Raises:
|
||||
ValueError: If tpid is non-numeric, tx contains invalid chars, or
|
||||
@@ -133,10 +99,7 @@ def build_outbound_filename(
|
||||
# Format: yyyymmddhhmmssSSS — 17 digits total
|
||||
ts = now_mt.strftime("%Y%m%d%H%M%S") + f"{now_mt.microsecond // 1000:03d}"
|
||||
assert len(ts) == 17
|
||||
# Per HCPF outbound spec, prefix is "tp" + tpid. Matches the format
|
||||
# we receive from HPE inbound (which uses uppercase TP) and the
|
||||
# historical outbound prodfile naming (e.g. tp11525703-837P-...).
|
||||
return f"tp{tpid}-{tx}-{ts}-1of1.{ext}"
|
||||
return f"{tpid}-{tx}-{ts}-1of1.{ext}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -147,49 +110,16 @@ def build_outbound_filename(
|
||||
def parse_inbound_filename(name: str) -> InboundFilename:
|
||||
"""Parse an inbound HCPF filename.
|
||||
|
||||
Accepts both forms (Gainwell ships both):
|
||||
* Spec form with ``_{file_type}.x12`` suffix:
|
||||
``tp11525703-837P_M019048402-20260520231513488-1of1_999.x12``
|
||||
* Suffix-less form (SP27 Task 7): the token between ``-`` and
|
||||
``_M`` doubles as both ``orig_tx`` and ``file_type``:
|
||||
``tp11525703-835_M019110219-20260525001606050-1of1.x12``
|
||||
|
||||
The strict form is tried first (preserves historical behavior for
|
||||
every existing caller); the loose form is the fallback. The
|
||||
``.x12`` extension and ``ALLOWED_FILE_TYPES`` set are enforced in
|
||||
both forms.
|
||||
|
||||
Args:
|
||||
name: Filename like ``tp11525703-837P_M019048402-20260520231513488-1of1_999.x12``
|
||||
(case-insensitive on the ``TP`` prefix; both ``TP`` and
|
||||
``tp`` are accepted.)
|
||||
name: Filename like "TP11525703-837P_M019048402-20260520231513488-1of1_999.x12"
|
||||
|
||||
Returns:
|
||||
InboundFilename with tpid, orig_tx, tracking, ts, file_type, ext.
|
||||
|
||||
Raises:
|
||||
ValueError: If the filename doesn't match either HCPF inbound
|
||||
form, or if the derived file_type isn't in
|
||||
ALLOWED_FILE_TYPES.
|
||||
ValueError: If the filename doesn't match the HCPF inbound format.
|
||||
"""
|
||||
m = INBOUND_RE.match(name)
|
||||
if m:
|
||||
file_type = m.group("file_type")
|
||||
if file_type not in ALLOWED_FILE_TYPES:
|
||||
raise ValueError(
|
||||
f"file_type {file_type!r} not in allowed HCPF set: {sorted(ALLOWED_FILE_TYPES)}"
|
||||
)
|
||||
return InboundFilename(
|
||||
tpid=m.group("tpid"),
|
||||
orig_tx=m.group("orig_tx"),
|
||||
tracking=m.group("tracking"),
|
||||
ts=m.group("ts"),
|
||||
file_type=file_type,
|
||||
ext=m.group("ext"),
|
||||
)
|
||||
|
||||
# Fall back to the suffix-less form.
|
||||
m = INBOUND_RE_LOOSE.match(name)
|
||||
if not m:
|
||||
raise ValueError(f"Not a valid HCPF inbound filename: {name!r}")
|
||||
file_type = m.group("file_type")
|
||||
@@ -199,7 +129,7 @@ def parse_inbound_filename(name: str) -> InboundFilename:
|
||||
)
|
||||
return InboundFilename(
|
||||
tpid=m.group("tpid"),
|
||||
orig_tx=m.group("file_type"), # preserve the historical shape
|
||||
orig_tx=m.group("orig_tx"),
|
||||
tracking=m.group("tracking"),
|
||||
ts=m.group("ts"),
|
||||
file_type=file_type,
|
||||
@@ -218,12 +148,5 @@ def is_outbound_filename(name: str) -> bool:
|
||||
|
||||
|
||||
def is_inbound_filename(name: str) -> bool:
|
||||
"""True if the given string matches either HCPF inbound form.
|
||||
|
||||
Accepts both the spec form (with ``_{file_type}.x12`` suffix) and
|
||||
the suffix-less form (SP27 Task 7). Cheap fast-path check used by
|
||||
callers that want to pre-filter a directory listing before invoking
|
||||
:func:`parse_inbound_filename`; mirrors the parser's own fallback
|
||||
so the two never disagree on what counts as an HCPF inbound file.
|
||||
"""
|
||||
return INBOUND_RE.match(name) is not None or INBOUND_RE_LOOSE.match(name) is not None
|
||||
"""True if the given string matches the HCPF inbound filename regex."""
|
||||
return INBOUND_RE.match(name) is not None
|
||||
|
||||
@@ -1,113 +0,0 @@
|
||||
"""File-type handlers for inbound MFT files (SP27).
|
||||
|
||||
Each handler is a pure function that parses + persists + dispatches
|
||||
events for one file type. The scheduler and the FastAPI endpoints
|
||||
both delegate here; the ``HANDLERS`` registry maps ``file_type`` →
|
||||
handler function.
|
||||
|
||||
Public API:
|
||||
HandleResult — dataclass returned by every handler
|
||||
HANDLERS — ``{"999": handle_999, "835": handle_835, ...}``
|
||||
handle_999, handle_ta1, handle_277ca, handle_835
|
||||
— call signatures: ``handle(text: str, source_file: str) -> HandleResult``
|
||||
|
||||
The handlers own their own DB session lifecycle. They emit pubsub
|
||||
events via the optional ``event_bus`` parameter (the FastAPI endpoint
|
||||
injects ``app.state.event_bus``; the scheduler passes ``None``).
|
||||
They never raise on per-segment problems; per-segment issues are
|
||||
logged and folded into the result. Whole-document failures
|
||||
(missing ISA, bad encoding) surface as ``CycloneParseError``, which
|
||||
the caller catches and records as ``STATUS_ERROR``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import logging
|
||||
|
||||
from ._ack_id import (
|
||||
ack_count_summary,
|
||||
ack_synthetic_source_batch_id,
|
||||
two77ca_synthetic_source_batch_id,
|
||||
)
|
||||
from .handle_result import HandleResult
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# Module-level HANDLERS dict populated lazily once handler modules
|
||||
# ship. Keys are file_type strings, values are the ``handle``
|
||||
# callable for that type.
|
||||
HANDLERS: dict[str, object] = {}
|
||||
|
||||
|
||||
# Candidate handlers, in registration order. Each tuple is
|
||||
# (module-path, file_type). The 277/277CA mapping is added explicitly
|
||||
# after registration when the 277CA handler is present.
|
||||
_CANDIDATES: list[tuple[str, str]] = [
|
||||
("cyclone.handlers.handle_999", "999"),
|
||||
("cyclone.handlers.handle_ta1", "TA1"),
|
||||
("cyclone.handlers.handle_277ca", "277CA"),
|
||||
("cyclone.handlers.handle_835", "835"),
|
||||
]
|
||||
|
||||
|
||||
def register_handlers() -> None:
|
||||
"""Populate ``HANDLERS`` from the per-type handler modules.
|
||||
|
||||
Tolerates missing or broken handler modules so the package can be
|
||||
imported incrementally as each handler ships (Tasks 2-5), and so
|
||||
a partial-modification error in one handler module can't break
|
||||
scheduler / API import. Safe to call multiple times.
|
||||
"""
|
||||
if HANDLERS:
|
||||
return
|
||||
for mod_path, file_type in _CANDIDATES:
|
||||
try:
|
||||
mod = importlib.import_module(mod_path)
|
||||
fn = getattr(mod, "handle")
|
||||
except Exception as exc: # noqa: BLE001 — best-effort registry
|
||||
log.debug(
|
||||
"handler %s unavailable: %s",
|
||||
mod_path, exc,
|
||||
)
|
||||
continue
|
||||
HANDLERS[file_type] = fn
|
||||
# The 277 filename maps to the same 277CA handler.
|
||||
if file_type == "277CA":
|
||||
HANDLERS["277"] = fn
|
||||
|
||||
|
||||
register_handlers()
|
||||
|
||||
|
||||
# Re-export handler functions on this package so callers can use the
|
||||
# flat import (``from cyclone.handlers import handle_999``) once each
|
||||
# module ships. Set after registration so we know what's present.
|
||||
def _reexport_handlers() -> None:
|
||||
"""Re-export each handler's ``handle`` fn as ``handle_<type>``.
|
||||
|
||||
No-op for absent handlers. Re-run on each import so freshly-
|
||||
installed handler modules (e.g. Tasks 2-5 commits) are visible
|
||||
after ``register_handlers()`` without a process restart.
|
||||
"""
|
||||
for file_type, fn in list(HANDLERS.items()):
|
||||
if file_type in ("277",): # alias of 277CA; don't re-export twice
|
||||
continue
|
||||
globals()[f"handle_{file_type.lower()}"] = fn
|
||||
|
||||
|
||||
_reexport_handlers()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"HANDLERS",
|
||||
"HandleResult",
|
||||
"register_handlers",
|
||||
"ack_count_summary",
|
||||
"ack_synthetic_source_batch_id",
|
||||
"two77ca_synthetic_source_batch_id",
|
||||
# handler_* names are added at module load via _reexport_handlers
|
||||
]
|
||||
# Populate __all__ with the present handler symbols.
|
||||
for _h in ("handle_999", "handle_ta1", "handle_277ca", "handle_835"):
|
||||
if _h in globals():
|
||||
__all__.append(_h) # noqa: PYI056
|
||||
@@ -1,87 +0,0 @@
|
||||
"""Ack ID helpers shared between the scheduler and the FastAPI
|
||||
endpoints (SP27 Task 1).
|
||||
|
||||
Lifted from ``scheduler.py:_ack_count_summary`` +
|
||||
``_ack_synthetic_source_batch_id`` + ``_277ca_synthetic_source_batch_id`` —
|
||||
all three had inline copies in both ``scheduler.py`` and ``api.py``
|
||||
prior to this SP. Both callers now import from this module.
|
||||
|
||||
Helpers
|
||||
-------
|
||||
``ack_count_summary(result)``
|
||||
Aggregate ``(received, accepted, rejected, ack_code)`` from a
|
||||
parsed ``ParseResult999``, trusting set-level IK5 over the
|
||||
functional-group AK9. Gainwell's MFT ships AK9 segments that
|
||||
contradict the per-set IK5 (e.g. ``AK9*A*1*1*1`` with
|
||||
``IK5*A``), so trusting AK9's rejected count over-reports.
|
||||
See scheduler commit ``6507a8c`` for the operational context.
|
||||
|
||||
``ack_synthetic_source_batch_id(icn, *, pcn, source_filename)``
|
||||
Build a unique-per-file ``batches.id`` for a 999 that ships
|
||||
without its own source batch. Falls back to a hash suffix of
|
||||
the filename so daily pulls don't all collapse onto the same
|
||||
ICN. Gainwell's MFT ships every 999 with the default ICN
|
||||
``000000001`` (per the scheduler docstring).
|
||||
|
||||
``two77ca_synthetic_source_batch_id(icn)``
|
||||
Same idea for a 277CA without its own source batch.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from typing import Any
|
||||
|
||||
|
||||
def ack_count_summary(result: Any) -> tuple[int, int, int, str]:
|
||||
"""Aggregate ``(received, accepted, rejected, ack_code)`` from
|
||||
a ``ParseResult999``.
|
||||
|
||||
Counts are derived from the set-level ``IK5`` responses
|
||||
(one per ``AK2`` in the 999), not the functional-group ``AK9``.
|
||||
Gainwell's MFT ships contradictory AK9 segments; the per-set
|
||||
IK5 is the authoritative per-claim accept/reject signal.
|
||||
"""
|
||||
sets = result.set_responses
|
||||
received = len(sets)
|
||||
accepted = sum(1 for s in sets if s.set_accept_reject.code == "A")
|
||||
rejected = received - accepted
|
||||
if rejected == 0:
|
||||
code = "A"
|
||||
elif accepted == 0:
|
||||
code = "R"
|
||||
else:
|
||||
code = "P"
|
||||
return (received, accepted, rejected, code)
|
||||
|
||||
|
||||
def ack_synthetic_source_batch_id(
|
||||
interchange_control_number: str,
|
||||
*,
|
||||
pcn: str | None = None,
|
||||
source_filename: str | None = None,
|
||||
) -> str:
|
||||
"""Synthetic ``batches.id`` for a received 999 with no source batch.
|
||||
|
||||
Precedence for the human-readable part of the id (column is
|
||||
``VARCHAR(32)``):
|
||||
|
||||
1. ``999-{pcn}-{hash8}`` if ``AK2`` set_control_number is
|
||||
present (the common case). 4 + 9 + 1 + 8 = 22 chars max.
|
||||
2. ``999-{icn}-{hash8}`` if no AK2 (envelope-only 999).
|
||||
3. ``999-{hash12}`` if no filename either (shouldn't happen
|
||||
in production).
|
||||
"""
|
||||
short_hash = ""
|
||||
if source_filename:
|
||||
short_hash = hashlib.sha1(source_filename.encode("utf-8")).hexdigest()[:8]
|
||||
if pcn and pcn.strip():
|
||||
return f"999-{pcn.strip()}-{short_hash}"
|
||||
icn = (interchange_control_number or "").strip() or "000000001"
|
||||
if short_hash:
|
||||
return f"999-{icn}-{short_hash}"
|
||||
return f"999-{short_hash or icn}"
|
||||
|
||||
|
||||
def two77ca_synthetic_source_batch_id(interchange_control_number: str) -> str:
|
||||
"""Synthetic ``batches.id`` for a received 277CA with no source batch."""
|
||||
return f"277CA-{(interchange_control_number or '').strip() or '000000001'}"
|
||||
@@ -1,152 +0,0 @@
|
||||
"""Handle a 277CA Claim Acknowledgment file (SP27 Task 4).
|
||||
|
||||
Lifted verbatim from ``scheduler.py:_handle_277ca``. The handler owns
|
||||
its own DB session, dispatches to ``parse_277ca_text``, persists the
|
||||
277CA ack row, applies 277CA rejections to matched claims via
|
||||
``inbox_state.apply_277ca_rejections``, and emits
|
||||
``claim.payer_rejected`` audit events for each newly-stamped claim.
|
||||
|
||||
The actor tag (``"277ca-parser-scheduler"``) is preserved so the
|
||||
audit log keeps tracing back to the same source after extraction.
|
||||
Both the FastAPI endpoint and the scheduler path (re)use this module
|
||||
— the API migration drops the inline copy in Task 6.
|
||||
|
||||
``claim.rejected_after_remit`` audit emission (when a 277CA rejects
|
||||
a claim that already has ``matched_remittance_id`` set) is deferred
|
||||
to SP27 Task 13. Today the handler only emits ``claim.payer_rejected``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.audit_log import AuditEvent, append_event
|
||||
from cyclone.claim_acks import apply_277ca_acks
|
||||
from cyclone.handlers._ack_id import two77ca_synthetic_source_batch_id
|
||||
from cyclone.inbox_state_277ca import apply_277ca_rejections
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
from cyclone.parsers.parse_277ca import parse_277ca_text
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def handle(
|
||||
text: str,
|
||||
source_file: str,
|
||||
) -> tuple[str, int]:
|
||||
"""Parse a 277CA, persist ack + stamp payer-rejected claims.
|
||||
|
||||
Args:
|
||||
text: Raw 277CA document bytes (decoded).
|
||||
source_file: Filename the 277CA came from.
|
||||
|
||||
Returns:
|
||||
``(parser_used, claim_count)`` tuple where ``claim_count`` is
|
||||
the number of STC statuses in the file (one per claim).
|
||||
|
||||
Raises:
|
||||
ValueError: on parser-level failure (wraps CycloneParseError).
|
||||
|
||||
SP25: the store (``cyclone.store.acks.add_277ca_ack``) owns the
|
||||
publish-from-store contract; the handler no longer needs an
|
||||
``event_bus`` kwarg.
|
||||
"""
|
||||
try:
|
||||
result = parse_277ca_text(text, input_file=source_file)
|
||||
except CycloneParseError as exc:
|
||||
raise ValueError(f"277CA parse error: {exc}") from exc
|
||||
|
||||
icn = result.envelope.control_number
|
||||
synthetic_id = two77ca_synthetic_source_batch_id(icn)
|
||||
accepted = sum(
|
||||
1 for s in result.claim_statuses if s.classification == "accepted"
|
||||
)
|
||||
paid = sum(
|
||||
1 for s in result.claim_statuses if s.classification == "paid"
|
||||
)
|
||||
rejected = sum(
|
||||
1 for s in result.claim_statuses if s.classification == "rejected"
|
||||
)
|
||||
pended = sum(
|
||||
1 for s in result.claim_statuses if s.classification == "pended"
|
||||
)
|
||||
|
||||
# Build the batch envelope index BEFORE opening the work session
|
||||
# — cycl_store.batch_envelope_index opens its own short-lived
|
||||
# session, and SQLite + concurrent sessions causes "database is
|
||||
# locked" errors.
|
||||
batch_index = cycl_store.batch_envelope_index()
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
row = cycl_store.add_277ca_ack(
|
||||
source_batch_id=synthetic_id,
|
||||
control_number=icn,
|
||||
accepted_count=accepted,
|
||||
rejected_count=rejected,
|
||||
paid_count=paid,
|
||||
pended_count=pended,
|
||||
raw_json=json.loads(result.model_dump_json()),
|
||||
)
|
||||
def _lookup(pcn: str):
|
||||
return (
|
||||
session.query(db.Claim)
|
||||
.filter(db.Claim.patient_control_number == pcn)
|
||||
.first()
|
||||
)
|
||||
apply_result = apply_277ca_rejections(
|
||||
session, result, claim_lookup=_lookup, two77ca_id=row.id,
|
||||
)
|
||||
if apply_result.matched:
|
||||
for cid in apply_result.matched:
|
||||
append_event(session, AuditEvent(
|
||||
event_type="claim.payer_rejected",
|
||||
entity_type="claim",
|
||||
entity_id=cid,
|
||||
payload={"source_batch_id": synthetic_id, "277ca_id": row.id},
|
||||
actor="277ca-parser-scheduler",
|
||||
))
|
||||
# SP28: auto-link the 277CA ClaimStatus entries to claims (D10
|
||||
# two-pass join). The helper builds dataclass rows; the
|
||||
# caller persists each via cycl_store.add_claim_ack so the
|
||||
# publish-from-store contract owns the live-tail event.
|
||||
def _pcn_lookup(pcn: str):
|
||||
return (
|
||||
session.query(db.Claim)
|
||||
.filter(db.Claim.patient_control_number == pcn)
|
||||
.first()
|
||||
)
|
||||
|
||||
link_result = apply_277ca_acks(
|
||||
session, result, ack_id=row.id,
|
||||
batch_envelope_index=batch_index,
|
||||
pc_claim_lookup=_pcn_lookup,
|
||||
)
|
||||
# Snapshot the rows before closing the work session — SQLite
|
||||
# + concurrent sessions from the same thread cause "database
|
||||
# is locked" errors when add_claim_ack opens its own session
|
||||
# while this one is still open.
|
||||
link_rows = list(link_result.linked)
|
||||
orphans = list(link_result.orphans)
|
||||
session.commit()
|
||||
|
||||
for link_row in link_rows:
|
||||
cycl_store.add_claim_ack(
|
||||
claim_id=link_row.claim_id,
|
||||
batch_id=link_row.batch_id,
|
||||
ack_id=row.id,
|
||||
ack_kind="277ca",
|
||||
ak2_index=link_row.ak2_index,
|
||||
set_control_number=link_row.set_control_number,
|
||||
set_accept_reject_code=link_row.set_accept_reject_code,
|
||||
linked_by="auto",
|
||||
)
|
||||
if orphans:
|
||||
log.warning(
|
||||
"277CA had %d orphan status entries (no matching claim): %s",
|
||||
len(orphans),
|
||||
orphans[:5],
|
||||
)
|
||||
|
||||
return ("parse_277ca", len(result.claim_statuses))
|
||||
@@ -1,107 +0,0 @@
|
||||
"""Handle an 835 ERA Remittance file (SP27 Task 5).
|
||||
|
||||
Lifted verbatim from ``scheduler.py:_handle_835``. The handler owns
|
||||
its own state, dispatches to ``parse_835`` (raises
|
||||
``CycloneParseError`` on bad EDI), runs the per-payer 835 validator,
|
||||
stamps the validation report into ``result.summary``, and persists
|
||||
the BatchRecord via ``cycl_store.add``.
|
||||
|
||||
The 835 handler is the largest of the four because the schema
|
||||
covers per-claim remittances + CAS adjustments + validation.
|
||||
``PAYER_FACTORIES_835`` lives in ``cyclone.api`` (it was always
|
||||
intended to live in ``cyclone.payers`` — a TODO pre-dating SP27 —
|
||||
but moving it is out of Task 5 scope). We import it lazily inside
|
||||
``handle`` to avoid a module-load-time cycle; the cyclic risk is
|
||||
acceptable because api.py also imports scheduler lazily (inside its
|
||||
lifespan handler).
|
||||
|
||||
Two-phase ingest closed in SP27 Task 10: the placeholder
|
||||
``adjustment_amount`` is overwritten by reconcile in the same DB
|
||||
session as the insert (before commit), so a reader never sees a
|
||||
half-reconciled Remittance row. If reconcile raises, the whole
|
||||
ingest rolls back and the scheduler records the per-file error.
|
||||
|
||||
``event_bus`` is best-effort and follows the same TODO(sp27-task-6)
|
||||
sync/async gap as handle_999 / handle_ta1 / handle_277ca.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
from cyclone.parsers.parse_835 import parse as parse_835
|
||||
from cyclone.parsers.validator_835 import validate as validate_835
|
||||
from cyclone.store import BatchRecord
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def handle(
|
||||
text: str,
|
||||
source_file: str,
|
||||
) -> tuple[str, int]:
|
||||
"""Parse an 835, run validation, persist batch + remittances.
|
||||
|
||||
Args:
|
||||
text: Raw 835 document bytes (decoded).
|
||||
source_file: Filename the 835 came from.
|
||||
|
||||
Returns:
|
||||
``(parser_used, claim_count)`` tuple. ``claim_count`` is the
|
||||
number of CLP claims parsed; the BatchRecord's summary's
|
||||
``passed`` / ``failed`` fields are derived from the
|
||||
validator's ``report.passed`` flag.
|
||||
|
||||
Raises:
|
||||
ValueError: on parser-level failure (wraps CycloneParseError).
|
||||
Validation failures don't raise — they're stamped into
|
||||
``summary.passed = 0``.
|
||||
|
||||
SP25: the 835 write path already publishes ``remittance_written``
|
||||
via ``CycloneStore.add`` (SP21 split). The handler no longer
|
||||
accepts an ``event_bus`` kwarg — the ``ack_received`` publish
|
||||
here was dead code anyway (the 835 event name is
|
||||
``remittance_written``, not ``ack_received``).
|
||||
"""
|
||||
# TODO(sp27-pre-t5): move PAYER_FACTORIES_835 out of api.py into
|
||||
# ``cyclone.payers`` to remove the lazy cyclic import below. The
|
||||
# import works today because api.py also imports scheduler lazily.
|
||||
from cyclone.api import PAYER_FACTORIES_835
|
||||
|
||||
config = PAYER_FACTORIES_835["co_medicaid_835"]()
|
||||
try:
|
||||
result = parse_835(text, config, input_file=source_file)
|
||||
except CycloneParseError as exc:
|
||||
raise ValueError(f"835 parse error: {exc}") from exc
|
||||
|
||||
# Validation report (mirrors the API endpoint).
|
||||
report = validate_835(result, config)
|
||||
n = len(result.claims)
|
||||
if report.passed:
|
||||
passed, failed, failed_claim_ids = n, 0, []
|
||||
else:
|
||||
passed, failed, failed_claim_ids = 0, n, [
|
||||
c.payer_claim_control_number for c in result.claims
|
||||
]
|
||||
result = result.model_copy(update={
|
||||
"validation": report,
|
||||
"summary": result.summary.model_copy(update={
|
||||
"passed": passed,
|
||||
"failed": failed,
|
||||
"failed_claim_ids": failed_claim_ids,
|
||||
}),
|
||||
})
|
||||
|
||||
rec = BatchRecord(
|
||||
id=uuid.uuid4().hex,
|
||||
kind="835",
|
||||
input_filename=source_file,
|
||||
parsed_at=datetime.now(timezone.utc),
|
||||
result=result,
|
||||
)
|
||||
cycl_store.add(rec)
|
||||
|
||||
return ("parse_835", len(result.claims))
|
||||
@@ -1,157 +0,0 @@
|
||||
"""Handle a 999 Implementation Acknowledgment file (SP27 Task 2).
|
||||
|
||||
Lifted verbatim from ``scheduler.py:_handle_999``. The handler owns
|
||||
its own DB session, dispatches to ``parse_999_text``, applies 999
|
||||
rejections to any matched claims via ``inbox_state.apply_999_rejections``,
|
||||
persists the ack row, and returns a ``(parser_used, claim_count)``
|
||||
tuple.
|
||||
|
||||
The actor tag (``"999-parser-scheduler"``) is preserved so the audit
|
||||
log keeps tracing back to the same source after extraction. Both
|
||||
the FastAPI endpoint and the scheduler path (re)use this module —
|
||||
see Task 6 for the API migration that drops the inline copy.
|
||||
|
||||
``claim_count`` mirrors ``parsed.received`` — the count of AK2
|
||||
(set-level) responses in the 999, which is the authoritative
|
||||
per-claim accept/reject signal (see ``_ack_id`` for why AK9 is
|
||||
ignored).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.audit_log import AuditEvent, append_event
|
||||
from cyclone.claim_acks import apply_999_acceptances
|
||||
from cyclone.handlers._ack_id import (
|
||||
ack_count_summary,
|
||||
ack_synthetic_source_batch_id,
|
||||
)
|
||||
from cyclone.inbox_state import apply_999_rejections
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
from cyclone.parsers.parse_999 import parse_999_text
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def handle(
|
||||
text: str,
|
||||
source_file: str,
|
||||
) -> tuple[str, int]:
|
||||
"""Parse a 999, apply rejections, persist ack row.
|
||||
|
||||
Args:
|
||||
text: Raw 999 document bytes (decoded).
|
||||
source_file: Filename the 999 came from. Used to derive a
|
||||
unique synthetic ``batches.id`` (see ``_ack_id``).
|
||||
|
||||
Returns:
|
||||
``(parser_used, claim_count)`` tuple. Matches the scheduler
|
||||
``_download_and_parse`` destructure; the HandleResult
|
||||
migration happens in Task 7.
|
||||
|
||||
Raises:
|
||||
ValueError: on parser-level failure (wraps CycloneParseError).
|
||||
|
||||
SP25: the store (``cyclone.store.acks.add_999_ack``) now owns the
|
||||
publish-from-store contract; the handler no longer needs an
|
||||
``event_bus`` kwarg. Both the scheduler path (no bus) and the
|
||||
FastAPI endpoint path (passes the bus to the store directly) see
|
||||
the same row, the same event, and the same payload.
|
||||
"""
|
||||
try:
|
||||
result = parse_999_text(text, input_file=source_file)
|
||||
except CycloneParseError as exc:
|
||||
raise ValueError(f"999 parse error: {exc}") from exc
|
||||
|
||||
received, accepted, rejected, ack_code = ack_count_summary(result)
|
||||
icn = result.envelope.control_number
|
||||
pcn = (
|
||||
result.set_responses[0].set_control_number
|
||||
if result.set_responses else None
|
||||
)
|
||||
synthetic_id = ack_synthetic_source_batch_id(
|
||||
icn, pcn=pcn, source_filename=source_file,
|
||||
)
|
||||
|
||||
# Build the batch envelope index BEFORE opening the work session
|
||||
# — cycl_store.batch_envelope_index opens its own short-lived
|
||||
# session, and SQLite + concurrent sessions causes "database is
|
||||
# locked" errors.
|
||||
batch_index = cycl_store.batch_envelope_index()
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
def _lookup(pcn: str):
|
||||
return (
|
||||
session.query(db.Claim)
|
||||
.filter_by(patient_control_number=pcn)
|
||||
.first()
|
||||
)
|
||||
rejection_result = apply_999_rejections(
|
||||
session, result,
|
||||
claim_lookup=_lookup,
|
||||
batch_envelope_index=batch_index,
|
||||
)
|
||||
if rejection_result.matched:
|
||||
for cid in rejection_result.matched:
|
||||
append_event(session, AuditEvent(
|
||||
event_type="claim.rejected",
|
||||
entity_type="claim",
|
||||
entity_id=cid,
|
||||
payload={"source_batch_id": synthetic_id},
|
||||
actor="999-parser-scheduler",
|
||||
))
|
||||
row = cycl_store.add_ack(
|
||||
source_batch_id=synthetic_id,
|
||||
accepted_count=accepted,
|
||||
rejected_count=rejected,
|
||||
received_count=received,
|
||||
ack_code=ack_code,
|
||||
raw_json=json.loads(result.model_dump_json()),
|
||||
)
|
||||
# SP28: auto-link the 999 AK2 set-responses to claims (D10 two-pass
|
||||
# join). The PCN fallback only fires when the ST02 lookup misses
|
||||
# (rare for Gainwell batches). Each created ClaimAck row is
|
||||
# persisted via cycl_store.add_claim_ack so the publish-from-store
|
||||
# contract owns the live-tail event.
|
||||
def _pcn_lookup(pcn: str):
|
||||
return (
|
||||
session.query(db.Claim)
|
||||
.filter_by(patient_control_number=pcn)
|
||||
.first()
|
||||
)
|
||||
|
||||
link_result = apply_999_acceptances(
|
||||
session, result, ack_id=row.id,
|
||||
batch_envelope_index=batch_index,
|
||||
pc_claim_lookup=_pcn_lookup,
|
||||
)
|
||||
# Snapshot the rows before closing the work session — SQLite
|
||||
# + concurrent sessions from the same thread cause "database
|
||||
# is locked" errors when add_claim_ack opens its own session
|
||||
# while this one is still open.
|
||||
link_rows = list(link_result.linked)
|
||||
orphans = list(link_result.orphans)
|
||||
session.commit()
|
||||
|
||||
for link_row in link_rows:
|
||||
cycl_store.add_claim_ack(
|
||||
claim_id=link_row.claim_id,
|
||||
batch_id=link_row.batch_id,
|
||||
ack_id=row.id,
|
||||
ack_kind="999",
|
||||
ak2_index=link_row.ak2_index,
|
||||
set_control_number=link_row.set_control_number,
|
||||
set_accept_reject_code=link_row.set_accept_reject_code,
|
||||
linked_by="auto",
|
||||
)
|
||||
if orphans:
|
||||
log.warning(
|
||||
"999 had %d orphan set refs (no matching claim): %s",
|
||||
len(orphans),
|
||||
orphans[:5],
|
||||
)
|
||||
|
||||
return ("parse_999", received)
|
||||
@@ -1,37 +0,0 @@
|
||||
"""Shared ``HandleResult`` dataclass for handlers (SP27).
|
||||
|
||||
Every per-file-type handler returns the same shape so the
|
||||
scheduler's ``_handle_one`` and the FastAPI parse endpoints can
|
||||
process them uniformly.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class HandleResult:
|
||||
"""Outcome of one handler invocation.
|
||||
|
||||
Attributes:
|
||||
parser_used: The parser name written to
|
||||
``processed_inbound_files.parser_used`` (e.g. ``"parse_999"``)
|
||||
and surfaced in the UI.
|
||||
claim_count: The number of claim rows persisted (or batch
|
||||
records, depending on the file type). For 999/TA1 this
|
||||
is the receipt count; for 835 this is the per-claim
|
||||
remittance count; for 277CA this is the per-claim
|
||||
status count.
|
||||
batch_id: The persisted batch id (when the handler creates a
|
||||
row in ``batches``). ``None`` for handlers that persist
|
||||
into per-file-type ack tables (999/TA1/277CA) rather
|
||||
than into the unified ``batches`` table.
|
||||
matched_count: For 835, the number of remits that were
|
||||
matched to an existing claim by ``reconcile.match``.
|
||||
Zero for other handlers.
|
||||
"""
|
||||
|
||||
parser_used: str
|
||||
claim_count: int
|
||||
batch_id: str | None = None
|
||||
matched_count: int = 0
|
||||
@@ -1,123 +0,0 @@
|
||||
"""Handle a TA1 Interchange Acknowledgment file (SP27 Task 3).
|
||||
|
||||
Lifted verbatim from ``scheduler.py:_handle_ta1``. The handler owns
|
||||
its own DB session, dispatches to ``parse_ta1_text``, persists the
|
||||
interchange ack row in ``ta1_acks``, and returns a
|
||||
``(parser_used, claim_count)`` tuple.
|
||||
|
||||
Unlike the 999 handler, TA1 is envelope-only — there is one TA1 per
|
||||
ISA/IEA interchange, no set-level (AK2) or claim-level matching.
|
||||
The claim_count is always 1 (one TA1 ack row per file).
|
||||
|
||||
The actor tag is implicit (no audit event here — TA1 doesn't tag
|
||||
anything in the activity log; it's an infrastructure-level ack).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.claim_acks import apply_ta1_envelope_link
|
||||
from cyclone.parsers.exceptions import CycloneParseError
|
||||
from cyclone.parsers.parse_ta1 import parse_ta1_text
|
||||
from cyclone.store import store as cycl_store
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def handle(
|
||||
text: str,
|
||||
source_file: str,
|
||||
) -> tuple[str, int]:
|
||||
"""Parse a TA1, persist the interchange ack row.
|
||||
|
||||
Args:
|
||||
text: Raw TA1 document bytes (decoded).
|
||||
source_file: Filename the TA1 came from. Used for audit
|
||||
attribution; the ``batches.id`` for TA1 rows is derived
|
||||
internally from the parsed envelope's control number
|
||||
(``TA1-{ICN}``).
|
||||
|
||||
Returns:
|
||||
``(parser_used, claim_count)`` tuple. TA1 always returns
|
||||
``claim_count=1`` (one TA1 ack row per interchange file).
|
||||
|
||||
Raises:
|
||||
ValueError: on parser-level failure (wraps CycloneParseError).
|
||||
|
||||
SP25: the store (``cyclone.store.acks.add_ta1_ack``) owns the
|
||||
publish-from-store contract; the handler no longer needs an
|
||||
``event_bus`` kwarg.
|
||||
"""
|
||||
try:
|
||||
result = parse_ta1_text(text, input_file=source_file)
|
||||
except CycloneParseError as exc:
|
||||
raise ValueError(f"TA1 parse error: {exc}") from exc
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
ta1_ack_row = cycl_store.add_ta1_ack(
|
||||
source_batch_id=result.source_batch_id,
|
||||
control_number=result.ta1.control_number,
|
||||
interchange_date=result.ta1.interchange_date,
|
||||
interchange_time=result.ta1.interchange_time,
|
||||
ack_code=result.ta1.ack_code,
|
||||
note_code=result.ta1.note_code,
|
||||
ack_generated_date=result.ta1.ack_generated_date,
|
||||
sender_id=result.envelope.sender_id,
|
||||
receiver_id=result.envelope.receiver_id,
|
||||
raw_json=json.loads(result.model_dump_json()),
|
||||
)
|
||||
# SP28: TA1 envelope-level link to the originating Batch. The
|
||||
# closure here matches the most-recent Batch whose envelope
|
||||
# sender_id/receiver_id matches the TA1 — see spec §D4.
|
||||
def _batch_lookup(sender_id, receiver_id):
|
||||
rows = (
|
||||
session.query(db.Batch)
|
||||
.filter(
|
||||
db.Batch.kind == "837p",
|
||||
db.Batch.raw_result_json.isnot(None),
|
||||
)
|
||||
.order_by(db.Batch.parsed_at.desc())
|
||||
.all()
|
||||
)
|
||||
for row in rows:
|
||||
env = (row.raw_result_json or {}).get("envelope") or {}
|
||||
if (
|
||||
env.get("sender_id") == sender_id
|
||||
and env.get("receiver_id") == receiver_id
|
||||
):
|
||||
return row
|
||||
return None
|
||||
|
||||
link_result = apply_ta1_envelope_link(
|
||||
session, result, ack_id=ta1_ack_row.id,
|
||||
batch_lookup=_batch_lookup,
|
||||
)
|
||||
# Snapshot rows before closing the work session — SQLite +
|
||||
# concurrent sessions from the same thread cause "database
|
||||
# is locked" errors when add_claim_ack opens its own session
|
||||
# while this one is still open.
|
||||
link_rows = list(link_result.linked)
|
||||
orphans = list(link_result.orphans)
|
||||
session.commit()
|
||||
|
||||
for link_row in link_rows:
|
||||
cycl_store.add_claim_ack(
|
||||
claim_id=link_row.claim_id,
|
||||
batch_id=link_row.batch_id,
|
||||
ack_id=ta1_ack_row.id,
|
||||
ack_kind="ta1",
|
||||
ak2_index=link_row.ak2_index,
|
||||
set_control_number=link_row.set_control_number,
|
||||
set_accept_reject_code=link_row.set_accept_reject_code,
|
||||
linked_by="auto",
|
||||
)
|
||||
if orphans:
|
||||
log.warning(
|
||||
"TA1 had %d orphan envelope refs (no matching batch): %s",
|
||||
len(orphans),
|
||||
orphans[:5],
|
||||
)
|
||||
|
||||
return ("parse_ta1", 1)
|
||||
@@ -175,82 +175,6 @@ def _line_count_lookup(session: Session, claims: list[Claim]) -> tuple[dict, dic
|
||||
return matched_counts, total_lines_by_claim
|
||||
|
||||
|
||||
def _ack_summary_for_claims(
|
||||
session: Session, claim_ids: list[str],
|
||||
) -> dict[str, dict]:
|
||||
"""Build a {claim_id: {total, rejected, items: [...]}} map for 999 acks.
|
||||
|
||||
SP29: the Inbox `rejected` lane needs to render AK2 evidence
|
||||
inline per row, plus a per-row Resubmit button. The data lives
|
||||
in ``claim_acks`` (SP28) — we don't want to N+1 fetch per row,
|
||||
so the whole rejected-claim set is summarized in one batched
|
||||
query here.
|
||||
|
||||
Filters to ``ack_kind='999'`` because the rejected lane is the
|
||||
999 envelope reject lane; the 277CA STC A4/A6/A7 evidence flows
|
||||
through the ``payer_rejected_*`` fields on a separate lane and
|
||||
isn't part of this scope (see SP29 spec D4 / scope).
|
||||
|
||||
Returns:
|
||||
``{claim_id: {"total": int, "rejected": int, "items": [...]}, ...}``
|
||||
Claims with zero linked 999 acks are NOT in the returned
|
||||
dict — the caller maps via ``.get(cid)`` and treats absence
|
||||
as "no 999 acks linked" (renders as ``null`` in the
|
||||
payload, ``999 not linked`` in the UI).
|
||||
|
||||
Args:
|
||||
session: SQLAlchemy session the caller owns.
|
||||
claim_ids: list of claim.id values to summarize. Typically
|
||||
the rejected-lane claim ids. Empty list → empty dict.
|
||||
"""
|
||||
if not claim_ids:
|
||||
return {}
|
||||
from cyclone.db import ClaimAck # late import — DB model registered
|
||||
rows = (
|
||||
session.query(
|
||||
ClaimAck.claim_id,
|
||||
ClaimAck.ack_id,
|
||||
ClaimAck.set_control_number,
|
||||
ClaimAck.set_accept_reject_code,
|
||||
ClaimAck.ak2_index,
|
||||
ClaimAck.linked_at,
|
||||
)
|
||||
.filter(
|
||||
ClaimAck.claim_id.in_(claim_ids),
|
||||
ClaimAck.ack_kind == "999",
|
||||
)
|
||||
.order_by(ClaimAck.linked_at.desc(), ClaimAck.id.desc())
|
||||
.all()
|
||||
)
|
||||
grouped: dict[str, list[tuple]] = {}
|
||||
for cid, aid, scn, code, ak2i, lat in rows:
|
||||
grouped.setdefault(cid, []).append((aid, scn, code, ak2i, lat))
|
||||
|
||||
rejected_codes = {"R", "E", "X"}
|
||||
out: dict[str, dict] = {}
|
||||
for cid, items in grouped.items():
|
||||
total = len(items)
|
||||
rejected_count = sum(1 for it in items if (it[2] or "") in rejected_codes)
|
||||
# Keep 5 most recent items for the chip column. The full count
|
||||
# is in ``total`` so the UI can show ``+N more`` honestly.
|
||||
trimmed = items[:5]
|
||||
out[cid] = {
|
||||
"total": total,
|
||||
"rejected": rejected_count,
|
||||
"items": [
|
||||
{
|
||||
"ack_id": aid,
|
||||
"set_control_number": scn,
|
||||
"set_accept_reject_code": code or "",
|
||||
"ak2_index": ak2i,
|
||||
"linked_at": _isoformat(lat),
|
||||
}
|
||||
for (aid, scn, code, ak2i, lat) in trimmed
|
||||
],
|
||||
}
|
||||
return out
|
||||
|
||||
|
||||
def compute_lanes(session: Session, *, dismissed_pairs: Iterable[frozenset]) -> Lanes:
|
||||
lanes = Lanes()
|
||||
dismissed = set(dismissed_pairs)
|
||||
@@ -268,17 +192,6 @@ def compute_lanes(session: Session, *, dismissed_pairs: Iterable[frozenset]) ->
|
||||
),
|
||||
))
|
||||
|
||||
# SP29: attach the 999 ack-evidence summary (total / rejected /
|
||||
# 5 most recent AK2 set_responses) to every rejected row so the
|
||||
# Inbox can render AK2 chips inline + a per-row Resubmit button
|
||||
# without an extra round-trip. One batched query, keyed off the
|
||||
# rejected-claim id set.
|
||||
rejected_ack_summary = _ack_summary_for_claims(
|
||||
session, [r["id"] for r in lanes.rejected]
|
||||
)
|
||||
for row in lanes.rejected:
|
||||
row["claim_acks"] = rejected_ack_summary.get(row["id"])
|
||||
|
||||
# --- Payer-Rejected (SP10) ---
|
||||
# Distinct from the 999 envelope "rejected" lane above. A claim
|
||||
# lands here when a 277CA STC category code is A4/A6/A7 (rejected
|
||||
|
||||
@@ -33,63 +33,42 @@ def apply_999_rejections(
|
||||
parsed_999,
|
||||
*,
|
||||
claim_lookup: Callable[[str], Claim | None],
|
||||
batch_envelope_index: dict[str, list[str]] | None = None,
|
||||
) -> Apply999Result:
|
||||
"""For each set response with code R, E, or X, look up the matching claim and
|
||||
"""For each set response with code R or E, look up the matching claim and
|
||||
move it to REJECTED. Idempotent on already-rejected claims.
|
||||
|
||||
Args:
|
||||
session: SQLAlchemy session.
|
||||
parsed_999: a ParseResult999 (or any object with .set_responses).
|
||||
claim_lookup: callable from patient_control_number → Claim or None.
|
||||
Legacy fallback; rarely hits when batch_envelope_index is present.
|
||||
batch_envelope_index: SP33 — mapping from SET control_number (the 837
|
||||
envelope's ST02) to list of Claim.id for the claims in that SET.
|
||||
Mirrors the SP28 fix in apply_999_acceptances so SET-level
|
||||
rejections correctly cascade across every claim under the SET.
|
||||
|
||||
Returns:
|
||||
Apply999Result with lists of matched claim ids and orphan PCNs.
|
||||
"""
|
||||
result = Apply999Result()
|
||||
now = datetime.now(timezone.utc)
|
||||
index = batch_envelope_index or {}
|
||||
|
||||
for sr in parsed_999.set_responses:
|
||||
code = sr.set_accept_reject.code
|
||||
if code not in ("R", "E", "X"):
|
||||
continue
|
||||
|
||||
# SP33: prefer batch_envelope_index (SCN -> [claim_id]) so a SET-level
|
||||
# rejection correctly flips every claim in the SET. Fall back to
|
||||
# the legacy claim_lookup when the index is empty for this SCN.
|
||||
candidate_ids = index.get(sr.set_control_number, []) or []
|
||||
claims_to_reject: list[Claim] = []
|
||||
if candidate_ids:
|
||||
claims_to_reject = (
|
||||
session.query(Claim)
|
||||
.filter(Claim.id.in_(candidate_ids))
|
||||
.all()
|
||||
)
|
||||
else:
|
||||
legacy = claim_lookup(sr.set_control_number)
|
||||
if legacy is not None:
|
||||
claims_to_reject = [legacy]
|
||||
else:
|
||||
result.orphans.append(sr.set_control_number)
|
||||
continue
|
||||
claim = claim_lookup(sr.set_control_number)
|
||||
if claim is None:
|
||||
result.orphans.append(sr.set_control_number)
|
||||
continue
|
||||
|
||||
for claim in claims_to_reject:
|
||||
if claim.state == ClaimState.REJECTED:
|
||||
# Idempotent: don't double-mutate.
|
||||
continue
|
||||
claim.state = ClaimState.REJECTED
|
||||
claim.state_changed_at = now
|
||||
claim.rejected_at = now
|
||||
claim.rejection_reason = _build_reason(
|
||||
code, len(sr.segment_errors or [])
|
||||
)
|
||||
result.matched.append(claim.id)
|
||||
if claim.state == ClaimState.REJECTED:
|
||||
# Idempotent: don't double-mutate.
|
||||
continue
|
||||
|
||||
claim.state = ClaimState.REJECTED
|
||||
claim.state_changed_at = now
|
||||
claim.rejected_at = now
|
||||
claim.rejection_reason = _build_reason(
|
||||
code, len(sr.segment_errors or [])
|
||||
)
|
||||
result.matched.append(claim.id)
|
||||
|
||||
if result.matched or result.orphans:
|
||||
session.commit()
|
||||
|
||||
@@ -1,379 +0,0 @@
|
||||
"""SP18 — Structured JSON logging.
|
||||
|
||||
Wraps Python's stdlib ``logging`` to emit newline-delimited JSON
|
||||
(or a dev-friendly tabular format) and to scrub obvious PHI
|
||||
patterns (NPIs, SSNs, DOBs, patient names) from the message +
|
||||
extra fields.
|
||||
|
||||
Design choices
|
||||
--------------
|
||||
|
||||
* **No third-party deps.** stdlib ``logging`` + ``json`` + ``re``
|
||||
is enough. ``loguru`` / ``structlog`` were considered; both add
|
||||
a dependency for marginal gain.
|
||||
|
||||
* **JSON by default.** Operators running Cyclone in production
|
||||
almost certainly want logs in a format their aggregator
|
||||
(Loki/ELK/Vector) can parse. The dev format (``CycloneDevFormatter``)
|
||||
is the opt-out for ``tail -f`` in dev.
|
||||
|
||||
* **Conservative PII scrubber.** Redacts unambiguous PHI patterns
|
||||
only. False positives are not free — an operator's diagnostic
|
||||
dump that says ``<redacted:npi>`` instead of the actual NPI
|
||||
makes root-causing a parse failure harder. The scrubber can be
|
||||
disabled with ``CYCLONE_LOG_NO_PII_SCRUB=1`` for tests /
|
||||
forensic mode.
|
||||
|
||||
* **Idempotent setup.** :func:`setup_logging` can be called
|
||||
multiple times (CLI re-invocation, FastAPI lifespan re-entry
|
||||
under TestClient). Each call clears existing handlers on the
|
||||
root logger before attaching fresh ones — so the format toggle
|
||||
actually takes effect on the second call.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from logging.handlers import RotatingFileHandler
|
||||
from typing import Any, Optional
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Formatters
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# Stdlib LogRecord attributes we don't want to dump into the
|
||||
# structured payload (they're noise for log consumers).
|
||||
_RESERVED_LOGRECORD_ATTRS = frozenset({
|
||||
"args", "asctime", "created", "exc_info", "exc_text", "filename",
|
||||
"funcName", "levelname", "levelno", "lineno", "module", "msecs",
|
||||
"message", "msg", "name", "pathname", "process", "processName",
|
||||
"relativeCreated", "stack_info", "thread", "threadName",
|
||||
"taskName",
|
||||
})
|
||||
|
||||
|
||||
class JsonFormatter(logging.Formatter):
|
||||
"""Format a LogRecord as a single JSON line.
|
||||
|
||||
Fields:
|
||||
ts — ISO 8601 UTC timestamp with milliseconds.
|
||||
level — uppercase level name (INFO, WARNING, etc.).
|
||||
logger — the logger name (e.g. "cyclone.scheduler").
|
||||
msg — the formatted log message (after %-substitution).
|
||||
extra — dict of any non-reserved LogRecord attributes.
|
||||
|
||||
If ``exc_info`` is set, the formatter appends a ``traceback``
|
||||
field with the formatted exception text (NOT a serialized
|
||||
object — just the stdlib-rendered string).
|
||||
"""
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
ts = datetime.fromtimestamp(record.created, tz=timezone.utc).isoformat(
|
||||
timespec="milliseconds",
|
||||
)
|
||||
payload: dict[str, Any] = {
|
||||
"ts": ts,
|
||||
"level": record.levelname,
|
||||
"logger": record.name,
|
||||
"msg": record.getMessage(),
|
||||
}
|
||||
# Collect user-provided extras.
|
||||
extras = {
|
||||
k: v
|
||||
for k, v in record.__dict__.items()
|
||||
if k not in _RESERVED_LOGRECORD_ATTRS and not k.startswith("_")
|
||||
}
|
||||
if extras:
|
||||
payload["extra"] = extras
|
||||
if record.exc_info:
|
||||
payload["traceback"] = self.formatException(record.exc_info)
|
||||
if record.stack_info:
|
||||
payload["stack"] = self.formatStack(record.stack_info)
|
||||
return json.dumps(payload, default=str, sort_keys=True)
|
||||
|
||||
|
||||
class CycloneDevFormatter(logging.Formatter):
|
||||
"""Dev-friendly tabular format.
|
||||
|
||||
Example:
|
||||
2026-06-21T15:30:00.123Z INFO cyclone.scheduler Processed inbound foo.x12 parser=parse_999 claims=3
|
||||
|
||||
Same fields as ``JsonFormatter`` but human-readable. Useful for
|
||||
``tail -f cyclone.log`` in dev.
|
||||
"""
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
ts = datetime.fromtimestamp(record.created, tz=timezone.utc).isoformat(
|
||||
timespec="milliseconds",
|
||||
)
|
||||
extras = {
|
||||
k: v
|
||||
for k, v in record.__dict__.items()
|
||||
if k not in _RESERVED_LOGRECORD_ATTRS and not k.startswith("_")
|
||||
}
|
||||
extra_str = ""
|
||||
if extras:
|
||||
pairs = " ".join(f"{k}={v!r}" for k, v in extras.items())
|
||||
extra_str = " " + pairs
|
||||
base = f"{ts} {record.levelname:<7s} {record.name} {record.getMessage()}{extra_str}"
|
||||
if record.exc_info:
|
||||
base += "\n" + self.formatException(record.exc_info)
|
||||
return base
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PII scrubber
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# Conservative PHI patterns. Each pattern is (label, compiled regex,
|
||||
# replacement). Some patterns use a backreference so the field name
|
||||
# (e.g. "dob=") is preserved and only the value is redacted — that
|
||||
# keeps the surrounding context readable in the log line.
|
||||
_PII_PATTERNS: tuple[tuple[str, "re.Pattern[str]", str], ...] = (
|
||||
# 10-digit NPI. Word-boundary anchored so we don't redact, e.g.,
|
||||
# the "10" in "10 claims processed".
|
||||
("npi", re.compile(r"\b\d{10}\b"), "<redacted:npi>"),
|
||||
# SSN: NNN-NN-NNNN or NNNNNNNNN.
|
||||
(
|
||||
"ssn",
|
||||
re.compile(r"\b\d{3}-\d{2}-\d{4}\b|\b\d{9}\b(?=[\s,;)}])"),
|
||||
"<redacted:ssn>",
|
||||
),
|
||||
# DOB: "dob=YYYY-MM-DD" / "date_of_birth=YYYY-MM-DD". Capture the
|
||||
# field name + separator, redact only the date — keeps the
|
||||
# surrounding sentence readable.
|
||||
(
|
||||
"dob",
|
||||
re.compile(
|
||||
r"(?i)(\b(?:dob|date[ _]?of[ _]?birth)[:=]\s*)\d{4}-\d{2}-\d{2}"
|
||||
),
|
||||
r"\1<redacted:dob>",
|
||||
),
|
||||
# Patient name: explicit field marker, redact the whole
|
||||
# "patient_name=..." chunk so the value can't leak in a quoted form.
|
||||
(
|
||||
"patient_name",
|
||||
re.compile(
|
||||
r'(?i)\bpatient[_ ]?name[:=]\s*"?[^\",\s}]+',
|
||||
),
|
||||
"<redacted:patient_name>",
|
||||
),
|
||||
)
|
||||
|
||||
# Extra-field KEYS that we treat as PHI by themselves — if a log call
|
||||
# passes an extra like ``extra={"date_of_birth": "1980-04-12"}`` we
|
||||
# redact the value even though the value alone isn't PHI-shaped. The
|
||||
# key is the signal. Matched case-insensitively against the full key
|
||||
# (with underscores normalized to spaces for "date of birth").
|
||||
_PHI_EXTRA_KEYS: dict[str, str] = {
|
||||
"npi": "npi",
|
||||
"provider_npi": "npi",
|
||||
"rendering_npi": "npi",
|
||||
"billing_npi": "npi",
|
||||
"ssn": "ssn",
|
||||
"dob": "dob",
|
||||
"date_of_birth": "dob",
|
||||
"patient_name": "patient_name",
|
||||
"patient first name": "patient_name",
|
||||
"patient last name": "patient_name",
|
||||
}
|
||||
|
||||
# When an extra key matches one of these, redact any string value
|
||||
# wholesale (don't try to parse it — just replace).
|
||||
_PHI_EXTRA_WHOLE_VALUE = {"npi", "ssn", "dob", "patient_name"}
|
||||
|
||||
|
||||
class PiiScrubber(logging.Filter):
|
||||
"""Filter that redacts obvious PHI from log records.
|
||||
|
||||
Walks the formatted message + every ``extra`` field value (if
|
||||
it's a string) and rewrites matches to ``<redacted:<name>``.
|
||||
Non-string extras are left alone (we don't try to serialize and
|
||||
re-scrub dicts — too risky for false positives).
|
||||
"""
|
||||
|
||||
def __init__(self, name: str = "pii_scrubber") -> None:
|
||||
super().__init__(name)
|
||||
self._enabled = True
|
||||
|
||||
def disable(self) -> None:
|
||||
"""Disable scrubbing (for tests / forensic mode)."""
|
||||
self._enabled = False
|
||||
|
||||
def enable(self) -> None:
|
||||
self._enabled = True
|
||||
|
||||
def _scrub(self, text: str) -> str:
|
||||
for label, pat, repl in _PII_PATTERNS:
|
||||
text = pat.sub(repl, text)
|
||||
return text
|
||||
|
||||
@staticmethod
|
||||
def _normalize_extra_key(key: str) -> set[str]:
|
||||
"""Return all candidate normalizations of a key.
|
||||
|
||||
``date_of_birth`` should match a lookup table that uses either
|
||||
``date_of_birth`` or ``date of birth`` — so return both. Same
|
||||
for ``patient_name`` vs ``patient name``.
|
||||
"""
|
||||
norm = key.strip().lower()
|
||||
spaced = norm.replace("_", " ")
|
||||
return {norm, spaced}
|
||||
|
||||
def _redact_extra_value(self, key: str, value: Any) -> Any:
|
||||
"""Redact a single extra field value if its key signals PHI."""
|
||||
for norm in self._normalize_extra_key(key):
|
||||
label = _PHI_EXTRA_KEYS.get(norm)
|
||||
if label:
|
||||
if not isinstance(value, str):
|
||||
return value
|
||||
return f"<redacted:{label}>"
|
||||
return value
|
||||
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
if not self._enabled:
|
||||
return True
|
||||
# Scrub the formatted message.
|
||||
try:
|
||||
msg = record.getMessage()
|
||||
scrubbed_msg = self._scrub(msg)
|
||||
if scrubbed_msg != msg:
|
||||
record.msg = scrubbed_msg
|
||||
record.args = ()
|
||||
except Exception: # noqa: BLE001
|
||||
pass # never let the scrubber crash a log call
|
||||
# Scrub string extras in place. We mutate the record's
|
||||
# __dict__ directly so the formatter sees the scrubbed value.
|
||||
for k, v in list(record.__dict__.items()):
|
||||
if k in _RESERVED_LOGRECORD_ATTRS or k.startswith("_"):
|
||||
continue
|
||||
# First, key-based redaction (covers `extra={"dob": "..."}`).
|
||||
redacted = self._redact_extra_value(k, v)
|
||||
if redacted is not v:
|
||||
record.__dict__[k] = redacted
|
||||
continue
|
||||
# Second, value-pattern redaction (covers `extra={"note":
|
||||
# "patient_name=John Doe"}`).
|
||||
if isinstance(v, str):
|
||||
scrubbed = self._scrub(v)
|
||||
if scrubbed != v:
|
||||
record.__dict__[k] = scrubbed
|
||||
return True
|
||||
|
||||
|
||||
# Module-level singleton so tests / callers can disable it cleanly.
|
||||
_scrubber = PiiScrubber()
|
||||
|
||||
|
||||
def get_scrubber() -> PiiScrubber:
|
||||
"""Return the module-level PII scrubber singleton."""
|
||||
return _scrubber
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# setup_logging entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _resolve_level(level: str | int | None) -> int:
|
||||
"""Resolve a level string/int, falling back to INFO."""
|
||||
if level is None:
|
||||
return logging.INFO
|
||||
if isinstance(level, int):
|
||||
return level
|
||||
name = str(level).strip().upper()
|
||||
return logging.getLevelNamesMapping().get(name, logging.INFO)
|
||||
|
||||
|
||||
def setup_logging(
|
||||
*,
|
||||
level: str | int | None = None,
|
||||
log_file: str | None = None,
|
||||
json_format: bool = True,
|
||||
scrub_pii: bool = True,
|
||||
propagate_from: str | None = None,
|
||||
) -> logging.Logger:
|
||||
"""Configure the root logger + attach handlers.
|
||||
|
||||
Idempotent: re-calling clears existing handlers on the root
|
||||
logger before attaching fresh ones. Safe to call from
|
||||
``click.command`` invocations and the FastAPI lifespan.
|
||||
|
||||
Args:
|
||||
level: ``"DEBUG"`` / ``"INFO"`` / etc. or an int. ``None``
|
||||
means honor ``CYCLONE_LOG_LEVEL`` env var, then INFO.
|
||||
log_file: Path to a rotating log file. ``None`` means
|
||||
honor ``CYCLONE_LOG_FILE`` env var, then stderr.
|
||||
json_format: Emit JSON lines (default). ``False`` uses
|
||||
:class:`CycloneDevFormatter`.
|
||||
scrub_pii: Apply the PII scrubber (default). Honored via
|
||||
``CYCLONE_LOG_NO_PII_SCRUB=1`` to disable.
|
||||
propagate_from: Optional logger name to attach the scrubber
|
||||
to (defaults to root).
|
||||
|
||||
Returns:
|
||||
The configured root logger.
|
||||
"""
|
||||
# Resolve env-var defaults.
|
||||
if level is None:
|
||||
level = os.environ.get("CYCLONE_LOG_LEVEL", "INFO")
|
||||
if log_file is None:
|
||||
log_file = os.environ.get("CYCLONE_LOG_FILE") or None
|
||||
if not json_format and os.environ.get("CYCLONE_LOG_JSON", "").lower() in (
|
||||
"false", "0", "no",
|
||||
):
|
||||
json_format = True
|
||||
if os.environ.get("CYCLONE_LOG_NO_PII_SCRUB", "").lower() in ("1", "true", "yes"):
|
||||
scrub_pii = False
|
||||
|
||||
root = logging.getLogger()
|
||||
root.setLevel(_resolve_level(level))
|
||||
|
||||
# Clear existing handlers (idempotent re-setup).
|
||||
for h in list(root.handlers):
|
||||
root.removeHandler(h)
|
||||
# Also clear our scrubber so we don't add duplicates.
|
||||
target = logging.getLogger(propagate_from) if propagate_from else root
|
||||
for flt in list(target.filters):
|
||||
if isinstance(flt, PiiScrubber):
|
||||
target.removeFilter(flt)
|
||||
|
||||
# Build the formatter.
|
||||
fmt: logging.Formatter
|
||||
if json_format:
|
||||
fmt = JsonFormatter()
|
||||
else:
|
||||
fmt = CycloneDevFormatter()
|
||||
|
||||
# Build the handler.
|
||||
if log_file:
|
||||
handler: logging.Handler = RotatingFileHandler(
|
||||
log_file,
|
||||
maxBytes=10 * 1024 * 1024,
|
||||
backupCount=5,
|
||||
encoding="utf-8",
|
||||
)
|
||||
else:
|
||||
handler = logging.StreamHandler(stream=sys.stderr)
|
||||
handler.setFormatter(fmt)
|
||||
root.addHandler(handler)
|
||||
|
||||
# Attach the scrubber.
|
||||
if scrub_pii:
|
||||
_scrubber.enable()
|
||||
else:
|
||||
_scrubber.disable()
|
||||
target.addFilter(_scrubber)
|
||||
|
||||
# Quiet down noisy third-party libs.
|
||||
for noisy in ("urllib3", "paramiko", "sqlalchemy.engine"):
|
||||
logging.getLogger(noisy).setLevel(max(root.level, logging.WARNING))
|
||||
|
||||
return root
|
||||
@@ -1,47 +0,0 @@
|
||||
-- version: 11
|
||||
-- SP16: Inbound MFT polling scheduler
|
||||
--
|
||||
-- Tracks every file the background scheduler has downloaded from
|
||||
-- the Gainwell MFT inbound path so a re-tick (or a restart) does not
|
||||
-- re-process the same file. Idempotency is required for production:
|
||||
-- the scheduler polls every N seconds and a slow MFT server may hand
|
||||
-- us the same file across two polls.
|
||||
--
|
||||
-- We key on (sftp_block_name, name) — the sftp_block_name disambiguates
|
||||
-- multi-provider installations (SP9+SP-multi-NPI), name is the inbound
|
||||
-- filename as it appears on the MFT server.
|
||||
--
|
||||
-- Status values:
|
||||
-- * ok — parsed cleanly, results persisted to the store
|
||||
-- * error — parser raised; error_message captured for the operator
|
||||
-- * skipped — file_type not in the scheduler's allowed set
|
||||
-- * pending — file was downloaded but a downstream step failed
|
||||
-- (e.g. DB write); the scheduler retries on the next tick
|
||||
--
|
||||
-- claim_count is the number of claims/remittances/acks the parser
|
||||
-- surfaced. Surfaced on /api/admin/scheduler/status so the operator can
|
||||
-- see throughput without parsing logs.
|
||||
--
|
||||
-- Compliance: not part of the HIPAA audit chain (SP11). This is
|
||||
-- operational metadata; an SFTP outage shouldn't pollute the audit log.
|
||||
|
||||
CREATE TABLE processed_inbound_files (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
sftp_block_name TEXT NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
size INTEGER NOT NULL,
|
||||
modified_at TEXT,
|
||||
file_type TEXT,
|
||||
processed_at TEXT NOT NULL,
|
||||
parser_used TEXT,
|
||||
claim_count INTEGER NOT NULL DEFAULT 0,
|
||||
status TEXT NOT NULL,
|
||||
error_message TEXT
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX ux_processed_inbound_files_block_name
|
||||
ON processed_inbound_files(sftp_block_name, name);
|
||||
CREATE INDEX ix_processed_inbound_files_processed_at
|
||||
ON processed_inbound_files(processed_at DESC);
|
||||
CREATE INDEX ix_processed_inbound_files_status
|
||||
ON processed_inbound_files(status);
|
||||
@@ -1,29 +0,0 @@
|
||||
-- version: 12
|
||||
-- SP17: encrypted DB backup metadata
|
||||
--
|
||||
-- Tracks every backup the BackupService has taken. The actual
|
||||
-- encrypted blob lives in a directory outside the DB (default
|
||||
-- ~/.local/share/cyclone/backups/); this table is just the index
|
||||
-- the operator queries via GET /api/admin/backup/list.
|
||||
--
|
||||
-- Status values:
|
||||
-- pending - row inserted, .backup() in progress or crashed before commit
|
||||
-- ok - encrypted blob + sidecar written successfully
|
||||
-- error - creation failed; error_message populated
|
||||
-- pruned - retention policy removed the file; row kept for audit
|
||||
|
||||
CREATE TABLE db_backups (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
filename TEXT NOT NULL,
|
||||
backup_dir TEXT NOT NULL,
|
||||
size_bytes INTEGER NOT NULL DEFAULT 0,
|
||||
db_fingerprint TEXT,
|
||||
table_count INTEGER NOT NULL DEFAULT 0,
|
||||
created_at TEXT NOT NULL,
|
||||
completed_at TEXT,
|
||||
status TEXT NOT NULL,
|
||||
error_message TEXT
|
||||
);
|
||||
CREATE UNIQUE INDEX ux_db_backups_filename ON db_backups(backup_dir, filename);
|
||||
CREATE INDEX ix_db_backups_created_at ON db_backups(created_at DESC);
|
||||
CREATE INDEX ix_db_backups_status ON db_backups(status);
|
||||
@@ -1,31 +0,0 @@
|
||||
-- version: 13
|
||||
-- Auth (SP-auth): users + sessions tables.
|
||||
--
|
||||
-- `users` holds the local credential store: bcrypt-hashed password,
|
||||
-- role enum ('admin' | 'user' | 'viewer'), and a soft-delete column
|
||||
-- (disabled_at) so admins can revoke access without losing history.
|
||||
--
|
||||
-- `sessions` holds the server-side session rows; the browser only
|
||||
-- carries an opaque token cookie (cyclone_session) that points here.
|
||||
-- expires_at index lets us cheaply reap stale sessions.
|
||||
|
||||
CREATE TABLE users (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
username TEXT NOT NULL UNIQUE,
|
||||
password_hash TEXT NOT NULL,
|
||||
role TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
disabled_at TEXT
|
||||
);
|
||||
|
||||
CREATE INDEX idx_users_username ON users(username);
|
||||
|
||||
CREATE TABLE sessions (
|
||||
id TEXT PRIMARY KEY,
|
||||
user_id INTEGER NOT NULL REFERENCES users(id),
|
||||
expires_at TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE INDEX idx_sessions_user_id ON sessions(user_id);
|
||||
CREATE INDEX idx_sessions_expires_at ON sessions(expires_at);
|
||||
@@ -1,10 +0,0 @@
|
||||
-- version: 14
|
||||
-- Auth (SP-auth): record the acting user_id on every audit_log entry.
|
||||
--
|
||||
-- Backwards-compatible: existing rows get NULL user_id (they were
|
||||
-- written by the pre-auth `system` actor). Going forward, the FastAPI
|
||||
-- get_current_user dependency injects the id into every audit log call.
|
||||
|
||||
ALTER TABLE audit_log ADD COLUMN user_id INTEGER;
|
||||
|
||||
CREATE INDEX idx_audit_log_user_id ON audit_log(user_id);
|
||||
@@ -1,74 +0,0 @@
|
||||
-- version: 15
|
||||
-- Drop the inline UNIQUE(batch_id, patient_control_number) on claims.
|
||||
--
|
||||
-- Migration 0003 attempted DROP INDEX IF EXISTS uq_claims_batch_pcn but
|
||||
-- the constraint is inline in CREATE TABLE, so the drop was a no-op.
|
||||
-- The only way to remove an inline UNIQUE in SQLite is table recreation.
|
||||
--
|
||||
-- Discovery 2026-06-23: the inline UNIQUE does NOT exist in the current
|
||||
-- production DB at user_version=14 (or in main's fresh-DB schema). The
|
||||
-- 32 "Duplicate claim" warnings in /tmp/cyclone-uvicorn.log are PK
|
||||
-- collisions on claims.id (CLM01) when an operator re-uploads the same
|
||||
-- file — not UNIQUE violations. This migration is therefore a defensive
|
||||
-- no-op against the current schema, but keeps the 0003 intent alive
|
||||
-- (drop the constraint if it ever reappears) and lets the SP22 spec
|
||||
-- ship as designed.
|
||||
--
|
||||
-- X12 837P allows any number of CLM segments per 2000B subscriber loop;
|
||||
-- claim identity is provided by the primary key (claims.id = CLM01).
|
||||
-- The remittances table had a parallel constraint already removed in 0003
|
||||
-- (because that one WAS a named index), so this migration only touches
|
||||
-- claims.
|
||||
--
|
||||
-- The migration runner (db_migrate.py) wraps each .sql in an implicit
|
||||
-- transaction via engine.begin(), so we MUST NOT use BEGIN/COMMIT.
|
||||
-- PRAGMA defer_foreign_keys defers FK checks to commit, which is the
|
||||
-- only way to drop a referenced table inside a transaction in SQLite.
|
||||
-- Other tables referencing claims:
|
||||
-- remittances.claim_id
|
||||
-- matches.claim_id
|
||||
-- line_reconciliations.claim_id
|
||||
-- activity_events.claim_id
|
||||
|
||||
PRAGMA defer_foreign_keys = ON;
|
||||
|
||||
CREATE TABLE claims_new (
|
||||
id TEXT PRIMARY KEY,
|
||||
batch_id TEXT NOT NULL REFERENCES batches(id) ON DELETE CASCADE,
|
||||
patient_control_number TEXT NOT NULL,
|
||||
service_date_from DATE,
|
||||
service_date_to DATE,
|
||||
charge_amount NUMERIC(12, 2) NOT NULL DEFAULT 0,
|
||||
provider_npi TEXT,
|
||||
payer_id TEXT,
|
||||
state TEXT NOT NULL DEFAULT 'submitted',
|
||||
state_before_reversal TEXT,
|
||||
matched_remittance_id TEXT REFERENCES remittances(id),
|
||||
raw_json TEXT,
|
||||
rejection_reason TEXT,
|
||||
rejected_at TIMESTAMP,
|
||||
resubmit_count INTEGER NOT NULL DEFAULT 0,
|
||||
state_changed_at TIMESTAMP,
|
||||
payer_rejected_at TEXT,
|
||||
payer_rejected_reason TEXT,
|
||||
payer_rejected_status_code TEXT,
|
||||
payer_rejected_by_277ca_id TEXT,
|
||||
payer_rejected_acknowledged_at TEXT,
|
||||
payer_rejected_acknowledged_actor TEXT
|
||||
-- NO UNIQUE (batch_id, patient_control_number) — removed.
|
||||
);
|
||||
|
||||
INSERT INTO claims_new SELECT * FROM claims;
|
||||
DROP TABLE claims;
|
||||
ALTER TABLE claims_new RENAME TO claims;
|
||||
|
||||
-- Recreate secondary indexes (same names, same columns as initial schema
|
||||
-- plus later migrations).
|
||||
CREATE INDEX ix_claims_state ON claims(state);
|
||||
CREATE INDEX ix_claims_patient_control_number ON claims(patient_control_number);
|
||||
CREATE INDEX ix_claims_service_date_from ON claims(service_date_from);
|
||||
CREATE INDEX ix_claims_state_changed_at ON claims(state, state_changed_at);
|
||||
CREATE INDEX idx_claims_payer_rejected_at ON claims(payer_rejected_at);
|
||||
CREATE INDEX idx_claims_payer_rejected_unack
|
||||
ON claims(payer_rejected_at)
|
||||
WHERE payer_rejected_acknowledged_at IS NULL;
|
||||
@@ -1,15 +0,0 @@
|
||||
-- version: 16
|
||||
-- Add the missing index on claims.matched_remittance_id.
|
||||
--
|
||||
-- SP27 Task 11 added ``check_matched_pair_drift`` at startup, which
|
||||
-- scans ``WHERE Claim.matched_remittance_id IS NOT NULL``. Without an
|
||||
-- index this is a full-table scan; becomes a noticeable boot
|
||||
-- latency cost past ~10k claims. The companion index on the
|
||||
-- reverse side (``remittances.claim_id``) was added in 0007.
|
||||
--
|
||||
-- A plain index is enough — neither side is unique (reversals
|
||||
-- re-reference the original PCN, and a claim without a match is
|
||||
-- fine).
|
||||
|
||||
CREATE INDEX ix_claims_matched_remittance_id
|
||||
ON claims(matched_remittance_id);
|
||||
@@ -1,28 +0,0 @@
|
||||
-- version: 17
|
||||
-- Backfill claims.patient_control_number = claims.id.
|
||||
--
|
||||
-- SP27 Task 17 fixed the 837 ingest in store.py:_claim_837_row so
|
||||
-- ``Claim.patient_control_number`` is populated from
|
||||
-- ``claim.claim_id`` (CLM01) instead of ``claim.subscriber.member_id``
|
||||
-- (the 2010BA NM109). The reconcile matcher joins on
|
||||
-- ``Claim.patient_control_number == Remittance.payer_claim_control_number``
|
||||
-- and the 835 echoes CLM01 in CLP01 per X12 spec, so the wrong field
|
||||
-- silently broke every auto-match in production.
|
||||
--
|
||||
-- For rows written BEFORE the fix, the stored value is the member_id
|
||||
-- (e.g. "W953474") which never matches any remit's CLP01. This
|
||||
-- migration backfills those rows by aligning
|
||||
-- ``patient_control_number`` with the row's own ``id`` (= CLM01).
|
||||
-- After this, every claim row's PCN is consistent with the value the
|
||||
-- 837 actually sent, and any newly-ingested 835 whose CLP01 echoes
|
||||
-- that CLM01 will auto-pair.
|
||||
--
|
||||
-- Idempotent: only touches rows where the stored PCN doesn't already
|
||||
-- match the row's id, so re-running on already-fixed rows is a no-op.
|
||||
--
|
||||
-- Reversal safety: the migration does NOT clear matched_remittance_id,
|
||||
-- so any pre-existing manual_match pairs stay intact.
|
||||
|
||||
UPDATE claims
|
||||
SET patient_control_number = id
|
||||
WHERE patient_control_number IS DISTINCT FROM id;
|
||||
@@ -1,52 +0,0 @@
|
||||
-- version: 18
|
||||
-- SP28: per-ACK auto-link join table.
|
||||
--
|
||||
-- Closes the operator gap where every inbound 999 / 277CA / TA1 ack was
|
||||
-- persisted but never durably linked back to the claim it
|
||||
-- acknowledges. One row per AK2 set-response for 999, per ClaimStatus
|
||||
-- for 277CA, per TA1 envelope (with claim_id NULL + batch_id set).
|
||||
--
|
||||
-- Granularity (per-AK2) is preserved by ``ak2_index`` and the unique
|
||||
-- index ``ux_claim_acks_dedup`` — an auto-link of
|
||||
-- (claim, 999, ak2_index=3) is idempotent on re-ingest of the same
|
||||
-- 999 file (the index enforces this at the DB layer; the helper
|
||||
-- pre-checks to avoid IntegrityError log noise).
|
||||
--
|
||||
-- Notes:
|
||||
-- * ``claim_id`` is nullable so TA1 envelope-level links to the
|
||||
-- originating Batch can land here (FK to batches.id). The
|
||||
-- CHECK constraint makes sure at least one of (claim_id,
|
||||
-- batch_id) is set on every row — see spec §3.1.
|
||||
-- * ``set_control_number`` records the value the upstream ACK
|
||||
-- ACTUALLY CARRIED (== source 837 ST02 for Gainwell batches). It
|
||||
-- is the orphan-traceability field — the link survives even when
|
||||
-- the join had to fall back from ST02 to PCN matching.
|
||||
-- * ``set_accept_reject_code`` carries the AK5 code (A/E/R/X) for
|
||||
-- 999 or the STC category code (A1/A2/A3/A4/A6/A7 etc.) for 277CA.
|
||||
-- TA1 stores the envelope-level ack code here (A/R/E).
|
||||
-- * No FK constraint on ``(ack_kind, ack_id)`` — there are three
|
||||
-- separate ack tables (``acks``, ``ta1_acks``, ``two77ca_acks``).
|
||||
-- Application code enforces the discriminator.
|
||||
|
||||
CREATE TABLE claim_acks (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
claim_id TEXT REFERENCES claims(id) ON DELETE CASCADE,
|
||||
batch_id TEXT REFERENCES batches(id) ON DELETE CASCADE,
|
||||
ack_id INTEGER NOT NULL,
|
||||
ack_kind TEXT NOT NULL CHECK (ack_kind IN ('999', '277ca', 'ta1')),
|
||||
ak2_index INTEGER,
|
||||
set_control_number TEXT,
|
||||
set_accept_reject_code TEXT,
|
||||
linked_at DATETIME NOT NULL,
|
||||
linked_by TEXT NOT NULL CHECK (linked_by IN ('auto', 'manual')),
|
||||
CHECK ((claim_id IS NOT NULL) OR (batch_id IS NOT NULL))
|
||||
);
|
||||
|
||||
CREATE INDEX ix_claim_acks_claim_id ON claim_acks(claim_id);
|
||||
CREATE INDEX ix_claim_acks_batch_id ON claim_acks(batch_id);
|
||||
CREATE INDEX ix_claim_acks_ack ON claim_acks(ack_kind, ack_id);
|
||||
|
||||
-- Dedup: an auto-link of (claim, 999, ak2_index=3) is idempotent on re-ingest.
|
||||
CREATE UNIQUE INDEX ux_claim_acks_dedup
|
||||
ON claim_acks(claim_id, ack_kind, ack_id, ak2_index)
|
||||
WHERE claim_id IS NOT NULL AND ak2_index IS NOT NULL;
|
||||
@@ -1,7 +0,0 @@
|
||||
-- version: 19
|
||||
-- SP32: render & service-provider NPI extraction.
|
||||
-- Nullable: existing rows stay NULL until backfill runs.
|
||||
-- No indexes (used for set-equality, not range queries; nullable).
|
||||
|
||||
ALTER TABLE claims ADD COLUMN rendering_provider_npi TEXT;
|
||||
ALTER TABLE remittances ADD COLUMN rendering_provider_npi TEXT;
|
||||
@@ -1,159 +0,0 @@
|
||||
"""SP20 — NPI checksum + Tax ID format validation.
|
||||
|
||||
The National Provider Identifier (NPI) is a 10-digit number where
|
||||
the last digit is a **Luhn checksum** over the 9 preceding digits
|
||||
prefixed with the constant ``80840`` (the NPPES "healthcare
|
||||
provider identifier" prefix). See CMS / HHS NPI Standard:
|
||||
|
||||
https://www.cms.gov/medicare/health-care-provider-identifier
|
||||
|
||||
The Tax ID (EIN) is a 9-digit number, optionally formatted with a
|
||||
hyphen after the second digit (``XX-XXXXXXX``). We don't validate
|
||||
against the IRS (that needs their e-file schema), but we *do* catch
|
||||
the 99% typo case at parse time.
|
||||
|
||||
Everything in this module is local — no NPPES, no network. Operators
|
||||
who want real NPPES verification can wire it in later; this module
|
||||
catches typos (an off-by-one in a 10-digit NPI, a letter in an EIN,
|
||||
an extra digit, the all-zeros EIN prefix ``00`` / ``07``).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
# NPPES prefix per the NPI Luhn algorithm. Prepended to the 9-digit
|
||||
# NPI body before running the Luhn check.
|
||||
_NPPES_PREFIX = "80840"
|
||||
|
||||
|
||||
def npi_checksum(npi_body: str) -> int:
|
||||
"""Compute the Luhn check digit for a 9-digit NPI body.
|
||||
|
||||
``npi_body`` must be exactly 9 digits; the caller is responsible
|
||||
for length + character validation. Returns the check digit (0–9).
|
||||
"""
|
||||
if not npi_body.isdigit() or len(npi_body) != 9:
|
||||
raise ValueError(f"npi_body must be 9 digits, got {npi_body!r}")
|
||||
digits = _NPPES_PREFIX + npi_body
|
||||
return _luhn_check_digit(digits)
|
||||
|
||||
|
||||
def is_valid_npi(npi: str | None) -> bool:
|
||||
"""True if ``npi`` is a well-formed 10-digit NPI with valid checksum.
|
||||
|
||||
Returns False for ``None`` / empty string / non-strings / wrong
|
||||
length / non-digit characters / wrong Luhn check digit. Doesn't
|
||||
call NPPES — see module docstring for why.
|
||||
|
||||
>>> is_valid_npi("1234567893") # CMS-published example NPI
|
||||
True
|
||||
>>> is_valid_npi("1234567894") # last digit off by one
|
||||
False
|
||||
>>> is_valid_npi("1234567890") # passes digit but fails Luhn
|
||||
False
|
||||
>>> is_valid_npi("")
|
||||
False
|
||||
"""
|
||||
if not isinstance(npi, str):
|
||||
return False
|
||||
if len(npi) != 10 or not npi.isdigit():
|
||||
return False
|
||||
return npi[-1] == str(npi_checksum(npi[:-1]))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tax ID (EIN)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# EIN prefix table (subset). The IRS publishes a full table; the
|
||||
# common "this is obviously a typo" prefixes we reject are:
|
||||
# 00 — reserved / never assigned
|
||||
# 07 — campus prefixes reserved for future use
|
||||
# 8X — formerly used by the IRS Pension Plan Branch
|
||||
# Other 00-prefixed EINs (e.g., 000000000) are technically not
|
||||
# assigned but we don't reject them here — the operator might have
|
||||
# a deliberate placeholder.
|
||||
_EIN_FORBIDDEN_PREFIXES = {"00", "07"}
|
||||
_EIN_RESERVED_PREFIX_8X = re.compile(r"^8\d$")
|
||||
|
||||
# 9 digits, optionally formatted as XX-XXXXXXX.
|
||||
_EIN_FORMATTED = re.compile(r"^\d{2}-\d{7}$")
|
||||
_EIN_PLAIN = re.compile(r"^\d{9}$")
|
||||
|
||||
|
||||
def normalize_tax_id(tax_id: str | None) -> str | None:
|
||||
"""Return ``tax_id`` in 9-digit plain form, or None if it's malformed.
|
||||
|
||||
>>> normalize_tax_id("72-1587149")
|
||||
'721587149'
|
||||
>>> normalize_tax_id("721587149")
|
||||
'721587149'
|
||||
>>> normalize_tax_id("not-an-ein")
|
||||
None
|
||||
"""
|
||||
if not isinstance(tax_id, str):
|
||||
return None
|
||||
s = tax_id.strip()
|
||||
if _EIN_FORMATTED.match(s):
|
||||
return s.replace("-", "")
|
||||
if _EIN_PLAIN.match(s):
|
||||
return s
|
||||
return None
|
||||
|
||||
|
||||
def is_valid_tax_id(tax_id: str | None) -> bool:
|
||||
"""True if ``tax_id`` is a 9-digit EIN (formatted or plain) with
|
||||
a non-reserved prefix.
|
||||
|
||||
>>> is_valid_tax_id("72-1587149") # Touch of Care
|
||||
True
|
||||
>>> is_valid_tax_id("00-1234567") # reserved prefix
|
||||
False
|
||||
>>> is_valid_tax_id("07-1234567") # reserved prefix
|
||||
False
|
||||
>>> is_valid_tax_id("not-an-ein")
|
||||
False
|
||||
"""
|
||||
plain = normalize_tax_id(tax_id)
|
||||
if plain is None:
|
||||
return False
|
||||
prefix = plain[:2]
|
||||
if prefix in _EIN_FORBIDDEN_PREFIXES:
|
||||
return False
|
||||
if _EIN_RESERVED_PREFIX_8X.match(prefix):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Luhn internals
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _luhn_check_digit(digits: str) -> int:
|
||||
"""Return the Luhn check digit for ``digits``.
|
||||
|
||||
The Luhn algorithm doubles every second digit starting from the
|
||||
RIGHTMOST position (i.e., the first digit doubled is the rightmost
|
||||
character of ``digits``). If the doubled value exceeds 9, subtract
|
||||
9. Sum all digits; the check digit is ``(10 - sum % 10) % 10``.
|
||||
|
||||
``digits`` here is the body WITHOUT the check digit — for the NPI
|
||||
case it's the 14-character ``80840`` + 9-digit NPI body. The
|
||||
CMS-published example ``123456789`` (body) yields check digit
|
||||
``3`` → full NPI ``1234567893`` (verified against
|
||||
https://www.cms.gov/.../NPIcheckdigit.pdf).
|
||||
"""
|
||||
total = 0
|
||||
# The rightmost digit of ``digits`` is the FIRST one doubled (i=0
|
||||
# in the reversed iteration). Per CMS, doubling starts at the
|
||||
# rightmost and alternates leftward.
|
||||
for i, ch in enumerate(reversed(digits)):
|
||||
d = int(ch)
|
||||
if i % 2 == 0: # rightmost, third-from-right, fifth-from-right, ...
|
||||
d *= 2
|
||||
if d > 9:
|
||||
d -= 9
|
||||
total += d
|
||||
return (10 - total % 10) % 10
|
||||
@@ -63,7 +63,6 @@ class ClaimHeader(_Base):
|
||||
frequency_code: str | None = None
|
||||
provider_signature: str | None = None
|
||||
assignment: str | None = None
|
||||
benefits_assignment_certification: str | None = None # CLM08 (Y/N)
|
||||
release_of_info: str | None = None
|
||||
prior_auth: str | None = None
|
||||
|
||||
@@ -88,14 +87,6 @@ class ServiceLine(_Base):
|
||||
place_of_service: str | None = None
|
||||
service_date: date | None = None
|
||||
provider_reference: str | None = None
|
||||
# SV1-07 — Diagnosis Code Pointer. Points to one or more
|
||||
# diagnosis codes in the parent claim's HI segment ("1".. "12",
|
||||
# space-separated when multiple). For 837P with a non-empty HI
|
||||
# segment, SV1-07 is required by HCPF / Gainwell. The parser
|
||||
# captures it from the source; the serializer defaults to "1"
|
||||
# when the claim has at least one diagnosis and no explicit
|
||||
# pointer was captured (matches the common single-dx case).
|
||||
dx_pointer: str | None = None
|
||||
|
||||
|
||||
class ValidationIssue(_Base):
|
||||
@@ -142,7 +133,6 @@ class ClaimOutput(_Base):
|
||||
subscriber: Subscriber
|
||||
payer: Payer
|
||||
claim: ClaimHeader
|
||||
rendering_provider_npi: str | None = None # NM1*82 NM109 (Loop 2420A)
|
||||
diagnoses: list[Diagnosis] = Field(default_factory=list)
|
||||
service_lines: list[ServiceLine] = Field(default_factory=list)
|
||||
validation: ValidationReport
|
||||
|
||||
@@ -160,7 +160,6 @@ class ClaimPayment(_Base):
|
||||
ref_benefit_plan: str | None = None # REF*CE per the CO guide
|
||||
service_payments: list[ServicePayment] = Field(default_factory=list)
|
||||
raw_segments: list[list[str]] = Field(default_factory=list)
|
||||
service_provider_npi: str | None = None # NM1*1P NM109 (Loop 2100 service provider)
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
|
||||
@@ -376,7 +376,6 @@ def _consume_claim_payment(segments: list[list[str]], idx: int) -> tuple[ClaimPa
|
||||
|
||||
service_payments: list[ServicePayment] = []
|
||||
ref_benefit_plan: str | None = None
|
||||
service_provider_npi: str | None = None
|
||||
per_diem: Decimal | None = None
|
||||
status_label = claim_status_label(status)
|
||||
|
||||
@@ -423,16 +422,9 @@ def _consume_claim_payment(segments: list[list[str]], idx: int) -> tuple[ClaimPa
|
||||
except ValueError:
|
||||
per_diem = None
|
||||
elif s[0] == "NM1":
|
||||
# SP32: capture service-provider NPI from NM1*1P (Loop 2100).
|
||||
# NM108 (idx 8) carries the ID qualifier (typically "XX");
|
||||
# NM109 (idx 9) is the value. Some senders omit NM108; accept
|
||||
# both forms but require a 10-digit ID.
|
||||
if len(s) > 9 and s[1] == "1P":
|
||||
if len(s) > 8 and s[8] == "XX" and s[9]:
|
||||
if s[9].isdigit() and len(s[9]) == 10:
|
||||
service_provider_npi = s[9]
|
||||
elif s[9] and s[9].isdigit() and len(s[9]) == 10:
|
||||
service_provider_npi = s[9]
|
||||
# Patient (QC) / service-provider (1P) — captured in raw_segments.
|
||||
# The 835 spec doesn't require a structured patient model in v1.
|
||||
pass
|
||||
elif s[0] == "DTM":
|
||||
# Claim-level dates — captured in raw_segments.
|
||||
pass
|
||||
@@ -454,7 +446,6 @@ def _consume_claim_payment(segments: list[list[str]], idx: int) -> tuple[ClaimPa
|
||||
ref_benefit_plan=ref_benefit_plan,
|
||||
service_payments=service_payments,
|
||||
raw_segments=raw,
|
||||
service_provider_npi=service_provider_npi,
|
||||
),
|
||||
idx,
|
||||
)
|
||||
|
||||
@@ -202,7 +202,6 @@ def _consume_claim(segments: list[list[str]], idx: int) -> tuple[ClaimOutput, in
|
||||
frequency_code=freq or None,
|
||||
provider_signature=clm[6] if len(clm) > 6 else None,
|
||||
assignment=clm[7] if len(clm) > 7 else None,
|
||||
benefits_assignment_certification=clm[8] if len(clm) > 8 else None,
|
||||
release_of_info=clm[9] if len(clm) > 9 else None,
|
||||
)
|
||||
|
||||
@@ -210,7 +209,6 @@ def _consume_claim(segments: list[list[str]], idx: int) -> tuple[ClaimOutput, in
|
||||
service_lines: list[ServiceLine] = []
|
||||
raw: list[list[str]] = [seg]
|
||||
prior_auth: str | None = None
|
||||
rendering_provider_npi: str | None = None
|
||||
|
||||
idx += 1
|
||||
while idx < len(segments) and segments[idx][0] not in {"HL", "CLM", "SE"}:
|
||||
@@ -227,11 +225,6 @@ def _consume_claim(segments: list[list[str]], idx: int) -> tuple[ClaimOutput, in
|
||||
for code in parts[1:]:
|
||||
if code:
|
||||
diagnoses.append(Diagnosis(code=code, qualifier=qualifier))
|
||||
elif s[0] == "NM1" and len(s) > 1 and s[1] == "82":
|
||||
# SP32: capture rendering provider NPI from Loop 2420A NM1*82.
|
||||
# NM108 (idx 8) is the qualifier (typically "XX"), NM109 (idx 9) is the NPI.
|
||||
if len(s) > 9 and s[8] == "XX" and len(s[9]) == 10 and s[9].isdigit():
|
||||
rendering_provider_npi = s[9]
|
||||
elif s[0] == "LX":
|
||||
line_no = int(s[1]) if len(s) > 1 and s[1].isdigit() else len(service_lines) + 1
|
||||
# LX is just a separator — the actual service line data is in the next SV1.
|
||||
@@ -261,7 +254,6 @@ def _consume_claim(segments: list[list[str]], idx: int) -> tuple[ClaimOutput, in
|
||||
subscriber=Subscriber(first_name="", last_name="", member_id="", address=Address(line1="", city="", state="", zip="")),
|
||||
payer=Payer(name="", id=""),
|
||||
claim=claim_header,
|
||||
rendering_provider_npi=rendering_provider_npi,
|
||||
diagnoses=diagnoses,
|
||||
service_lines=service_lines,
|
||||
validation=ValidationReport(passed=True, errors=[], warnings=[]),
|
||||
@@ -293,17 +285,11 @@ def _consume_service_line(segments: list[list[str]], idx: int, line_no: int) ->
|
||||
except Exception:
|
||||
units = None
|
||||
place_of_service = seg[5] if len(seg) > 5 else None
|
||||
# SV1-06 (Unit Basis of Measurement) is X12 "UN" for "units" — we
|
||||
# already use unit_type in SV1-03; SV1-06 is rarely populated and
|
||||
# is not required by HCPF.
|
||||
# SV1-07 — Diagnosis Code Pointer (e.g. "1" for the first HI
|
||||
# diagnosis). Required by HCPF when the claim has diagnoses.
|
||||
dx_pointer = seg[7] if len(seg) > 7 and seg[7] else None
|
||||
service_date: date | None = None
|
||||
provider_ref: str | None = None
|
||||
|
||||
idx += 1
|
||||
while idx < len(segments) and segments[idx][0] not in {"LX", "HL", "CLM", "SE", "NM1"}:
|
||||
while idx < len(segments) and segments[idx][0] not in {"LX", "HL", "CLM", "SE"}:
|
||||
s = segments[idx]
|
||||
raw.append(s)
|
||||
if s[0] == "DTP" and len(s) > 2 and s[1] == "472":
|
||||
@@ -325,7 +311,6 @@ def _consume_service_line(segments: list[list[str]], idx: int, line_no: int) ->
|
||||
place_of_service=place_of_service,
|
||||
service_date=service_date,
|
||||
provider_reference=provider_ref,
|
||||
dx_pointer=dx_pointer,
|
||||
),
|
||||
idx,
|
||||
)
|
||||
|
||||
@@ -8,12 +8,6 @@ Single-pass walker over the tokenized segment list:
|
||||
- AK3 (Segment Context) + AK4 (Element Context) — optional per-segment errors
|
||||
- AK5 (Transaction Set Response Status) — per-set accept/reject
|
||||
- AK9 (Functional Group Response Status) — per-group counts + ack code
|
||||
- IK5 — a non-standard synonym for ``AK5`` that Gainwell's MFT ships
|
||||
in place of the spec-defined ``AK5``. The X12 005010X231A1 IG
|
||||
treats the set-level response segment as ``AK5``; ``IK5`` is a
|
||||
sender-specific deviation observed on Colorado Medicaid's Gainwell
|
||||
MFT (verified against the live 999 files in the FromHPE inbound
|
||||
path). We accept either.
|
||||
- SE / GE / IEA
|
||||
|
||||
Errors at the file level raise :class:`CycloneParseError`. The parser
|
||||
@@ -152,11 +146,6 @@ def _consume_ak3_ak4(segments: list[list[str]], idx: int) -> tuple[list[SegmentE
|
||||
def _consume_ak2(segments: list[list[str]], idx: int) -> SetFunctionalGroupResponse | None:
|
||||
"""Read an AK2 + its child AK3*/AK4* + AK5 segments, return the SetResponse.
|
||||
|
||||
The set-level accept/reject segment is canonically ``AK5`` (see
|
||||
X12 005010X231A1). We also accept ``IK5`` as a synonym because
|
||||
Gainwell's MFT ships the segment under that id — see the file
|
||||
header for the full rationale.
|
||||
|
||||
Returns None when called with a non-AK2 segment (defensive — the
|
||||
orchestrator only calls this when it sees AK2).
|
||||
"""
|
||||
@@ -175,11 +164,8 @@ def _consume_ak2(segments: list[list[str]], idx: int) -> SetFunctionalGroupRespo
|
||||
if idx < len(segments) and segments[idx][0] == "AK3":
|
||||
seg_errors, idx = _consume_ak3_ak4(segments, idx)
|
||||
# AK5 (set accept/reject) — required by the spec; default to "R" if missing.
|
||||
# Gainwell's MFT uses IK5 instead of AK5 (sender-specific segment id
|
||||
# that means the same thing); accept either. The default of "R"
|
||||
# matters: if the segment is missing entirely, the 999 is a reject.
|
||||
accept_code = "R"
|
||||
if idx < len(segments) and segments[idx][0] in ("AK5", "IK5"):
|
||||
if idx < len(segments) and segments[idx][0] == "AK5":
|
||||
ak5 = segments[idx]
|
||||
if len(ak5) > 1 and ak5[1]:
|
||||
accept_code = ak5[1]
|
||||
@@ -270,11 +256,8 @@ def parse_999_text(text: str, *, input_file: str = "") -> ParseResult999:
|
||||
set_responses.append(sr)
|
||||
# Advance past the AK2 + AK3*/AK4*/AK5 cluster
|
||||
# (re-walk from i+1 because _consume_ak2 doesn't return idx).
|
||||
# ``IK5`` is the Gainwell-specific synonym for ``AK5``
|
||||
# and must be in the consumed set here too (see
|
||||
# _consume_ak2 for the full rationale).
|
||||
i += 1
|
||||
while i < len(segments) and segments[i][0] in {"AK3", "AK4", "AK5", "IK5"}:
|
||||
while i < len(segments) and segments[i][0] in {"AK3", "AK4", "AK5"}:
|
||||
i += 1
|
||||
else:
|
||||
i += 1
|
||||
|
||||
@@ -66,8 +66,8 @@ class PayerConfig(BaseModel):
|
||||
# Lenient in v1 — see spec §9 R031.
|
||||
require_ref_g1_for_adjustments=False,
|
||||
allowed_bht06={"CH"},
|
||||
payer_id="CO_TXIX",
|
||||
payer_name="CO_TXIX",
|
||||
payer_id="SKCO0",
|
||||
payer_name="COHCPF",
|
||||
no_patient_loop=True,
|
||||
encounter_claim_in_same_batch=False,
|
||||
allowed_facility_qualifiers={"B"},
|
||||
|
||||
@@ -112,10 +112,7 @@ def _build_gs(sender_id: str, receiver_id: str, group_control_number: str) -> st
|
||||
_FUNCTIONAL_ID_HEALTH_CARE,
|
||||
sender_id,
|
||||
receiver_id,
|
||||
# GS-04 must be CCYYMMDD (8 digits) per X12 — ISA uses YYMMDD
|
||||
# (6 digits) for the older format, but the GS segment is the
|
||||
# newer ANSI X12 format and requires the full year.
|
||||
_today_yyyymmdd(),
|
||||
_today_yymmdd(),
|
||||
_today_hhmm(),
|
||||
group_control_number,
|
||||
"X",
|
||||
@@ -189,30 +186,17 @@ def _build_nm1(entity_id_qualifier: str, entity_type: str, name: str,
|
||||
return _ELEM.join(parts) + _SEG
|
||||
|
||||
|
||||
def _build_per(
|
||||
contact_name: str | None,
|
||||
contact_phone: str | None,
|
||||
contact_email: str | None = None,
|
||||
email_qual: str = "EM",
|
||||
) -> str:
|
||||
"""PER segment — submitter contact (Loop 1000A).
|
||||
|
||||
X12 005010X222A1 *requires* at least one PER segment in Loop 1000A
|
||||
(Submitter Name) and at least PER01 must be present, so this
|
||||
builder always emits a segment. PER01 = "IC" (Information Contact).
|
||||
|
||||
The remaining elements are filled from the available contact info:
|
||||
name, then email (preferred — Gainwell/HCPF expect this), then phone.
|
||||
"""
|
||||
parts = ["PER", "IC"]
|
||||
if contact_name:
|
||||
parts.append(contact_name)
|
||||
if contact_email:
|
||||
parts.append(email_qual) # PER03 — email qualifier (default "EM")
|
||||
parts.append(contact_email) # PER04 — the email itself
|
||||
elif contact_phone:
|
||||
parts.append("TE") # PER03 — phone qualifier
|
||||
parts.append(contact_phone) # PER04 — the phone itself
|
||||
def _build_per(contact_name: str | None, contact_phone: str | None) -> str:
|
||||
"""PER segment — submitter contact. Returns empty when no contact info."""
|
||||
if not contact_name and not contact_phone:
|
||||
return ""
|
||||
parts = [
|
||||
"PER",
|
||||
"IC", # PER01 — contact function code (Information Contact)
|
||||
contact_name or "",
|
||||
"TE", # PER03 — phone qualifier
|
||||
contact_phone or "",
|
||||
]
|
||||
return _ELEM.join(parts) + _SEG
|
||||
|
||||
|
||||
@@ -252,40 +236,26 @@ def _build_hl(hl_id: str, parent_id: str, level_code: str, child_code: str) -> s
|
||||
return _ELEM.join(parts) + _SEG
|
||||
|
||||
|
||||
def _build_sbr(
|
||||
individual_relationship_code: str | None,
|
||||
claim_filing_indicator_code: str | None,
|
||||
) -> str:
|
||||
def _build_sbr(relationship_code: str | None, member_id: str | None,
|
||||
payer_name: str | None) -> str:
|
||||
"""SBR segment — subscriber information.
|
||||
|
||||
Slot layout (X12 005010X222A1):
|
||||
SBR01 — Payer Responsibility Sequence Number Code. Default ``"P"``
|
||||
(Patient = primary). The parser does not capture this
|
||||
field on ``ClaimOutput`` so we default it.
|
||||
SBR02 — Individual Relationship Code. ``"18"`` = self, ``"01"`` = spouse, etc.
|
||||
The parser does not capture this either; we default ``"18"``
|
||||
for the common self-pay case.
|
||||
SBR09 — Claim Filing Indicator Code. ``"MC"`` for Medicaid,
|
||||
``"16"`` for Medicare Part B, etc. The canonical
|
||||
PayerConfig837 carries ``sbr09_default``; we thread it in
|
||||
from the caller.
|
||||
|
||||
The member_id and payer name do NOT belong in SBR — the member_id
|
||||
lives in NM109 of the NM1*IL segment, and the payer name is in
|
||||
NM103 of NM1*PR. (Earlier revisions of this function put them in
|
||||
SBR06 / SBR09, which is wrong and rejected by HCPF.)
|
||||
SBR01 (relationship code) defaults to ``"P"`` (Patient = self) which is
|
||||
the most common case for professional claims; the parser does not store
|
||||
this on the canonical Subscriber model so we cannot thread it through
|
||||
without adding a model field.
|
||||
"""
|
||||
parts = [
|
||||
"SBR",
|
||||
"P", # SBR01 — primary
|
||||
individual_relationship_code or "18", # SBR02 — self
|
||||
"", # SBR03 — group number
|
||||
"", # SBR04 — group name
|
||||
"", # SBR05 — insurance type code
|
||||
"", # SBR06 — coordination of benefits
|
||||
"", # SBR07 — yes/no condition
|
||||
"", # SBR08 — employment status code
|
||||
claim_filing_indicator_code or "", # SBR09 — claim filing indicator
|
||||
relationship_code or "P",
|
||||
"", # SBR02 — group number
|
||||
"", # SBR03 — group name
|
||||
"", # SBR04 — claim filing indicator code
|
||||
"", # SBR05 — sequence number code
|
||||
payer_name or "", # SBR06 — claim filing indicator code (CO uses MC)
|
||||
"", # SBR07
|
||||
"", # SBR08
|
||||
member_id or "", # SBR09 — claim submitter's id
|
||||
]
|
||||
return _ELEM.join(parts) + _SEG
|
||||
|
||||
@@ -330,11 +300,7 @@ def _build_clm(claim) -> str:
|
||||
clm05, # CLM05 — composite POS:qualifier:frequency_code
|
||||
claim.provider_signature or "Y", # CLM06
|
||||
claim.assignment or "Y", # CLM07
|
||||
# CLM08 — Benefits Assignment Certification. X12 837P requires
|
||||
# this when CLM07 = "Y" (the common case for in-network
|
||||
# professional claims). Default to "Y" when the source did
|
||||
# not capture one — matches what 99% of HCPF files look like.
|
||||
claim.benefits_assignment_certification or "Y", # CLM08
|
||||
"", # CLM08 — benefit assignment certification
|
||||
claim.release_of_info or "Y", # CLM09
|
||||
]
|
||||
return _ELEM.join(parts) + _SEG
|
||||
@@ -359,41 +325,21 @@ def _build_lx(line_number: int) -> str:
|
||||
return _ELEM.join(["LX", str(line_number)]) + _SEG
|
||||
|
||||
|
||||
def _build_sv1(line, *, dx_pointer: str | None = None) -> str:
|
||||
"""SV1 segment — professional service line.
|
||||
|
||||
X12 005010X222A1 layout (837P):
|
||||
SV1-01 composite procedure identifier
|
||||
SV1-02 monetary amount (charge)
|
||||
SV1-03 unit of basis measurement (UN, MJ, etc.) — ``line.unit_type``
|
||||
SV1-04 service unit count — ``line.units``
|
||||
SV1-05 place of service code — ``line.place_of_service``
|
||||
SV1-06 **NOT USED** by this guide (must be empty)
|
||||
SV1-07 diagnosis code pointer — ``dx_pointer`` (required when the
|
||||
parent claim has an HI segment)
|
||||
|
||||
The parser captures the original SV1-07 pointer when present; the
|
||||
serializer defaults it to ``"1"`` (pointing at the first HI
|
||||
diagnosis) when the claim has diagnoses and no explicit pointer
|
||||
was captured. When the claim has no HI segment we leave SV1-07
|
||||
empty to match the spec.
|
||||
"""
|
||||
def _build_sv1(line) -> str:
|
||||
"""SV1 segment — professional service line."""
|
||||
proc = line.procedure
|
||||
code = proc.code if proc else ""
|
||||
mods = proc.modifiers if proc else []
|
||||
composite = "HC:" + code + "".join(f":{m}" for m in (mods or [])[:4])
|
||||
charge = f"{Decimal(line.charge or 0):.2f}"
|
||||
units = f"{Decimal(line.units):g}" if line.units is not None else "1"
|
||||
sv1_07 = dx_pointer or ""
|
||||
parts = [
|
||||
"SV1",
|
||||
composite, # SV1-01
|
||||
charge, # SV1-02
|
||||
line.unit_type or "UN", # SV1-03 — unit basis code
|
||||
units, # SV1-04
|
||||
line.place_of_service or "", # SV1-05
|
||||
"", # SV1-06 — NOT USED in 837P
|
||||
sv1_07, # SV1-07 — diagnosis pointer
|
||||
composite,
|
||||
charge,
|
||||
line.unit_type or "UN",
|
||||
units,
|
||||
line.place_of_service or "",
|
||||
]
|
||||
return _ELEM.join(parts) + _SEG
|
||||
|
||||
@@ -411,20 +357,15 @@ def _build_dtp_472(service_date: date | None) -> str:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_submitter_block(
|
||||
sender_id: str,
|
||||
submitter_name: str | None,
|
||||
contact_name: str | None,
|
||||
contact_phone: str | None,
|
||||
contact_email: str | None = None,
|
||||
email_qual: str = "EM",
|
||||
) -> list[str]:
|
||||
def _build_submitter_block(sender_id: str, submitter_name: str | None,
|
||||
contact_name: str | None,
|
||||
contact_phone: str | None) -> list[str]:
|
||||
out = [
|
||||
_build_nm1("41", "41", submitter_name or sender_id, "46", sender_id),
|
||||
]
|
||||
# PER is required by X12 (at least PER01). _build_per always emits
|
||||
# the segment; the submitter block always has exactly one.
|
||||
out.append(_build_per(contact_name, contact_phone, contact_email, email_qual))
|
||||
per = _build_per(contact_name, contact_phone)
|
||||
if per:
|
||||
out.append(per)
|
||||
return out
|
||||
|
||||
|
||||
@@ -454,14 +395,11 @@ def _build_billing_provider_block(provider) -> list[str]:
|
||||
return out
|
||||
|
||||
|
||||
def _build_subscriber_block(
|
||||
subscriber,
|
||||
claim_filing_indicator_code: str | None,
|
||||
) -> list[str]:
|
||||
def _build_subscriber_block(subscriber, payer_name: str | None) -> list[str]:
|
||||
"""HL*2 → SBR → NM1*IL → N3 → N4 → DMG. Subscriber has no children."""
|
||||
out = [
|
||||
_build_hl("2", "1", "22", "0"), # HL*2 — subscriber, 0 children
|
||||
_build_sbr("18", claim_filing_indicator_code),
|
||||
_build_sbr("18", subscriber.member_id, payer_name),
|
||||
_build_nm1(
|
||||
"IL", "IL",
|
||||
f"{subscriber.last_name} {subscriber.first_name}".strip(),
|
||||
@@ -489,24 +427,12 @@ def _build_payer_block(payer) -> list[str]:
|
||||
]
|
||||
|
||||
|
||||
def _build_service_lines_block(service_lines, *, has_diagnoses: bool = False) -> list[str]:
|
||||
"""Per line: LX / SV1 / DTP*472 / REF*6R.
|
||||
|
||||
``has_diagnoses`` is True when the parent claim emits an HI segment;
|
||||
in that case SV1-07 is required by X12 and we default each line's
|
||||
pointer to ``"1"`` (the first HI diagnosis) unless the source
|
||||
captured a different pointer on the line itself.
|
||||
"""
|
||||
def _build_service_lines_block(service_lines) -> list[str]:
|
||||
"""Per line: LX / SV1 / DTP*472 / REF*6R."""
|
||||
out: list[str] = []
|
||||
for idx, line in enumerate(service_lines or [], start=1):
|
||||
out.append(_build_lx(idx))
|
||||
# Prefer the line's captured pointer (parser pulled SV1-07
|
||||
# when present). Fall back to "1" only when the claim has
|
||||
# diagnoses and the source had no explicit pointer — the
|
||||
# common single-diagnosis case.
|
||||
line_pointer = getattr(line, "dx_pointer", None)
|
||||
effective_pointer = line_pointer or ("1" if has_diagnoses else "")
|
||||
out.append(_build_sv1(line, dx_pointer=effective_pointer))
|
||||
out.append(_build_sv1(line))
|
||||
dtp = _build_dtp_472(line.service_date)
|
||||
if dtp:
|
||||
out.append(dtp)
|
||||
@@ -524,10 +450,7 @@ def serialize_837(
|
||||
submitter_name: str | None = None,
|
||||
submitter_contact_name: str | None = None,
|
||||
submitter_contact_phone: str | None = None,
|
||||
submitter_contact_email: str | None = None,
|
||||
submitter_contact_email_qual: str = "EM",
|
||||
receiver_name: str | None = None,
|
||||
claim_filing_indicator_code: str | None = None,
|
||||
interchange_control_number: str = "000000001",
|
||||
group_control_number: str = "1",
|
||||
) -> str:
|
||||
@@ -539,16 +462,6 @@ def serialize_837(
|
||||
(``"CYCLONE"`` / ``"RECEIVER"``) but real deployments should pass
|
||||
the configured values.
|
||||
|
||||
The submitter block (Loop 1000A) always emits a PER segment per the
|
||||
X12 spec — the canonical clearhouse config provides the contact
|
||||
name and email so callers should pass them through.
|
||||
|
||||
The claim filing indicator (SBR09) is read from the per-payer
|
||||
config (``PayerConfig837.sbr09_default``); callers should pass it
|
||||
in. If not passed, SBR09 is left empty (which causes the
|
||||
:func:`cyclone.parsers.validator._r202_sbr09_allowed` rule to skip
|
||||
its check — degraded but not a hard error).
|
||||
|
||||
Editable fields (CLM, REF*G1, HI, service-line SV1, DTP*472) are
|
||||
emitted from the canonical ``ClaimOutput`` fields, so post-parse
|
||||
edits propagate to the output.
|
||||
@@ -568,13 +481,11 @@ def serialize_837(
|
||||
),
|
||||
]
|
||||
segments.extend(_build_submitter_block(
|
||||
sender_id, submitter_name,
|
||||
submitter_contact_name, submitter_contact_phone, submitter_contact_email,
|
||||
submitter_contact_email_qual,
|
||||
sender_id, submitter_name, submitter_contact_name, submitter_contact_phone,
|
||||
))
|
||||
segments.extend(_build_receiver_block(receiver_id, receiver_name))
|
||||
segments.extend(_build_billing_provider_block(claim.billing_provider))
|
||||
segments.extend(_build_subscriber_block(claim.subscriber, claim_filing_indicator_code))
|
||||
segments.extend(_build_subscriber_block(claim.subscriber, claim.payer.name))
|
||||
segments.extend(_build_payer_block(claim.payer))
|
||||
|
||||
# Claim-level editable segments.
|
||||
@@ -586,10 +497,7 @@ def serialize_837(
|
||||
segments.append(_build_hi(claim.diagnoses))
|
||||
|
||||
# Service lines (LX / SV1 / DTP*472 / REF*6R).
|
||||
segments.extend(_build_service_lines_block(
|
||||
claim.service_lines,
|
||||
has_diagnoses=bool(claim.diagnoses),
|
||||
))
|
||||
segments.extend(_build_service_lines_block(claim.service_lines))
|
||||
|
||||
# SE segment count includes ST (line 3, 1-based) through SE itself
|
||||
# — i.e. the entire ST..SE block inclusive.
|
||||
@@ -606,23 +514,15 @@ def serialize_837_for_resubmit(
|
||||
claim: ClaimOutput,
|
||||
*,
|
||||
interchange_index: int,
|
||||
**kwargs,
|
||||
) -> str:
|
||||
"""Like :func:`serialize_837` but assigns deterministic-but-unique
|
||||
interchange + group control numbers for a bundle position.
|
||||
|
||||
Interchange number = ``f"{interchange_index:09d}"``.
|
||||
Group number = ``str(interchange_index)``.
|
||||
|
||||
All other keyword arguments (sender_id, receiver_id, submitter_*
|
||||
and receiver_* contact info, claim_filing_indicator_code) are
|
||||
forwarded to :func:`serialize_837` unchanged so callers — like the
|
||||
export and SFTP-submit endpoints — can pass through clearhouse +
|
||||
payer config without copying the signature.
|
||||
"""
|
||||
return serialize_837(
|
||||
claim,
|
||||
interchange_control_number=f"{interchange_index:09d}",
|
||||
group_control_number=str(interchange_index),
|
||||
**kwargs,
|
||||
)
|
||||
@@ -35,36 +35,6 @@ def _r020_npi_format(claim: ClaimOutput, _: PayerConfig) -> Iterable[ValidationI
|
||||
yield ValidationIssue(rule="R020_npi_format", severity="error", message=f"Billing provider NPI must be 10 digits, got {claim.billing_provider.npi!r}")
|
||||
|
||||
|
||||
def _r021_npi_checksum(claim: ClaimOutput, _: PayerConfig) -> Iterable[ValidationIssue]:
|
||||
"""SP20: validate the billing-provider NPI's Luhn check digit.
|
||||
|
||||
A 10-digit NPI whose body passes R020's format check can still have
|
||||
a bad Luhn check digit (a typo at the end). Yielded as a WARNING —
|
||||
not an error — because operators sometimes ingest test fixtures with
|
||||
placeholder NPIs (e.g. all-same-digit) and we don't want to block
|
||||
that path. Local-only check, no NPPES round-trip.
|
||||
"""
|
||||
npi = claim.billing_provider.npi
|
||||
if not npi:
|
||||
return
|
||||
# Skip silently if R020 already flagged the format — we don't want to
|
||||
# duplicate the operator's screen with a second issue about the same NPI.
|
||||
if not NPI_RE.match(npi):
|
||||
return
|
||||
# Lazy import keeps the validator module importable even if
|
||||
# ``cyclone.npi`` is unavailable (e.g. in some legacy test setups).
|
||||
try:
|
||||
from cyclone.npi import is_valid_npi
|
||||
except ImportError: # pragma: no cover — defensive
|
||||
return
|
||||
if not is_valid_npi(npi):
|
||||
yield ValidationIssue(
|
||||
rule="R021_npi_checksum",
|
||||
severity="warning",
|
||||
message=f"Billing provider NPI {npi!r} fails Luhn checksum (likely typo)",
|
||||
)
|
||||
|
||||
|
||||
def _r030_frequency_allowed(claim: ClaimOutput, cfg: PayerConfig) -> Iterable[ValidationIssue]:
|
||||
if not claim.claim.frequency_code:
|
||||
return
|
||||
@@ -422,7 +392,6 @@ _RULES: list[Rule] = [
|
||||
_r010_clm01_present,
|
||||
_r011_total_charge_positive,
|
||||
_r020_npi_format,
|
||||
_r021_npi_checksum,
|
||||
_r030_frequency_allowed,
|
||||
_r031_ref_g1_optional,
|
||||
_r034_ref_g1_required,
|
||||
|
||||
@@ -100,19 +100,6 @@ class EventBus:
|
||||
yield await queue.get()
|
||||
queue.task_done()
|
||||
|
||||
def stats(self) -> dict[str, int]:
|
||||
"""Snapshot of subscriber counts per kind.
|
||||
|
||||
Used by ``/api/health`` (SP19) and the admin diagnostics page.
|
||||
Returns ``{kind: count}`` for every kind with at least one
|
||||
subscriber; kinds with zero subscribers are omitted.
|
||||
"""
|
||||
return {
|
||||
kind: len(subs)
|
||||
for kind, subs in self._subscribers.items()
|
||||
if subs
|
||||
}
|
||||
|
||||
|
||||
def get_event_bus() -> EventBus:
|
||||
"""Return the process-wide EventBus attached to the FastAPI app state.
|
||||
|
||||
@@ -17,7 +17,7 @@ Match algorithm:
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, datetime, timezone
|
||||
from decimal import Decimal
|
||||
from typing import Iterable, Optional, Protocol
|
||||
@@ -50,18 +50,6 @@ class Match:
|
||||
remittance: _RemitLike
|
||||
strategy: str
|
||||
is_reversal: bool
|
||||
# SP31: which content-keys the 2-of-3 (or 3-of-3) rule agreed on.
|
||||
# Populated for ``score-auto`` matches; empty for ``pcn-exact`` /
|
||||
# ``manual`` (PCN-exact only uses the PCN key by definition; manual
|
||||
# is operator-driven). Floats into the ``auto_matched_835``
|
||||
# ActivityEvent payload so the audit trail records *why* the link
|
||||
# fired. Transient — the ORM Match row does not persist these
|
||||
# (spec keeps the audit trail in ActivityEvent.payload_json only).
|
||||
keys_matched: set[str] = field(default_factory=set)
|
||||
# SP31: how many candidates were in the ±30-day window pool when
|
||||
# the score-auto match fired. Useful when an operator is reviewing
|
||||
# a borderline auto-link — 1-of-1 is unambiguous; 1-of-5 was lucky.
|
||||
candidate_count: int = 0
|
||||
|
||||
|
||||
def match(
|
||||
@@ -96,7 +84,7 @@ def match(
|
||||
matches.append(Match(
|
||||
claim=chosen,
|
||||
remittance=r,
|
||||
strategy="pcn-exact",
|
||||
strategy="auto",
|
||||
is_reversal=r.is_reversal,
|
||||
))
|
||||
used_claim_ids.add(chosen.id)
|
||||
@@ -129,163 +117,6 @@ def _pick_claim(
|
||||
return None
|
||||
|
||||
|
||||
# --- SP31: content-keys fallback matcher -----------------------------------
|
||||
|
||||
CHARGE_TOLERANCE = Decimal("0.01")
|
||||
KEYS_REQUIRED = 2
|
||||
|
||||
|
||||
def _content_keys_match(remit, claim) -> set[str]:
|
||||
"""Return the set of content-keys that agree between ``remit`` and ``claim``.
|
||||
|
||||
Compares {``pcn``, ``charge``, ``npi``} between the two sides and returns
|
||||
the subset whose values match. Caller checks ``len(returned) >= KEYS_REQUIRED``
|
||||
to decide whether to auto-link. Returning the matched keys (rather than a
|
||||
bool) lets the ActivityEvent payload record exactly which 2-of-3 (or 3-of-3)
|
||||
produced the auto-link — the audit trail for the SP31 content-keys rule.
|
||||
|
||||
Charge compared with ``CHARGE_TOLERANCE`` tolerance (rounding drift).
|
||||
NPI counts as "not matched" when the remit's NPI is empty.
|
||||
|
||||
Uses duck-typing via ``getattr`` so the helper works against both:
|
||||
- shim / dataclass test objects (planned names: total_charge_amount,
|
||||
total_charge, rendering_provider_npi), and
|
||||
- real SQLAlchemy ORM instances (Claim.charge_amount, Remittance.total_charge,
|
||||
Claim.provider_npi, Claim.rendering_provider_npi,
|
||||
Remittance.rendering_provider_npi).
|
||||
|
||||
Production note (SP32): the typed ``rendering_provider_npi`` column is
|
||||
the primary read for both sides (Claim = NM1*82, Remit = NM1*1P). Legacy
|
||||
rows pre-0019 still carry the value in ``raw_json``; the read order is:
|
||||
- NPI: typed column → raw_json (``service_provider_npi`` for remit,
|
||||
``rendering_provider_npi`` for claim) → claim ``provider_npi``
|
||||
(billing fallback for legacy 837p rows without NM1*82 extraction).
|
||||
- Charge: planned attribute name → real ORM column → raw_json.
|
||||
The ``raw_json`` fallback is what makes this helper safe against a
|
||||
raw ``Remittance`` ORM instance from ``reconcile.run()``.
|
||||
"""
|
||||
matched: set[str] = set()
|
||||
|
||||
# PCN: payer_claim_control_number ↔ patient_control_number (stripped).
|
||||
if (getattr(remit, "payer_claim_control_number", "") or "").strip() == \
|
||||
(getattr(claim, "patient_control_number", "") or "").strip():
|
||||
matched.add("pcn")
|
||||
|
||||
# Charge: prefer the planned attribute names, fall back to real ORM columns,
|
||||
# then to raw_json (where the 835 parser stores CLP03). Explicit ``is None``
|
||||
# checks avoid the ``Decimal("0")`` truthiness footgun — ``Decimal("0")``
|
||||
# is truthy in a boolean context but ``or`` would still let it through;
|
||||
# the more important property is that we don't accidentally replace a real
|
||||
# value with a default.
|
||||
_remit_charge = getattr(remit, "total_charge_amount", None)
|
||||
if _remit_charge is None:
|
||||
_remit_charge = getattr(remit, "total_charge", None)
|
||||
if _remit_charge is None:
|
||||
_remit_charge = (getattr(remit, "raw_json", None) or {}).get("total_charge_amount")
|
||||
|
||||
_claim_charge = getattr(claim, "total_charge", None)
|
||||
if _claim_charge is None:
|
||||
_claim_charge = getattr(claim, "charge_amount", None)
|
||||
if _claim_charge is None:
|
||||
_claim_charge = (getattr(claim, "raw_json", None) or {}).get("total_charge_amount")
|
||||
|
||||
if _remit_charge is not None and _claim_charge is not None and \
|
||||
abs(_remit_charge - _claim_charge) < CHARGE_TOLERANCE:
|
||||
matched.add("charge")
|
||||
|
||||
# NPI: typed-column primary path (SP32), raw_json fallback (legacy rows).
|
||||
# Remit reads rendering_provider_npi (single value, D4); claim reads
|
||||
# rendering_provider_npi first, then falls back to provider_npi (billing)
|
||||
# for legacy rows where the 837p parser hadn't yet extracted NM1*82.
|
||||
remit_npi = (
|
||||
(getattr(remit, "rendering_provider_npi", "") or "")
|
||||
or ((getattr(remit, "raw_json", None) or {}).get("service_provider_npi", "") or "")
|
||||
or ((getattr(remit, "raw_json", None) or {}).get("rendering_provider_npi", "") or "")
|
||||
).strip()
|
||||
claim_npi = (
|
||||
(getattr(claim, "rendering_provider_npi", "") or "")
|
||||
or (getattr(claim, "provider_npi", "") or "")
|
||||
or ((getattr(claim, "raw_json", None) or {}).get("rendering_provider_npi", "") or "")
|
||||
).strip()
|
||||
if remit_npi and remit_npi == claim_npi:
|
||||
matched.add("npi")
|
||||
|
||||
return matched
|
||||
|
||||
|
||||
# --- SP31: DB-side fallback candidate matcher -------------------------------
|
||||
|
||||
SCORE_FALLBACK_WINDOW_DAYS = 30
|
||||
|
||||
|
||||
def _score_fallback_candidates(session, remit) -> Optional[tuple[str, set[str], int]]:
|
||||
"""Return (claim_id, keys_matched, candidate_count) for a unique 2-of-3 match.
|
||||
|
||||
Queries a ±SCORE_FALLBACK_WINDOW_DAYS candidate pool filtered by:
|
||||
- claim.matched_remittance_id IS NULL (not yet linked)
|
||||
- claim.state NOT IN terminal set (PAID, DENIED, REJECTED,
|
||||
REVERSED, RECONCILED)
|
||||
- claim.service_date_from within window of remit.service_date
|
||||
|
||||
Returns ``(claim_id, keys_matched, candidate_count)`` when exactly one
|
||||
candidate passes the 2-of-3 rule (via :func:`_content_keys_match`),
|
||||
otherwise ``None``. ``keys_matched`` is the set returned by the helper
|
||||
(``{"pcn", "charge"}`` or any 2-of-3 / 3-of-3 combination) — the
|
||||
ActivityEvent payload records it as the audit trail for *why* the
|
||||
auto-link fired. ``candidate_count`` is the size of the window pool
|
||||
(independent of how many matched) so the operator can see whether
|
||||
the match was 1-of-N or 1-of-1.
|
||||
|
||||
Ambiguous matches (2+ candidates) also return ``None`` — they land
|
||||
in the Inbox Unlinked lane for manual review.
|
||||
|
||||
Note on payer_id: ``Remittance`` has no ``payer_id`` column
|
||||
(``Claim.payer_id`` does, and is the source of truth on the claim
|
||||
side; the 835 parser stores payer_id in ``Remittance.raw_json``).
|
||||
Filtering by payer would require either a raw_json read or a model
|
||||
column add — out of scope for SP31 Task 3. The PCN itself is
|
||||
expected to be unique within a payer's submission, so cross-payer
|
||||
collisions are unlikely. Revisit if production shows otherwise.
|
||||
"""
|
||||
from sqlalchemy import select, and_
|
||||
from datetime import timedelta
|
||||
from cyclone.db import Claim, ClaimState
|
||||
|
||||
TERMINAL = {
|
||||
ClaimState.PAID, ClaimState.DENIED, ClaimState.REJECTED,
|
||||
ClaimState.REVERSED, ClaimState.RECONCILED,
|
||||
}
|
||||
|
||||
if remit.service_date is None:
|
||||
return None # need a date for the window query
|
||||
|
||||
window_lo = remit.service_date - timedelta(days=SCORE_FALLBACK_WINDOW_DAYS)
|
||||
window_hi = remit.service_date + timedelta(days=SCORE_FALLBACK_WINDOW_DAYS)
|
||||
|
||||
candidates = list(session.execute(
|
||||
select(Claim).where(
|
||||
and_(
|
||||
Claim.matched_remittance_id.is_(None),
|
||||
Claim.service_date_from >= window_lo,
|
||||
Claim.service_date_from <= window_hi,
|
||||
Claim.state.notin_([s.value for s in TERMINAL]),
|
||||
)
|
||||
)
|
||||
).scalars().all())
|
||||
|
||||
matched_pairs = [
|
||||
(c.id, _content_keys_match(remit, c))
|
||||
for c in candidates
|
||||
]
|
||||
matched_pairs = [
|
||||
(cid, ks) for cid, ks in matched_pairs if len(ks) >= KEYS_REQUIRED
|
||||
]
|
||||
if len(matched_pairs) == 1:
|
||||
cid, keys_matched = matched_pairs[0]
|
||||
return (cid, keys_matched, len(candidates))
|
||||
return None # 0 matches OR 2+ (ambiguous)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ApplyIntent:
|
||||
"""Result of applying a match. The caller persists this.
|
||||
@@ -393,22 +224,17 @@ class ReconcileResult:
|
||||
def run(session, batch_id: str) -> ReconcileResult:
|
||||
"""Orchestrate reconciliation for an 835 batch.
|
||||
|
||||
Expects Batch + Remittance + CasAdjustment rows already added to
|
||||
``session`` (not yet committed). SP27 Task 10 calls this from
|
||||
inside ``CycloneStore.add``'s ingest session, BEFORE
|
||||
``s.commit()`` — so the whole 835 ingest (rows + reconcile) is
|
||||
a single transaction. If anything here raises, the ingest rolls
|
||||
back; no half-reconciled batch ever lands.
|
||||
|
||||
Loads the new remittances + all currently-unmatched Claims, calls
|
||||
match(), then applies each match's intent inside the caller's session.
|
||||
Expects Batch + Remittance + CasAdjustment rows already persisted
|
||||
(this is called from store.add() AFTER commit). Loads the new
|
||||
remittances + all currently-unmatched Claims, calls match(), then
|
||||
applies each match's intent inside the caller's session.
|
||||
|
||||
Returns counts. Does NOT commit; caller controls transaction.
|
||||
Raises on failure (caller catches and writes activity event).
|
||||
"""
|
||||
from sqlalchemy import select
|
||||
from cyclone.db import (
|
||||
Batch, Claim, Remittance, CasAdjustment, Match as MatchORM, ActivityEvent,
|
||||
Batch, Claim, Remittance, CasAdjustment, Match, ActivityEvent,
|
||||
ClaimState,
|
||||
)
|
||||
|
||||
@@ -426,34 +252,6 @@ def run(session, batch_id: str) -> ReconcileResult:
|
||||
|
||||
matches = match(unmatched_claims, new_remits)
|
||||
|
||||
# SP31: content-keys fallback for remits that PCN-exact couldn't pair.
|
||||
# Operates on the still-unmatched remits so we don't double-match.
|
||||
matched_remit_ids = {m.remittance.id for m in matches}
|
||||
used_claim_ids = {m.claim.id for m in matches}
|
||||
for remit in new_remits:
|
||||
if remit.id in matched_remit_ids:
|
||||
continue
|
||||
if getattr(remit, "claim_id", None) is not None:
|
||||
continue # already linked
|
||||
fallback_result = _score_fallback_candidates(session, remit)
|
||||
if fallback_result is None:
|
||||
continue
|
||||
matched_claim_id, keys_matched, candidate_count = fallback_result
|
||||
# Find the claim object in the unmatched_claims list (it was loaded).
|
||||
target_claim = next(
|
||||
(c for c in unmatched_claims if c.id == matched_claim_id),
|
||||
None,
|
||||
)
|
||||
if target_claim is None or target_claim.id in used_claim_ids:
|
||||
continue
|
||||
matches.append(Match(
|
||||
claim=target_claim, remittance=remit,
|
||||
strategy="score-auto", is_reversal=remit.is_reversal,
|
||||
keys_matched=keys_matched,
|
||||
candidate_count=candidate_count,
|
||||
))
|
||||
used_claim_ids.add(target_claim.id)
|
||||
|
||||
applied = 0
|
||||
skipped = 0
|
||||
for m in matches:
|
||||
@@ -477,7 +275,7 @@ def run(session, batch_id: str) -> ReconcileResult:
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
session.add(MatchORM(
|
||||
session.add(Match(
|
||||
claim_id=m.claim.id, remittance_id=m.remittance.id,
|
||||
strategy=m.strategy,
|
||||
matched_at=datetime.now(timezone.utc),
|
||||
@@ -495,25 +293,10 @@ def run(session, batch_id: str) -> ReconcileResult:
|
||||
m.remittance.claim_id = m.claim.id
|
||||
|
||||
session.add(ActivityEvent(
|
||||
ts=datetime.now(timezone.utc),
|
||||
kind=("auto_matched_835" if m.strategy == "score-auto" else intent.activity_kind),
|
||||
ts=datetime.now(timezone.utc), kind=intent.activity_kind,
|
||||
batch_id=batch_id, claim_id=m.claim.id,
|
||||
remittance_id=m.remittance.id,
|
||||
payload_json=(
|
||||
{
|
||||
"new_state": m.claim.state.value, "strategy": m.strategy,
|
||||
# SP31 spec D8: the auto_matched_835 payload records
|
||||
# which content-keys the 2-of-3 (or 3-of-3) rule
|
||||
# agreed on, plus how many candidates were in the
|
||||
# window pool. ``sorted()`` converts the set to a
|
||||
# JSON-serializable list and pins a deterministic
|
||||
# order for tests + audit reads.
|
||||
"keys_matched": sorted(m.keys_matched),
|
||||
"candidate_count": m.candidate_count,
|
||||
}
|
||||
if m.strategy == "score-auto"
|
||||
else {"new_state": m.claim.state.value}
|
||||
),
|
||||
payload_json={"new_state": m.claim.state.value},
|
||||
))
|
||||
applied += 1
|
||||
|
||||
|
||||
@@ -1,736 +0,0 @@
|
||||
"""Background inbound MFT polling scheduler (SP16).
|
||||
|
||||
Turns Cyclone from a manual upload tool into a live clearinghouse:
|
||||
a long-running asyncio task that periodically polls the Gainwell MFT
|
||||
inbound path, downloads each new file, and runs it through the
|
||||
appropriate parser. The operator no longer has to watch for inbound
|
||||
files and POST them to ``/api/parse-999`` etc. by hand.
|
||||
|
||||
Design constraints
|
||||
------------------
|
||||
|
||||
* **Idempotent.** A re-tick (or a process restart) must not re-parse
|
||||
the same inbound file. We persist a ``processed_inbound_files`` row
|
||||
per file and skip ones we've already seen.
|
||||
|
||||
* **Crash-safe.** If the parser raises or the DB write fails, the
|
||||
scheduler logs the error, records an ``error`` row, and moves on.
|
||||
The next tick continues from the next file.
|
||||
|
||||
* **Bounded blast radius.** A bad file must not stop the scheduler.
|
||||
Each file is wrapped in try/except so a 999 parser crash doesn't
|
||||
prevent us from processing the next inbound 835.
|
||||
|
||||
* **Operator-controlled.** The scheduler is OFF by default; the
|
||||
operator must explicitly start it (``POST /api/admin/scheduler/start``
|
||||
or ``CYCLONE_SCHEDULER_AUTOSTART=true``). When it's running, status
|
||||
is exposed via ``GET /api/admin/scheduler/status``.
|
||||
|
||||
* **No threading.** We use ``asyncio.create_task`` + ``asyncio.sleep``
|
||||
rather than APScheduler or threading because the rest of the
|
||||
codebase is asyncio-native (FastAPI). The whole polling loop runs
|
||||
in the FastAPI event loop on the main thread.
|
||||
|
||||
Compliance: SP16 is operational metadata only. Inbound file
|
||||
processing is NOT part of the HIPAA audit chain (SP11) — an SFTP
|
||||
outage shouldn't pollute the audit log with parser errors.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import traceback
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Optional
|
||||
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.store import store as cycl_store
|
||||
from cyclone.audit_log import AuditEvent, append_event
|
||||
from cyclone.clearhouse import InboundFile, SftpClient
|
||||
from cyclone.db import ProcessedInboundFile
|
||||
from cyclone.edi.filenames import parse_inbound_filename
|
||||
from cyclone.handlers import handle_277ca as _handle_277ca
|
||||
from cyclone.handlers import handle_835 as _handle_835
|
||||
from cyclone.handlers import handle_999 as _handle_999
|
||||
from cyclone.handlers import handle_ta1 as _handle_ta1
|
||||
from cyclone.inbox_state_277ca import apply_277ca_rejections
|
||||
from cyclone.providers import SftpBlock
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Status values for ProcessedInboundFile.status.
|
||||
STATUS_OK = "ok"
|
||||
STATUS_ERROR = "error"
|
||||
STATUS_SKIPPED = "skipped"
|
||||
STATUS_PENDING = "pending"
|
||||
|
||||
|
||||
# How long reconfigure_scheduler waits for the in-flight tick to
|
||||
# drain before cancelling the old task. Matches Scheduler.stop().
|
||||
_DRAIN_TIMEOUT_SECONDS = 30
|
||||
|
||||
|
||||
# File types we know how to route. The HCPF set is broader (270/271/
|
||||
# 276/277/278/820/834/ENCR) but Cyclone's parser only covers the
|
||||
# four below. Files with unknown types are recorded as ``skipped``
|
||||
# so the operator can see them in the audit table.
|
||||
ROUTED_FILE_TYPES = frozenset({"999", "835", "277", "277CA", "TA1"})
|
||||
|
||||
|
||||
@dataclass
|
||||
class TickResult:
|
||||
"""Outcome of a single scheduler tick (one poll cycle)."""
|
||||
|
||||
started_at: datetime
|
||||
finished_at: Optional[datetime] = None
|
||||
files_seen: int = 0
|
||||
files_processed: int = 0
|
||||
files_skipped: int = 0
|
||||
files_errored: int = 0
|
||||
errors: list[str] = field(default_factory=list)
|
||||
# SP27 Task 9: ``sftp_failed`` is the discriminator that separates
|
||||
# SFTP-side failures (the listing step) from per-file processing
|
||||
# errors. ``tick()`` keys off this flag — NOT ``result.errors`` —
|
||||
# so a single malformed 999 cannot flip the operator's destructive
|
||||
# pill. Set in ``_tick_impl``'s listing-error branches only.
|
||||
sftp_failed: bool = False
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"started_at": self.started_at.isoformat(),
|
||||
"finished_at": (
|
||||
self.finished_at.isoformat() if self.finished_at else None
|
||||
),
|
||||
"files_seen": self.files_seen,
|
||||
"files_processed": self.files_processed,
|
||||
"files_skipped": self.files_skipped,
|
||||
"files_errored": self.files_errored,
|
||||
"errors": list(self.errors),
|
||||
"sftp_failed": self.sftp_failed,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class SchedulerStatus:
|
||||
"""Snapshot of the scheduler's runtime state.
|
||||
|
||||
SP27 Task 9 added the four SFTP-error fields
|
||||
(``consecutive_failures``, ``last_error``, ``last_error_at``,
|
||||
``last_sftp_attempt_at``) so the operator's UI can tell "all
|
||||
quiet on the MFT front" from "we've been unable to reach the
|
||||
MFT server for 3 hours". The previous 06/25 incident went
|
||||
unnoticed because the status dict had no signal for a hung
|
||||
SFTP poll.
|
||||
"""
|
||||
|
||||
running: bool
|
||||
poll_interval_seconds: int
|
||||
sftp_block_name: str
|
||||
last_poll_at: Optional[datetime]
|
||||
poll_count: int
|
||||
total_processed: int
|
||||
total_skipped: int
|
||||
total_errored: int
|
||||
last_tick: Optional[TickResult] = None
|
||||
# SP27 Task 9: SFTP-error surfacing. The UI flips to a destructive
|
||||
# pill when ``consecutive_failures >= 3`` (``CYCLONE_SCHEDULER_FAIL_THRESHOLD``
|
||||
# in the operator pill config; not enforced server-side).
|
||||
consecutive_failures: int = 0
|
||||
last_error: Optional[str] = None
|
||||
last_error_at: Optional[datetime] = None
|
||||
last_sftp_attempt_at: Optional[datetime] = None
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"running": self.running,
|
||||
"poll_interval_seconds": self.poll_interval_seconds,
|
||||
"sftp_block_name": self.sftp_block_name,
|
||||
"last_poll_at": (
|
||||
self.last_poll_at.isoformat() if self.last_poll_at else None
|
||||
),
|
||||
"poll_count": self.poll_count,
|
||||
"total_processed": self.total_processed,
|
||||
"total_skipped": self.total_skipped,
|
||||
"total_errored": self.total_errored,
|
||||
"last_tick": self.last_tick.as_dict() if self.last_tick else None,
|
||||
"consecutive_failures": self.consecutive_failures,
|
||||
"last_error": self.last_error,
|
||||
"last_error_at": (
|
||||
self.last_error_at.isoformat() if self.last_error_at else None
|
||||
),
|
||||
"last_sftp_attempt_at": (
|
||||
self.last_sftp_attempt_at.isoformat()
|
||||
if self.last_sftp_attempt_at else None
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-file-type handlers. Each returns (parser_name, claim_count) and
|
||||
# persists its own DB rows. The scheduler records the outcome.
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# 999 (Task 2), TA1 (Task 3), 277CA (Task 4), and 835 (Task 5) now
|
||||
# live in ``cyclone.handlers``. The HANDLERS dict literal below
|
||||
# references the alias imports so the wiring stays identical.
|
||||
|
||||
|
||||
# The 277CA handler has moved to ``cyclone.handlers.handle_277ca``
|
||||
# (SP27 Task 4); the inline def was deleted. The HANDLERS dict literal
|
||||
# below still references ``_handle_277ca`` because the import-alias
|
||||
# line at the top of this module binds that name to the new function.
|
||||
# Only the 835 handler stays inline (Task 5).
|
||||
|
||||
|
||||
# Map file_type → handler. Mirrors ROUTED_FILE_TYPES.
|
||||
HANDLERS: dict[str, Callable[[str, str], tuple[str, int]]] = {
|
||||
"999": _handle_999,
|
||||
"835": _handle_835,
|
||||
"277": _handle_277ca, # filename uses 277; parser is the same
|
||||
"277CA": _handle_277ca,
|
||||
"TA1": _handle_ta1,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Light copies of helpers the API endpoints use, so the scheduler can
|
||||
# Run without depending on the FastAPI module.
|
||||
#
|
||||
# Note: ``_277ca_synthetic_source_batch_id`` lived here as a
|
||||
# scheduler-local duplicate of
|
||||
# ``cyclone.handlers._ack_id.two77ca_synthetic_source_batch_id`` until
|
||||
# SP27 Task 4 landed; the inline def has been deleted because
|
||||
# ``handle_277ca`` now imports the canonical copy. The historical
|
||||
# helper was the only remaining inline def in this section — the
|
||||
# surviving inline handler is ``_handle_835`` (lifts in Task 5).
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scheduler
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class Scheduler:
|
||||
"""Background polling loop for inbound MFT files.
|
||||
|
||||
Lifecycle:
|
||||
sched = Scheduler(sftp_block, poll_interval_seconds=60)
|
||||
await sched.start() # begin polling
|
||||
# ... later ...
|
||||
await sched.stop() # finish current tick, then exit
|
||||
status = sched.status() # snapshot
|
||||
|
||||
The scheduler is a single asyncio task. ``tick()`` does one full
|
||||
poll cycle and is exposed for tests + the ``/api/admin/scheduler/tick``
|
||||
endpoint so the operator can force a poll without waiting.
|
||||
|
||||
Threading: NOT thread-safe. All access (start/stop/tick/status)
|
||||
must happen on the same event loop. The FastAPI app satisfies
|
||||
this trivially because endpoints run on the loop.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
sftp_block: SftpBlock,
|
||||
*,
|
||||
poll_interval_seconds: int = 60,
|
||||
sftp_block_name: str = "default",
|
||||
sftp_client_factory: Optional[Callable[[SftpBlock], Any]] = None,
|
||||
) -> None:
|
||||
self._sftp_block = sftp_block
|
||||
self._poll_interval = poll_interval_seconds
|
||||
self._sftp_block_name = sftp_block_name
|
||||
# Factory indirection lets tests substitute a fake client
|
||||
# without monkey-patching the module-level SftpClient.
|
||||
self._sftp_client_factory = sftp_client_factory or SftpClient
|
||||
|
||||
self._task: Optional[asyncio.Task[None]] = None
|
||||
self._stop_event = asyncio.Event()
|
||||
self._last_poll_at: Optional[datetime] = None
|
||||
self._poll_count = 0
|
||||
self._total_processed = 0
|
||||
self._total_skipped = 0
|
||||
self._total_errored = 0
|
||||
self._last_tick: Optional[TickResult] = None
|
||||
# Coalesce overlapping ticks (a slow MFT server shouldn't let
|
||||
# ticks stack up; the next tick fires only after the previous
|
||||
# one finishes).
|
||||
self._tick_in_progress = False
|
||||
# SP27 Task 9: SFTP-error surfacing. ``_consecutive_failures``
|
||||
# counts back-to-back tick failures (cleared on the next
|
||||
# successful tick). The other three fields are the most
|
||||
# recent failure for the operator's pill — preserved across
|
||||
# successes so the audit trail doesn't disappear the moment
|
||||
# the MFT server recovers.
|
||||
self._consecutive_failures = 0
|
||||
self._last_error: Optional[str] = None
|
||||
self._last_error_at: Optional[datetime] = None
|
||||
self._last_sftp_attempt_at: Optional[datetime] = None
|
||||
|
||||
def _record_sftp_outcome(
|
||||
self, *, success: bool, error: Optional[str] = None,
|
||||
) -> None:
|
||||
"""Update the SFTP-error state after a tick.
|
||||
|
||||
``success=True`` resets the consecutive-failure counter; the
|
||||
last-error fields are preserved as an audit trail (the
|
||||
operator's UI can still surface "last failure: 2 hours ago"
|
||||
after a recovery).
|
||||
|
||||
``success=False`` bumps the counter and records the error
|
||||
message + timestamp. ``last_sftp_attempt_at`` is bumped in
|
||||
both cases so the operator can tell when the scheduler last
|
||||
*tried* to reach the MFT (not just when it last failed).
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
self._last_sftp_attempt_at = now
|
||||
if success:
|
||||
self._consecutive_failures = 0
|
||||
return
|
||||
self._consecutive_failures += 1
|
||||
self._last_error = error
|
||||
self._last_error_at = now
|
||||
|
||||
# ---- Public API -------------------------------------------------------
|
||||
|
||||
async def start(self) -> None:
|
||||
"""Begin polling. Idempotent."""
|
||||
if self._task is not None and not self._task.done():
|
||||
log.info("Scheduler already running; start() is a no-op")
|
||||
return
|
||||
self._stop_event.clear()
|
||||
self._task = asyncio.create_task(self._run(), name="mft-scheduler")
|
||||
log.info(
|
||||
"Scheduler started",
|
||||
extra={
|
||||
"poll_interval_s": self._poll_interval,
|
||||
"sftp_block": self._sftp_block_name,
|
||||
},
|
||||
)
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""Stop polling. Waits for the current tick to finish."""
|
||||
if self._task is None or self._task.done():
|
||||
return
|
||||
self._stop_event.set()
|
||||
try:
|
||||
await asyncio.wait_for(self._task, timeout=30)
|
||||
except asyncio.TimeoutError:
|
||||
log.warning("Scheduler did not stop within 30s; cancelling")
|
||||
self._task.cancel()
|
||||
try:
|
||||
await self._task
|
||||
except (asyncio.CancelledError, Exception): # noqa: BLE001
|
||||
pass
|
||||
self._task = None
|
||||
log.info("Scheduler stopped")
|
||||
|
||||
def status(self) -> SchedulerStatus:
|
||||
"""Return a snapshot of the scheduler's state."""
|
||||
return SchedulerStatus(
|
||||
running=self.is_running(),
|
||||
poll_interval_seconds=self._poll_interval,
|
||||
sftp_block_name=self._sftp_block_name,
|
||||
last_poll_at=self._last_poll_at,
|
||||
poll_count=self._poll_count,
|
||||
total_processed=self._total_processed,
|
||||
total_skipped=self._total_skipped,
|
||||
total_errored=self._total_errored,
|
||||
last_tick=self._last_tick,
|
||||
consecutive_failures=self._consecutive_failures,
|
||||
last_error=self._last_error,
|
||||
last_error_at=self._last_error_at,
|
||||
last_sftp_attempt_at=self._last_sftp_attempt_at,
|
||||
)
|
||||
|
||||
def is_running(self) -> bool:
|
||||
return self._task is not None and not self._task.done()
|
||||
|
||||
async def tick(self) -> TickResult:
|
||||
"""Run a single poll cycle and return the outcome.
|
||||
|
||||
Concurrent ticks are coalesced: if a tick is already in
|
||||
progress, the second caller waits for it. This protects the
|
||||
SFTP server from a stampede when the operator hits
|
||||
``/api/admin/scheduler/tick`` while a scheduled tick is
|
||||
already running.
|
||||
|
||||
SP27 Task 9: the tick outcome is also recorded on the
|
||||
scheduler's SFTP-error state via
|
||||
:meth:`_record_sftp_outcome`. The discriminator is
|
||||
``TickResult.sftp_failed`` (set by ``_tick_impl``'s
|
||||
listing-error branches only), not ``TickResult.errors``
|
||||
which is also populated by per-file processing errors.
|
||||
"""
|
||||
while self._tick_in_progress:
|
||||
await asyncio.sleep(0.05)
|
||||
self._tick_in_progress = True
|
||||
try:
|
||||
result = await self._tick_impl()
|
||||
self._last_tick = result
|
||||
self._last_poll_at = result.finished_at or result.started_at
|
||||
self._poll_count += 1
|
||||
self._total_processed += result.files_processed
|
||||
self._total_skipped += result.files_skipped
|
||||
self._total_errored += result.files_errored
|
||||
# SFTP-error surfacing: discriminator is ``result.sftp_failed``,
|
||||
# NOT ``result.errors``. ``sftp_failed`` is set by
|
||||
# ``_tick_impl``'s listing-error branches (timeout, connect
|
||||
# refused). Per-file processing errors append to
|
||||
# ``result.errors`` but do NOT set ``sftp_failed`` — a
|
||||
# single malformed 999 in a 5-file batch must not flip
|
||||
# the operator's pill to destructive.
|
||||
if result.sftp_failed:
|
||||
self._record_sftp_outcome(
|
||||
success=False,
|
||||
error="; ".join(result.errors),
|
||||
)
|
||||
else:
|
||||
self._record_sftp_outcome(success=True)
|
||||
return result
|
||||
finally:
|
||||
self._tick_in_progress = False
|
||||
|
||||
async def process_inbound_files(
|
||||
self, files: list[InboundFile],
|
||||
) -> TickResult:
|
||||
"""Run the per-file processing pipeline over a pre-fetched list.
|
||||
|
||||
Unlike :meth:`tick`, this does **not** call SFTP — the caller
|
||||
is expected to have already downloaded (or arranged the local
|
||||
copy of) each ``InboundFile.local_path``. Used by the
|
||||
``/api/admin/scheduler/pull-inbound`` admin endpoint and the
|
||||
``cyclone pull-inbound`` CLI command to process a date-filtered
|
||||
subset of the inbound MFT path without paying the cost of a
|
||||
full poll.
|
||||
|
||||
Honors ``_stop_event`` between files. Concurrent calls are
|
||||
coalesced the same way :meth:`tick` does, so a slow handler
|
||||
can't be stampeded by a parallel admin invocation.
|
||||
|
||||
SP27 Task 9: deliberately does NOT update the SFTP-error
|
||||
state (no listing, no SFTP). The consecutive-failure counter
|
||||
is the signal for scheduled SFTP polling; operator-initiated
|
||||
batches shouldn't be able to flip it.
|
||||
"""
|
||||
while self._tick_in_progress:
|
||||
await asyncio.sleep(0.05)
|
||||
self._tick_in_progress = True
|
||||
try:
|
||||
started = datetime.now(timezone.utc)
|
||||
result = TickResult(started_at=started, files_seen=len(files))
|
||||
for f in files:
|
||||
if self._stop_event.is_set():
|
||||
break
|
||||
await self._handle_one(f, result)
|
||||
result.finished_at = datetime.now(timezone.utc)
|
||||
self._last_tick = result
|
||||
self._last_poll_at = result.finished_at
|
||||
self._poll_count += 1
|
||||
self._total_processed += result.files_processed
|
||||
self._total_skipped += result.files_skipped
|
||||
self._total_errored += result.files_errored
|
||||
return result
|
||||
finally:
|
||||
self._tick_in_progress = False
|
||||
|
||||
# ---- Internals --------------------------------------------------------
|
||||
|
||||
async def _run(self) -> None:
|
||||
"""Main loop. Runs until ``stop()`` is called."""
|
||||
# Stagger the first tick so we don't hammer the MFT server on
|
||||
# startup if multiple operators restart Cyclone in lockstep.
|
||||
await asyncio.sleep(1)
|
||||
while not self._stop_event.is_set():
|
||||
try:
|
||||
await self.tick()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# tick() should never raise — it catches per-file
|
||||
# exceptions. This is the safety net for SFTP outages
|
||||
# or DB connectivity issues.
|
||||
log.exception("Scheduler tick raised", extra={"error": str(exc)})
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
self._stop_event.wait(),
|
||||
timeout=self._poll_interval,
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
pass # poll interval elapsed; loop again
|
||||
|
||||
async def _tick_impl(self) -> TickResult:
|
||||
"""One poll cycle: list → filter already-processed → route each."""
|
||||
started = datetime.now(timezone.utc)
|
||||
result = TickResult(started_at=started)
|
||||
|
||||
# SP27 Task 8: use the async-wrapped SFTP client so a hung
|
||||
# ``listdir_attr`` (the 06/25 silent-hang failure mode) is
|
||||
# bounded by ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` instead of
|
||||
# waiting on paramiko forever. ``asyncio.TimeoutError`` is
|
||||
# handled explicitly so the tick surfaces a clear error in
|
||||
# ``Scheduler.status()`` (Task 9) rather than a generic
|
||||
# ``Exception`` catch-all.
|
||||
client = self._sftp_client_factory(self._sftp_block)
|
||||
try:
|
||||
files = await client.async_list_inbound()
|
||||
except asyncio.TimeoutError:
|
||||
log.exception("SFTP list_inbound timed out")
|
||||
result.errors.append("list_inbound: timeout")
|
||||
result.sftp_failed = True
|
||||
result.finished_at = datetime.now(timezone.utc)
|
||||
return result
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.exception("SFTP list_inbound failed")
|
||||
result.errors.append(f"list_inbound: {exc}")
|
||||
result.sftp_failed = True
|
||||
result.finished_at = datetime.now(timezone.utc)
|
||||
return result
|
||||
|
||||
result.files_seen = len(files)
|
||||
for f in files:
|
||||
if self._stop_event.is_set():
|
||||
break
|
||||
await self._handle_one(f, result)
|
||||
|
||||
result.finished_at = datetime.now(timezone.utc)
|
||||
return result
|
||||
|
||||
async def _handle_one(self, f: InboundFile, result: TickResult) -> None:
|
||||
"""Process one inbound file: skip-if-seen, classify, parse, record."""
|
||||
if await self._already_processed(f.name):
|
||||
return
|
||||
|
||||
try:
|
||||
inbound = parse_inbound_filename(f.name)
|
||||
file_type = inbound.file_type
|
||||
except ValueError:
|
||||
file_type = None
|
||||
|
||||
if file_type not in HANDLERS:
|
||||
await self._record(
|
||||
name=f.name, size=f.size, modified_at=f.modified_at,
|
||||
file_type=file_type, parser_used=None, claim_count=0,
|
||||
status=STATUS_SKIPPED,
|
||||
error_message=(
|
||||
f"file_type {file_type!r} not in {sorted(HANDLERS)}"
|
||||
if file_type else "filename does not match HCPF inbound format"
|
||||
),
|
||||
)
|
||||
result.files_skipped += 1
|
||||
return
|
||||
|
||||
try:
|
||||
_path, parser_used, claim_count = await asyncio.to_thread(
|
||||
self._download_and_parse, f, file_type,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.exception("Failed to process inbound file", extra={"input_filename": f.name})
|
||||
await self._record(
|
||||
name=f.name, size=f.size, modified_at=f.modified_at,
|
||||
file_type=file_type, parser_used=None, claim_count=0,
|
||||
status=STATUS_ERROR,
|
||||
error_message=(
|
||||
f"{type(exc).__name__}: {exc}\n"
|
||||
f"{traceback.format_exc()[-500:]}"
|
||||
),
|
||||
)
|
||||
result.files_errored += 1
|
||||
result.errors.append(f"{f.name}: {exc}")
|
||||
return
|
||||
|
||||
await self._record(
|
||||
name=f.name, size=f.size, modified_at=f.modified_at,
|
||||
file_type=file_type, parser_used=parser_used, claim_count=claim_count,
|
||||
status=STATUS_OK, error_message=None,
|
||||
)
|
||||
result.files_processed += 1
|
||||
log.info(
|
||||
"Processed inbound file",
|
||||
extra={
|
||||
"input_filename": f.name,
|
||||
"parser": parser_used,
|
||||
"claims": claim_count,
|
||||
},
|
||||
)
|
||||
|
||||
async def _already_processed(self, name: str) -> bool:
|
||||
with db.SessionLocal()() as session:
|
||||
row = (
|
||||
session.query(ProcessedInboundFile)
|
||||
.filter_by(sftp_block_name=self._sftp_block_name, name=name)
|
||||
.filter(ProcessedInboundFile.status != STATUS_PENDING)
|
||||
.first()
|
||||
)
|
||||
return row is not None
|
||||
|
||||
async def _record(
|
||||
self,
|
||||
*,
|
||||
name: str,
|
||||
size: int,
|
||||
modified_at: datetime,
|
||||
file_type: Optional[str],
|
||||
parser_used: Optional[str],
|
||||
claim_count: int,
|
||||
status: str,
|
||||
error_message: Optional[str],
|
||||
) -> None:
|
||||
"""Persist a processed_inbound_files row. Idempotent."""
|
||||
with db.SessionLocal()() as session:
|
||||
row = ProcessedInboundFile(
|
||||
sftp_block_name=self._sftp_block_name,
|
||||
name=name,
|
||||
size=size,
|
||||
modified_at=modified_at,
|
||||
file_type=file_type,
|
||||
processed_at=datetime.now(timezone.utc),
|
||||
parser_used=parser_used,
|
||||
claim_count=claim_count,
|
||||
status=status,
|
||||
error_message=error_message,
|
||||
)
|
||||
session.add(row)
|
||||
try:
|
||||
session.commit()
|
||||
except IntegrityError:
|
||||
# A concurrent scheduler (or a retry after a partial
|
||||
# failure) already recorded this file. That's fine —
|
||||
# the latest row wins; we just skip the dup.
|
||||
session.rollback()
|
||||
|
||||
def _download_and_parse(
|
||||
self, f: InboundFile, file_type: str,
|
||||
) -> tuple[Path, str, int]:
|
||||
"""Run the right handler on one inbound file. Returns (path, parser, count).
|
||||
|
||||
Both stub and real modes read from ``f.local_path`` — the
|
||||
inbound file is already on disk:
|
||||
* Stub mode: ``_list_inbound_stub`` points ``local_path`` at
|
||||
the operator-dropped staging file.
|
||||
* Real mode: ``_list_inbound_paramiko`` downloads each
|
||||
``listdir_attr`` entry into the local cache as part of the
|
||||
listing pass. Re-reading from the MFT would require
|
||||
``SftpClient.read_file`` with a full remote path, which the
|
||||
scheduler was passing just ``f.name`` for (i.e. a bare
|
||||
filename at the SFTP root, not the inbound dir). Use the
|
||||
cached bytes instead.
|
||||
"""
|
||||
content = f.local_path.read_bytes()
|
||||
text = content.decode("utf-8")
|
||||
handler = HANDLERS[file_type]
|
||||
parser_used, claim_count = handler(text, f.name)
|
||||
return f.local_path, parser_used, claim_count
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module-level singleton — only one scheduler per process.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
_scheduler: Optional[Scheduler] = None
|
||||
|
||||
|
||||
def configure_scheduler(
|
||||
sftp_block: SftpBlock,
|
||||
*,
|
||||
poll_interval_seconds: int = 60,
|
||||
sftp_block_name: str = "default",
|
||||
force: bool = False,
|
||||
) -> Scheduler:
|
||||
"""Create the module-level scheduler singleton (or return the existing one).
|
||||
|
||||
Called from the FastAPI lifespan handler. Tests pre-configure the
|
||||
scheduler before the TestClient opens the lifespan; in that case
|
||||
we leave the existing singleton alone (``force=False``). Pass
|
||||
``force=True`` to replace unconditionally.
|
||||
"""
|
||||
global _scheduler
|
||||
if _scheduler is not None and not force:
|
||||
return _scheduler
|
||||
poll = int(
|
||||
os.environ.get("CYCLONE_SCHEDULER_POLL_SECONDS", poll_interval_seconds),
|
||||
)
|
||||
_scheduler = Scheduler(
|
||||
sftp_block,
|
||||
poll_interval_seconds=poll,
|
||||
sftp_block_name=sftp_block_name,
|
||||
)
|
||||
return _scheduler
|
||||
|
||||
|
||||
def get_scheduler() -> Scheduler:
|
||||
"""Return the module-level scheduler.
|
||||
|
||||
Raises:
|
||||
RuntimeError: if ``configure_scheduler`` hasn't been called.
|
||||
"""
|
||||
if _scheduler is None:
|
||||
raise RuntimeError(
|
||||
"scheduler not configured; call configure_scheduler() first",
|
||||
)
|
||||
return _scheduler
|
||||
|
||||
|
||||
def reset_scheduler_for_tests() -> None:
|
||||
"""Clear the module-level scheduler. Test-only."""
|
||||
global _scheduler
|
||||
_scheduler = None
|
||||
|
||||
|
||||
async def reconfigure_scheduler(
|
||||
sftp_block: SftpBlock,
|
||||
*,
|
||||
poll_interval_seconds: int = 60,
|
||||
sftp_block_name: str = "default",
|
||||
) -> Scheduler:
|
||||
"""Replace the singleton scheduler; restart if it was running. SP25.
|
||||
|
||||
Called from ``PATCH /api/clearhouse`` so the next scheduler tick
|
||||
uses the new SftpBlock without a process restart.
|
||||
|
||||
Lifecycle:
|
||||
* If the previous scheduler is running, ``stop()`` it (waits
|
||||
up to ``_DRAIN_TIMEOUT_SECONDS`` for the current tick to
|
||||
finish — matches the existing ``Scheduler.stop()`` timeout).
|
||||
* Replace the module-level ``_scheduler`` singleton with a
|
||||
fresh ``Scheduler`` against the new block.
|
||||
* If the previous one was running, ``start()`` the new one.
|
||||
|
||||
Threading: same constraint as ``configure_scheduler`` — must be
|
||||
called from the FastAPI event loop, not a worker thread.
|
||||
"""
|
||||
global _scheduler
|
||||
was_running = False
|
||||
if _scheduler is not None:
|
||||
was_running = _scheduler.is_running()
|
||||
if was_running:
|
||||
# Drain the in-flight tick. Scheduler.stop() already
|
||||
# applies the _DRAIN_TIMEOUT_SECONDS timeout internally;
|
||||
# we just await it.
|
||||
await _scheduler.stop()
|
||||
poll = int(
|
||||
os.environ.get("CYCLONE_SCHEDULER_POLL_SECONDS", poll_interval_seconds),
|
||||
)
|
||||
_scheduler = Scheduler(
|
||||
sftp_block,
|
||||
poll_interval_seconds=poll,
|
||||
sftp_block_name=sftp_block_name,
|
||||
)
|
||||
if was_running:
|
||||
await _scheduler.start()
|
||||
log.info(
|
||||
"Scheduler reconfigured",
|
||||
extra={
|
||||
"was_running": was_running,
|
||||
"sftp_block": sftp_block_name,
|
||||
"host": sftp_block.host,
|
||||
"stub": sftp_block.stub,
|
||||
},
|
||||
)
|
||||
return _scheduler
|
||||
@@ -14,36 +14,11 @@ Setup (one-time, by the operator):
|
||||
|
||||
Verification:
|
||||
security find-generic-password -s cyclone -a sftp.gainwell.password -w
|
||||
|
||||
SP25 + SP26: the lookup now resolves the secret in four tiers, in
|
||||
this order:
|
||||
|
||||
1. ``<env_name>_FILE`` env var set — highest priority. Reads the
|
||||
file at that path, strips whitespace, returns the contents.
|
||||
This is the standard Docker-secrets pattern: mount a file at
|
||||
``/run/secrets/<name>`` and point the env var at it. An
|
||||
operator who explicitly sets ``_FILE`` is making a positive
|
||||
statement about where the secret lives, so a missing file
|
||||
surfaces as a ``RuntimeError`` rather than silently falling
|
||||
through. Empty string is treated as unset.
|
||||
2. Plain env var named exactly ``<env_name>`` — second priority.
|
||||
Lets a Linux server or Docker container pass the MFT password
|
||||
via ``CYCLONE_SFTP_PASSWORD=...`` without touching the Keychain
|
||||
or a file mount. Trailing/leading whitespace (including the
|
||||
``\\n`` that .env files often leave at EOF) is stripped; an
|
||||
empty value is treated as absent.
|
||||
3. macOS Keychain via ``keyring`` — third priority. Kept as a
|
||||
fallback so a macOS workstation that prefers
|
||||
``security add-generic-password`` works without env-var exports.
|
||||
4. ``None`` — caller decides what to do (``SftpClient._connect``
|
||||
raises a precise ``RuntimeError`` on real-mode auth).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
@@ -61,68 +36,24 @@ except ImportError:
|
||||
_HAS_KEYRING = False
|
||||
|
||||
|
||||
def _env_var_for(name: str) -> str:
|
||||
"""Resolve the operator-facing env-var name for a secret.
|
||||
|
||||
Returns the entry from ``_ENV_NAME_FOR`` if present, otherwise
|
||||
returns ``name`` verbatim.
|
||||
"""
|
||||
return _ENV_NAME_FOR.get(name, name)
|
||||
|
||||
|
||||
def get_secret(name: str) -> Optional[str]:
|
||||
"""Fetch a secret by name. Four-tier lookup: _FILE, env var, Keychain, None.
|
||||
"""Fetch a secret from macOS Keychain by name.
|
||||
|
||||
Args:
|
||||
name: The secret name. Matches the Keychain account name
|
||||
(e.g. ``"sftp.gainwell.password"``). The operator-facing
|
||||
env-var form is mapped via the ``_ENV_NAME_FOR`` table at
|
||||
the bottom of this module.
|
||||
name: The Keychain account (e.g. "sftp.gainwell.password").
|
||||
|
||||
Returns:
|
||||
The secret string (stripped), or ``None`` if no tier produced
|
||||
a value.
|
||||
|
||||
Raises:
|
||||
RuntimeError: if ``<env_name>_FILE`` is set but the file at
|
||||
that path is missing or unreadable. The operator made a
|
||||
positive statement about where the secret lives; silent
|
||||
fall-through would mask a misconfiguration.
|
||||
The secret string, or None if the entry is missing or keyring
|
||||
is not installed.
|
||||
"""
|
||||
env_name = _env_var_for(name)
|
||||
file_env = env_name + "_FILE"
|
||||
file_path_raw = os.environ.get(file_env)
|
||||
if file_path_raw:
|
||||
# An empty string is treated as unset (matches the SP25 rule
|
||||
# for plain env vars).
|
||||
stripped_path = file_path_raw.strip()
|
||||
if stripped_path:
|
||||
try:
|
||||
value = Path(stripped_path).read_text().strip()
|
||||
except OSError as exc:
|
||||
raise RuntimeError(
|
||||
f"failed to read {file_env}={stripped_path}: {exc}"
|
||||
) from exc
|
||||
if value:
|
||||
log.debug("Secret %r resolved from file %r", name, stripped_path)
|
||||
return value
|
||||
|
||||
raw = os.environ.get(env_name)
|
||||
if raw is not None:
|
||||
stripped = raw.strip()
|
||||
if stripped:
|
||||
log.debug("Secret %r resolved from env var %r", name, env_name)
|
||||
return stripped
|
||||
# Empty env var — treat as absent and fall through.
|
||||
|
||||
if _HAS_KEYRING:
|
||||
try:
|
||||
value = keyring.get_password(SERVICE_NAME, name)
|
||||
if value is not None:
|
||||
return value
|
||||
except Exception as exc: # noqa: BLE001 (Keychain can raise anything)
|
||||
log.warning("Keychain get_secret(%r) failed: %s", name, exc)
|
||||
return None
|
||||
if not _HAS_KEYRING:
|
||||
log.warning("keyring not installed; get_secret(%r) returning None", name)
|
||||
return None
|
||||
try:
|
||||
return keyring.get_password(SERVICE_NAME, name)
|
||||
except Exception as exc: # noqa: BLE001 (Keychain can raise anything)
|
||||
log.warning("Keychain get_secret(%r) failed: %s", name, exc)
|
||||
return None
|
||||
|
||||
|
||||
def set_secret(name: str, value: str) -> bool:
|
||||
@@ -146,13 +77,3 @@ def has_keyring() -> bool:
|
||||
"""True if the ``keyring`` library is importable (regardless of whether
|
||||
the Keychain entry actually exists)."""
|
||||
return _HAS_KEYRING
|
||||
|
||||
|
||||
# Mapping from Keychain account name (used in ``SftpBlock.auth``) to
|
||||
# the operator-facing env var. Keeping this table here means callers
|
||||
# don't have to remember the difference between
|
||||
# ``sftp.gainwell.password`` (Keychain account) and
|
||||
# ``CYCLONE_SFTP_PASSWORD`` (env var).
|
||||
_ENV_NAME_FOR: dict[str, str] = {
|
||||
"sftp.gainwell.password": "CYCLONE_SFTP_PASSWORD",
|
||||
}
|
||||
|
||||
@@ -1,485 +0,0 @@
|
||||
"""SP19 — Security middleware + health probe.
|
||||
|
||||
Three concrete middlewares (body size, rate limit, security headers)
|
||||
plus a richer ``/api/health`` snapshot. Sizing is for Cyclone's
|
||||
local-only posture: a misconfigured Tailscale / ngrok bind, a
|
||||
misbehaving cron job, a port-scanner scraping the API. Anything more
|
||||
aggressive (auth, mTLS, WAF) is out of scope.
|
||||
|
||||
Design choices
|
||||
--------------
|
||||
|
||||
* **In-memory rate limiter.** Cyclone is single-process; a dict
|
||||
keyed by IP is enough. If we ever go multi-worker, swap for
|
||||
Redis. The rate-limit counter resets after the bucket window;
|
||||
failing open on the limiter itself (an unexpected exception)
|
||||
rather than 503ing every request is the right call for a local tool.
|
||||
|
||||
* **Body-size check by Content-Length first, then chunked-read
|
||||
guard.** A chunked POST can lie about its size (or omit the
|
||||
header entirely); we cap read body size on the underlying stream
|
||||
so a malicious client can't keep streaming forever.
|
||||
|
||||
* **Security headers on every response.** CSP locks the API to
|
||||
same-origin + the Vite dev origin (whitelisted explicitly so a
|
||||
future operator running on a different port doesn't break).
|
||||
|
||||
* **Health snapshot is best-effort.** Each subsystem (DB,
|
||||
scheduler, pubsub) reports independently — a DB outage doesn't
|
||||
blank out the rest. ``status: "degraded"`` if any subsystem is
|
||||
unhappy; ``"ok"`` only when everything is.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
from collections import deque
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Callable
|
||||
|
||||
from fastapi import Request, Response
|
||||
from fastapi.responses import JSONResponse
|
||||
from starlette.types import ASGIApp, Message, Receive, Scope, Send
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Knobs (env-var driven)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
DEFAULT_MAX_BODY_BYTES = 50 * 1024 * 1024 # 50 MB — generous for X12 EDI
|
||||
DEFAULT_RATE_LIMIT_PER_MIN = 300
|
||||
DEFAULT_RATE_LIMIT_WINDOW_S = 60
|
||||
|
||||
# CSP: API responses are JSON, not HTML. ``default-src 'none'`` is the
|
||||
# strictest setting; it forbids the API from being a vector for
|
||||
# injected scripts in case an operator opens a JSON viewer with an
|
||||
# HTML renderer.
|
||||
_SECURITY_HEADERS: dict[str, str] = {
|
||||
"X-Content-Type-Options": "nosniff",
|
||||
"X-Frame-Options": "DENY",
|
||||
"Referrer-Policy": "same-origin",
|
||||
"Permissions-Policy": "geolocation=(), microphone=(), camera=()",
|
||||
"Content-Security-Policy": "default-src 'none'; frame-ancestors 'none'",
|
||||
}
|
||||
|
||||
|
||||
def _env_int(name: str, default: int) -> int:
|
||||
raw = os.environ.get(name)
|
||||
if not raw:
|
||||
return default
|
||||
try:
|
||||
return int(raw)
|
||||
except ValueError:
|
||||
log.warning("SP19: %s=%r is not an int; using default %d", name, raw, default)
|
||||
return default
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Body-size middleware
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class BodySizeLimitMiddleware:
|
||||
"""Reject requests whose body exceeds ``max_bytes``.
|
||||
|
||||
Pure ASGI middleware (not BaseHTTPMiddleware — that one breaks
|
||||
FastAPI's ``request.body()`` introspection). Two-stage guard:
|
||||
1. If the request declares a ``Content-Length`` larger than
|
||||
``max_bytes``, reject immediately with ``413``.
|
||||
2. While reading the body chunks, cap accumulated bytes at
|
||||
``max_bytes``. If we cross the cap, return 413 instead of
|
||||
letting the handler read the rest.
|
||||
"""
|
||||
|
||||
def __init__(self, app: ASGIApp, max_bytes: int | None = None) -> None:
|
||||
self.app = app
|
||||
self.max_bytes = max_bytes or _env_int(
|
||||
"CYCLONE_MAX_BODY_BYTES", DEFAULT_MAX_BODY_BYTES,
|
||||
)
|
||||
|
||||
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
|
||||
if scope["type"] != "http":
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
|
||||
# Stage 1: declared length.
|
||||
cl_header = None
|
||||
for k, v in scope.get("headers", []):
|
||||
if k == b"content-length":
|
||||
cl_header = v.decode("latin-1")
|
||||
break
|
||||
if cl_header is not None:
|
||||
try:
|
||||
if int(cl_header) > self.max_bytes:
|
||||
await _send_rejection(
|
||||
scope, send,
|
||||
code=413,
|
||||
reason="body_too_large",
|
||||
detail=f"Content-Length {cl_header} exceeds limit {self.max_bytes}",
|
||||
)
|
||||
return
|
||||
except ValueError:
|
||||
await _send_rejection(
|
||||
scope, send,
|
||||
code=400, reason="bad_content_length",
|
||||
detail=f"Content-Length {cl_header!r} is not an integer",
|
||||
)
|
||||
return
|
||||
|
||||
# Stage 2: chunked read guard.
|
||||
seen = 0
|
||||
over_limit = False
|
||||
|
||||
async def wrapped_receive() -> Message:
|
||||
nonlocal seen, over_limit
|
||||
if over_limit:
|
||||
# Drain any remaining bytes so the upstream ASGI
|
||||
# server doesn't see a truncated stream.
|
||||
msg = await receive()
|
||||
if msg.get("type") == "http.request":
|
||||
return {"type": "http.request", "body": b"", "more_body": False}
|
||||
return msg
|
||||
msg = await receive()
|
||||
if msg.get("type") == "http.request":
|
||||
body = msg.get("body", b"") or b""
|
||||
seen += len(body)
|
||||
if seen > self.max_bytes:
|
||||
over_limit = True
|
||||
return {"type": "http.request", "body": b"", "more_body": False}
|
||||
return msg
|
||||
|
||||
if cl_header is None:
|
||||
# Chunked / unknown length — guard with wrapped receive.
|
||||
await self.app(scope, wrapped_receive, send)
|
||||
else:
|
||||
# Fixed-length known to be safe; pass through.
|
||||
await self.app(scope, receive, send)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Rate-limit middleware
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class _Bucket:
|
||||
"""Sliding-window counter for one IP."""
|
||||
timestamps: deque = field(default_factory=deque)
|
||||
|
||||
def hit(self, window_s: int, now: float) -> bool:
|
||||
"""Record one hit; return True if under the limit, False if over."""
|
||||
# Drop expired entries.
|
||||
cutoff = now - window_s
|
||||
while self.timestamps and self.timestamps[0] < cutoff:
|
||||
self.timestamps.popleft()
|
||||
return True # we always record; the dispatcher decides to reject
|
||||
|
||||
def count_in_window(self, now: float, window_s: int) -> int:
|
||||
cutoff = now - window_s
|
||||
while self.timestamps and self.timestamps[0] < cutoff:
|
||||
self.timestamps.popleft()
|
||||
return len(self.timestamps)
|
||||
|
||||
|
||||
class RateLimitMiddleware:
|
||||
"""Per-IP sliding-window rate limiter (pure ASGI).
|
||||
|
||||
Defaults to ``CYCLONE_RATE_LIMIT_PER_MIN`` requests/minute per IP.
|
||||
Health-check probes and the ``/api/health`` endpoint are exempt
|
||||
so a load balancer's frequent probes don't trip the limiter.
|
||||
|
||||
On unexpected errors the limiter fails OPEN — better to serve a
|
||||
few extra requests than to 503 every request because of a bug.
|
||||
"""
|
||||
|
||||
EXEMPT_PATHS = ("/api/health", "/healthz", "/readyz")
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
app: ASGIApp,
|
||||
per_minute: int | None = None,
|
||||
window_s: int | None = None,
|
||||
) -> None:
|
||||
self.app = app
|
||||
self.per_minute = per_minute or _env_int(
|
||||
"CYCLONE_RATE_LIMIT_PER_MIN", DEFAULT_RATE_LIMIT_PER_MIN,
|
||||
)
|
||||
self.window_s = window_s or DEFAULT_RATE_LIMIT_WINDOW_S
|
||||
self._buckets: dict[str, _Bucket] = {}
|
||||
self._lock = threading.Lock()
|
||||
|
||||
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
|
||||
if scope["type"] != "http":
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
path = scope.get("path", "")
|
||||
if path in self.EXEMPT_PATHS:
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
ip = _client_ip_from_scope(scope)
|
||||
now = time.monotonic()
|
||||
try:
|
||||
with self._lock:
|
||||
bucket = self._buckets.setdefault(ip, _Bucket())
|
||||
bucket.timestamps.append(now)
|
||||
count = bucket.count_in_window(now, self.window_s)
|
||||
if count > self.per_minute:
|
||||
await _send_rejection(
|
||||
scope, send,
|
||||
code=429,
|
||||
reason="rate_limited",
|
||||
detail=(
|
||||
f"IP {ip} exceeded {self.per_minute} req/"
|
||||
f"{self.window_s}s window"
|
||||
),
|
||||
)
|
||||
return
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.warning("SP19: rate limiter failed open: %s", exc)
|
||||
await self.app(scope, receive, send)
|
||||
|
||||
|
||||
def _client_ip_from_scope(scope: Scope) -> str:
|
||||
"""Best-effort client IP from the ASGI scope. Falls back to ``"unknown"``."""
|
||||
for k, v in scope.get("headers", []):
|
||||
if k == b"x-forwarded-for":
|
||||
return v.decode("latin-1").split(",")[0].strip()
|
||||
client = scope.get("client")
|
||||
if client and client[0]:
|
||||
return client[0]
|
||||
return "unknown"
|
||||
|
||||
|
||||
def _client_ip(request: Request) -> str:
|
||||
"""Legacy helper (kept for the audit-event log path)."""
|
||||
return _client_ip_from_scope(request.scope)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Security-headers middleware
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class SecurityHeadersMiddleware:
|
||||
"""Stamp the static security headers on every response (pure ASGI).
|
||||
|
||||
CSP / X-Content-Type-Options / X-Frame-Options / Referrer-Policy /
|
||||
Permissions-Policy. The headers are static for now; per-route
|
||||
overrides can be added later if a route needs to relax them.
|
||||
"""
|
||||
|
||||
def __init__(self, app: ASGIApp, extra: dict[str, str] | None = None) -> None:
|
||||
self.app = app
|
||||
self.headers = [(k.lower().encode("latin-1"), v.encode("latin-1"))
|
||||
for k, v in _SECURITY_HEADERS.items()]
|
||||
if extra:
|
||||
self.headers.extend(
|
||||
(k.lower().encode("latin-1"), v.encode("latin-1"))
|
||||
for k, v in extra.items()
|
||||
)
|
||||
|
||||
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
|
||||
if scope["type"] != "http":
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
|
||||
async def wrapped_send(message: Message) -> None:
|
||||
if message["type"] == "http.response.start":
|
||||
headers = list(message.get("headers", []))
|
||||
existing = {k for k, _ in headers}
|
||||
for k, v in self.headers:
|
||||
if k not in existing:
|
||||
headers.append((k, v))
|
||||
message["headers"] = headers
|
||||
await send(message)
|
||||
|
||||
await self.app(scope, receive, wrapped_send)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Reject helper (also writes an audit event when DB is available)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def _send_rejection(
|
||||
scope: Scope,
|
||||
send: Send,
|
||||
*,
|
||||
code: int,
|
||||
reason: str,
|
||||
detail: str,
|
||||
) -> None:
|
||||
"""Build a 413/429 JSON response, send it, and emit a log + audit event."""
|
||||
method = scope.get("method", "GET")
|
||||
path = scope.get("path", "/")
|
||||
ip = _client_ip_from_scope(scope)
|
||||
log.warning(
|
||||
"api.request_rejected",
|
||||
extra={
|
||||
"status": code,
|
||||
"reason": reason,
|
||||
"path": path,
|
||||
"method": method,
|
||||
"ip": ip,
|
||||
"detail": detail,
|
||||
},
|
||||
)
|
||||
payload = {"error": reason, "detail": detail, "status": code}
|
||||
body = json.dumps(payload).encode("utf-8")
|
||||
await send({
|
||||
"type": "http.response.start",
|
||||
"status": code,
|
||||
"headers": [
|
||||
(b"content-type", b"application/json"),
|
||||
(b"content-length", str(len(body)).encode("latin-1")),
|
||||
],
|
||||
})
|
||||
await send({"type": "http.response.body", "body": body, "more_body": False})
|
||||
# Best-effort audit-log append. Don't block the response on a DB
|
||||
# outage (the rejection is the more important signal anyway).
|
||||
try:
|
||||
from cyclone import db
|
||||
from cyclone.audit_log import AuditEvent, append_event
|
||||
|
||||
with db.SessionLocal()() as session:
|
||||
append_event(
|
||||
session,
|
||||
AuditEvent(
|
||||
event_type="api.request_rejected",
|
||||
entity_type="http_request",
|
||||
entity_id=f"{method} {path}",
|
||||
payload={
|
||||
"status": code,
|
||||
"reason": reason,
|
||||
"path": path,
|
||||
"method": method,
|
||||
"ip": ip,
|
||||
},
|
||||
actor=f"api:{ip}",
|
||||
),
|
||||
)
|
||||
session.commit()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.debug("SP19: audit-log append failed for rejection: %s", exc)
|
||||
|
||||
|
||||
def _reject(
|
||||
request: Request,
|
||||
*,
|
||||
code: int,
|
||||
reason: str,
|
||||
detail: str,
|
||||
) -> JSONResponse:
|
||||
"""Sync helper kept for back-compat (the ``audit_log`` payload path)."""
|
||||
return JSONResponse(
|
||||
{"error": reason, "detail": detail, "status": code},
|
||||
status_code=code,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Health snapshot
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class HealthSnapshot:
|
||||
status: str
|
||||
version: str
|
||||
db: dict[str, Any] = field(default_factory=dict)
|
||||
scheduler: dict[str, Any] = field(default_factory=dict)
|
||||
pubsub: dict[str, Any] = field(default_factory=dict)
|
||||
batch: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"status": self.status,
|
||||
"version": self.version,
|
||||
"db": self.db,
|
||||
"scheduler": self.scheduler,
|
||||
"pubsub": self.pubsub,
|
||||
"batch": self.batch,
|
||||
}
|
||||
|
||||
|
||||
def get_health_snapshot() -> HealthSnapshot:
|
||||
"""Gather a best-effort snapshot of every Cyclone subsystem.
|
||||
|
||||
Returns ``HealthSnapshot`` with ``status="ok"`` only if every
|
||||
subsystem check passes. ``"degraded"`` if any subsystem is
|
||||
unhappy but the API itself is responsive. Each subsystem reports
|
||||
independently so one outage doesn't blank out the rest.
|
||||
"""
|
||||
from cyclone import __version__, db
|
||||
|
||||
snap = HealthSnapshot(status="ok", version=__version__)
|
||||
|
||||
# DB connectivity.
|
||||
try:
|
||||
with db.SessionLocal()() as session:
|
||||
session.execute(db.text("SELECT 1"))
|
||||
snap.db = {"ok": True}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
snap.db = {"ok": False, "error": str(exc)}
|
||||
snap.status = "degraded"
|
||||
|
||||
# Scheduler state.
|
||||
try:
|
||||
from cyclone import scheduler as scheduler_mod
|
||||
sched = scheduler_mod.get_scheduler()
|
||||
snap.scheduler = {
|
||||
"running": sched.is_running(),
|
||||
"interval_s": sched._poll_interval, # noqa: SLF001
|
||||
"sftp_block": sched._sftp_block_name, # noqa: SLF001
|
||||
}
|
||||
except RuntimeError:
|
||||
snap.scheduler = {"running": False, "configured": False}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
snap.scheduler = {"ok": False, "error": str(exc)}
|
||||
snap.status = "degraded"
|
||||
|
||||
# Backup scheduler.
|
||||
try:
|
||||
from cyclone import backup_scheduler as bks_mod
|
||||
bks = bks_mod.get_backup_scheduler()
|
||||
snap.scheduler["backup_scheduler_running"] = bks.is_running()
|
||||
snap.scheduler["backup_interval_hours"] = bks.interval_hours
|
||||
except (RuntimeError, ImportError):
|
||||
snap.scheduler["backup_scheduler_running"] = False
|
||||
except Exception: # noqa: BLE001
|
||||
pass # secondary subsystem; don't degrade the overall status
|
||||
|
||||
# Pubsub bus stats — placeholder. The /api/health handler fills
|
||||
# in the real subscriber counts using request.app.state.event_bus.
|
||||
snap.pubsub = {"note": "filled in by health router"}
|
||||
|
||||
# Last batch timestamp + count.
|
||||
try:
|
||||
from cyclone import db as db_mod
|
||||
from cyclone.db import Batch
|
||||
with db_mod.SessionLocal()() as session:
|
||||
row = (
|
||||
session.query(Batch)
|
||||
.order_by(Batch.parsed_at.desc())
|
||||
.first()
|
||||
)
|
||||
if row is not None:
|
||||
snap.batch = {
|
||||
"last_batch_id": row.id,
|
||||
"last_batch_kind": row.kind,
|
||||
"last_batch_at": row.parsed_at.isoformat() if row.parsed_at else None,
|
||||
"last_batch_filename": row.input_filename,
|
||||
}
|
||||
else:
|
||||
snap.batch = {"last_batch_id": None, "note": "no batches yet"}
|
||||
except Exception as exc: # noqa: BLE001
|
||||
snap.batch = {"ok": False, "error": str(exc)}
|
||||
snap.status = "degraded"
|
||||
|
||||
return snap
|
||||
@@ -1,439 +0,0 @@
|
||||
"""CLI subcommand: ``python -m cyclone seed``.
|
||||
|
||||
Populates the local DB with a deterministic batch of sample claims
|
||||
plus matching activity events, so the Dashboard / Claims / Activity
|
||||
Log pages have something to render in a fresh dev environment.
|
||||
|
||||
This is a dev-only convenience — the production DB never sees this
|
||||
command (no ``[env: dev]`` gate, but it's never wired into any
|
||||
container image). It writes rows with a recognizable batch id prefix
|
||||
(``SEED-``) so ``--reset`` can clean them up without touching real
|
||||
ingested data.
|
||||
|
||||
Why the data shape is hand-rolled here
|
||||
--------------------------------------
|
||||
|
||||
The ``Claim`` ORM table stores its billing_provider / payer /
|
||||
subscriber / service_lines payloads in a ``raw_json`` blob that the
|
||||
read path (``store.to_ui_claim_from_orm``) parses back into the UI
|
||||
shape. Mirroring the 837 parser's structure keeps the UI rendering
|
||||
identical to a real ingestion — wire format parity, not shortcuts.
|
||||
|
||||
Activity rows are similar: ``payload_json`` carries the message
|
||||
string the Dashboard's activity card shows.
|
||||
|
||||
Subcommands
|
||||
-----------
|
||||
|
||||
``python -m cyclone seed`` — insert the default batch.
|
||||
``python -m cyclone seed --count N`` — insert N claims (default 96).
|
||||
``python -m cyclone seed --reset`` — wipe previously-seeded rows
|
||||
first, then insert a fresh batch.
|
||||
``python -m cyclone seed --status`` — print row counts without
|
||||
inserting anything.
|
||||
|
||||
Exit codes
|
||||
----------
|
||||
|
||||
* 0 — success (or already seeded and not asked to reset).
|
||||
* 1 — DB error during insert.
|
||||
* 2 — usage error (bad flag value).
|
||||
|
||||
The command is idempotent: re-running without ``--reset`` is a no-op
|
||||
once a seed batch exists. This keeps ``cyclone``-driven boot scripts
|
||||
safe to re-run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import random
|
||||
import sys
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import click
|
||||
|
||||
from cyclone.db import ActivityEvent, Batch, Claim, ClaimState, Remittance, SessionLocal
|
||||
|
||||
SEED_BATCH_PREFIX = "SEED-"
|
||||
SEED_CLAIM_PREFIX = "CLM-S"
|
||||
SEED_REMIT_PREFIX = "REM-S"
|
||||
SEED_DEFAULT_COUNT = 96
|
||||
SEED_ACTIVITY_LIMIT = 28
|
||||
|
||||
# Mirror src/data/sampleData.ts on the frontend so the Dashboard looks
|
||||
# the same in dev mode as it did with the in-memory fixtures.
|
||||
SAMPLE_PROVIDERS = [
|
||||
{
|
||||
"npi": "1730187395",
|
||||
"name": "Cedar Park Family Medicine",
|
||||
"tax_id": "47-3829104",
|
||||
"address": "1401 Medical Pkwy",
|
||||
"city": "Cedar Park",
|
||||
"state": "TX",
|
||||
"zip": "78613",
|
||||
"phone": "(512) 555-0142",
|
||||
},
|
||||
{
|
||||
"npi": "1528471902",
|
||||
"name": "Lakeside Orthopedics",
|
||||
"tax_id": "83-1172654",
|
||||
"address": "900 W Lake Dr",
|
||||
"city": "Austin",
|
||||
"state": "TX",
|
||||
"zip": "78746",
|
||||
"phone": "(512) 555-0188",
|
||||
},
|
||||
{
|
||||
"npi": "1982036471",
|
||||
"name": "Hill Country Pediatrics",
|
||||
"tax_id": "74-5520183",
|
||||
"address": "205 State Hwy 27",
|
||||
"city": "Marble Falls",
|
||||
"state": "TX",
|
||||
"zip": "78654",
|
||||
"phone": "(830) 555-0117",
|
||||
},
|
||||
]
|
||||
|
||||
SAMPLE_PAYERS = [
|
||||
"Blue Cross Blue Shield",
|
||||
"United Healthcare",
|
||||
"Aetna",
|
||||
"Cigna",
|
||||
"Humana",
|
||||
"Medicare",
|
||||
"Medicaid TX",
|
||||
]
|
||||
|
||||
SAMPLE_CPTS = ["99213", "99214", "99203", "93000", "85025", "80053", "73721", "20610"]
|
||||
|
||||
SAMPLE_FIRST_NAMES = [
|
||||
"Avery", "Jordan", "Riley", "Casey", "Morgan",
|
||||
"Quinn", "Reese", "Sasha", "Drew", "Hayden",
|
||||
]
|
||||
SAMPLE_LAST_NAMES = [
|
||||
"Nguyen", "Patel", "Garcia", "Cohen", "Okafor",
|
||||
"Martinez", "Hwang", "Brooks", "Singh", "Tanaka",
|
||||
]
|
||||
|
||||
# Frontend uses "accepted"/"pending" which the backend ClaimState
|
||||
# enum doesn't carry. Map them to the closest real states so the
|
||||
# Dashboard's filters (e.g. "status === 'submitted' || 'pending'")
|
||||
# still find a match for in-flight work.
|
||||
STATUS_WEIGHTS: list[tuple[ClaimState, int]] = [
|
||||
(ClaimState.SUBMITTED, 1),
|
||||
(ClaimState.RECEIVED, 1),
|
||||
(ClaimState.DENIED, 1),
|
||||
(ClaimState.PAID, 3),
|
||||
(ClaimState.PAID, 3),
|
||||
(ClaimState.PARTIAL, 1),
|
||||
]
|
||||
|
||||
DENIAL_REASONS = [
|
||||
"CO-97: Service included in another service",
|
||||
"CO-16: Claim lacks information",
|
||||
"CO-50: Non-covered service",
|
||||
"PR-1: Deductible amount",
|
||||
]
|
||||
|
||||
|
||||
def _now_utc() -> datetime:
|
||||
return datetime.now(timezone.utc).replace(tzinfo=None)
|
||||
|
||||
|
||||
def _pick(rng: random.Random, items):
|
||||
return items[rng.randint(0, len(items) - 1)]
|
||||
|
||||
|
||||
def _build_seed_rows(
|
||||
count: int, *, seed: int = 42,
|
||||
) -> tuple[Batch, list[Claim], list[ActivityEvent], list[Remittance]]:
|
||||
"""Build a deterministic Batch + N Claims + matching activity events.
|
||||
|
||||
PAID and PARTIAL claims get a paired ``Remittance`` row (with a
|
||||
realistic ``total_paid`` derived from the billed amount) so the
|
||||
Dashboard's "Received" KPI lights up; the wire shape mirrors what
|
||||
``CycloneStore.iter_claims`` expects to find via
|
||||
``Claim.matched_remittance_id`` → ``Remittance.total_paid``.
|
||||
|
||||
The seed is fixed so re-running produces identical data — keeps
|
||||
screenshots and dev environments stable.
|
||||
"""
|
||||
rng = random.Random(seed)
|
||||
now = _now_utc()
|
||||
parsed_at = now - timedelta(minutes=5)
|
||||
|
||||
batch = Batch(
|
||||
id=f"{SEED_BATCH_PREFIX}{now.strftime('%Y%m%d-%H%M%S')}",
|
||||
kind="837",
|
||||
input_filename="seed/sample-837.edi",
|
||||
parsed_at=parsed_at,
|
||||
totals_json={"claim_count": count},
|
||||
validation_json={"errors": [], "warnings": []},
|
||||
raw_result_json={},
|
||||
)
|
||||
|
||||
claims: list[Claim] = []
|
||||
activity: list[ActivityEvent] = []
|
||||
remittances: list[Remittance] = []
|
||||
seq = 10428 # Match the frontend fixture's id range
|
||||
|
||||
for i in range(count):
|
||||
provider = _pick(rng, SAMPLE_PROVIDERS)
|
||||
payer = _pick(rng, SAMPLE_PAYERS)
|
||||
cpt = _pick(rng, SAMPLE_CPTS)
|
||||
first = _pick(rng, SAMPLE_FIRST_NAMES)
|
||||
last = _pick(rng, SAMPLE_LAST_NAMES)
|
||||
state, _ = _pick(rng, STATUS_WEIGHTS)
|
||||
|
||||
days_back = rng.randint(0, 200)
|
||||
submitted = now - timedelta(days=days_back, hours=rng.randint(0, 23))
|
||||
|
||||
billed = 80 + rng.randint(0, 1400)
|
||||
if state == ClaimState.PAID:
|
||||
received = int(billed * (0.6 + rng.random() * 0.4))
|
||||
elif state == ClaimState.PARTIAL:
|
||||
received = int(billed * (0.2 + rng.random() * 0.3))
|
||||
elif state == ClaimState.DENIED:
|
||||
received = 0
|
||||
else:
|
||||
received = 0
|
||||
|
||||
claim_id = f"{SEED_CLAIM_PREFIX}{(seq + i):05d}"
|
||||
denial_reason = _pick(rng, DENIAL_REASONS) if state == ClaimState.DENIED else None
|
||||
|
||||
# Build a paired Remittance for any claim with received > 0. Status
|
||||
# code 1 = "Primary payer forward" — the 835 CAS code that the
|
||||
# remittance mapper turns into "received". Mirror the 835 parser's
|
||||
# raw_json shape (a stripped provider/payer/service_lines block)
|
||||
# so downstream debug views still render something useful.
|
||||
matched_remit_id: str | None = None
|
||||
if received > 0:
|
||||
matched_remit_id = f"{SEED_REMIT_PREFIX}{(seq + i):05d}"
|
||||
remittances.append(Remittance(
|
||||
id=matched_remit_id,
|
||||
batch_id=batch.id,
|
||||
payer_claim_control_number=f"PCN-{(seq + i):05d}",
|
||||
claim_id=claim_id,
|
||||
status_code="1",
|
||||
status_label="Primary payer forward",
|
||||
total_charge=float(billed),
|
||||
total_paid=float(received),
|
||||
adjustment_amount=float(billed - received),
|
||||
received_at=submitted + timedelta(days=rng.randint(2, 14)),
|
||||
raw_json={
|
||||
"payer": {"name": payer},
|
||||
"provider": {
|
||||
"npi": provider["npi"],
|
||||
"name": provider["name"],
|
||||
},
|
||||
"service_lines": [
|
||||
{
|
||||
"procedure": {"code": cpt},
|
||||
"charge_amount": float(billed),
|
||||
"paid_amount": float(received),
|
||||
}
|
||||
],
|
||||
},
|
||||
))
|
||||
|
||||
raw_json = {
|
||||
"billing_provider": {
|
||||
"npi": provider["npi"],
|
||||
"name": provider["name"],
|
||||
"tax_id": provider["tax_id"],
|
||||
# 837 parser produces ``address`` as a structured dict so
|
||||
# the claim-detail drawer can render line1/line2/city/state/zip.
|
||||
# A flat string here crashes ``_address_to_ui`` with
|
||||
# ``AttributeError: 'str' object has no attribute 'get'``.
|
||||
"address": {
|
||||
"line1": provider["address"],
|
||||
"line2": None,
|
||||
"city": provider["city"],
|
||||
"state": provider["state"],
|
||||
"zip": provider["zip"],
|
||||
},
|
||||
"phone": provider["phone"],
|
||||
},
|
||||
"payer": {"name": payer},
|
||||
"subscriber": {"first_name": first, "last_name": last},
|
||||
"service_lines": [
|
||||
{
|
||||
"procedure": {"code": cpt},
|
||||
"charge_amount": float(billed),
|
||||
"service_date": submitted.date().isoformat(),
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
claims.append(Claim(
|
||||
id=claim_id,
|
||||
batch_id=batch.id,
|
||||
patient_control_number=claim_id.replace(SEED_CLAIM_PREFIX, "PCN-"),
|
||||
service_date_from=submitted.date(),
|
||||
service_date_to=submitted.date(),
|
||||
charge_amount=billed,
|
||||
provider_npi=provider["npi"],
|
||||
payer_id=None,
|
||||
state=state,
|
||||
state_changed_at=submitted,
|
||||
rejection_reason=denial_reason,
|
||||
resubmit_count=0,
|
||||
matched_remittance_id=matched_remit_id,
|
||||
raw_json=raw_json,
|
||||
))
|
||||
|
||||
# First SEED_ACTIVITY_LIMIT claims get an activity event. Mirrors
|
||||
# the frontend buildActivity() shape (claim_paid / claim_denied /
|
||||
# claim_accepted / claim_submitted) but maps frontend statuses
|
||||
# onto the backend's wire enum.
|
||||
if i < SEED_ACTIVITY_LIMIT:
|
||||
if state == ClaimState.PAID:
|
||||
kind = "claim_paid"
|
||||
verb = "Paid"
|
||||
elif state == ClaimState.DENIED:
|
||||
kind = "claim_denied"
|
||||
verb = "Denied"
|
||||
elif state in (ClaimState.RECEIVED, ClaimState.PARTIAL):
|
||||
kind = "claim_accepted"
|
||||
verb = "Accepted"
|
||||
else:
|
||||
kind = "claim_submitted"
|
||||
verb = "Submitted"
|
||||
|
||||
payload = {
|
||||
"message": f"{verb} {claim_id} · {first} {last}",
|
||||
"npi": provider["npi"],
|
||||
"amount": float(billed),
|
||||
}
|
||||
activity.append(ActivityEvent(
|
||||
ts=submitted,
|
||||
kind=kind,
|
||||
batch_id=batch.id,
|
||||
claim_id=claim_id,
|
||||
remittance_id=None,
|
||||
payload_json=payload,
|
||||
))
|
||||
|
||||
return batch, claims, activity, remittances
|
||||
|
||||
|
||||
def _existing_seed_batch_ids(s) -> list[str]:
|
||||
rows = s.query(Batch.id).filter(Batch.id.like(f"{SEED_BATCH_PREFIX}%")).all()
|
||||
return [r[0] for r in rows]
|
||||
|
||||
|
||||
def _delete_seed_rows(s) -> int:
|
||||
"""Delete every row that was inserted by the seed.
|
||||
|
||||
The seed writes rows whose ids begin with ``SEED-`` (batches),
|
||||
``CLM-S`` (claims), or ``REM-S`` (remittances). Activity events
|
||||
are joined to seeded batches when the batch is still around, but
|
||||
we also clean up orphan activity rows (no FK from
|
||||
``activity_events.claim_id`` to ``claims.id``) by ``claim_id``
|
||||
prefix. Returns the number of batch rows deleted.
|
||||
"""
|
||||
# Activity first — its rows reference both claims and batches, so
|
||||
# delete the events tied to seed claims first, then any that were
|
||||
# only tied to a now-orphaned seed batch.
|
||||
activity = s.query(ActivityEvent).filter(
|
||||
ActivityEvent.claim_id.like(f"{SEED_CLAIM_PREFIX}%")
|
||||
).delete(synchronize_session=False)
|
||||
activity2 = s.query(ActivityEvent).filter(
|
||||
ActivityEvent.batch_id.like(f"{SEED_BATCH_PREFIX}%")
|
||||
).delete(synchronize_session=False)
|
||||
# Remittances (FK to claims; cascade may not fire without FK pragma,
|
||||
# so we delete them explicitly to clear the symmetric FK first).
|
||||
remits = s.query(Remittance).filter(Remittance.id.like(f"{SEED_REMIT_PREFIX}%")).delete(synchronize_session=False)
|
||||
claims = s.query(Claim).filter(Claim.id.like(f"{SEED_CLAIM_PREFIX}%")).delete(synchronize_session=False)
|
||||
batches = s.query(Batch).filter(Batch.id.like(f"{SEED_BATCH_PREFIX}%")).delete(synchronize_session=False)
|
||||
s.commit()
|
||||
click.echo(f" Removed {batches} seeded batch(es), {claims} claim(s), "
|
||||
f"{remits} remittance(s), {activity + activity2} activity event(s).")
|
||||
return batches
|
||||
|
||||
|
||||
def _insert(
|
||||
batch: Batch,
|
||||
claims: list[Claim],
|
||||
activity: list[ActivityEvent],
|
||||
remittances: list[Remittance],
|
||||
) -> None:
|
||||
with SessionLocal()() as s:
|
||||
s.add(batch)
|
||||
s.add_all(claims)
|
||||
s.add_all(activity)
|
||||
s.add_all(remittances)
|
||||
s.commit()
|
||||
|
||||
|
||||
def _print_status(s) -> None:
|
||||
from sqlalchemy import func
|
||||
seed_batches = s.query(func.count(Batch.id)).filter(Batch.id.like(f"{SEED_BATCH_PREFIX}%")).scalar() or 0
|
||||
seed_claims = s.query(func.count(Claim.id)).filter(Claim.id.like(f"{SEED_CLAIM_PREFIX}%")).scalar() or 0
|
||||
seed_remits = s.query(func.count(Remittance.id)).filter(Remittance.id.like(f"{SEED_REMIT_PREFIX}%")).scalar() or 0
|
||||
total_claims = s.query(func.count(Claim.id)).scalar() or 0
|
||||
total_remits = s.query(func.count(Remittance.id)).scalar() or 0
|
||||
total_activity = s.query(func.count(ActivityEvent.id)).scalar() or 0
|
||||
click.echo(f" Seed batches: {seed_batches}")
|
||||
click.echo(f" Seed claims: {seed_claims}")
|
||||
click.echo(f" Seed remits: {seed_remits}")
|
||||
click.echo(f" Total claims: {total_claims}")
|
||||
click.echo(f" Total remits: {total_remits}")
|
||||
click.echo(f" Total activity: {total_activity}")
|
||||
|
||||
|
||||
@click.command("seed")
|
||||
@click.option(
|
||||
"--count",
|
||||
default=SEED_DEFAULT_COUNT,
|
||||
show_default=True,
|
||||
type=click.IntRange(min=1, max=10_000),
|
||||
help="Number of claims to insert in the new batch.",
|
||||
)
|
||||
@click.option(
|
||||
"--reset",
|
||||
is_flag=True,
|
||||
help="Delete any existing seeded batch (and its claims/activity) before inserting.",
|
||||
)
|
||||
@click.option(
|
||||
"--status",
|
||||
"show_status",
|
||||
is_flag=True,
|
||||
help="Print current seeded row counts and exit.",
|
||||
)
|
||||
def seed_cli(count: int, reset: bool, show_status: bool) -> None:
|
||||
"""Populate the local DB with a deterministic batch of sample claims."""
|
||||
with SessionLocal()() as s:
|
||||
if show_status:
|
||||
_print_status(s)
|
||||
return
|
||||
|
||||
existing = _existing_seed_batch_ids(s)
|
||||
if existing and not reset:
|
||||
click.echo(
|
||||
f"Seed already present (batch ids: {', '.join(existing)}). "
|
||||
f"Re-run with --reset to replace, or --status to inspect.",
|
||||
err=True,
|
||||
)
|
||||
sys.exit(0)
|
||||
|
||||
if reset and existing:
|
||||
deleted = _delete_seed_rows(s)
|
||||
click.echo(f" Removed {deleted} seeded batch(es).")
|
||||
|
||||
try:
|
||||
batch, claims, activity, remittances = _build_seed_rows(count)
|
||||
except Exception as exc:
|
||||
click.echo(f"Failed to build seed rows: {exc}", err=True)
|
||||
sys.exit(1)
|
||||
try:
|
||||
_insert(batch, claims, activity, remittances)
|
||||
except Exception as exc:
|
||||
click.echo(f"Failed to insert seed rows: {exc}", err=True)
|
||||
sys.exit(1)
|
||||
|
||||
click.echo(f" Inserted batch {batch.id} with {len(claims)} claims, "
|
||||
f"{len(remittances)} remittances, {len(activity)} activity events.")
|
||||
_print_status(s)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,377 +0,0 @@
|
||||
"""SQLAlchemy-backed batch store for parsed X12 files.
|
||||
|
||||
The package exposes a single ``CycloneStore`` class and a module-level
|
||||
singleton (``store``). All persistence flows through SQLAlchemy
|
||||
sessions via ``db.SessionLocal()()`` (the double-paren function-style
|
||||
accessor).
|
||||
|
||||
This ``__init__.py`` is the facade — it re-exports every public name
|
||||
plus the 3 private helpers imported by tests (``_claim_status_from_validation``,
|
||||
``_persist_835_remit``, ``_remittance_835_row``) from the sibling modules:
|
||||
|
||||
exceptions AlreadyMatchedError, NotMatchedError, InvalidStateError
|
||||
records BatchKind, BatchRecord, BatchRecord837, BatchRecord835
|
||||
orm_builders ORM row builders + the 3 private-helper re-exports
|
||||
ui UI serializers (to_ui_*)
|
||||
write add_record + private event/reconcile helpers
|
||||
batches Batch reads + _BatchesShim (test-cleanup shim)
|
||||
claim_detail Single-claim reads + matched-pair drift audit
|
||||
kpis Dashboard aggregation
|
||||
acks 999 / TA1 / 277CA ACK persistence
|
||||
backups Backup-pending marker inserts
|
||||
inbox Manual match/unmatch + lane listing
|
||||
providers Provider/payer/clearhouse config
|
||||
|
||||
The ``CycloneStore`` class below keeps its current method signatures as
|
||||
thin delegations for back-compat with existing callers and tests.
|
||||
|
||||
Public API (preserved verbatim):
|
||||
CycloneStore, store, BatchRecord, BatchRecord837, BatchRecord835,
|
||||
BatchKind, AlreadyMatchedError, NotMatchedError, InvalidStateError,
|
||||
utcnow, dashboard_kpis, check_matched_pair_drift
|
||||
|
||||
Backward-compat shims for tests:
|
||||
``_lock`` — a threading.RLock used by 7 test files for cleanup.
|
||||
``_batches.clear()`` — wipes all rows from the DB tables; the
|
||||
``_BatchesShim`` class lives in ``batches.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
def utcnow() -> datetime:
|
||||
"""tz-aware UTC `datetime` (replaces the old `utcnow_iso` string helper)."""
|
||||
return datetime.now(timezone.utc)
|
||||
|
||||
|
||||
from . import write
|
||||
from .batches import (
|
||||
_BatchesShim,
|
||||
_row_to_record,
|
||||
all_batches,
|
||||
get_batch,
|
||||
get_record,
|
||||
list_batches,
|
||||
load_two_for_diff,
|
||||
)
|
||||
from .claim_detail import (
|
||||
check_matched_pair_drift,
|
||||
count_claims,
|
||||
count_remittances,
|
||||
distinct_providers,
|
||||
get_claim_detail,
|
||||
get_remittance,
|
||||
iter_claims,
|
||||
iter_remittances,
|
||||
recent_activity,
|
||||
summarize_remittances,
|
||||
)
|
||||
from .acks import (
|
||||
add_277ca_ack,
|
||||
add_999_ack,
|
||||
add_ta1_ack,
|
||||
get_277ca_ack,
|
||||
get_ack,
|
||||
get_ta1_ack,
|
||||
list_277ca_acks,
|
||||
list_acks,
|
||||
list_ta1_acks,
|
||||
)
|
||||
from .claim_acks import (
|
||||
add_claim_ack as _add_claim_ack,
|
||||
batch_envelope_index as _batch_envelope_index,
|
||||
find_ack_orphans as _find_ack_orphans,
|
||||
list_acks_for_claim as _list_acks_for_claim,
|
||||
list_claims_for_ack as _list_claims_for_ack,
|
||||
remove_claim_ack as _remove_claim_ack,
|
||||
)
|
||||
from .backups import add_backup_pending
|
||||
from .exceptions import AlreadyMatchedError, InvalidStateError, NotMatchedError
|
||||
from .inbox import list_unmatched, manual_match, manual_unmatch
|
||||
from .kpis import dashboard_kpis
|
||||
from .orm_builders import (
|
||||
_claim_status_from_validation,
|
||||
_persist_835_remit,
|
||||
_remittance_835_row,
|
||||
)
|
||||
from .providers import (
|
||||
ensure_clearhouse_seeded,
|
||||
get_clearhouse,
|
||||
get_payer_config,
|
||||
get_provider,
|
||||
list_payers,
|
||||
list_providers,
|
||||
update_clearhouse,
|
||||
upsert_provider,
|
||||
)
|
||||
from .records import BatchKind, BatchRecord, BatchRecord837, BatchRecord835
|
||||
from .ui import (
|
||||
to_ui_ack,
|
||||
to_ui_claim_ack,
|
||||
to_ui_claim_detail,
|
||||
to_ui_claim_from_orm,
|
||||
to_ui_provider,
|
||||
to_ui_remittance_from_orm,
|
||||
to_ui_remittance_with_adjustments,
|
||||
to_ui_ta1_ack,
|
||||
to_ui_two77ca_ack,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# UI mappers: ORM rows → simpler UI types.
|
||||
# ---------------------------------------------------------------------------
|
||||
# Backward-compat shim: tests called ``_batches.clear()`` on the in-memory
|
||||
# store. The DB-backed store doesn't have an in-memory list, so we expose
|
||||
# a tiny shim object whose ``.clear()`` wipes the DB. The class itself
|
||||
# lives in ``batches.py`` (imported above as ``_BatchesShim``); the
|
||||
# ``CycloneStore.__init__`` below instantiates that imported class.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CycloneStore: the SQLAlchemy-backed facade.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class CycloneStore:
|
||||
"""SQLAlchemy-backed facade over the parsed X12 store.
|
||||
|
||||
Each public method opens a short-lived session via
|
||||
``db.SessionLocal()()`` so callers don't have to manage session
|
||||
lifecycles. Concurrency is handled by the SQLAlchemy engine; the
|
||||
``_lock`` attribute is a no-op ``RLock`` retained for backward
|
||||
compatibility with code that wrapped cleanup in a lock context.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._lock = threading.RLock()
|
||||
self._batches = _BatchesShim()
|
||||
|
||||
# -- write path -----------------------------------------------------
|
||||
|
||||
def add(self, record: BatchRecord, *, event_bus=None):
|
||||
"""Persist a parsed batch (837P or 835)."""
|
||||
return write.add_record(record, event_bus=event_bus)
|
||||
|
||||
def _publish_events_sync(self, event_bus, record, claim_ids, remit_ids):
|
||||
return write.publish_events_sync(event_bus, record, claim_ids, remit_ids)
|
||||
|
||||
@staticmethod
|
||||
def _sync_publish(event_bus, kind: str, payload: dict) -> None:
|
||||
return write._sync_publish(event_bus, kind, payload)
|
||||
|
||||
# -- read path ------------------------------------------------------
|
||||
|
||||
def get_batch(self, batch_id: str) -> dict | None:
|
||||
return get_batch(batch_id)
|
||||
|
||||
def get(self, batch_id: str) -> BatchRecord | None:
|
||||
return get_record(batch_id)
|
||||
|
||||
def list(self, *, limit: int = 100) -> list[BatchRecord]:
|
||||
return list_batches(limit=limit)
|
||||
|
||||
def get_remittance(self, remittance_id: str) -> dict | None:
|
||||
return get_remittance(remittance_id)
|
||||
|
||||
def get_claim_detail(self, claim_id: str) -> dict | None:
|
||||
return get_claim_detail(claim_id)
|
||||
|
||||
def all(self) -> list[BatchRecord]:
|
||||
return all_batches()
|
||||
|
||||
def load_two_for_diff(
|
||||
self,
|
||||
a_id: str,
|
||||
b_id: str,
|
||||
) -> tuple[BatchRecord, BatchRecord]:
|
||||
return load_two_for_diff(a_id, b_id)
|
||||
|
||||
def iter_claims(self, **kwargs):
|
||||
return iter_claims(**kwargs)
|
||||
|
||||
def iter_remittances(self, **kwargs):
|
||||
return iter_remittances(**kwargs)
|
||||
|
||||
# -- count helpers (SP27 Task 13b) ---------------------------------
|
||||
#
|
||||
# The list endpoints (``/api/claims`` and ``/api/remittances``)
|
||||
# previously computed ``total`` by calling ``iter_*`` with default
|
||||
# ``limit=100`` and taking ``len(...)``. With 60k+ claims and 800+
|
||||
# remits in production, the reported total silently capped at 100
|
||||
# even when the UI rendered a "100" KPI tile and a 100-row table —
|
||||
# the page looked complete but the population was 600× larger. These
|
||||
# helpers reuse the iter's filter pipeline with an effectively
|
||||
# unbounded ``limit`` so the count reflects the true DB population,
|
||||
# not a 100-row sample. Mirrors Dashboard's "server-aggregated
|
||||
# counts" fix from commit 59c3275.
|
||||
def count_claims(
|
||||
self,
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
status: str | None = None,
|
||||
provider_npi: str | None = None,
|
||||
payer: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
) -> int:
|
||||
"""Count claims that would be returned by ``iter_claims``."""
|
||||
return count_claims(
|
||||
batch_id=batch_id, status=status, provider_npi=provider_npi,
|
||||
payer=payer, date_from=date_from, date_to=date_to,
|
||||
)
|
||||
|
||||
def count_remittances(
|
||||
self,
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
payer: str | None = None,
|
||||
claim_id: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
) -> int:
|
||||
"""Count remittances that would be returned by ``iter_remittances``."""
|
||||
return count_remittances(
|
||||
batch_id=batch_id, payer=payer, claim_id=claim_id,
|
||||
date_from=date_from, date_to=date_to,
|
||||
)
|
||||
|
||||
def summarize_remittances(
|
||||
self,
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
payer: str | None = None,
|
||||
claim_id: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
) -> dict:
|
||||
"""Return ``{count, total_paid, total_adjustments}`` summed over
|
||||
the remittance population that ``iter_remittances`` would return.
|
||||
"""
|
||||
return summarize_remittances(
|
||||
batch_id=batch_id, payer=payer, claim_id=claim_id,
|
||||
date_from=date_from, date_to=date_to,
|
||||
)
|
||||
|
||||
def distinct_providers(self) -> list[dict]:
|
||||
return distinct_providers()
|
||||
|
||||
def recent_activity(self, *, limit: int = 200) -> list[dict]:
|
||||
return recent_activity(limit=limit)
|
||||
|
||||
# -- 999 ACKs (SP3 P3 T13) -------------------------------------------
|
||||
|
||||
def add_ack(self, **kwargs):
|
||||
return add_999_ack(**kwargs)
|
||||
|
||||
def list_acks(self):
|
||||
return list_acks()
|
||||
|
||||
def get_ack(self, ack_id):
|
||||
return get_ack(ack_id)
|
||||
|
||||
# -- TA1 (Interchange Acknowledgment) -------------------------------
|
||||
|
||||
def add_ta1_ack(self, **kwargs):
|
||||
return add_ta1_ack(**kwargs)
|
||||
|
||||
def list_ta1_acks(self):
|
||||
return list_ta1_acks()
|
||||
|
||||
def get_ta1_ack(self, ack_id):
|
||||
return get_ta1_ack(ack_id)
|
||||
|
||||
# -- 277CA (SP10) --------------------------------------------------
|
||||
|
||||
def add_277ca_ack(self, **kwargs):
|
||||
return add_277ca_ack(**kwargs)
|
||||
|
||||
def list_277ca_acks(self):
|
||||
return list_277ca_acks()
|
||||
|
||||
def get_277ca_ack(self, ack_id):
|
||||
return get_277ca_ack(ack_id)
|
||||
|
||||
# -- SP28: claim↔ack auto-link join table --------------------------
|
||||
|
||||
def add_claim_ack(self, **kwargs):
|
||||
"""Persist one claim_acks link row.
|
||||
|
||||
Mirrors the publish-from-store contract used by the ACK paths:
|
||||
when ``event_bus`` is passed, the matching ``claim_ack_written``
|
||||
event fires after commit so live-tail subscribers on both the
|
||||
claim and the ack side see the row immediately.
|
||||
"""
|
||||
return _add_claim_ack(**kwargs)
|
||||
|
||||
def list_acks_for_claim(self, claim_id):
|
||||
"""Return every ClaimAck row for one claim (newest first)."""
|
||||
return _list_acks_for_claim(claim_id)
|
||||
|
||||
def list_claims_for_ack(self, kind, ack_id):
|
||||
"""Return every ClaimAck row for one ack."""
|
||||
return _list_claims_for_ack(kind, ack_id)
|
||||
|
||||
def find_ack_orphans(self, kind):
|
||||
"""Return acks with no resolvable link (Inbox ack-orphans lane)."""
|
||||
return _find_ack_orphans(kind)
|
||||
|
||||
def remove_claim_ack(self, link_id, *, event_bus=None):
|
||||
"""Unlink one row. Publishes ``claim_ack_dropped`` on the bus."""
|
||||
return _remove_claim_ack(link_id, event_bus=event_bus)
|
||||
|
||||
def batch_envelope_index(self):
|
||||
"""Return a {envelope.control_number: batch.id} map for D10 Pass 1."""
|
||||
return _batch_envelope_index()
|
||||
|
||||
# -- SP17: encrypted DB backups -------------------------------------
|
||||
|
||||
def add_backup_pending(self, *, filename: str, backup_dir: str):
|
||||
return add_backup_pending(filename=filename, backup_dir=backup_dir)
|
||||
|
||||
# -- manual reconciliation (T12) -----------------------------------
|
||||
|
||||
def list_unmatched(self, *, kind="both"):
|
||||
return list_unmatched(kind=kind)
|
||||
|
||||
def manual_match(self, claim_id, remit_id):
|
||||
return manual_match(claim_id, remit_id)
|
||||
|
||||
def manual_unmatch(self, claim_id):
|
||||
return manual_unmatch(claim_id)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# SP9: providers / payers / payer_configs / clearhouse
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def list_providers(self, *, is_active=True):
|
||||
return list_providers(is_active=is_active)
|
||||
|
||||
def get_provider(self, npi):
|
||||
return get_provider(npi)
|
||||
|
||||
def upsert_provider(self, provider):
|
||||
return upsert_provider(provider)
|
||||
|
||||
def list_payers(self, *, is_active=True):
|
||||
return list_payers(is_active=is_active)
|
||||
|
||||
def get_payer_config(self, payer_id, transaction_type):
|
||||
return get_payer_config(payer_id, transaction_type)
|
||||
|
||||
def get_clearhouse(self):
|
||||
return get_clearhouse()
|
||||
|
||||
def update_clearhouse(self, block):
|
||||
return update_clearhouse(block)
|
||||
|
||||
def ensure_clearhouse_seeded(self):
|
||||
return ensure_clearhouse_seeded()
|
||||
|
||||
|
||||
# Module-level singleton — same import path the old InMemoryStore used.
|
||||
store = CycloneStore()
|
||||
@@ -1,242 +0,0 @@
|
||||
"""ACK persistence — 999 / TA1 / 277CA acknowledgements.
|
||||
|
||||
Each ACK type has its own ORM row (Ack / Ta1Ack / Two77caAck). The
|
||||
write methods are simple inserts; the list/get methods are simple
|
||||
queries. Fail-soft: persistence errors are logged but do not block
|
||||
parsing.
|
||||
|
||||
SP25: every write path publishes one event on the EventBus so the
|
||||
Acks page live-tail can subscribe — mirrors the existing
|
||||
``claim_written`` / ``remittance_written`` / ``activity_recorded``
|
||||
publish pattern (see ``cyclone.store.write``). Publish is best-effort:
|
||||
a failing subscriber MUST NOT roll back the persisted row.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import date
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import Ack
|
||||
|
||||
from . import utcnow
|
||||
from .ui import to_ui_ack, to_ui_ta1_ack, to_ui_two77ca_ack
|
||||
from .write import _sync_publish
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from cyclone.pubsub import EventBus
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _safe_publish(event_bus: "EventBus | None", kind: str, payload: dict) -> None:
|
||||
"""Best-effort publish wrapped in try/except.
|
||||
|
||||
Mirrors ``CycloneStore._publish_events_sync`` — never raises, never
|
||||
rolls back the persisted row. Falls back to skipping silently when
|
||||
the bus doesn't implement the private ``_sync_publish`` interface
|
||||
(e.g. test stubs).
|
||||
"""
|
||||
if event_bus is None:
|
||||
return
|
||||
try:
|
||||
_sync_publish(event_bus, kind, payload)
|
||||
except Exception: # noqa: BLE001
|
||||
log.exception("acks store: %s publish failed", kind)
|
||||
|
||||
|
||||
# -- 999 ACKs (SP3 P3 T13) -------------------------------------------
|
||||
|
||||
def add_999_ack(
|
||||
*,
|
||||
source_batch_id: str,
|
||||
accepted_count: int,
|
||||
rejected_count: int,
|
||||
received_count: int,
|
||||
ack_code: str,
|
||||
raw_json: dict,
|
||||
event_bus: "EventBus | None" = None,
|
||||
) -> db.Ack:
|
||||
"""Persist a 999 ACK row and return it.
|
||||
|
||||
``source_batch_id`` must reference an existing ``batches.id``.
|
||||
For received 999s with no source batch the caller should pass a
|
||||
synthetic id (e.g. ``"999-<interchange_control_number>"``) —
|
||||
see ``/api/parse-999`` for that policy.
|
||||
|
||||
``raw_json`` is the full ``ParseResult999`` model dump; the
|
||||
detail endpoint surfaces it without re-parsing the original
|
||||
X12 text.
|
||||
|
||||
SP25: when ``event_bus`` is provided, publishes one
|
||||
``ack_received`` event (with the full ``to_ui_ack`` row shape)
|
||||
after the row commits so the Acks page live-tail sees new 999s
|
||||
the moment they land.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = Ack(
|
||||
source_batch_id=source_batch_id,
|
||||
accepted_count=accepted_count,
|
||||
rejected_count=rejected_count,
|
||||
received_count=received_count,
|
||||
ack_code=ack_code,
|
||||
parsed_at=utcnow(),
|
||||
raw_json=raw_json,
|
||||
)
|
||||
s.add(row)
|
||||
s.commit()
|
||||
s.refresh(row)
|
||||
row_id = row.id
|
||||
|
||||
if event_bus is not None:
|
||||
with db.SessionLocal()() as s:
|
||||
payload = to_ui_ack(s.get(Ack, row_id))
|
||||
_safe_publish(event_bus, "ack_received", payload)
|
||||
|
||||
return row
|
||||
|
||||
def list_acks() -> list[db.Ack]:
|
||||
"""Return every 999 ACK row, newest first (auto-increment id desc)."""
|
||||
with db.SessionLocal()() as s:
|
||||
return (
|
||||
s.query(Ack)
|
||||
.order_by(Ack.id.desc())
|
||||
.all()
|
||||
)
|
||||
|
||||
def get_ack( ack_id: int) -> db.Ack | None:
|
||||
"""Return a single ACK row by id, or ``None`` if not found."""
|
||||
with db.SessionLocal()() as s:
|
||||
return s.get(Ack, ack_id)
|
||||
|
||||
# -- TA1 (Interchange Acknowledgment) -------------------------------
|
||||
|
||||
def add_ta1_ack(
|
||||
*,
|
||||
source_batch_id: str,
|
||||
control_number: str,
|
||||
interchange_date: date | None,
|
||||
interchange_time: str | None,
|
||||
ack_code: str,
|
||||
note_code: str | None,
|
||||
ack_generated_date: date | None,
|
||||
sender_id: str,
|
||||
receiver_id: str,
|
||||
raw_json: dict,
|
||||
event_bus: "EventBus | None" = None,
|
||||
) -> db.Ta1Ack:
|
||||
"""Persist a TA1 (Interchange Acknowledgment) row and return it.
|
||||
|
||||
Mirrors :meth:`add_999_ack` for the lower-level envelope ack. The
|
||||
flat columns are promoted out of ``raw_json`` so the list
|
||||
endpoint can sort/filter without a JSON parse; the full
|
||||
``ParseResultTa1`` stays in ``raw_json`` for the detail endpoint.
|
||||
|
||||
SP25: when ``event_bus`` is provided, publishes one
|
||||
``ta1_ack_received`` event after the row commits.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = db.Ta1Ack(
|
||||
source_batch_id=source_batch_id,
|
||||
control_number=control_number,
|
||||
interchange_date=interchange_date,
|
||||
interchange_time=interchange_time,
|
||||
ack_code=ack_code,
|
||||
note_code=note_code,
|
||||
ack_generated_date=ack_generated_date,
|
||||
sender_id=sender_id,
|
||||
receiver_id=receiver_id,
|
||||
parsed_at=utcnow(),
|
||||
raw_json=raw_json,
|
||||
)
|
||||
s.add(row)
|
||||
s.commit()
|
||||
s.refresh(row)
|
||||
row_id = row.id
|
||||
|
||||
if event_bus is not None:
|
||||
with db.SessionLocal()() as s:
|
||||
payload = to_ui_ta1_ack(s.get(db.Ta1Ack, row_id))
|
||||
_safe_publish(event_bus, "ta1_ack_received", payload)
|
||||
|
||||
return row
|
||||
|
||||
def list_ta1_acks() -> list[db.Ta1Ack]:
|
||||
"""Return every TA1 ACK row, newest first (auto-increment id desc).
|
||||
|
||||
Mirrors :meth:`list_acks` — the API endpoint slices to its own
|
||||
``limit`` so the ``total`` field reflects the full row count.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
return (
|
||||
s.query(db.Ta1Ack)
|
||||
.order_by(db.Ta1Ack.id.desc())
|
||||
.all()
|
||||
)
|
||||
|
||||
def get_ta1_ack( ack_id: int) -> db.Ta1Ack | None:
|
||||
"""Return a single TA1 ACK row by id, or ``None`` if not found."""
|
||||
with db.SessionLocal()() as s:
|
||||
return s.get(db.Ta1Ack, ack_id)
|
||||
|
||||
# -- 277CA (SP10) --------------------------------------------------
|
||||
|
||||
def add_277ca_ack(
|
||||
*,
|
||||
source_batch_id: str,
|
||||
control_number: str,
|
||||
accepted_count: int,
|
||||
rejected_count: int,
|
||||
paid_count: int,
|
||||
pended_count: int,
|
||||
raw_json: dict,
|
||||
event_bus: "EventBus | None" = None,
|
||||
) -> db.Two77caAck:
|
||||
"""Persist a 277CA (Claim Acknowledgment) row and return it.
|
||||
|
||||
Mirrors :meth:`add_999_ack` but for the claim-level ack. The
|
||||
per-claim status detail stays in ``raw_json``; only the four
|
||||
counts are promoted so the list endpoint stays fast.
|
||||
|
||||
SP25: when ``event_bus`` is provided, publishes one
|
||||
``two77ca_ack_received`` event after the row commits.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = db.Two77caAck(
|
||||
source_batch_id=source_batch_id,
|
||||
control_number=control_number,
|
||||
accepted_count=accepted_count,
|
||||
rejected_count=rejected_count,
|
||||
paid_count=paid_count,
|
||||
pended_count=pended_count,
|
||||
parsed_at=utcnow(),
|
||||
raw_json=raw_json,
|
||||
)
|
||||
s.add(row)
|
||||
s.commit()
|
||||
s.refresh(row)
|
||||
row_id = row.id
|
||||
|
||||
if event_bus is not None:
|
||||
with db.SessionLocal()() as s:
|
||||
payload = to_ui_two77ca_ack(s.get(db.Two77caAck, row_id))
|
||||
_safe_publish(event_bus, "two77ca_ack_received", payload)
|
||||
|
||||
return row
|
||||
|
||||
def list_277ca_acks() -> list[db.Two77caAck]:
|
||||
"""Return every 277CA ACK row, newest first (auto-increment id desc)."""
|
||||
with db.SessionLocal()() as s:
|
||||
return (
|
||||
s.query(db.Two77caAck)
|
||||
.order_by(db.Two77caAck.id.desc())
|
||||
.all()
|
||||
)
|
||||
|
||||
def get_277ca_ack( ack_id: int) -> db.Two77caAck | None:
|
||||
"""Return a single 277CA ACK row by id, or ``None`` if not found."""
|
||||
with db.SessionLocal()() as s:
|
||||
return s.get(db.Two77caAck, ack_id)
|
||||
|
||||
@@ -1,340 +0,0 @@
|
||||
"""SP32 Task 6: backfill rendering/service-provider NPIs from on-disk files.
|
||||
|
||||
The T4 writers populate ``Claim.rendering_provider_npi`` (from NM1*82 in
|
||||
837P Loop 2420A) and ``Remittance.rendering_provider_npi`` (from the
|
||||
NM1*1P service-provider segment in 835 Loop 2100). Rows ingested before
|
||||
T4 was wired (or ingested via a path that bypasses the writer — e.g. an
|
||||
ad-hoc ``store.add`` from a notebook) still have a NULL column.
|
||||
|
||||
This module re-parses on-disk X12 files and patches up those columns on
|
||||
matching rows. It is **idempotent**: rows whose
|
||||
``rendering_provider_npi`` is already non-NULL are left untouched, and
|
||||
re-running the same file twice is a clean no-op.
|
||||
|
||||
Public surface
|
||||
--------------
|
||||
|
||||
* :func:`backfill_rendering_provider_npi` — entry point used by the CLI.
|
||||
* :class:`BackfillSummary` — counts dataclass echoed back as a one-line
|
||||
summary by the CLI.
|
||||
|
||||
Reconcile is run once at the end so the new typed NPI arm (T5) can fire
|
||||
retroactively across the open claim/remit pairs the backfill touched.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.parsers.parse_835 import parse as parse_835
|
||||
from cyclone.parsers.parse_837 import parse as parse_837
|
||||
from cyclone.parsers.payer import PayerConfig, PayerConfig835
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 837P fixture uses NM1*82 → rendering_provider_npi.
|
||||
# 835 fixture uses NM1*1P → service_provider_npi (mapped onto the same
|
||||
# Remittance.rendering_provider_npi column by T4 _remittance_835_row).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class BackfillSummary:
|
||||
"""Counts emitted by ``backfill_rendering_provider_npi``.
|
||||
|
||||
The CLI echoes a one-line summary (`claims_updated=N remits_updated=N
|
||||
…`) at the end of the run.
|
||||
"""
|
||||
|
||||
claims_updated: int = 0
|
||||
remits_updated: int = 0
|
||||
files_processed: int = 0
|
||||
files_skipped: int = 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Parser wrappers — keep exceptions local so one bad file can't abort
|
||||
# the whole backfill run.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _parse_837_file(path: Path) -> tuple[list, Path | None]:
|
||||
"""Re-parse a single 837P file. Returns ``(claims, broken_alias)``.
|
||||
|
||||
On success, ``claims`` is the list of parsed ``ClaimOutput`` rows
|
||||
and ``broken_alias`` is None. On any failure, ``claims`` is empty
|
||||
and ``broken_alias`` is the same ``path`` so the caller can log
|
||||
which file was skipped.
|
||||
"""
|
||||
try:
|
||||
text = path.read_text()
|
||||
except OSError as exc:
|
||||
log.warning("backfill: cannot read %s: %s", path, exc)
|
||||
return [], path
|
||||
try:
|
||||
result = parse_837(text, PayerConfig.co_medicaid(), input_file=str(path))
|
||||
except Exception as exc: # noqa: BLE001 — failure isolated per-file
|
||||
log.warning("backfill: 837 parse failed for %s: %s", path, exc)
|
||||
return [], path
|
||||
return list(result.claims), None
|
||||
|
||||
|
||||
def _parse_835_file(path: Path) -> tuple[list, Path | None]:
|
||||
"""Re-parse a single 835 file. Returns ``(claims, broken_alias)``.
|
||||
|
||||
``claims`` is the list of parsed ``ClaimPayment`` rows.
|
||||
"""
|
||||
try:
|
||||
text = path.read_text()
|
||||
except OSError as exc:
|
||||
log.warning("backfill: cannot read %s: %s", path, exc)
|
||||
return [], path
|
||||
try:
|
||||
result = parse_835(text, PayerConfig835.co_medicaid_835(), input_file=str(path))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.warning("backfill: 835 parse failed for %s: %s", path, exc)
|
||||
return [], path
|
||||
return list(result.claims), None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DB-patching helpers — keep the per-row update logic in one place.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _patch_claim_rendering_npi(parsed_claims: list) -> int:
|
||||
"""Update ``Claim.rendering_provider_npi`` for any parsed 837 claim.
|
||||
|
||||
Only rows whose column is currently NULL are touched. Returns the
|
||||
count of rows updated.
|
||||
"""
|
||||
from cyclone.db import Claim, SessionLocal
|
||||
|
||||
updated = 0
|
||||
with SessionLocal()() as session:
|
||||
for parsed in parsed_claims:
|
||||
npi = getattr(parsed, "rendering_provider_npi", None)
|
||||
if not npi:
|
||||
continue
|
||||
row = session.get(Claim, parsed.claim_id)
|
||||
if row is None:
|
||||
continue
|
||||
if row.rendering_provider_npi is not None:
|
||||
# Already populated (e.g. by a prior backfill run, or by
|
||||
# the T4 writer if the claim was ingested afterwards).
|
||||
continue
|
||||
row.rendering_provider_npi = npi
|
||||
updated += 1
|
||||
if updated:
|
||||
session.commit()
|
||||
return updated
|
||||
|
||||
|
||||
def _patch_remit_rendering_npi(parsed_claims: list) -> int:
|
||||
"""Update ``Remittance.rendering_provider_npi`` for any parsed 835 claim.
|
||||
|
||||
The 835 NM1*1P segment maps onto ``ClaimPayment.service_provider_npi``
|
||||
(which the T4 writer copies into ``Remittance.rendering_provider_npi``).
|
||||
We match on ``payer_claim_control_number``; only NULL columns are
|
||||
touched. Returns the count of rows updated.
|
||||
"""
|
||||
from cyclone.db import Remittance, SessionLocal
|
||||
|
||||
updated = 0
|
||||
with SessionLocal()() as session:
|
||||
for parsed in parsed_claims:
|
||||
npi = getattr(parsed, "service_provider_npi", None)
|
||||
if not npi:
|
||||
continue
|
||||
target_pcn = getattr(parsed, "payer_claim_control_number", None)
|
||||
if not target_pcn:
|
||||
continue
|
||||
row = session.execute(
|
||||
select(Remittance).where(
|
||||
Remittance.payer_claim_control_number == target_pcn,
|
||||
Remittance.rendering_provider_npi.is_(None),
|
||||
)
|
||||
).scalars().first()
|
||||
if row is None:
|
||||
continue
|
||||
row.rendering_provider_npi = npi
|
||||
updated += 1
|
||||
if updated:
|
||||
session.commit()
|
||||
return updated
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Reconcile pass — run once across every 835 batch so the T5 scoring arm
|
||||
# can fire retroactively on pairs the backfill touched.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _run_reconcile_sweep() -> int:
|
||||
"""Run :func:`cyclone.reconcile.run` over every 835 batch. Returns count.
|
||||
|
||||
Failures are isolated per-batch so one bad batch can't abort the
|
||||
sweep — the goal is "best-effort retroactive reconcile".
|
||||
"""
|
||||
from sqlalchemy import select as _select
|
||||
|
||||
from cyclone import reconcile as _reconcile
|
||||
from cyclone.db import Batch, SessionLocal
|
||||
|
||||
completed = 0
|
||||
with SessionLocal()() as session:
|
||||
batch_ids = [
|
||||
row[0] for row in session.execute(
|
||||
_select(Batch.id).where(Batch.kind == "835")
|
||||
).all()
|
||||
]
|
||||
# Each batch gets its own session — ``reconcile.run`` does not commit,
|
||||
# so an exception in one batch must not orphan a half-flushed
|
||||
# transaction.
|
||||
for bid in batch_ids:
|
||||
try:
|
||||
with SessionLocal()() as s:
|
||||
_reconcile.run(s, bid)
|
||||
s.commit()
|
||||
completed += 1
|
||||
except Exception as exc: # noqa: BLE001
|
||||
log.warning("backfill: reconcile failed for batch %s: %s", bid, exc)
|
||||
return completed
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public entry point.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
# Supported ``--type`` flag values; also ``None`` means auto-detect.
|
||||
TransactionType = str | None
|
||||
|
||||
|
||||
def backfill_rendering_provider_npi(
|
||||
*,
|
||||
files: Iterable[Path] | None = None,
|
||||
input_dir: Path | None = None,
|
||||
transaction_type: TransactionType = None,
|
||||
) -> BackfillSummary:
|
||||
"""Re-parse on-disk 837p + 835 files and populate the typed NPI columns.
|
||||
|
||||
Args:
|
||||
files: explicit list of file paths to re-parse.
|
||||
input_dir: directory to scan for ``*.txt`` / ``*.edi`` files.
|
||||
Files are scanned one level deep.
|
||||
transaction_type: ``"837p"`` or ``"835"``. ``None`` auto-detects
|
||||
by attempting the 837p parser first and falling back to
|
||||
the 835 parser.
|
||||
|
||||
Returns:
|
||||
:class:`BackfillSummary` with populated counts.
|
||||
|
||||
Idempotent: only writes columns that are currently NULL; re-running
|
||||
on the same files is a clean no-op. After patching, runs
|
||||
:func:`cyclone.reconcile.run` over every 835 batch so the T5
|
||||
scoring arm can re-fire on the touched pairs.
|
||||
"""
|
||||
summary = BackfillSummary()
|
||||
|
||||
# 1. Resolve the candidate file set.
|
||||
candidates: list[Path] = []
|
||||
if files:
|
||||
candidates.extend(Path(f) for f in files)
|
||||
if input_dir is not None:
|
||||
for ext in ("*.txt", "*.edi", "*.835", "*.837", "*.x12"):
|
||||
candidates.extend(input_dir.glob(ext))
|
||||
candidates = [p for p in candidates if p.is_file()]
|
||||
|
||||
# 2. Re-parse each file and patch matching rows.
|
||||
for path in candidates:
|
||||
kind = transaction_type or _sniff_kind(path)
|
||||
if kind == "837p":
|
||||
parsed_claims, broken = _parse_837_file(path)
|
||||
if broken is not None:
|
||||
summary.files_skipped += 1
|
||||
continue
|
||||
summary.files_processed += 1
|
||||
summary.claims_updated += _patch_claim_rendering_npi(parsed_claims)
|
||||
elif kind == "835":
|
||||
parsed_claims, broken = _parse_835_file(path)
|
||||
if broken is not None:
|
||||
summary.files_skipped += 1
|
||||
continue
|
||||
summary.files_processed += 1
|
||||
summary.remits_updated += _patch_remit_rendering_npi(parsed_claims)
|
||||
else:
|
||||
# Auto-detect tried both parsers and both failed → skip.
|
||||
summary.files_skipped += 1
|
||||
|
||||
# 3. Reconcile sweep — let the T5 NPI arm fire retroactively.
|
||||
if summary.claims_updated or summary.remits_updated:
|
||||
try:
|
||||
_run_reconcile_sweep()
|
||||
except Exception: # noqa: BLE001 — sweep is best-effort
|
||||
log.exception("backfill: reconcile sweep failed")
|
||||
|
||||
return summary
|
||||
|
||||
|
||||
def _sniff_kind(path: Path) -> TransactionType:
|
||||
"""Best-effort transaction-type sniff (content + filename).
|
||||
|
||||
Returns ``"837p"``, ``"835"``, or ``None`` if both parsers fail.
|
||||
Used only when the caller didn't pin ``transaction_type`` explicitly
|
||||
via the CLI.
|
||||
"""
|
||||
try:
|
||||
text = path.read_text()
|
||||
except OSError:
|
||||
return None
|
||||
# Cheap filename hint.
|
||||
name = path.name.lower()
|
||||
if name.endswith(".835") or "835" in name:
|
||||
return _try_both(path, text, prefer="835")
|
||||
if name.endswith(".837") or "837" in name or "837p" in name:
|
||||
return _try_both(path, text, prefer="837p")
|
||||
# Content: ISA + ST. 837 starts with "ST*837", 835 starts with "ST*835".
|
||||
if "ST*837*" in text:
|
||||
return "837p"
|
||||
if "ST*835*" in text:
|
||||
return "835"
|
||||
# Fallback: try 837p parser, then 835.
|
||||
return _try_both(path, text, prefer="837p")
|
||||
|
||||
|
||||
def _try_both(path: Path, text: str, *, prefer: str) -> TransactionType:
|
||||
"""Try the preferred parser first, then the other; return first winner."""
|
||||
if prefer == "837p":
|
||||
try:
|
||||
parse_837(text, PayerConfig.co_medicaid(), input_file=str(path))
|
||||
return "837p"
|
||||
except Exception:
|
||||
try:
|
||||
parse_835(text, PayerConfig835.co_medicaid_835(), input_file=str(path))
|
||||
return "835"
|
||||
except Exception:
|
||||
return None
|
||||
try:
|
||||
parse_835(text, PayerConfig835.co_medicaid_835(), input_file=str(path))
|
||||
return "835"
|
||||
except Exception:
|
||||
try:
|
||||
parse_837(text, PayerConfig.co_medicaid(), input_file=str(path))
|
||||
return "837p"
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BackfillSummary",
|
||||
"backfill_rendering_provider_npi",
|
||||
]
|
||||
@@ -1,39 +0,0 @@
|
||||
"""Backup-pending marker inserts — SP17 encrypted DB backups.
|
||||
|
||||
``add_backup_pending()`` inserts a ``pending`` row for a backup that is
|
||||
about to start; the BackupService updates ``status`` / ``size_bytes``
|
||||
/ ``db_fingerprint`` / ``table_count`` / ``completed_at`` after the
|
||||
encrypted blob lands on disk.
|
||||
"""
|
||||
|
||||
from cyclone import db
|
||||
|
||||
from . import utcnow
|
||||
|
||||
|
||||
# -- SP17: encrypted DB backups -------------------------------------
|
||||
|
||||
def add_backup_pending(*, filename: str, backup_dir: str) -> db.DbBackup:
|
||||
"""Insert a ``pending`` row for a backup that is about to start.
|
||||
|
||||
The BackupService fills in ``status`` / ``size_bytes`` /
|
||||
``db_fingerprint`` / ``table_count`` / ``completed_at`` after
|
||||
the encrypted blob lands on disk.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = db.DbBackup(
|
||||
filename=filename,
|
||||
backup_dir=backup_dir,
|
||||
size_bytes=0,
|
||||
db_fingerprint=None,
|
||||
table_count=0,
|
||||
created_at=utcnow(),
|
||||
completed_at=None,
|
||||
status="pending",
|
||||
error_message=None,
|
||||
)
|
||||
s.add(row)
|
||||
s.commit()
|
||||
s.refresh(row)
|
||||
return row
|
||||
|
||||
@@ -1,155 +0,0 @@
|
||||
"""Batch read APIs and the legacy _BatchesShim.
|
||||
|
||||
The ``_BatchesShim`` class is retained for back-compat with the
|
||||
``with store._lock: store._batches.clear()`` test-cleanup idiom used
|
||||
in 7 test files. Its ``clear()`` method wipes all rows from the DB
|
||||
tables so tests that depended on a fresh in-memory list per-test get
|
||||
a fresh DB state per-test.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import timezone
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import (
|
||||
ActivityEvent,
|
||||
Batch,
|
||||
CasAdjustment,
|
||||
Claim,
|
||||
Match,
|
||||
Remittance,
|
||||
)
|
||||
from cyclone.parsers.models import ParseResult
|
||||
from cyclone.parsers.models_835 import ParseResult835
|
||||
|
||||
from .records import BatchRecord, BatchRecord837, BatchRecord835
|
||||
|
||||
|
||||
class _BatchesShim:
|
||||
"""Drop-in replacement for the old in-memory ``_batches`` list.
|
||||
|
||||
``clear()`` removes every row from the DB tables in FK-safe order.
|
||||
Other list operations are not implemented because the only call site
|
||||
is the ``clear()`` inside the test fixtures (``test_api_gets.py`` and
|
||||
``test_api_parse_persists.py``).
|
||||
"""
|
||||
|
||||
def clear(self) -> None: # type: ignore[no-untyped-def]
|
||||
with db.SessionLocal()() as s:
|
||||
s.query(ActivityEvent).delete()
|
||||
s.query(Match).delete()
|
||||
s.query(CasAdjustment).delete()
|
||||
s.query(Remittance).delete()
|
||||
s.query(Claim).delete()
|
||||
s.query(Batch).delete()
|
||||
s.commit()
|
||||
|
||||
|
||||
def _row_to_record(row: Batch) -> BatchRecord:
|
||||
"""Rehydrate a ``BatchRecord`` (837 or 835) from a Batch ORM row.
|
||||
|
||||
The full ``ParseResult`` / ``ParseResult835`` lives in
|
||||
``raw_result_json`` (stashed at insert time). Re-parsing JSON
|
||||
here means callers get the same typed Pydantic object the old
|
||||
in-memory store handed out, so api.py and tests that do
|
||||
``rec.result.claims`` keep working unchanged.
|
||||
|
||||
SQLite drops tz info on round-trip even though the column type
|
||||
is ``DateTime(timezone=True)``. We re-attach UTC so the
|
||||
``BatchRecord`` validator (``parsed_at must be tz-aware``)
|
||||
passes.
|
||||
"""
|
||||
if row.kind == "835":
|
||||
result_cls = ParseResult835
|
||||
else:
|
||||
result_cls = ParseResult
|
||||
payload = row.raw_result_json or {}
|
||||
result = result_cls.model_validate(payload)
|
||||
parsed_at = row.parsed_at
|
||||
if parsed_at is not None and parsed_at.tzinfo is None:
|
||||
parsed_at = parsed_at.replace(tzinfo=timezone.utc)
|
||||
record_cls = BatchRecord835 if row.kind == "835" else BatchRecord837
|
||||
return record_cls(
|
||||
id=row.id,
|
||||
kind=row.kind,
|
||||
input_filename=row.input_filename,
|
||||
parsed_at=parsed_at,
|
||||
result=result,
|
||||
)
|
||||
|
||||
|
||||
def get_batch(batch_id: str) -> dict | None:
|
||||
"""Return a summary dict for ``batch_id`` or ``None`` if missing.
|
||||
|
||||
The dict shape matches what ``/api/batches/{id}`` callers need:
|
||||
``id``, ``kind``, ``input_filename``, ``parsed_at``, and the
|
||||
full ``result`` (raw_result_json) as a dict.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(Batch, batch_id)
|
||||
if row is None:
|
||||
return None
|
||||
return {
|
||||
"id": row.id,
|
||||
"kind": row.kind,
|
||||
"input_filename": row.input_filename,
|
||||
"parsed_at": row.parsed_at,
|
||||
"result": row.raw_result_json,
|
||||
}
|
||||
|
||||
|
||||
def get_record(batch_id: str) -> BatchRecord | None:
|
||||
"""Return the ``BatchRecord`` for ``batch_id`` or ``None``.
|
||||
|
||||
Preserves the in-memory store contract: callers get a Pydantic
|
||||
``BatchRecord`` (subclass ``BatchRecord837`` / ``BatchRecord835``)
|
||||
with ``.id``, ``.kind``, ``.input_filename``, ``.parsed_at``,
|
||||
and ``.result`` (typed ``ParseResult`` / ``ParseResult835``).
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(Batch, batch_id)
|
||||
if row is None:
|
||||
return None
|
||||
return _row_to_record(row)
|
||||
|
||||
|
||||
def list_batches(*, limit: int = 100) -> list[BatchRecord]:
|
||||
"""Return up to ``limit`` ``BatchRecord``s, newest first."""
|
||||
with db.SessionLocal()() as s:
|
||||
rows = (
|
||||
s.query(Batch)
|
||||
.order_by(Batch.parsed_at.desc())
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
return [_row_to_record(r) for r in rows]
|
||||
|
||||
|
||||
def all_batches() -> list[BatchRecord]:
|
||||
"""Return every ``BatchRecord``, oldest first (no pagination)."""
|
||||
with db.SessionLocal()() as s:
|
||||
rows = s.query(Batch).order_by(Batch.parsed_at.asc()).all()
|
||||
return [_row_to_record(r) for r in rows]
|
||||
|
||||
|
||||
def load_two_for_diff(a_id: str, b_id: str) -> tuple[BatchRecord, BatchRecord]:
|
||||
"""Load two batches by id for the side-by-side diff view.
|
||||
|
||||
Returns ``(a, b)`` as ``BatchRecord`` objects. Raises
|
||||
:class:`LookupError` when either id is missing — the API layer
|
||||
catches it and maps it to ``404 Not Found`` (matching the
|
||||
``GET /api/batches/{id}`` contract). The two loads happen in
|
||||
independent sessions so a transient failure on one side can't
|
||||
poison the other.
|
||||
|
||||
Used exclusively by :mod:`cyclone.batch_diff` via the
|
||||
``/api/batch-diff`` endpoint.
|
||||
"""
|
||||
a = get_record(a_id)
|
||||
if a is None:
|
||||
raise LookupError(f"batch {a_id} not found")
|
||||
b = get_record(b_id)
|
||||
if b is None:
|
||||
raise LookupError(f"batch {b_id} not found")
|
||||
return a, b
|
||||
@@ -1,360 +0,0 @@
|
||||
"""SP28: claim_acks persistence + batch envelope index (D10).
|
||||
|
||||
Five methods on top of the new ``ClaimAck`` ORM table:
|
||||
|
||||
* ``add_claim_ack`` — insert one link row (manual or auto) and
|
||||
publish ``claim_ack_written`` on the bus.
|
||||
* ``list_acks_for_claim`` — every link row for one claim (per-claim
|
||||
only; TA1 batch-level rows are filtered out by the API layer).
|
||||
* ``list_claims_for_ack`` — every link row for one ack.
|
||||
* ``find_ack_orphans`` — acks with no resolvable claim (Inbox
|
||||
ack-orphans lane).
|
||||
* ``remove_claim_ack`` — unlink. Publishes ``claim_ack_dropped``.
|
||||
* ``batch_envelope_index`` — D10 in-memory map of
|
||||
``Batch.envelope.control_number → batch.id`` (cheap to rebuild;
|
||||
re-built once per ingest).
|
||||
|
||||
Each mutating method opens its own ``db.SessionLocal()()`` session
|
||||
so callers don't have to manage session lifecycles. Publishes are
|
||||
best-effort (wrapped in :func:`_safe_publish`) so a failing bus
|
||||
subscriber cannot roll back the persisted row.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import (
|
||||
Ack,
|
||||
Batch,
|
||||
Claim,
|
||||
ClaimAck,
|
||||
Ta1Ack,
|
||||
Two77caAck,
|
||||
)
|
||||
|
||||
from . import utcnow
|
||||
from .ui import to_ui_claim_ack
|
||||
from .write import _sync_publish
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from cyclone.pubsub import EventBus
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _safe_publish(event_bus: "EventBus | None", kind: str, payload: dict) -> None:
|
||||
"""Best-effort publish wrapped in try/except.
|
||||
|
||||
Mirrors ``cyclone.store.acks._safe_publish`` — never raises,
|
||||
never rolls back the persisted row.
|
||||
"""
|
||||
if event_bus is None:
|
||||
return
|
||||
try:
|
||||
_sync_publish(event_bus, kind, payload)
|
||||
except Exception: # noqa: BLE001
|
||||
log.exception("claim_acks store: %s publish failed", kind)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# D10 batch envelope index
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def batch_envelope_index() -> dict[str, str]:
|
||||
"""Build a ``{envelope.control_number: batch.id}`` map.
|
||||
|
||||
D10 primary join key (spec §D10). Built once per ingest from
|
||||
every Batch row whose ``raw_result_json`` carries an envelope —
|
||||
cost is O(N batches), currently ~16, so trivial. Kept as a
|
||||
plain dict so callers can ``index.get(scn)`` to resolve.
|
||||
"""
|
||||
out: dict[str, str] = {}
|
||||
with db.SessionLocal()() as s:
|
||||
rows = (
|
||||
s.query(Batch.id, Batch.raw_result_json)
|
||||
.filter(Batch.kind == "837p")
|
||||
.all()
|
||||
)
|
||||
for bid, raw in rows:
|
||||
env = (raw or {}).get("envelope") or {}
|
||||
ctrl = env.get("control_number")
|
||||
if isinstance(ctrl, str) and ctrl:
|
||||
out[ctrl] = bid
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Mutators
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def add_claim_ack(
|
||||
*,
|
||||
claim_id: str | None,
|
||||
batch_id: str | None,
|
||||
ack_id: int,
|
||||
ack_kind: str,
|
||||
ak2_index: int | None = None,
|
||||
set_control_number: str | None = None,
|
||||
set_accept_reject_code: str | None = None,
|
||||
linked_by: str = "auto",
|
||||
event_bus: "EventBus | None" = None,
|
||||
now: datetime | None = None,
|
||||
) -> ClaimAck:
|
||||
"""Persist one link row and return it.
|
||||
|
||||
The DB-level unique index
|
||||
``ux_claim_acks_dedup(claim_id, ack_kind, ack_id, ak2_index)
|
||||
WHERE claim_id IS NOT NULL AND ak2_index IS NOT NULL`` enforces
|
||||
idempotency for the per-AK2 case. The 277CA (ak2_index NULL)
|
||||
and TA1 (claim_id NULL) paths are deduplicated by application
|
||||
code — see :func:`cyclone.claim_acks._link_exists`.
|
||||
|
||||
When ``event_bus`` is provided, publishes one ``claim_ack_written``
|
||||
event (with the full :func:`cyclone.store.ui.to_ui_claim_ack`
|
||||
payload) so the live-tail subscribers on both
|
||||
``/api/claims/{id}/acks/stream`` and
|
||||
``/api/acks/{kind}/{id}/claims/stream`` see new rows the moment
|
||||
they land.
|
||||
"""
|
||||
if ack_kind not in ("999", "277ca", "ta1"):
|
||||
raise ValueError(f"add_claim_ack: unknown ack_kind={ack_kind!r}")
|
||||
if linked_by not in ("auto", "manual"):
|
||||
raise ValueError(f"add_claim_ack: linked_by={linked_by!r}")
|
||||
if claim_id is None and batch_id is None:
|
||||
raise ValueError(
|
||||
"add_claim_ack: at least one of claim_id/batch_id must be set"
|
||||
)
|
||||
ts = now or utcnow()
|
||||
with db.SessionLocal()() as s:
|
||||
row = ClaimAck(
|
||||
claim_id=claim_id,
|
||||
batch_id=batch_id,
|
||||
ack_id=ack_id,
|
||||
ack_kind=ack_kind,
|
||||
ak2_index=ak2_index,
|
||||
set_control_number=set_control_number,
|
||||
set_accept_reject_code=set_accept_reject_code,
|
||||
linked_at=ts,
|
||||
linked_by=linked_by,
|
||||
)
|
||||
s.add(row)
|
||||
s.commit()
|
||||
s.refresh(row)
|
||||
row_id = row.id
|
||||
|
||||
if event_bus is not None:
|
||||
with db.SessionLocal()() as s:
|
||||
payload = to_ui_claim_ack(s.get(ClaimAck, row_id))
|
||||
_safe_publish(event_bus, "claim_ack_written", payload)
|
||||
|
||||
return row
|
||||
|
||||
|
||||
def remove_claim_ack(
|
||||
link_id: int,
|
||||
*,
|
||||
event_bus: "EventBus | None" = None,
|
||||
) -> bool:
|
||||
"""Unlink. Returns ``True`` when a row was deleted.
|
||||
|
||||
Publishes ``claim_ack_dropped`` with ``{"id", "claim_id"}`` so
|
||||
the live-tail subscribers can remove the link from their local
|
||||
store. Does NOT touch ``Claim.state`` — the unlink is purely
|
||||
about the link row, per spec §D6.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(ClaimAck, link_id)
|
||||
if row is None:
|
||||
return False
|
||||
payload = {
|
||||
"id": row.id,
|
||||
"claim_id": row.claim_id,
|
||||
"ack_id": row.ack_id,
|
||||
"ack_kind": row.ack_kind,
|
||||
}
|
||||
s.delete(row)
|
||||
s.commit()
|
||||
|
||||
if event_bus is not None:
|
||||
_safe_publish(event_bus, "claim_ack_dropped", payload)
|
||||
return True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Readers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def list_acks_for_claim(claim_id: str) -> list[ClaimAck]:
|
||||
"""Return every ClaimAck row for one claim (newest first)."""
|
||||
with db.SessionLocal()() as s:
|
||||
return (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.claim_id == claim_id)
|
||||
.order_by(ClaimAck.id.desc())
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
def list_claims_for_ack(kind: str, ack_id: int) -> list[ClaimAck]:
|
||||
"""Return every ClaimAck row for one ack (any kind).
|
||||
|
||||
For 999 / 277CA: returns 0..N rows (one per AK2 / ClaimStatus).
|
||||
For TA1: returns 0..1 row (envelope-level, populated batch_id).
|
||||
"""
|
||||
if kind not in ("999", "277ca", "ta1"):
|
||||
raise ValueError(f"list_claims_for_ack: unknown kind={kind!r}")
|
||||
with db.SessionLocal()() as s:
|
||||
return (
|
||||
s.query(ClaimAck)
|
||||
.filter(
|
||||
ClaimAck.ack_kind == kind,
|
||||
ClaimAck.ack_id == ack_id,
|
||||
)
|
||||
.order_by(ClaimAck.id.asc())
|
||||
.all()
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Orphan detection (Inbox "Ack orphans" lane — spec §D7)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def find_ack_orphans(kind: str) -> list[dict]:
|
||||
"""Return acks with no resolvable Claim row of their own kind.
|
||||
|
||||
Used by the Inbox ack-orphans lane for the operator's manual
|
||||
reconciliation flow. The downstream ``/api/inbox/ack-orphans``
|
||||
endpoint calls this and returns the rendered shape.
|
||||
|
||||
"Orphan" means: for 999 / 277CA, ``ack_kind=kind AND ack_id IN
|
||||
(acks_with_no_claim_acks_link)``. For TA1, "orphan" means the
|
||||
TA1 row exists but no Batch with matching sender/receiver was
|
||||
resolved (so no link row was created).
|
||||
|
||||
Output dict shape (one row per orphan ack, rendered by
|
||||
:func:`to_ui_claim_ack`-style serialization):
|
||||
|
||||
* ``kind`` — "999" / "277ca" / "ta1"
|
||||
* ``ack_id`` — the ack row's id
|
||||
* ``control_number`` — for 999/277CA, envelope.control_number;
|
||||
for TA1, ta1.control_number
|
||||
* ``set_control_numbers`` — empty list when no claims match
|
||||
* ``raw_summary`` — flat copy of the ack's UI shape for the
|
||||
lane-header counts
|
||||
"""
|
||||
if kind not in ("999", "277ca", "ta1"):
|
||||
raise ValueError(f"find_ack_orphans: unknown kind={kind!r}")
|
||||
|
||||
out: list[dict] = []
|
||||
if kind == "999":
|
||||
ack_table = Ack
|
||||
ctrl_attr = None
|
||||
elif kind == "277ca":
|
||||
ack_table = Two77caAck
|
||||
ctrl_attr = "control_number"
|
||||
else:
|
||||
ack_table = Ta1Ack
|
||||
ctrl_attr = "control_number"
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
# Every ack row of the given kind, with a LEFT JOIN against
|
||||
# any claim_acks link; orphan when NO link was created.
|
||||
if kind == "999":
|
||||
# For 999, the "ack has no link" means no ClaimAck row
|
||||
# was emitted at all (the auto-linker emits one per AK2
|
||||
# even when the AK2 is rejected, so 999 with at least
|
||||
# one AK2 that resolved to a claim is never an orphan).
|
||||
# We treat a 999 as orphan when it has zero ClaimAck
|
||||
# rows tied to its id.
|
||||
all_acks = s.query(Ack).order_by(Ack.id.desc()).all()
|
||||
for ack_row in all_acks:
|
||||
count = (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.ack_kind == "999",
|
||||
ClaimAck.ack_id == ack_row.id)
|
||||
.count()
|
||||
)
|
||||
if count == 0:
|
||||
out.append({
|
||||
"kind": "999",
|
||||
"ack_id": ack_row.id,
|
||||
"control_number": _ack_control_number(ack_row, "999"),
|
||||
"parsed_at": (
|
||||
ack_row.parsed_at.isoformat().replace(
|
||||
"+00:00", "Z"
|
||||
) if ack_row.parsed_at else None
|
||||
),
|
||||
})
|
||||
elif kind == "277ca":
|
||||
all_acks = s.query(Two77caAck).order_by(Two77caAck.id.desc()).all()
|
||||
for ack_row in all_acks:
|
||||
count = (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.ack_kind == "277ca",
|
||||
ClaimAck.ack_id == ack_row.id)
|
||||
.count()
|
||||
)
|
||||
if count == 0:
|
||||
out.append({
|
||||
"kind": "277ca",
|
||||
"ack_id": ack_row.id,
|
||||
"control_number": ack_row.control_number or "",
|
||||
"parsed_at": (
|
||||
ack_row.parsed_at.isoformat().replace(
|
||||
"+00:00", "Z"
|
||||
) if ack_row.parsed_at else None
|
||||
),
|
||||
})
|
||||
else:
|
||||
all_acks = s.query(Ta1Ack).order_by(Ta1Ack.id.desc()).all()
|
||||
for ack_row in all_acks:
|
||||
count = (
|
||||
s.query(ClaimAck)
|
||||
.filter(ClaimAck.ack_kind == "ta1",
|
||||
ClaimAck.ack_id == ack_row.id)
|
||||
.count()
|
||||
)
|
||||
if count == 0:
|
||||
out.append({
|
||||
"kind": "ta1",
|
||||
"ack_id": ack_row.id,
|
||||
"control_number": ack_row.control_number or "",
|
||||
"parsed_at": (
|
||||
ack_row.parsed_at.isoformat().replace(
|
||||
"+00:00", "Z"
|
||||
) if ack_row.parsed_at else None
|
||||
),
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def _ack_control_number(ack_row: Ack, kind: str) -> str:
|
||||
"""Best-effort control-number lookup for a 999 ack row.
|
||||
|
||||
The 999 ORM row doesn't carry the envelope's control_number in
|
||||
a dedicated column; we re-derive it from ``raw_json`` (the same
|
||||
source :func:`cyclone.store.ui.to_ui_ack` uses for the patient
|
||||
control number).
|
||||
"""
|
||||
raw = ack_row.raw_json or {}
|
||||
env = raw.get("envelope") or {}
|
||||
return env.get("control_number") or ""
|
||||
|
||||
|
||||
__all__ = [
|
||||
"add_claim_ack",
|
||||
"batch_envelope_index",
|
||||
"find_ack_orphans",
|
||||
"list_acks_for_claim",
|
||||
"list_claims_for_ack",
|
||||
"remove_claim_ack",
|
||||
]
|
||||
@@ -1,710 +0,0 @@
|
||||
"""Claim- and remittance-detail read APIs.
|
||||
|
||||
Includes the only large non-write query in the store:
|
||||
``get_claim_detail`` which joins Claim + CasAdjustment + ActivityEvent
|
||||
history for the right-drawer UI.
|
||||
|
||||
Also hosts ``check_matched_pair_drift()`` — the SP27 startup invariant
|
||||
audit that returns the count of Claim↔Remit pairs where
|
||||
``matched_remittance_id`` doesn't agree with the FK in ``remittances.claim_id``.
|
||||
It lives here (not in its own module) because it reads the same pair of
|
||||
tables as ``get_claim_detail``'s matched-remittance summary.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import timezone
|
||||
from decimal import Decimal
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import (
|
||||
ActivityEvent,
|
||||
CasAdjustment,
|
||||
Claim,
|
||||
ClaimState,
|
||||
Remittance,
|
||||
)
|
||||
from .ui import (
|
||||
CLAIM_DETAIL_HISTORY_LIMIT,
|
||||
_date_in_bounds,
|
||||
_iso_z,
|
||||
_svc_to_wire_dict,
|
||||
to_ui_claim_detail,
|
||||
to_ui_claim_from_orm,
|
||||
to_ui_provider,
|
||||
to_ui_remittance_from_orm,
|
||||
to_ui_remittance_with_adjustments,
|
||||
)
|
||||
|
||||
|
||||
def get_remittance(remittance_id: str) -> dict | None:
|
||||
"""Return a UI-shaped remittance dict with ``adjustments`` array.
|
||||
|
||||
Joins the persisted ``CasAdjustment`` rows for ``remittance_id``
|
||||
and labels each via :mod:`cyclone.parsers.cas_codes`. Returns
|
||||
``None`` when the remittance is not found so the API layer can
|
||||
map that to a 404.
|
||||
|
||||
SP7: also returns the per-line SVC composites
|
||||
(``serviceLinePayments``) and the CLP-level (claim-level) CAS
|
||||
bucket (``claimLevelAdjustments``) so the remit drawer can show
|
||||
per-line payments + adjustments without a second fetch.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
row = s.get(Remittance, remittance_id)
|
||||
if row is None:
|
||||
return None
|
||||
cas_rows = (
|
||||
s.query(CasAdjustment)
|
||||
.filter(CasAdjustment.remittance_id == remittance_id)
|
||||
.all()
|
||||
)
|
||||
parsed_at = (
|
||||
row.batch.parsed_at if row.batch is not None else row.received_at
|
||||
)
|
||||
if parsed_at is not None and parsed_at.tzinfo is None:
|
||||
parsed_at = parsed_at.replace(tzinfo=timezone.utc)
|
||||
body = to_ui_remittance_with_adjustments(
|
||||
row,
|
||||
batch_id=row.batch_id,
|
||||
parsed_at=parsed_at,
|
||||
cas_rows=cas_rows,
|
||||
)
|
||||
# SP7: per-line SVC composites + claim-level CAS bucket.
|
||||
from cyclone.db import ServiceLinePayment as SLP
|
||||
slps = (
|
||||
s.query(SLP)
|
||||
.filter(SLP.remittance_id == remittance_id)
|
||||
.order_by(SLP.line_number)
|
||||
.all()
|
||||
)
|
||||
body["serviceLinePayments"] = [
|
||||
_svc_to_wire_dict(svc) for svc in slps
|
||||
]
|
||||
body["claimLevelAdjustments"] = [
|
||||
{
|
||||
"id": c.id,
|
||||
"group_code": c.group_code,
|
||||
"reason_code": c.reason_code,
|
||||
"amount": str(Decimal(str(c.amount))),
|
||||
"quantity": (
|
||||
str(Decimal(str(c.quantity)))
|
||||
if c.quantity is not None
|
||||
else None
|
||||
),
|
||||
}
|
||||
for c in cas_rows
|
||||
if c.service_line_payment_id is None
|
||||
]
|
||||
return body
|
||||
|
||||
|
||||
def get_claim_detail(claim_id: str) -> dict | None:
|
||||
"""Return the SP4 detail-drawer shape for one claim, or ``None``.
|
||||
|
||||
Drives ``GET /api/claims/{claim_id}``. Returns the spec-shaped
|
||||
dict from :func:`to_ui_claim_detail` (header + state + parties +
|
||||
validation + service lines + diagnoses + raw segments) stitched
|
||||
with the claim's recent activity history and, if paired, a
|
||||
matched-remittance summary.
|
||||
|
||||
Returns ``None`` when ``claim_id`` is not in the DB so the API
|
||||
layer can map that to a 404 — the URL-driven drawer
|
||||
distinguishes "claim doesn't exist" from "fetch failed" (the
|
||||
spec §3.4 calls for a distinct 404 state in the drawer).
|
||||
|
||||
The history is capped at :data:`CLAIM_DETAIL_HISTORY_LIMIT`
|
||||
(50, per the spec) and ordered ``ts DESC`` so the most recent
|
||||
event is first. The status string in ``matchedRemittance``
|
||||
follows the same ``reconciled``/``received`` mapping used by
|
||||
:func:`to_ui_remittance_from_orm`.
|
||||
"""
|
||||
# Lazy import — same pattern used throughout this module to
|
||||
# avoid a circular store ↔ db import on cold start.
|
||||
from cyclone import db as _db
|
||||
|
||||
with _db.SessionLocal()() as s:
|
||||
row = s.get(Claim, claim_id)
|
||||
if row is None:
|
||||
return None
|
||||
|
||||
history_rows = (
|
||||
s.query(ActivityEvent)
|
||||
.filter(ActivityEvent.claim_id == claim_id)
|
||||
.order_by(ActivityEvent.ts.desc())
|
||||
.limit(CLAIM_DETAIL_HISTORY_LIMIT)
|
||||
.all()
|
||||
)
|
||||
|
||||
# Claim.batch_id is FK NOT NULL with ON DELETE CASCADE, so
|
||||
# ``row.batch`` is always populated in normal flow. Re-attach
|
||||
# UTC only when SQLite drops the tzinfo on read.
|
||||
parsed_at = row.batch.parsed_at
|
||||
if parsed_at.tzinfo is None:
|
||||
parsed_at = parsed_at.replace(tzinfo=timezone.utc)
|
||||
|
||||
detail = to_ui_claim_detail(
|
||||
row,
|
||||
batch_id=row.batch_id,
|
||||
parsed_at=parsed_at,
|
||||
)
|
||||
|
||||
detail["stateHistory"] = [
|
||||
{
|
||||
"kind": ev.kind,
|
||||
# SQLite drops tzinfo on read; rows are stored UTC
|
||||
# at write time (see ``add`` / ``manual_match``),
|
||||
# so re-attach UTC if needed to keep the spec
|
||||
# contract that ``ts`` ends in Z.
|
||||
"ts": _iso_z(ev.ts),
|
||||
"batchId": ev.batch_id,
|
||||
"remittanceId": ev.remittance_id,
|
||||
}
|
||||
for ev in history_rows
|
||||
]
|
||||
|
||||
if row.matched_remittance_id is not None:
|
||||
remit = s.get(Remittance, row.matched_remittance_id)
|
||||
if remit is not None:
|
||||
status = (
|
||||
"reconciled"
|
||||
if remit.status_code in ("21", "22")
|
||||
else "received"
|
||||
)
|
||||
detail["matchedRemittance"] = {
|
||||
"id": remit.id,
|
||||
"totalPaid": float(remit.total_paid or 0),
|
||||
"status": status,
|
||||
"receivedAt": _iso_z(remit.received_at),
|
||||
}
|
||||
# If the remittance was deleted out from under the FK
|
||||
# (the FK is ``ON DELETE SET NULL`` so the column is
|
||||
# already cleared in normal flow), the matched_remittance_id
|
||||
# would be None here and we wouldn't enter this branch.
|
||||
# If the FK is non-null but the row is gone (e.g. tests
|
||||
# that bypass the cascade), fall through with the
|
||||
# default ``None`` — the UI shows "no match" rather
|
||||
# than crashing.
|
||||
|
||||
# SP7 §5.2: slim per-line projection so the ServiceLinesTable
|
||||
# can show Paid + Adjustments columns without a second fetch.
|
||||
# The 837 side is keyed by ``claim_service_line_number`` (the
|
||||
# 1-based line number from raw_json) since 837 service lines
|
||||
# are not a separate ORM table.
|
||||
from cyclone.db import (
|
||||
LineReconciliation, ServiceLinePayment, CasAdjustment,
|
||||
)
|
||||
slim_lrs = list(
|
||||
s.query(LineReconciliation)
|
||||
.filter(LineReconciliation.claim_id == claim_id)
|
||||
.all()
|
||||
)
|
||||
svc_ids_for_cas = [
|
||||
lr.service_line_payment_id
|
||||
for lr in slim_lrs
|
||||
if lr.service_line_payment_id is not None
|
||||
]
|
||||
cas_sums_by_svc: dict = {}
|
||||
svc_by_id_slim: dict = {}
|
||||
if svc_ids_for_cas:
|
||||
cas_rows = (
|
||||
s.query(CasAdjustment.service_line_payment_id, CasAdjustment.amount)
|
||||
.filter(CasAdjustment.service_line_payment_id.in_(svc_ids_for_cas))
|
||||
.all()
|
||||
)
|
||||
from collections import defaultdict
|
||||
agg = defaultdict(lambda: Decimal("0"))
|
||||
for svc_id, amount in cas_rows:
|
||||
agg[svc_id] += Decimal(str(amount))
|
||||
cas_sums_by_svc = {k: str(v) for k, v in agg.items()}
|
||||
for svc in (
|
||||
s.query(ServiceLinePayment)
|
||||
.filter(ServiceLinePayment.id.in_(svc_ids_for_cas))
|
||||
.all()
|
||||
):
|
||||
svc_by_id_slim[svc.id] = svc
|
||||
|
||||
slim_by_num: dict = {
|
||||
lr.claim_service_line_number: lr
|
||||
for lr in slim_lrs
|
||||
if lr.claim_service_line_number is not None
|
||||
}
|
||||
line_reconciliation_slim: list = []
|
||||
for sl in detail["serviceLines"]:
|
||||
ln = sl.get("lineNumber")
|
||||
lr = slim_by_num.get(ln)
|
||||
if lr is None:
|
||||
line_reconciliation_slim.append({
|
||||
"lineNumber": ln,
|
||||
"status": "unmatched_837_only",
|
||||
"paid": None,
|
||||
"adjustmentsSum": None,
|
||||
})
|
||||
continue
|
||||
svc = (
|
||||
svc_by_id_slim.get(lr.service_line_payment_id)
|
||||
if lr.service_line_payment_id
|
||||
else None
|
||||
)
|
||||
line_reconciliation_slim.append({
|
||||
"lineNumber": ln,
|
||||
"status": lr.status,
|
||||
"paid": str(Decimal(str(svc.payment))) if svc else None,
|
||||
"adjustmentsSum": (
|
||||
cas_sums_by_svc.get(lr.service_line_payment_id)
|
||||
if lr.service_line_payment_id
|
||||
else None
|
||||
),
|
||||
})
|
||||
detail["lineReconciliation"] = line_reconciliation_slim
|
||||
|
||||
return detail
|
||||
|
||||
|
||||
def iter_claims(
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
status: str | None = None,
|
||||
provider_npi: str | None = None,
|
||||
payer: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
sort: str | None = None,
|
||||
order: str = "desc",
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
) -> list[dict]:
|
||||
"""Return UI-shaped claim dicts from the DB.
|
||||
|
||||
Filters mirror the in-memory version. The ``payer`` filter is
|
||||
a case-insensitive substring on the payer's ``name``, recovered
|
||||
from each claim's ``raw_json`` payload (the DB stores it there
|
||||
because ``Claim`` itself only carries ``payer_id``).
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
q = s.query(Claim)
|
||||
if batch_id is not None:
|
||||
q = q.filter(Claim.batch_id == batch_id)
|
||||
if status is not None:
|
||||
q = q.filter(Claim.state == ClaimState(status))
|
||||
if provider_npi is not None:
|
||||
q = q.filter(Claim.provider_npi == provider_npi)
|
||||
|
||||
rows = q.all()
|
||||
# Bulk-load matched-remittance totals so the UI's "Received"
|
||||
# KPI + per-claim received_amount reflect real paid amounts
|
||||
# rather than always-0. One SQL roundtrip for the whole page
|
||||
# rather than per-claim lookups.
|
||||
matched_ids = [
|
||||
r.matched_remittance_id
|
||||
for r in rows
|
||||
if r.matched_remittance_id
|
||||
]
|
||||
received_by_remit: dict[str, float] = {}
|
||||
if matched_ids:
|
||||
for rid, total_paid in (
|
||||
s.query(Remittance.id, Remittance.total_paid)
|
||||
.filter(Remittance.id.in_(matched_ids))
|
||||
.all()
|
||||
):
|
||||
received_by_remit[rid] = float(total_paid or 0)
|
||||
|
||||
out: list[dict] = []
|
||||
for r in rows:
|
||||
raw = r.raw_json or {}
|
||||
bp = raw.get("billing_provider", {})
|
||||
payer_obj = raw.get("payer", {})
|
||||
sub = raw.get("subscriber", {})
|
||||
claim_hdr = raw.get("claim", {})
|
||||
service_lines = raw.get("service_lines", [])
|
||||
parsed_at_iso = (
|
||||
r.batch.parsed_at.isoformat().replace("+00:00", "Z")
|
||||
if r.batch is not None
|
||||
else ""
|
||||
)
|
||||
cpt = (
|
||||
service_lines[0].get("procedure", {}).get("code", "")
|
||||
if service_lines
|
||||
else ""
|
||||
)
|
||||
out.append({
|
||||
"id": r.id,
|
||||
"patientName": (
|
||||
f"{sub.get('first_name', '')} "
|
||||
f"{sub.get('last_name', '')}".strip()
|
||||
),
|
||||
"providerNpi": bp.get("npi") or r.provider_npi or "",
|
||||
"payerName": payer_obj.get("name") or "",
|
||||
"cptCode": cpt,
|
||||
"billedAmount": float(r.charge_amount or 0),
|
||||
"receivedAmount": received_by_remit.get(
|
||||
r.matched_remittance_id, 0.0
|
||||
),
|
||||
"status": r.state.value if hasattr(r.state, "value") else str(r.state),
|
||||
"state": r.state.value if hasattr(r.state, "value") else str(r.state),
|
||||
"denialReason": None,
|
||||
"submissionDate": parsed_at_iso,
|
||||
"batchId": r.batch_id,
|
||||
"parsedAt": parsed_at_iso,
|
||||
# Keep these so we can sort on them in-memory below.
|
||||
"_sort_billedAmount": float(r.charge_amount or 0),
|
||||
"_sort_submissionDate": parsed_at_iso,
|
||||
})
|
||||
|
||||
if payer is not None:
|
||||
needle = payer.casefold()
|
||||
out = [
|
||||
c for c in out
|
||||
if needle in (c.get("payerName") or "").casefold()
|
||||
]
|
||||
out = [
|
||||
c for c in out
|
||||
if _date_in_bounds(c, "submissionDate", date_from, date_to)
|
||||
]
|
||||
if sort is not None:
|
||||
out.sort(
|
||||
key=lambda c: c.get(f"_sort_{sort}", 0) or 0,
|
||||
reverse=(order == "desc"),
|
||||
)
|
||||
# Drop the private sort keys before returning.
|
||||
for c in out:
|
||||
c.pop("_sort_billedAmount", None)
|
||||
c.pop("_sort_submissionDate", None)
|
||||
return out[offset:offset + limit]
|
||||
|
||||
|
||||
def iter_remittances(
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
payer: str | None = None,
|
||||
claim_id: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
sort: str | None = None,
|
||||
order: str = "desc",
|
||||
limit: int = 100,
|
||||
offset: int = 0,
|
||||
) -> list[dict]:
|
||||
"""Return UI-shaped remittance dicts from the DB."""
|
||||
with db.SessionLocal()() as s:
|
||||
q = s.query(Remittance)
|
||||
if batch_id is not None:
|
||||
q = q.filter(Remittance.batch_id == batch_id)
|
||||
if claim_id is not None:
|
||||
q = q.filter(Remittance.claim_id == claim_id)
|
||||
|
||||
rows = q.all()
|
||||
# Bulk-fetch all CAS rows for these remittances in one query
|
||||
# (SP3 P2 follow-up — fixes the list-view's empty adjustments
|
||||
# expansion). N+1-free.
|
||||
cas_by_remit: dict[str, list] = {}
|
||||
if rows:
|
||||
from cyclone.parsers.cas_codes import reason_label
|
||||
cas_rows = (
|
||||
s.query(CasAdjustment)
|
||||
.filter(CasAdjustment.remittance_id.in_([r.id for r in rows]))
|
||||
.all()
|
||||
)
|
||||
for c in cas_rows:
|
||||
cas_by_remit.setdefault(c.remittance_id, []).append(c)
|
||||
|
||||
out: list[dict] = []
|
||||
for r in rows:
|
||||
raw = r.raw_json or {}
|
||||
parsed_at_iso = (
|
||||
r.batch.parsed_at.isoformat().replace("+00:00", "Z")
|
||||
if r.batch is not None
|
||||
else r.received_at.isoformat().replace("+00:00", "Z")
|
||||
)
|
||||
payer_name = ""
|
||||
if r.batch is not None and r.batch.raw_result_json:
|
||||
payer_name = (
|
||||
r.batch.raw_result_json.get("payer", {}).get("name", "")
|
||||
)
|
||||
adjustments = [
|
||||
{
|
||||
"group": c.group_code,
|
||||
"reason": c.reason_code,
|
||||
"label": reason_label(c.group_code, c.reason_code),
|
||||
"amount": float(c.amount),
|
||||
"quantity": float(c.quantity) if c.quantity is not None else None,
|
||||
}
|
||||
for c in cas_by_remit.get(r.id, [])
|
||||
]
|
||||
out.append({
|
||||
"id": r.id,
|
||||
"claimId": r.claim_id or "",
|
||||
"payerName": payer_name,
|
||||
"paidAmount": float(r.total_paid or 0),
|
||||
"adjustmentAmount": float(r.adjustment_amount or 0),
|
||||
"status": (
|
||||
"reconciled" if r.status_code in ("21", "22")
|
||||
else "received"
|
||||
),
|
||||
"denialReason": None,
|
||||
"validationWarnings": [],
|
||||
"receivedDate": r.received_at.isoformat().replace("+00:00", "Z"),
|
||||
"batchId": r.batch_id,
|
||||
"parsedAt": parsed_at_iso,
|
||||
"adjustments": adjustments,
|
||||
"_sort_receivedDate": r.received_at.isoformat().replace("+00:00", "Z"),
|
||||
})
|
||||
|
||||
if payer is not None:
|
||||
out = [r for r in out if r.get("payerName") == payer]
|
||||
out = [
|
||||
r for r in out
|
||||
if _date_in_bounds(r, "receivedDate", date_from, date_to)
|
||||
]
|
||||
if sort is not None:
|
||||
out.sort(
|
||||
key=lambda r: r.get(f"_sort_{sort}", 0) or 0,
|
||||
reverse=(order == "desc"),
|
||||
)
|
||||
for r in out:
|
||||
r.pop("_sort_receivedDate", None)
|
||||
return out[offset:offset + limit]
|
||||
|
||||
|
||||
def distinct_providers() -> list[dict]:
|
||||
"""Group claims by NPI and return one row per provider."""
|
||||
with db.SessionLocal()() as s:
|
||||
rows = s.query(Claim).all()
|
||||
by_npi: dict[str, dict] = {}
|
||||
for r in rows:
|
||||
npi = r.provider_npi or ""
|
||||
if npi not in by_npi:
|
||||
raw = r.raw_json or {}
|
||||
bp = raw.get("billing_provider", {})
|
||||
by_npi[npi] = to_ui_provider(
|
||||
npi=npi,
|
||||
name=bp.get("name") or "",
|
||||
tax_id=bp.get("tax_id"),
|
||||
address=None,
|
||||
city=None,
|
||||
state=None,
|
||||
zip=None,
|
||||
phone=None,
|
||||
claim_count=0,
|
||||
outstanding_ar=0.0,
|
||||
)
|
||||
by_npi[npi]["claimCount"] += 1
|
||||
return list(by_npi.values())
|
||||
|
||||
|
||||
def recent_activity(*, limit: int = 200) -> list[dict]:
|
||||
"""Return recent activity events from the DB, newest first.
|
||||
|
||||
SP21 Task 2.5: each row also carries ``claimId`` and
|
||||
``remittanceId`` (read from the ORM columns) so the Dashboard's
|
||||
Recent-activity card can route clicks to the right entity
|
||||
drawer via ``src/lib/event-routing.ts``. Both are nullable
|
||||
strings; the wire shape uses camelCase keys to match the
|
||||
existing ``npi`` / ``amount`` fields and the frontend
|
||||
``Activity`` interface.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
rows = (
|
||||
s.query(ActivityEvent)
|
||||
.order_by(ActivityEvent.ts.desc())
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
return [
|
||||
{
|
||||
"id": f"ae-{r.id}",
|
||||
"kind": r.kind,
|
||||
"message": (r.payload_json or {}).get("message", ""),
|
||||
"timestamp": r.ts.isoformat().replace("+00:00", "Z"),
|
||||
"npi": (r.payload_json or {}).get("npi"),
|
||||
"amount": (r.payload_json or {}).get("amount"),
|
||||
"claimId": r.claim_id,
|
||||
"remittanceId": r.remittance_id,
|
||||
}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
|
||||
def check_matched_pair_drift() -> int:
|
||||
"""Audit the ``Claim.matched_remittance_id`` ↔ ``Remittance.claim_id``
|
||||
FK pair at startup. Non-blocking (SP27 Task 11).
|
||||
|
||||
The matched pair is a denormalized FK pair maintained transactionally
|
||||
by ``manual_match``, ``manual_unmatch``, and ``reconcile.run``. A
|
||||
pre-existing mismatch (e.g. a row written before a state migration
|
||||
that added one column but not the other) would otherwise stay
|
||||
invisible until the next operator pair attempt fails confusingly.
|
||||
This check logs the count + up to N examples so operators can
|
||||
investigate without booting the system.
|
||||
|
||||
Returns the number of drifted rows (0 means clean). Does not
|
||||
raise; bootstrap continues even if drift is detected.
|
||||
|
||||
Count semantics: this returns *drifted rows*, not *drifted pairs*.
|
||||
A single broken pair (``Claim.matched_remittance_id = X`` AND
|
||||
``Remittance.claim_id = Y != nil`` with neither pointing back)
|
||||
can produce TWO drifted rows — one in case A (the claim) and
|
||||
one in case B (the remit). In practice drift is almost always
|
||||
asymmetric (one side NULL), so count == count(pairs); for the
|
||||
fully-symmetric minority, divide by ~2 when alerting. Real drift
|
||||
should be fixed by repairing the writer path, not by counting.
|
||||
|
||||
Cases:
|
||||
A. Claim ``matched_remittance_id = X`` but the paired remit X's
|
||||
``claim_id`` is either NULL or doesn't point back to the claim.
|
||||
B. Remit ``claim_id = A`` but the paired claim A's
|
||||
``matched_remittance_id`` is either NULL or doesn't point back.
|
||||
"""
|
||||
import logging
|
||||
from sqlalchemy import select
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
# Case A: claim says X is paired, but X.claim_id doesn't point back.
|
||||
case_a = list(
|
||||
s.execute(
|
||||
select(
|
||||
Claim.id.label("claim_id"),
|
||||
Claim.matched_remittance_id.label("claimed_remit_id"),
|
||||
Remittance.claim_id.label("remit_points_to"),
|
||||
)
|
||||
.outerjoin(
|
||||
Remittance,
|
||||
Remittance.id == Claim.matched_remittance_id,
|
||||
)
|
||||
.where(Claim.matched_remittance_id.is_not(None))
|
||||
.where(
|
||||
(Remittance.claim_id.is_(None))
|
||||
| (Remittance.claim_id != Claim.id)
|
||||
)
|
||||
).all()
|
||||
)
|
||||
# Case B: remit says A is paired, but A.matched_remittance_id
|
||||
# doesn't point back.
|
||||
case_b = list(
|
||||
s.execute(
|
||||
select(
|
||||
Remittance.id.label("remit_id"),
|
||||
Remittance.claim_id.label("claimed_claim_id"),
|
||||
Claim.matched_remittance_id.label("claim_points_to"),
|
||||
)
|
||||
.outerjoin(
|
||||
Claim,
|
||||
Claim.id == Remittance.claim_id,
|
||||
)
|
||||
.where(Remittance.claim_id.is_not(None))
|
||||
.where(
|
||||
(Claim.matched_remittance_id.is_(None))
|
||||
| (Claim.matched_remittance_id != Remittance.id)
|
||||
)
|
||||
).all()
|
||||
)
|
||||
|
||||
total = len(case_a) + len(case_b)
|
||||
if total == 0:
|
||||
log.info("matched-pair drift check: 0 mismatches (clean)")
|
||||
return 0
|
||||
|
||||
log.warning(
|
||||
"matched-pair drift check: %d mismatched pair(s) (showing up to 5 "
|
||||
"of each). Investigate via SELECT against claim / remittance; "
|
||||
"manual re-pair via /api/claims/{id}/manual-match will repair.",
|
||||
total,
|
||||
)
|
||||
for r in case_a[:5]:
|
||||
log.warning(
|
||||
" case A: claim %s -> remit %s, but remit.claim_id=%r",
|
||||
r.claim_id,
|
||||
r.claimed_remit_id,
|
||||
r.remit_points_to,
|
||||
)
|
||||
for r in case_b[:5]:
|
||||
log.warning(
|
||||
" case B: remit %s -> claim %s, but claim.matched_remittance_id=%r",
|
||||
r.remit_id,
|
||||
r.claimed_claim_id,
|
||||
r.claim_points_to,
|
||||
)
|
||||
return total
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Aggregate counters (SP25 / SP27): full-population counts and sums that
|
||||
# the /api/* endpoints expose so page-local reductions (25 rows + live-tail
|
||||
# delta) can never silently understate the true population.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_ITER_UNBOUNDED = 2**31 - 1
|
||||
|
||||
|
||||
def count_claims(
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
status: str | None = None,
|
||||
provider_npi: str | None = None,
|
||||
payer: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
) -> int:
|
||||
"""Count claims that would be returned by ``iter_claims``.
|
||||
|
||||
Same filter parameters as ``iter_claims`` (excluding
|
||||
``sort``/``order``/``limit``/``offset``, which don't affect cardinality).
|
||||
"""
|
||||
rows = iter_claims(
|
||||
batch_id=batch_id, status=status, provider_npi=provider_npi,
|
||||
payer=payer, date_from=date_from, date_to=date_to,
|
||||
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def count_remittances(
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
payer: str | None = None,
|
||||
claim_id: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
) -> int:
|
||||
"""Count remittances that would be returned by ``iter_remittances``.
|
||||
|
||||
Same filter parameters as ``iter_remittances`` (excluding
|
||||
``sort``/``order``/``limit``/``offset``).
|
||||
"""
|
||||
rows = iter_remittances(
|
||||
batch_id=batch_id, payer=payer, claim_id=claim_id,
|
||||
date_from=date_from, date_to=date_to,
|
||||
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def summarize_remittances(
|
||||
*,
|
||||
batch_id: str | None = None,
|
||||
payer: str | None = None,
|
||||
claim_id: str | None = None,
|
||||
date_from: str | None = None,
|
||||
date_to: str | None = None,
|
||||
) -> dict:
|
||||
"""Return ``{count, total_paid, total_adjustments}`` summed over the
|
||||
remittance population that ``iter_remittances`` would return under
|
||||
the same filters. Backs ``GET /api/remittances/summary`` — the
|
||||
Remittances page's KPI tiles (added in SP27).
|
||||
"""
|
||||
rows = iter_remittances(
|
||||
batch_id=batch_id, payer=payer, claim_id=claim_id,
|
||||
date_from=date_from, date_to=date_to,
|
||||
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
|
||||
)
|
||||
total_paid = 0.0
|
||||
total_adjustments = 0.0
|
||||
for r in rows:
|
||||
total_paid += float(r.get("paidAmount") or 0)
|
||||
total_adjustments += float(r.get("adjustmentAmount") or 0)
|
||||
return {
|
||||
"count": len(rows),
|
||||
"total_paid": total_paid,
|
||||
"total_adjustments": total_adjustments,
|
||||
}
|
||||
@@ -1,41 +0,0 @@
|
||||
"""Exception types raised by CycloneStore and its domain modules.
|
||||
|
||||
These are part of the public API — callers (API endpoints) catch them
|
||||
and translate to HTTP 409 Conflict responses.
|
||||
"""
|
||||
|
||||
|
||||
class AlreadyMatchedError(Exception):
|
||||
"""Raised by ``CycloneStore.manual_match`` when the claim is already paired.
|
||||
|
||||
The claim's ``matched_remittance_id`` is set, so any new pairing would
|
||||
clobber an existing match. Callers (the T15 API endpoint) should surface
|
||||
this as a 409 Conflict.
|
||||
"""
|
||||
|
||||
|
||||
class NotMatchedError(Exception):
|
||||
"""Raised by ``CycloneStore.manual_unmatch`` when the claim has no match.
|
||||
|
||||
Mirrors ``AlreadyMatchedError`` for the unpair operation. Same HTTP
|
||||
treatment: 409 Conflict at the API layer.
|
||||
"""
|
||||
|
||||
|
||||
class InvalidStateError(Exception):
|
||||
"""Raised when an apply_* pure fn returns a skipped ApplyIntent.
|
||||
|
||||
``reconcile.apply_payment`` / ``apply_reversal`` may return
|
||||
``skipped=True`` (e.g. claim already in a terminal state, or reversal
|
||||
on a non-paid claim). The store surfaces that as ``InvalidStateError``
|
||||
rather than silently pairing. The T15 API endpoint maps this to a
|
||||
409 Conflict and echoes ``current_state`` and ``activity_kind`` so
|
||||
the UI can render a precise message.
|
||||
"""
|
||||
|
||||
def __init__(self, current_state: str, activity_kind: str = "invalid_state"):
|
||||
self.current_state = current_state
|
||||
self.activity_kind = activity_kind
|
||||
super().__init__(
|
||||
f"invalid state {current_state} for apply (kind={activity_kind})"
|
||||
)
|
||||
@@ -1,315 +0,0 @@
|
||||
"""Inbox lane state and manual match/unmatch operations.
|
||||
|
||||
``list_unmatched`` returns the 5-lane inbox view (unmatched claims +
|
||||
unmatched remits). ``manual_match`` and ``manual_unmatch`` invoke the
|
||||
reconcile pure functions and persist the resulting Match row (or
|
||||
remove it). Invalid state transitions surface as ``InvalidStateError``.
|
||||
"""
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import ActivityEvent, Claim, ClaimState, Match, Remittance
|
||||
|
||||
from .exceptions import AlreadyMatchedError, InvalidStateError, NotMatchedError
|
||||
from . import utcnow
|
||||
from .ui import to_ui_claim_from_orm, to_ui_remittance_from_orm
|
||||
|
||||
|
||||
# -- manual reconciliation (T12) -----------------------------------
|
||||
|
||||
def list_unmatched(*, kind: str = "both") -> dict:
|
||||
"""Return unmatched claims and/or remittances.
|
||||
|
||||
An unmatched claim is one with ``matched_remittance_id IS NULL`` —
|
||||
either auto-match never paired it, or it was unpaired by
|
||||
``manual_unmatch``. An unmatched remittance is one with
|
||||
``claim_id IS NULL`` — symmetric FK on the remittance side; we
|
||||
update this in ``manual_match`` so the filter reflects the pair.
|
||||
|
||||
Note: T10's ``reconcile.run`` only writes the claim-side FK
|
||||
(``Claim.matched_remittance_id``) when auto-pairing. Auto-matched
|
||||
remittances therefore still show as "unmatched" here until the
|
||||
pair is touched (re-ingest, manual unmatch + rematch). That gap
|
||||
is intentional for T12 — fixing it requires modifying T10.
|
||||
|
||||
``kind`` selects which side(s) to return:
|
||||
- "claims": only claims
|
||||
- "remittances": only remittances
|
||||
- "both": both (default)
|
||||
|
||||
Returns ``{"claims": [...], "remittances": [...]}`` with the
|
||||
unused side always an empty list (never absent) so callers can
|
||||
unconditionally index.
|
||||
"""
|
||||
if kind not in ("claims", "remittances", "both"):
|
||||
raise ValueError(
|
||||
f"list_unmatched: unknown kind={kind!r} "
|
||||
"(expected 'claims', 'remittances', or 'both')"
|
||||
)
|
||||
|
||||
result: dict = {"claims": [], "remittances": []}
|
||||
with db.SessionLocal()() as s:
|
||||
if kind in ("claims", "both"):
|
||||
rows = (
|
||||
s.query(Claim)
|
||||
.filter(Claim.matched_remittance_id.is_(None))
|
||||
.order_by(Claim.id.asc())
|
||||
.all()
|
||||
)
|
||||
for r in rows:
|
||||
parsed_at = (
|
||||
r.batch.parsed_at
|
||||
if r.batch is not None
|
||||
else r.service_date_from or utcnow()
|
||||
)
|
||||
result["claims"].append(
|
||||
to_ui_claim_from_orm(
|
||||
r, batch_id=r.batch_id, parsed_at=parsed_at,
|
||||
# list_unmatched filters matched_remittance_id IS NULL,
|
||||
# so every row has no remittance yet.
|
||||
received_total=0.0,
|
||||
)
|
||||
)
|
||||
|
||||
if kind in ("remittances", "both"):
|
||||
rows = (
|
||||
s.query(Remittance)
|
||||
.filter(Remittance.claim_id.is_(None))
|
||||
.order_by(Remittance.id.asc())
|
||||
.all()
|
||||
)
|
||||
for r in rows:
|
||||
parsed_at = (
|
||||
r.batch.parsed_at
|
||||
if r.batch is not None
|
||||
else r.received_at
|
||||
)
|
||||
result["remittances"].append(
|
||||
to_ui_remittance_from_orm(
|
||||
r, batch_id=r.batch_id, parsed_at=parsed_at,
|
||||
)
|
||||
)
|
||||
return result
|
||||
|
||||
def manual_match(claim_id: str, remit_id: str) -> dict:
|
||||
"""Pair a claim with a remittance manually (operator override).
|
||||
|
||||
Steps:
|
||||
1. Load the claim; raise ``AlreadyMatchedError`` if it already
|
||||
has a match (we never silently overwrite an existing pair).
|
||||
2. Load the remittance; raise ``LookupError`` if missing.
|
||||
3. Compute the new claim state via ``reconcile.apply_payment``
|
||||
(or ``apply_reversal`` for status codes 21/22).
|
||||
4. Insert a ``Match`` row with ``strategy="manual"``.
|
||||
5. Update the claim (``state``, ``matched_remittance_id``) AND
|
||||
the remittance (``claim_id``) so the symmetric FK reflects
|
||||
the pair — required for ``list_unmatched`` to drop them.
|
||||
6. Record an ``ActivityEvent(kind="manual_match", ...)``.
|
||||
7. Commit; return ``{"claim": <ui>, "match": <ui>}``.
|
||||
|
||||
``reconcile.apply_payment`` may return a noop (claim in terminal
|
||||
state); we surface that as ``InvalidStateError`` rather than
|
||||
silently pairing, because the operator clearly intended a state
|
||||
change. The T15 API endpoint maps this to a 409 Conflict.
|
||||
"""
|
||||
from cyclone import reconcile as _reconcile
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
claim = s.get(Claim, claim_id)
|
||||
if claim is None:
|
||||
raise LookupError(f"claim {claim_id} not found")
|
||||
if claim.matched_remittance_id is not None:
|
||||
raise AlreadyMatchedError(
|
||||
f"claim {claim_id} already matched to "
|
||||
f"{claim.matched_remittance_id}"
|
||||
)
|
||||
|
||||
remit = s.get(Remittance, remit_id)
|
||||
if remit is None:
|
||||
raise LookupError(f"remittance {remit_id} not found")
|
||||
|
||||
prior_state = claim.state
|
||||
if remit.is_reversal:
|
||||
intent = _reconcile.apply_reversal(claim, remit)
|
||||
else:
|
||||
intent = _reconcile.apply_payment(
|
||||
claim, remit,
|
||||
charge=claim.charge_amount,
|
||||
paid=remit.total_paid,
|
||||
status_code=remit.status_code,
|
||||
)
|
||||
|
||||
if intent.skipped or intent.new_state is None:
|
||||
current = (
|
||||
claim.state.value
|
||||
if hasattr(claim.state, "value")
|
||||
else str(claim.state)
|
||||
)
|
||||
raise InvalidStateError(
|
||||
current_state=current,
|
||||
activity_kind=intent.activity_kind,
|
||||
)
|
||||
|
||||
new_state = intent.new_state
|
||||
now = utcnow()
|
||||
|
||||
s.add(Match(
|
||||
claim_id=claim_id,
|
||||
remittance_id=remit_id,
|
||||
strategy="manual",
|
||||
matched_at=now,
|
||||
prior_claim_state=prior_state,
|
||||
is_reversal=remit.is_reversal,
|
||||
))
|
||||
claim.state = new_state
|
||||
claim.matched_remittance_id = remit_id
|
||||
# Symmetric FK update — see list_unmatched docstring for why
|
||||
# we don't rely on T10 to do this.
|
||||
remit.claim_id = claim_id
|
||||
|
||||
# SP7: line-level reconciliation + claim-level CAS aggregate.
|
||||
# Skipped for reversals — they don't have SV1↔SVC line pairs.
|
||||
if not remit.is_reversal:
|
||||
_reconcile._reconcile_pair(s, claim, remit)
|
||||
|
||||
s.add(ActivityEvent(
|
||||
ts=now,
|
||||
kind="manual_match",
|
||||
batch_id=remit.batch_id,
|
||||
claim_id=claim_id,
|
||||
remittance_id=remit_id,
|
||||
payload_json={
|
||||
"strategy": "manual",
|
||||
"new_state": new_state.value,
|
||||
"prior_state": prior_state.value,
|
||||
"is_reversal": remit.is_reversal,
|
||||
},
|
||||
))
|
||||
|
||||
s.commit()
|
||||
|
||||
parsed_at = (
|
||||
claim.batch.parsed_at
|
||||
if claim.batch is not None
|
||||
else now
|
||||
)
|
||||
claim_dict = to_ui_claim_from_orm(
|
||||
claim, batch_id=claim.batch_id, parsed_at=parsed_at,
|
||||
received_total=float(remit.total_paid or 0),
|
||||
)
|
||||
matched_at_iso = now.isoformat().replace("+00:00", "Z")
|
||||
return {
|
||||
"claim": claim_dict,
|
||||
"match": {
|
||||
"strategy": "manual",
|
||||
"claimId": claim_id,
|
||||
"remittanceId": remit_id,
|
||||
"matchedAt": matched_at_iso,
|
||||
"isReversal": remit.is_reversal,
|
||||
"priorState": prior_state.value,
|
||||
"newState": new_state.value,
|
||||
},
|
||||
}
|
||||
|
||||
def manual_unmatch(claim_id: str) -> dict:
|
||||
"""Unpair a previously matched claim and restore its prior state.
|
||||
|
||||
Reverses ``manual_match`` (and auto-match as a side-effect of
|
||||
clearing the FK). Strategy:
|
||||
|
||||
1. Load the claim; raise ``NotMatchedError`` if it isn't
|
||||
currently matched.
|
||||
2. Delete every ``Match`` row for the claim (there may be more
|
||||
than one — reversals create a 2nd row, see T10 spec).
|
||||
3. Restore ``claim.state`` from the latest Match's
|
||||
``prior_claim_state``; fall back to ``SUBMITTED`` when
|
||||
``prior_claim_state`` is NULL (auto-match doesn't set it for
|
||||
non-reversal payments — see reconcile.run line ~278).
|
||||
4. Clear ``claim.matched_remittance_id`` and the symmetric
|
||||
``remit.claim_id`` so ``list_unmatched`` surfaces the pair
|
||||
again.
|
||||
5. Record ``ActivityEvent(kind="manual_unmatch", ...)`` and
|
||||
commit.
|
||||
6. Return ``{"claim": <ui>, "deletedMatches": <count>}``.
|
||||
"""
|
||||
with db.SessionLocal()() as s:
|
||||
claim = s.get(Claim, claim_id)
|
||||
if claim is None:
|
||||
raise LookupError(f"claim {claim_id} not found")
|
||||
if claim.matched_remittance_id is None:
|
||||
raise NotMatchedError(
|
||||
f"claim {claim_id} has no active match"
|
||||
)
|
||||
|
||||
matches = (
|
||||
s.query(Match)
|
||||
.filter(Match.claim_id == claim_id)
|
||||
.order_by(Match.matched_at.desc())
|
||||
.all()
|
||||
)
|
||||
if not matches:
|
||||
# Defensive: matched_remittance_id was set but no Match
|
||||
# rows exist. Shouldn't happen, but if it does, fall back
|
||||
# to clearing the FK and starting fresh.
|
||||
latest = None
|
||||
paired_remit = None
|
||||
restored_state = ClaimState.SUBMITTED
|
||||
else:
|
||||
latest = matches[0]
|
||||
paired_remit = s.get(Remittance, latest.remittance_id)
|
||||
restored_state = (
|
||||
latest.prior_claim_state
|
||||
if latest.prior_claim_state is not None
|
||||
else ClaimState.SUBMITTED
|
||||
)
|
||||
|
||||
deleted_count = len(matches)
|
||||
for m in matches:
|
||||
s.delete(m)
|
||||
|
||||
claim.state = restored_state
|
||||
claim.matched_remittance_id = None
|
||||
# Clear the symmetric FK on the remittance so list_unmatched
|
||||
# surfaces the pair again. The remittance may have been
|
||||
# deleted between the match and this call — guard with a
|
||||
# None check so we don't blow up on a stale FK.
|
||||
if paired_remit is not None:
|
||||
paired_remit.claim_id = None
|
||||
|
||||
now = utcnow()
|
||||
s.add(ActivityEvent(
|
||||
ts=now,
|
||||
kind="manual_unmatch",
|
||||
claim_id=claim_id,
|
||||
payload_json={
|
||||
"restored_state": restored_state.value,
|
||||
"deleted_matches": deleted_count,
|
||||
},
|
||||
))
|
||||
|
||||
s.commit()
|
||||
|
||||
parsed_at = (
|
||||
claim.batch.parsed_at
|
||||
if claim.batch is not None
|
||||
else now
|
||||
)
|
||||
# ``paired_remit`` is the matched remittance we cleared in
|
||||
# the unmatch; use its ``total_paid`` for the response shape
|
||||
# so the UI sees what was paid before the unpair. May be
|
||||
# ``None`` if the remittance was deleted since the match —
|
||||
# default to 0.0 in that case.
|
||||
received_total = (
|
||||
float(paired_remit.total_paid or 0)
|
||||
if paired_remit is not None
|
||||
else 0.0
|
||||
)
|
||||
claim_dict = to_ui_claim_from_orm(
|
||||
claim, batch_id=claim.batch_id, parsed_at=parsed_at,
|
||||
received_total=received_total,
|
||||
)
|
||||
return {
|
||||
"claim": claim_dict,
|
||||
"deletedMatches": deleted_count,
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -1,307 +0,0 @@
|
||||
"""Dashboard KPI aggregation — cross-table reads consumed by /api/dashboard/kpis.
|
||||
|
||||
``dashboard_kpis()`` returns a single dict that the React Dashboard page
|
||||
consumes as the source of truth for the 4 KPI tiles + the recent-activity
|
||||
panel. It reads from claims + remittances + batches in a single pass.
|
||||
|
||||
``_claim_state_str`` is a small mapping helper that translates ORM
|
||||
status enum values to UI-friendly strings. It's private to this module.
|
||||
"""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import Claim, Remittance
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SP27 Task 13: Dashboard aggregate KPIs.
|
||||
#
|
||||
# The Dashboard's "Billed / Received / Denial rate / Pending AR" tiles are
|
||||
# computed from the *whole* claim population, not a sample. With 60k+ claims
|
||||
# in production, fetching ``/api/claims?limit=100`` and reducing in JS would
|
||||
# silently produce wrong numbers (denial rate sampled, billed summed from
|
||||
# 100 rows). This module-level function does the aggregation server-side in
|
||||
# a single session and returns a small structured payload the Dashboard
|
||||
# can render directly.
|
||||
#
|
||||
# Performance: one ``SELECT * FROM claims`` (no pagination) + one
|
||||
# ``SELECT id, total_paid FROM remittances WHERE id IN (...)`` for matched
|
||||
# remits. SQLite returns 60k claim rows in ~30ms on the development
|
||||
# machine; the Python reduce is microseconds. If the dataset grows past
|
||||
# ~500k claims we'd want a SQL-side ``GROUP BY month`` instead — but at
|
||||
# that volume the dashboard should probably be backed by a materialized
|
||||
# view, not a live query.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _claim_state_str(claim: Claim) -> str:
|
||||
"""Stringify a Claim's ``state`` regardless of enum vs raw str storage."""
|
||||
st = claim.state
|
||||
return st.value if hasattr(st, "value") else str(st)
|
||||
|
||||
|
||||
# Claim states counted toward the Dashboard's "pending" tile. SUBMITTED
|
||||
# is the initial post-parse state; REJECTED is the 999 envelope-level
|
||||
# rejection that means the payer never saw the claim. Both are
|
||||
# "outstanding adjudication" from the operator's POV; ``denied`` is
|
||||
# payer adjudication and lives in its own tile.
|
||||
_DASHBOARD_PENDING_STATES: frozenset[str] = frozenset({"submitted", "rejected"})
|
||||
|
||||
|
||||
def dashboard_kpis(
|
||||
*,
|
||||
months: int = 6,
|
||||
top_n_providers: int = 4,
|
||||
top_n_denials: int = 5,
|
||||
) -> dict:
|
||||
"""Compute Dashboard KPIs over the entire claim/remittance population.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
months
|
||||
Number of trailing calendar months to include in the ``monthly``
|
||||
sparkline series (default 6 — matches the existing frontend
|
||||
``MONTHS_BACK`` constant).
|
||||
top_n_providers
|
||||
How many providers to include in the ``topProviders`` array
|
||||
(default 4 — matches the existing Dashboard layout).
|
||||
top_n_denials
|
||||
How many most-recent denied claims to include in the
|
||||
``topDenials`` array (default 5).
|
||||
|
||||
Returns
|
||||
-------
|
||||
dict with keys:
|
||||
|
||||
- ``totals``: aggregate counts + dollar sums + rates for the whole DB.
|
||||
- ``monthly``: list of ``{month, label, count, billed, received,
|
||||
denied, denialRate, ar}`` dicts, oldest-first, length = ``months``.
|
||||
``ar`` is the running outstanding accounts-receivable (billed -
|
||||
received) carried forward across months, clamped at zero.
|
||||
- ``topProviders``: list of ``{npi, label, claimCount, billed,
|
||||
denied}`` dicts, sorted by claimCount desc.
|
||||
- ``topDenials``: list of ``{id, patientName, billedAmount,
|
||||
denialReason, submissionDate}`` dicts, sorted by submissionDate
|
||||
desc, capped at ``top_n_denials``.
|
||||
|
||||
Notes
|
||||
-----
|
||||
- Empty DB returns zero-filled aggregates, an empty ``monthly`` array
|
||||
(one entry per requested month), an empty ``topProviders`` array,
|
||||
and an empty ``topDenials`` array.
|
||||
- ``received`` for a claim is sourced from the matched Remittance's
|
||||
``total_paid`` column. Claims without a matched remittance
|
||||
contribute 0 to the received sum (and thus inflate the
|
||||
outstanding-AR figure, which is the correct operator-visible
|
||||
semantic — "money we haven't been told was paid yet").
|
||||
"""
|
||||
# Pre-build the trailing-N-months skeleton so the response always has
|
||||
# exactly ``months`` entries even when the DB is empty or sparse.
|
||||
now = datetime.now(timezone.utc)
|
||||
skeleton: list[dict] = []
|
||||
for i in range(months - 1, -1, -1):
|
||||
d = now.replace(day=1)
|
||||
# Walk backwards N months without ``relativedelta``.
|
||||
for _ in range(i):
|
||||
prev_month = d.month - 1
|
||||
if prev_month == 0:
|
||||
d = d.replace(year=d.year - 1, month=12)
|
||||
else:
|
||||
d = d.replace(month=prev_month)
|
||||
skeleton.append({
|
||||
"month": f"{d.year:04d}-{d.month:02d}",
|
||||
"label": d.strftime("%b"),
|
||||
"count": 0,
|
||||
"billed": 0.0,
|
||||
"received": 0.0,
|
||||
"denied": 0,
|
||||
"ar": 0.0,
|
||||
})
|
||||
skeleton_index = {entry["month"]: entry for entry in skeleton}
|
||||
|
||||
with db.SessionLocal()() as s:
|
||||
# ``Claim.batch`` is a lazy ``relationship`` (default
|
||||
# ``lazy="select"``); without ``selectinload`` each access to
|
||||
# ``r.batch`` in the reduce loop below issues a fresh
|
||||
# ``SELECT ... FROM batches WHERE id=?``. ``selectinload``
|
||||
# pulls every distinct batch in one round-trip instead of N+1
|
||||
# — critical for the 60k-claim dataset.
|
||||
from sqlalchemy.orm import selectinload
|
||||
claims: list[Claim] = (
|
||||
s.query(Claim)
|
||||
.options(selectinload(Claim.batch))
|
||||
.all()
|
||||
)
|
||||
|
||||
# Bulk-load matched-remit total_paid so a 60k-claim DB doesn't
|
||||
# produce a 60k-query N+1.
|
||||
matched_ids = [
|
||||
r.matched_remittance_id
|
||||
for r in claims
|
||||
if r.matched_remittance_id
|
||||
]
|
||||
received_by_remit: dict[str, float] = {}
|
||||
if matched_ids:
|
||||
for rid, total_paid in (
|
||||
s.query(Remittance.id, Remittance.total_paid)
|
||||
.filter(Remittance.id.in_(matched_ids))
|
||||
.all()
|
||||
):
|
||||
received_by_remit[rid] = float(total_paid or 0)
|
||||
|
||||
# Per-provider accumulator for the topProviders leaderboard.
|
||||
provider_counts: dict[str, int] = {}
|
||||
provider_billed: dict[str, float] = {}
|
||||
provider_denied: dict[str, int] = {}
|
||||
|
||||
# Per-month accumulator + totals in a single pass.
|
||||
total_count = 0
|
||||
total_billed = 0.0
|
||||
total_received = 0.0
|
||||
denied_count = 0
|
||||
pending_count = 0
|
||||
|
||||
# Collect denied candidates as we walk so we don't issue a
|
||||
# second pass for the topDenials array.
|
||||
denied_candidates: list[dict] = []
|
||||
|
||||
for r in claims:
|
||||
billed = float(r.charge_amount or 0)
|
||||
received = received_by_remit.get(r.matched_remittance_id, 0.0)
|
||||
state_str = _claim_state_str(r)
|
||||
|
||||
total_count += 1
|
||||
total_billed += billed
|
||||
total_received += received
|
||||
if state_str == "denied":
|
||||
denied_count += 1
|
||||
# Drop denied claims with no ``submissionDate`` — they'd
|
||||
# sort first under reverse-lex (empty string < ISO) and
|
||||
# render as "Invalid Date" in the Dashboard. A denial
|
||||
# without a batch is exceptional and operator-irrelevant
|
||||
# for the "recent denials" widget.
|
||||
if r.batch is None or r.batch.parsed_at is None:
|
||||
pass
|
||||
else:
|
||||
raw = r.raw_json or {}
|
||||
sub = raw.get("subscriber", {})
|
||||
denied_candidates.append({
|
||||
"id": r.id,
|
||||
"patientName": (
|
||||
f"{sub.get('first_name', '')} "
|
||||
f"{sub.get('last_name', '')}".strip()
|
||||
),
|
||||
"billedAmount": billed,
|
||||
"denialReason": r.rejection_reason,
|
||||
"submissionDate": (
|
||||
r.batch.parsed_at
|
||||
.isoformat()
|
||||
.replace("+00:00", "Z")
|
||||
),
|
||||
})
|
||||
if state_str in _DASHBOARD_PENDING_STATES:
|
||||
pending_count += 1
|
||||
|
||||
# Monthly bin. ``submissionDate`` lives on the parent Batch
|
||||
# (all claims in a batch share a parsed_at). Use UTC year-month
|
||||
# to match the skeleton.
|
||||
if r.batch is not None and r.batch.parsed_at is not None:
|
||||
pa = r.batch.parsed_at
|
||||
key = f"{pa.year:04d}-{pa.month:02d}"
|
||||
bucket = skeleton_index.get(key)
|
||||
if bucket is not None:
|
||||
bucket["count"] += 1
|
||||
bucket["billed"] += billed
|
||||
bucket["received"] += received
|
||||
if state_str == "denied":
|
||||
bucket["denied"] += 1
|
||||
|
||||
# Provider bin (keyed on NPI).
|
||||
npi = r.provider_npi or ""
|
||||
if npi:
|
||||
provider_counts[npi] = provider_counts.get(npi, 0) + 1
|
||||
provider_billed[npi] = provider_billed.get(npi, 0.0) + billed
|
||||
if state_str == "denied":
|
||||
provider_denied[npi] = provider_denied.get(npi, 0) + 1
|
||||
|
||||
# Compute denial rate per month + running AR.
|
||||
running_ar = 0.0
|
||||
for entry in skeleton:
|
||||
if entry["count"] > 0:
|
||||
entry["denialRate"] = (entry["denied"] / entry["count"]) * 100.0
|
||||
else:
|
||||
entry["denialRate"] = 0.0
|
||||
running_ar = max(0.0, running_ar + entry["billed"] - entry["received"])
|
||||
entry["ar"] = running_ar
|
||||
|
||||
# Resolve top provider labels from the Provider table in one
|
||||
# round-trip. NPIs without a Provider row still appear in the
|
||||
# leaderboard with an empty label so operators can see
|
||||
# "unknown-NPI" claims are coming from somewhere.
|
||||
# Local import keeps the module-level ``cyclone.providers.Provider``
|
||||
# Pydantic DTO and the SQLAlchemy ``cyclone.db.Provider`` ORM
|
||||
# separate (same pattern as ``list_providers`` / ``upsert_provider``).
|
||||
provider_labels: dict[str, str] = {}
|
||||
if provider_counts:
|
||||
from cyclone.db import Provider as ProviderORM
|
||||
for npi, label in (
|
||||
s.query(ProviderORM.npi, ProviderORM.label).filter(
|
||||
ProviderORM.npi.in_(provider_counts.keys())
|
||||
).all()
|
||||
):
|
||||
provider_labels[npi] = label or ""
|
||||
|
||||
top_providers = sorted(
|
||||
provider_counts.items(),
|
||||
key=lambda kv: kv[1],
|
||||
reverse=True,
|
||||
)[: max(0, top_n_providers)]
|
||||
top_providers_out = [
|
||||
{
|
||||
"npi": npi,
|
||||
"label": provider_labels.get(npi, ""),
|
||||
"claimCount": count,
|
||||
"billed": round(provider_billed.get(npi, 0.0), 2),
|
||||
"denied": provider_denied.get(npi, 0),
|
||||
}
|
||||
for npi, count in top_providers
|
||||
]
|
||||
|
||||
# Top denials = most recently submitted denied claims, capped at
|
||||
# ``top_n_denials``. submissionDate is ISO-8601 so lex sort ==
|
||||
# chronological sort when timestamps share a tz.
|
||||
denied_candidates.sort(key=lambda d: d["submissionDate"], reverse=True)
|
||||
top_denials = denied_candidates[: max(0, top_n_denials)]
|
||||
|
||||
total_denial_rate = (
|
||||
(denied_count / total_count) * 100.0 if total_count > 0 else 0.0
|
||||
)
|
||||
return {
|
||||
"totals": {
|
||||
"count": total_count,
|
||||
"billed": round(total_billed, 2),
|
||||
"received": round(total_received, 2),
|
||||
"outstandingAr": round(max(0.0, total_billed - total_received), 2),
|
||||
"denied": denied_count,
|
||||
"denialRate": round(total_denial_rate, 4),
|
||||
"pending": pending_count,
|
||||
},
|
||||
"monthly": [
|
||||
{
|
||||
"month": e["month"],
|
||||
"label": e["label"],
|
||||
"count": e["count"],
|
||||
"billed": round(e["billed"], 2),
|
||||
"received": round(e["received"], 2),
|
||||
"denied": e["denied"],
|
||||
"denialRate": round(e["denialRate"], 4),
|
||||
"ar": round(e["ar"], 2),
|
||||
}
|
||||
for e in skeleton
|
||||
],
|
||||
"topProviders": top_providers_out,
|
||||
"topDenials": top_denials,
|
||||
}
|
||||
|
||||
|
||||
@@ -1,180 +0,0 @@
|
||||
"""ORM row builders — convert parser output into SQLAlchemy ORM rows.
|
||||
|
||||
Each function returns an unsaved ORM object (or inserts related rows
|
||||
within an existing session). They never commit or close the session —
|
||||
the caller (``write.add_record``, ``acks.add_ack``, etc.) owns the
|
||||
transaction boundary.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from decimal import Decimal
|
||||
|
||||
from cyclone import db
|
||||
from cyclone.db import Claim, ClaimState, Remittance
|
||||
from cyclone.parsers.models import ClaimOutput
|
||||
from cyclone.parsers.models_835 import ClaimPayment
|
||||
|
||||
from . import utcnow
|
||||
|
||||
|
||||
def _service_dates_from_claim(claim: ClaimOutput) -> tuple[date | None, date | None]:
|
||||
"""Extract (service_date_from, service_date_to) from a ClaimOutput.
|
||||
|
||||
The 837P model has ``service_lines[*].service_date`` (one per SV1).
|
||||
We use the earliest as ``from`` and the latest as ``to``; if there
|
||||
are no service lines, both are ``None``.
|
||||
"""
|
||||
dates: list[date] = []
|
||||
for sl in claim.service_lines:
|
||||
if sl.service_date is not None:
|
||||
dates.append(sl.service_date)
|
||||
if not dates:
|
||||
return None, None
|
||||
return min(dates), max(dates)
|
||||
|
||||
|
||||
def _claim_837_row(claim: ClaimOutput, batch_id: str) -> Claim:
|
||||
"""Build a Claim ORM row from a ClaimOutput. NOT yet persisted."""
|
||||
d_from, d_to = _service_dates_from_claim(claim)
|
||||
return Claim(
|
||||
id=claim.claim_id,
|
||||
batch_id=batch_id,
|
||||
# SP27 Task 17: Claim.patient_control_number must hold the CLM01
|
||||
# claim_submittr's_identifier the 837 sent — that's the value the
|
||||
# 835 echoes in CLP01, which the reconcile matcher joins on
|
||||
# (reconcile.py:by_pcn), and what 999 / 277CA ACK lookups also
|
||||
# use to cross-reference the original claim. Storing
|
||||
# subscriber.member_id here (the 2010BA NM109) silently broke
|
||||
# every auto-match in production.
|
||||
patient_control_number=claim.claim_id or "",
|
||||
service_date_from=d_from,
|
||||
service_date_to=d_to,
|
||||
charge_amount=Decimal(claim.claim.total_charge or 0),
|
||||
provider_npi=claim.billing_provider.npi,
|
||||
# SP32: wire NM1*82 (Loop 2420A) rendering provider NPI from
|
||||
# ClaimOutput (T3 parser extraction) into the typed ORM column
|
||||
# (T1 migration 0019). Falls back to None if absent.
|
||||
rendering_provider_npi=claim.rendering_provider_npi,
|
||||
payer_id=claim.payer.id,
|
||||
state=ClaimState.SUBMITTED,
|
||||
raw_json=json.loads(claim.model_dump_json()),
|
||||
)
|
||||
|
||||
|
||||
def _remittance_835_row(cp: ClaimPayment, batch_id: str) -> Remittance:
|
||||
"""Build a Remittance ORM row from a ClaimPayment. NOT yet persisted."""
|
||||
received_at = utcnow()
|
||||
# Adjustment amount: sum the CAS rows for the first service line.
|
||||
# NOTE: This is a best-effort placeholder used until the reconciliation
|
||||
# pass (T10) overwrites it from the persisted CasAdjustment rows. The
|
||||
# authoritative value comes from `reconcile.run()`, which sums
|
||||
# ``CasAdjustment.amount`` per ``remittance_id`` and writes the result
|
||||
# back to ``Remittance.adjustment_amount``. We keep this stub so the
|
||||
# row has a sane value if reconciliation is disabled or fails.
|
||||
adjustment = Decimal("0")
|
||||
if cp.service_payments:
|
||||
sp = cp.service_payments[0]
|
||||
for adj in sp.adjustments:
|
||||
adjustment += adj.amount
|
||||
# Use the first service line's service_date as the remit service_date.
|
||||
service_date: date | None = None
|
||||
if cp.service_payments and cp.service_payments[0].service_date is not None:
|
||||
service_date = cp.service_payments[0].service_date
|
||||
return Remittance(
|
||||
id=cp.payer_claim_control_number,
|
||||
batch_id=batch_id,
|
||||
payer_claim_control_number=cp.payer_claim_control_number,
|
||||
claim_id=None,
|
||||
status_code=cp.status_code,
|
||||
status_label=cp.status_label,
|
||||
total_charge=Decimal(cp.total_charge or 0),
|
||||
total_paid=Decimal(cp.total_paid or 0),
|
||||
patient_responsibility=cp.patient_responsibility,
|
||||
adjustment_amount=adjustment,
|
||||
# SP32: wire NM1*1P (Loop 2100) service provider NPI from
|
||||
# ClaimPayment (T2 parser extraction) into the typed ORM column
|
||||
# (T1 migration 0019). Falls back to None if absent.
|
||||
rendering_provider_npi=cp.service_provider_npi,
|
||||
received_at=received_at,
|
||||
service_date=service_date,
|
||||
is_reversal=cp.status_code in ("21", "22"),
|
||||
raw_json=json.loads(cp.model_dump_json()),
|
||||
)
|
||||
|
||||
|
||||
def _persist_835_remit(session, cp: "ClaimPayment", remittance_id: str) -> None:
|
||||
"""SP7: persist ServiceLinePayment + CAS rows for one CLP composite.
|
||||
|
||||
For each 835 SVC composite in ``cp.service_payments``:
|
||||
- insert a ServiceLinePayment row (line_number, procedure, modifiers,
|
||||
charge, payment, units, service_date).
|
||||
- flush to populate slp.id.
|
||||
- insert each per-SVC CAS adjustment with ``service_line_payment_id``
|
||||
set to slp.id.
|
||||
|
||||
For CLP-level CAS adjustments (``cp.claim_adjustments``, a future
|
||||
extension; not produced by today's 835 parser but allowed by the spec):
|
||||
- insert CAS rows with ``service_line_payment_id IS NULL``.
|
||||
|
||||
The caller controls the transaction; this function does not commit.
|
||||
The 835 ingest site calls this after ``_remittance_835_row`` is
|
||||
flushed so the FK target is populated.
|
||||
"""
|
||||
import json as _json
|
||||
from cyclone.db import ServiceLinePayment, CasAdjustment
|
||||
|
||||
for svc in cp.service_payments:
|
||||
slp = ServiceLinePayment(
|
||||
remittance_id=remittance_id,
|
||||
line_number=svc.line_number,
|
||||
procedure_qualifier=svc.procedure_qualifier,
|
||||
procedure_code=svc.procedure_code,
|
||||
modifiers_json=_json.dumps(svc.modifiers or []),
|
||||
charge=Decimal(str(svc.charge)),
|
||||
payment=Decimal(str(svc.payment)),
|
||||
units=Decimal(str(svc.units)) if svc.units is not None else None,
|
||||
unit_type=svc.unit_type,
|
||||
service_date=svc.service_date,
|
||||
ref_benefit_plan=svc.ref_benefit_plan,
|
||||
)
|
||||
session.add(slp)
|
||||
session.flush() # populate slp.id for the FK below
|
||||
|
||||
for adj in svc.adjustments:
|
||||
quantity = getattr(adj, "quantity", None)
|
||||
session.add(CasAdjustment(
|
||||
remittance_id=remittance_id,
|
||||
group_code=adj.group_code,
|
||||
reason_code=adj.reason_code,
|
||||
amount=Decimal(str(adj.amount)),
|
||||
quantity=Decimal(str(quantity)) if quantity is not None else None,
|
||||
service_line_payment_id=slp.id,
|
||||
))
|
||||
|
||||
# CLP-level CAS (no SVC composite to attach to). Today's parser does
|
||||
# not produce these; the branch is forward-compatible.
|
||||
for adj in getattr(cp, "claim_adjustments", []) or []:
|
||||
quantity = getattr(adj, "quantity", None)
|
||||
session.add(CasAdjustment(
|
||||
remittance_id=remittance_id,
|
||||
group_code=adj.group_code,
|
||||
reason_code=adj.reason_code,
|
||||
amount=Decimal(str(adj.amount)),
|
||||
quantity=Decimal(str(quantity)) if quantity is not None else None,
|
||||
service_line_payment_id=None,
|
||||
))
|
||||
|
||||
|
||||
def _claim_status_from_validation(claim: ClaimOutput) -> str:
|
||||
"""Re-implement the in-memory status rules (sub-project 1 §6.2)."""
|
||||
v = claim.validation
|
||||
if not v.passed:
|
||||
has_r050 = any(e.rule == "R050_diagnosis_present" for e in v.errors)
|
||||
return "draft" if has_r050 else "denied"
|
||||
if claim.claim.frequency_code == "1":
|
||||
return "submitted"
|
||||
if v.warnings:
|
||||
return "pending"
|
||||
return "draft"
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user