Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Tyler
|
d25f00ac58 |
@@ -1,13 +0,0 @@
|
|||||||
# Repo-root .dockerignore — applies to `docker compose build` (which builds
|
|
||||||
# both backend and frontend contexts from the repo root). Excludes anything
|
|
||||||
# that should never end up in a build context.
|
|
||||||
|
|
||||||
.worktrees/
|
|
||||||
.git/
|
|
||||||
.github/
|
|
||||||
docs/prodfiles/
|
|
||||||
*.production.txt
|
|
||||||
node_modules/
|
|
||||||
dist/
|
|
||||||
.venv/
|
|
||||||
.superpowers/brainstorm/
|
|
||||||
+4
-17
@@ -1,20 +1,7 @@
|
|||||||
# Cyclone — environment configuration
|
# Cyclone — environment configuration
|
||||||
# Copy this file to `.env.local` and fill in values for your environment.
|
# Copy this file to `.env.local` and fill in values for your environment.
|
||||||
|
|
||||||
# Required on first boot. Cyclone refuses to start without these unless
|
# Base URL for the Python (FastAPI) backend that powers the Upload page and
|
||||||
# at least one user already exists (e.g. seeded via `python -m cyclone users create`).
|
# the real /api/parse-837 + /api/parse-835 endpoints. Leave empty to keep
|
||||||
# Min 12 chars for password.
|
# the in-memory sample data store and disable real EDI parsing.
|
||||||
CYCLONE_ADMIN_USERNAME=admin
|
VITE_API_BASE_URL=http://localhost:8000
|
||||||
CYCLONE_ADMIN_PASSWORD=change-me-to-a-strong-password-min-12-chars
|
|
||||||
|
|
||||||
# Base URL for the Python (FastAPI) backend. Leave empty for the
|
|
||||||
# Docker deployment (nginx proxies /api/* to backend on compose network).
|
|
||||||
VITE_API_BASE_URL=
|
|
||||||
|
|
||||||
# Optional. Set to 1 if you're behind an HTTPS reverse proxy and want
|
|
||||||
# the session cookie to include the Secure flag.
|
|
||||||
# CYCLONE_BEHIND_HTTPS=1
|
|
||||||
|
|
||||||
# Optional. Set to 1 to disable auth entirely (DEV ONLY). When set,
|
|
||||||
# the backend auto-grants admin access without checking credentials.
|
|
||||||
# CYCLONE_AUTH_DISABLED=0
|
|
||||||
|
|||||||
+2
-3
@@ -35,6 +35,5 @@ claims_output/
|
|||||||
# Worktrees (subagent-driven development)
|
# Worktrees (subagent-driven development)
|
||||||
.worktrees/
|
.worktrees/
|
||||||
|
|
||||||
# Brainstorm session artifacts (visual companion mockups, events, server state).
|
# Brainstorm session artifacts (visual companion mockups, events, server state)
|
||||||
# Skills under .superpowers/skills/ are committed project-scoped guidance.
|
.superpowers/
|
||||||
.superpowers/brainstorm/
|
|
||||||
|
|||||||
@@ -1,193 +0,0 @@
|
|||||||
---
|
|
||||||
name: cyclone-api-router
|
|
||||||
description: "Cyclone FastAPI router conventions (api_routers/, api_helpers.py, response shapes, error envelopes). Use when: adding or changing an HTTP endpoint, splitting a route out of api.py, or wiring a new helper into api_helpers.py."
|
|
||||||
---
|
|
||||||
|
|
||||||
# cyclone-api-router
|
|
||||||
|
|
||||||
Cyclone splits its FastAPI surface two ways: small resource-group
|
|
||||||
routers live in `backend/src/cyclone/api_routers/<topic>.py` and are
|
|
||||||
mounted bare into `api.py`; the high-traffic streaming and parse
|
|
||||||
endpoints still live as top-level decorators in `api.py` itself. This
|
|
||||||
skill codifies the conventions so additions stay consistent with the
|
|
||||||
four routers already shipped (`acks`, `admin`, `health`, `ta1_acks`).
|
|
||||||
|
|
||||||
As of this writing: **4 router modules** under
|
|
||||||
`backend/src/cyclone/api_routers/`, **one shared helpers module** at
|
|
||||||
`backend/src/cyclone/api_helpers.py` (248 lines, NDJSON primitives +
|
|
||||||
content negotiation + `tail_events`), and ~30 routes still inlined
|
|
||||||
in `backend/src/cyclone/api.py`. The next refactor target is the
|
|
||||||
parse endpoints.
|
|
||||||
|
|
||||||
## Auth gate (SP24)
|
|
||||||
|
|
||||||
Every router declared in `backend/src/cyclone/api_routers/` **must** carry `dependencies=[Depends(matrix_gate)]` at the `APIRouter(...)` declaration — not on each individual endpoint. The gate lives at `backend/src/cyclone/auth/deps.py:107` and the role matrix is at `backend/src/cyclone/auth/permissions.py`. The roles are `admin / user / viewer`; `matrix_gate` returns 401 when there's no session and 403 when the role is below the endpoint's required role. When `AUTH_DISABLED` is True (conftest autouse fixture flips it; `CYCLONE_AUTH_DISABLED=1` in prod-by-mistake), the gate short-circuits to a synthetic admin — see the SP24 spec for the threat-model implications. New routers get the gate by default; the auth-aware convention is `router = APIRouter(dependencies=[Depends(matrix_gate)])`.
|
|
||||||
|
|
||||||
## When to use
|
|
||||||
|
|
||||||
- **Adding an endpoint.** You're adding a new GET / POST handler —
|
|
||||||
you need to know whether it belongs in `api.py` (parse / streaming)
|
|
||||||
or in a new router under `api_routers/`, and what the response
|
|
||||||
shape and test file conventions look like.
|
|
||||||
- **Splitting a route.** You're moving a route out of `api.py` into a
|
|
||||||
dedicated `api_routers/<topic>.py` module and need the import /
|
|
||||||
mounting rules (`from cyclone.api_routers import <topic>` then
|
|
||||||
`app.include_router(<topic>.router)`).
|
|
||||||
- **Adding a helper.** You're wiring a new function into
|
|
||||||
`api_helpers.py` (NDJSON primitive, content-negotiation probe,
|
|
||||||
tail-event helper) and need to keep it private to the API layer.
|
|
||||||
- **Defining an error response.** You're raising from a route handler
|
|
||||||
and need the conventional `HTTPException(status_code=..., detail=...)`
|
|
||||||
shape used everywhere else in the API surface.
|
|
||||||
|
|
||||||
## Conventions
|
|
||||||
|
|
||||||
1. **No new top-level routes in `api.py` for resource groups.** Any
|
|
||||||
endpoint grouped under a resource (`/api/<resource>` and its
|
|
||||||
`/{id}` detail) lives in `backend/src/cyclone/api_routers/<topic>.py`
|
|
||||||
as an `APIRouter`. The streaming list endpoints (`/api/claims/stream`,
|
|
||||||
`/api/remittances/stream`, `/api/activity/stream`) and the parse
|
|
||||||
endpoints (`/api/parse-*`) currently stay in `api.py` because they
|
|
||||||
span multiple store modules — don't move them unless you're also
|
|
||||||
restructuring the store split.
|
|
||||||
2. **Reuse `api_helpers.py`.** NDJSON primitives (`ndjson_line`,
|
|
||||||
`ndjson_stream_list`, `ndjson_stream_837`, `ndjson_stream_835`),
|
|
||||||
content negotiation (`client_wants_json`, `wants_ndjson`), the
|
|
||||||
strict / `raw_segments` rewrites (`strict_rewrite_837`,
|
|
||||||
`strict_rewrite_835`, `drop_raw_segments_837`, `drop_raw_segments_835`),
|
|
||||||
and the shared live-tail generator (`tail_events`,
|
|
||||||
`heartbeat_seconds`) all live there. Don't duplicate them in a
|
|
||||||
router. The module's docstring (`api_helpers.py:1-18`) declares it
|
|
||||||
private to the API layer — no business logic, no DB writes.
|
|
||||||
3. **Response shape.** Every successful response is a **plain dict**
|
|
||||||
produced by a per-router `<entity>_to_ui(row)` helper (see
|
|
||||||
`_ack_to_ui` at `api_routers/acks.py:29-48` and `_ta1_to_ui` at
|
|
||||||
`api_routers/ta1_acks.py:23-37`). This dict shape **must** match
|
|
||||||
the matching `<entity>_written` event payload so live-tail pages
|
|
||||||
don't drift (see `cyclone-store` for the serializer contract).
|
|
||||||
Errors use FastAPI's `HTTPException` with a `detail` dict of the
|
|
||||||
form `{"error": "<Title>", "detail": "<message>"}` —
|
|
||||||
`acks.py:85-88`, `ta1_acks.py:72`. There is **no** shared
|
|
||||||
`ErrorEnvelope` Pydantic model; the `detail` dict is the contract.
|
|
||||||
4. **Mounting.** Routers are mounted bare in `api.py:251-256` —
|
|
||||||
`app.include_router(<name>.router)` with **no** `prefix=`
|
|
||||||
argument. Each `@router.<verb>` decorator carries the **full**
|
|
||||||
`/api/<resource>` path itself (see `acks.py:51,75`,
|
|
||||||
`ta1_acks.py:49,67`, `admin.py:23`, `health.py:28`). The
|
|
||||||
`router = APIRouter()` declaration carries no `tags=` either —
|
|
||||||
keep it minimal.
|
|
||||||
5. **Streaming endpoints.** Use
|
|
||||||
`StreamingResponse(media_type="application/x-ndjson")` and feed it
|
|
||||||
either `ndjson_stream_list(items, total, returned, has_more)` (for
|
|
||||||
list pages) or `tail_events(request, bus, kinds)` (for live-tail
|
|
||||||
pages). See `acks.py:62-66` for the list-stream skeleton and
|
|
||||||
`api.py:1357-1401` for the full live-tail pattern (snapshot →
|
|
||||||
`snapshot_end` → subscription → heartbeats). See `cyclone-tail`
|
|
||||||
for the wire format.
|
|
||||||
6. **Tests.** Every new endpoint gets a `test_api_<topic>_<verb>.py`
|
|
||||||
under `backend/tests/` (see `cyclone-tests` for the naming
|
|
||||||
convention + autouse `conftest.py`). Existing examples:
|
|
||||||
`test_api_validate_provider.py` (admin),
|
|
||||||
`test_api_parse_persists_ack.py` (acks).
|
|
||||||
|
|
||||||
## Patterns
|
|
||||||
|
|
||||||
### A new `APIRouter` skeleton
|
|
||||||
|
|
||||||
Pattern from `backend/src/cyclone/api_routers/acks.py:1-72`. Module
|
|
||||||
docstring names the resource, imports the shared helpers, declares
|
|
||||||
`router = APIRouter()` with no prefix, and defines a `_foo_to_ui(row)`
|
|
||||||
mapper at module scope.
|
|
||||||
|
|
||||||
```python
|
|
||||||
"""``/api/foo`` — list & detail endpoints for <topic>."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException, Query, Request
|
|
||||||
from fastapi.responses import StreamingResponse
|
|
||||||
|
|
||||||
from cyclone.api_helpers import ndjson_stream_list, wants_ndjson
|
|
||||||
from cyclone.store import store
|
|
||||||
|
|
||||||
router = APIRouter()
|
|
||||||
|
|
||||||
|
|
||||||
def _foo_to_ui(row) -> dict:
|
|
||||||
"""Map a Foo ORM row to the UI shape used by ``/api/foo``."""
|
|
||||||
return {"id": row.id, "name": row.name}
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/api/foo")
|
|
||||||
def list_foo(
|
|
||||||
request: Request,
|
|
||||||
limit: int = Query(100, ge=1, le=1000),
|
|
||||||
):
|
|
||||||
"""Return the list of persisted Foo rows, newest first."""
|
|
||||||
rows = store.list_foo()
|
|
||||||
items = [_foo_to_ui(r) for r in rows[:limit]]
|
|
||||||
total = len(rows)
|
|
||||||
returned = len(items)
|
|
||||||
has_more = total > returned
|
|
||||||
if wants_ndjson(request):
|
|
||||||
return StreamingResponse(
|
|
||||||
ndjson_stream_list(items, total, returned, has_more),
|
|
||||||
media_type="application/x-ndjson",
|
|
||||||
)
|
|
||||||
return {"items": items, "total": total, "returned": returned, "has_more": has_more}
|
|
||||||
```
|
|
||||||
|
|
||||||
### A `get_<topic>` detail endpoint with 404
|
|
||||||
|
|
||||||
Pattern from `api_routers/acks.py:75-104` and `ta1_acks.py:67-76`.
|
|
||||||
Path param is `<entity>_id` (not `id`) so it doesn't shadow
|
|
||||||
FastAPI's internal `id` and the OpenAPI docs stay self-describing.
|
|
||||||
|
|
||||||
```python
|
|
||||||
@router.get("/api/foo/{foo_id}")
|
|
||||||
def get_foo(foo_id: int) -> dict:
|
|
||||||
"""Return one persisted Foo row with its parsed detail.
|
|
||||||
|
|
||||||
Path param is ``foo_id`` (not ``id``) to avoid shadowing
|
|
||||||
FastAPI's internal ``id`` name and to keep OpenAPI docs
|
|
||||||
self-describing. Returns 404 when the row is missing — never 500.
|
|
||||||
"""
|
|
||||||
row = store.get_foo(foo_id)
|
|
||||||
if row is None:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=404,
|
|
||||||
detail={"error": "Not found", "detail": f"Foo {foo_id} not found"},
|
|
||||||
)
|
|
||||||
return _foo_to_ui(row)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Mounting in `api.py`
|
|
||||||
|
|
||||||
Pattern from `backend/src/cyclone/api.py:246-256`. The block lives
|
|
||||||
just after middleware registration and just before the first
|
|
||||||
`@app.<verb>` decorator.
|
|
||||||
|
|
||||||
```python
|
|
||||||
# Resource-group routers. Each module owns its own APIRouter and is
|
|
||||||
# registered below. New resources go in `cyclone.api_routers.<name>`
|
|
||||||
# and are wired in here.
|
|
||||||
from cyclone.api_routers import acks, admin, health, ta1_acks # noqa: E402
|
|
||||||
|
|
||||||
app.include_router(health.router)
|
|
||||||
app.include_router(acks.router)
|
|
||||||
app.include_router(ta1_acks.router)
|
|
||||||
app.include_router(admin.router)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Anti-patterns
|
|
||||||
|
|
||||||
- **Don't import from `cyclone.api` into a router — the dependency runs the other way.** Routers are mounted *into* `api.py` (`api_routers/acks.py` etc. know nothing about `cyclone.api`). A circular import would silently break the `from cyclone.api_routers import ...` block at `api.py:251`.
|
|
||||||
- **Don't introduce Pydantic response models where the codebase returns dicts.** Every existing list / detail endpoint returns a plain dict produced by a `<entity>_to_ui(row)` helper (see `acks.py:29-48`, `ta1_acks.py:23-37`). That dict shape **is** the event payload that `<entity>_written` carries — see `cyclone-store`. Introducing a Pydantic model on one side drifts the event payload from the list shape and silently breaks live-tail dedup.
|
|
||||||
- **Don't bypass `CycloneStore` to query the ORM directly from a route.** Always call `store.<method>(...)` (`store.list_acks()`, `store.get_ta1_ack(ack_id)`) so the read path picks up the same session + snapshot serializer as the live-tail subscriber. A raw `with db.SessionLocal()() as s: s.get(Foo, foo_id)` in a handler bypasses the serializer contract and breaks the event-payload match. See `cyclone-store`.
|
|
||||||
- **Don't set `prefix=` on `APIRouter`.** Mount the router bare (`app.include_router(<name>.router)`) and put the full `/api/<resource>` path in the decorator. Mixing the two styles scatters the URL across two files and breaks `grep "/api/foo"` audits.
|
|
||||||
|
|
||||||
## Related skills
|
|
||||||
|
|
||||||
- **`cyclone-store`** — most routes call `store.<method>(...)` and the dict payload **is** the `<entity>_written` event payload; load when adding a route so the read path stays aligned with the pubsub contract.
|
|
||||||
- **`cyclone-tail`** — streaming endpoints (`/api/<resource>/stream`) and the NDJSON wire format; load when adding a live-tail route or changing the wire format.
|
|
||||||
- **`cyclone-edi`** — parse endpoints (`/api/parse-837`, `/api/parse-835`, `/api/parse-999`, `/api/parse-ta1`, `/api/parse-277ca`) currently live in `api.py`; load when adding or changing a parse endpoint.
|
|
||||||
- **`cyclone-tests`** — endpoint tests follow the `test_api_<topic>_<verb>.py` naming under `backend/tests/`; load when writing the test for a new endpoint.
|
|
||||||
@@ -1,187 +0,0 @@
|
|||||||
---
|
|
||||||
name: cyclone-cli
|
|
||||||
description: "Cyclone CLI subcommand conventions (cli.py — Click group + subcommands parse-837/parse-835/validate-npi/validate-tax-id/backup, --yes + click.confirm for destructive ops, exit codes 0/1/2, CliRunner smoke tests in backend/tests/test_cli_*.py). Use when: adding a CLI subcommand, changing an exit code, adding a smoke test, or wiring a security-sensitive command (backup, key rotation, anything touching secrets.py)."
|
|
||||||
---
|
|
||||||
|
|
||||||
# cyclone-cli
|
|
||||||
|
|
||||||
The operator-facing CLI is a **Click** group at `cli.py:45` (`@click.group()` for `main`), mounted as the `cyclone` console script in `pyproject.toml:54` (`cyclone = "cyclone.cli:main"`). Seven subcommands ship today: `parse-837`, `parse-835`, `validate-npi`, `validate-tax-id`, plus the `backup` group (`init-passphrase`, `create`, `list`, `verify`, `restore`, `prune`, `status`). `serve` lives separately in `__main__.py:19` (dispatches `uvicorn cyclone.api:app`).
|
|
||||||
|
|
||||||
## When to use
|
|
||||||
|
|
||||||
- **Adding a subcommand.** You need a new operator command (e.g. `cyclone rotate-key`) and want to match the existing Click decorator + smoke-test rhythm.
|
|
||||||
- **Changing an exit code.** You're tweaking which `sys.exit(N)` a subcommand raises and need the 0/1/2 contract used by `parse_837`, `parse_835`, `validate_npi_cmd`, `validate_tax_id_cmd`, and the `backup` group.
|
|
||||||
- **Adding a smoke test.** You need `backend/tests/test_cli_<name>.py` using `click.testing.CliRunner` (NOT `subprocess.run`) and want the canonical fixture + monkeypatch layout (Keychain stub, fresh SQLite, `CYCLONE_BACKUP_DIR`).
|
|
||||||
- **Wiring a security-sensitive command.** Anything touching `cyclone/secrets.py` (Keychain writes), DB key rotation, or destructive restores needs the two-step confirm dance used by `backup restore` / `backup prune` (`cli.py:462,509`).
|
|
||||||
|
|
||||||
## Conventions
|
|
||||||
|
|
||||||
1. **Click decorator pattern, not argparse.** Each subcommand is a
|
|
||||||
top-level function decorated with `@main.command("<name>")` and
|
|
||||||
one `@click.option` / `@click.argument` per parameter. Group
|
|
||||||
dispatch is implicit — no `set_defaults(func=...)` and no
|
|
||||||
`cmd_<name>(args) -> int` signature. Top-level commands at
|
|
||||||
`cli.py:77,151,241,261,303`; the `backup` sub-group nests a
|
|
||||||
second `@main.group()` (`cli.py:303`) with its own
|
|
||||||
`@backup.command("<name>")` children.
|
|
||||||
2. **Long-form flags for safety.** Prefer `--rotate-key`,
|
|
||||||
`--backup-dir`, `--from-stdin` over positional args for anything
|
|
||||||
that mutates state or takes a secret. `init-passphrase`
|
|
||||||
(`cli.py:308-311`) demonstrates the canonical "flag OR stdin"
|
|
||||||
pattern: `--passphrase` for automation, `--from-stdin` for
|
|
||||||
interactive `getpass()` prompting.
|
|
||||||
3. **Exit codes: 0 / 1 / 2.** `0` = success. `1` = user / input
|
|
||||||
error (invalid NPI/EIN at `cli.py:258,277,285`; Keychain write
|
|
||||||
failure at `cli.py:341,351`; tampered-backup verify at
|
|
||||||
`cli.py:459`). `2` = operator / parse error
|
|
||||||
(`CycloneParseError` at `cli.py:106,178`; passphrase
|
|
||||||
empty/mismatch/short at `cli.py:331,337`). Document the codes
|
|
||||||
in the docstring (see `validate_npi_cmd` at `cli.py:244-250`).
|
|
||||||
Use `click.UsageError(...)` for usage mistakes; reserve
|
|
||||||
`sys.exit(2)` for "the file failed to parse" semantics.
|
|
||||||
4. **Smoke test with `click.testing.CliRunner`.** Every new
|
|
||||||
subcommand gets `backend/tests/test_cli_<name>.py` that
|
|
||||||
imports `from cyclone.cli import main` and invokes via
|
|
||||||
`CliRunner().invoke(main, [...], catch_exceptions=False)`. The
|
|
||||||
test must stub Keychain (`monkeypatch.setattr(secrets_mod,
|
|
||||||
"get_secret"/"set_secret", ...)`) and pin a temp SQLite DB via
|
|
||||||
`CYCLONE_DB_URL` + `db._reset_for_tests()` — see
|
|
||||||
`backend/tests/test_cli_backup.py:13-69` for the canonical
|
|
||||||
`_cli_env` fixture. CliRunner captures output and exit codes
|
|
||||||
in-process; do NOT shell out to `subprocess.run`.
|
|
||||||
5. **Destructive ops need `--yes` + `click.confirm(abort=True)`.**
|
|
||||||
`backup restore` (`cli.py:462-506`) and `backup prune`
|
|
||||||
(`cli.py:509-537`) both gate the destructive action behind a
|
|
||||||
`--yes` is_flag and an interactive `click.confirm(..., abort=True)`
|
|
||||||
prompt. CliRunner auto-aborts confirm prompts, so smoke tests
|
|
||||||
assert `exit_code != 0` when `--yes` is omitted
|
|
||||||
(`test_cli_backup.py:122-137`). A `--dry-run` flag is NOT yet
|
|
||||||
implemented anywhere; if needed, mirror the `--yes` pattern.
|
|
||||||
|
|
||||||
## Patterns
|
|
||||||
|
|
||||||
### A Click subcommand — `@main.command("<name>")` + options
|
|
||||||
|
|
||||||
From `cli.py:241-258` (smallest standalone subcommand):
|
|
||||||
|
|
||||||
```python
|
|
||||||
@main.command("validate-npi")
|
|
||||||
@click.argument("npi")
|
|
||||||
@click.option("--log-level", default="WARNING", show_default=True,
|
|
||||||
type=click.Choice(["DEBUG", "INFO", "WARNING", "ERROR"]))
|
|
||||||
def validate_npi_cmd(npi: str, log_level: str) -> None:
|
|
||||||
"""Validate a 10-digit NPI's Luhn checksum locally (SP20). Exit 0 valid, 1 invalid. PHI — don't log the value."""
|
|
||||||
setup_logging(level=log_level)
|
|
||||||
from cyclone.npi import is_valid_npi
|
|
||||||
if is_valid_npi(npi):
|
|
||||||
click.echo(f"OK: {len(npi)}-digit NPI passes Luhn checksum")
|
|
||||||
return
|
|
||||||
click.echo(f"INVALID: {npi!r} fails NPI Luhn checksum", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
```
|
|
||||||
|
|
||||||
### A smoke test — `CliRunner` + Keychain stub + temp SQLite
|
|
||||||
|
|
||||||
From `test_cli_backup.py:13-69`. Stable hex salt keeps multiple `CliRunner` invocations consistent within one test. The `Batch` seed at `cli_backup.py:27-35` is omitted — only the Keychain + DB plumbing is the convention:
|
|
||||||
|
|
||||||
```python
|
|
||||||
@pytest.fixture
|
|
||||||
def _cli_env(tmp_path, monkeypatch):
|
|
||||||
from cyclone import db, secrets as secrets_mod
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
|
|
||||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
|
|
||||||
db._reset_for_tests()
|
|
||||||
db.init_db()
|
|
||||||
# ... seed any DB rows the subcommand needs (see cli_backup.py:27-35) ...
|
|
||||||
|
|
||||||
store = {svc_mod.KEYCHAIN_BACKUP_PASSPHRASE_ACCOUNT: "cli-test-passphrase",
|
|
||||||
svc_mod.KEYCHAIN_BACKUP_SALT_ACCOUNT:
|
|
||||||
"0123456789abcdef0123456789abcdef"}
|
|
||||||
monkeypatch.setattr(secrets_mod, "get_secret", lambda n: store.get(n))
|
|
||||||
monkeypatch.setattr(secrets_mod, "set_secret",
|
|
||||||
lambda n, v: store.__setitem__(n, v) or True)
|
|
||||||
monkeypatch.setenv("CYCLONE_BACKUP_DIR", str(tmp_path / "backups"))
|
|
||||||
yield tmp_path / "backups"
|
|
||||||
db._reset_for_tests()
|
|
||||||
|
|
||||||
def test_backup_create_list_verify_status(_cli_env):
|
|
||||||
from cyclone.cli import main
|
|
||||||
runner = CliRunner()
|
|
||||||
r = runner.invoke(main, ["backup", "create"], catch_exceptions=False)
|
|
||||||
assert r.exit_code == 0, r.output
|
|
||||||
assert "created backup id=" in r.output
|
|
||||||
```
|
|
||||||
|
|
||||||
### A destructive subcommand — `--yes` + `click.confirm(abort=True)`
|
|
||||||
|
|
||||||
From `cli.py:462-506` (`backup restore`). Two-step: announce, prompt unless `--yes`, then execute. Restore uses an explicit init/confirm round-trip so the operator can back out between phases. Matching smoke test asserts the guard fires:
|
|
||||||
|
|
||||||
```python
|
|
||||||
@backup.command("restore")
|
|
||||||
@click.argument("backup_id", type=int)
|
|
||||||
@click.option("--yes", is_flag=True, help="Skip the interactive confirm prompt")
|
|
||||||
@click.option("--actor", default="operator-cli", show_default=True)
|
|
||||||
def backup_restore(backup_id: int, yes: bool, actor: str) -> None:
|
|
||||||
"""Restore the live DB from a backup (two-step, requires --yes)."""
|
|
||||||
# ... db.init_db() + service config omitted ...
|
|
||||||
click.echo(f"Initiating restore from backup {backup_id}...")
|
|
||||||
init = svc.restore_initiate(backup_id)
|
|
||||||
# ... echo init summary (filename, fp, table_count, ttl) ...
|
|
||||||
if not yes:
|
|
||||||
click.confirm(
|
|
||||||
"Replace the live DB with this backup? "
|
|
||||||
"This will dispose the engine and rebuild it.",
|
|
||||||
abort=True,
|
|
||||||
)
|
|
||||||
click.echo("Confirming restore...")
|
|
||||||
result = svc.restore_confirm(backup_id, init.restore_token, actor=actor)
|
|
||||||
|
|
||||||
def test_backup_restore_requires_yes_flag(_cli_env):
|
|
||||||
runner = CliRunner()
|
|
||||||
runner.invoke(main, ["backup", "create"], catch_exceptions=False)
|
|
||||||
r = runner.invoke(main, ["backup", "restore", "1"], catch_exceptions=False)
|
|
||||||
# CliRunner auto-aborts confirm prompts → exit_code != 0.
|
|
||||||
assert r.exit_code != 0
|
|
||||||
```
|
|
||||||
|
|
||||||
## Anti-patterns
|
|
||||||
|
|
||||||
- **Don't `sys.exit(2)` for usage errors.** Reserve code 2 for
|
|
||||||
parse/operator errors (`CycloneParseError`, Keychain not
|
|
||||||
initialized, passphrase policy violation). For "you passed the
|
|
||||||
wrong flag" use `raise click.UsageError(...)` — Click formats
|
|
||||||
it as a clean help message and exits 2 on its own.
|
|
||||||
- **Don't print errors to stdout.** Use `click.echo(msg, err=True)`
|
|
||||||
for all error output. `print(..., file=sys.stderr)` and bare
|
|
||||||
`logging.error(...)` bypass Click's stdout/stderr split and leak
|
|
||||||
into test `result.output` — breaking `assert "FAIL" in r.output`
|
|
||||||
style assertions.
|
|
||||||
- **Don't add a subcommand without a smoke test.** Every new
|
|
||||||
`@main.command(...)` ships a sibling
|
|
||||||
`backend/tests/test_cli_<name>.py` exercising the happy path
|
|
||||||
AND at least one error path (missing input, invalid arg,
|
|
||||||
tampered ciphertext — see `test_cli_backup.py:102-119`).
|
|
||||||
- **Don't reuse the `parse` command name.** Existing subcommands
|
|
||||||
are type-specific (`parse-837`, `parse-835`); a generic `parse`
|
|
||||||
would shadow them or force an `--type` flag — neither is the
|
|
||||||
codebase pattern.
|
|
||||||
|
|
||||||
## Related skills
|
|
||||||
|
|
||||||
- **`cyclone-store`** — `backup` subcommands and the parse
|
|
||||||
subcommands both round-trip through `CycloneStore` and the DB
|
|
||||||
session; load when the increment changes write paths or the
|
|
||||||
`<entity>_written` event contract.
|
|
||||||
- **`cyclone-api-router`** — `cyclone serve` (via `__main__.py:19`)
|
|
||||||
launches the FastAPI app; the CLI parse subcommands share the
|
|
||||||
same `CycloneParseError` exception and Pydantic result models
|
|
||||||
as the matching HTTP endpoints.
|
|
||||||
- **`cyclone-edi`** — `parse-837` / `parse-835` are the CLI smoke
|
|
||||||
entry points for the parser/validator surface; load when adding
|
|
||||||
a parser or R-code rule.
|
|
||||||
- **`cyclone-tests`** — every CLI subcommand gets a pytest smoke
|
|
||||||
case under `backend/tests/test_cli_<name>.py`; the `_cli_env`
|
|
||||||
fixture pattern (fresh SQLite + Keychain stub) is documented
|
|
||||||
there.
|
|
||||||
- **`cyclone-spec`** — load when the SP-N spec introduces a new operator command or reserves a new exit-code category.
|
|
||||||
@@ -1,193 +0,0 @@
|
|||||||
---
|
|
||||||
name: cyclone-edi
|
|
||||||
description: "Cyclone EDI parser/validator conventions (837P/835/999/270/271/277CA/TA1). Use when: adding or changing a parser, adding a validator rule (R010/R020/R100/R200-R210/R835_*/NPI Luhn/EIN/CAS), or mapping a new CAS adjustment reason code."
|
|
||||||
---
|
|
||||||
|
|
||||||
# cyclone-edi
|
|
||||||
|
|
||||||
Cyclone parses seven X12 EDI transaction types (837P, 835, 999, 270, 271, 277CA, TA1) into typed Pydantic models, then runs per-claim / per-batch validator rules that surface as R-coded `ValidationIssue` records. This skill codifies the conventions so new parsers and rules stay consistent with the seven that already exist.
|
|
||||||
|
|
||||||
As of this writing: **7 parser modules** under `backend/src/cyclone/parsers/parse_<edi>.py`, **~25 per-claim rules** numbered `R010`–`R100` and `R200`–`R210` in `validator.py`, plus a parallel set of **835-specific rules** prefixed `R835_*` in `validator_835.py`. The next increment is **SP22**.
|
|
||||||
|
|
||||||
## When to use
|
|
||||||
|
|
||||||
- **Adding or changing a parser.** You are about to touch `backend/src/cyclone/parsers/parse_<edi>.py` or its paired `models_<edi>.py` and need the orchestrator signature, the segment walker convention, and the re-export in `parsers/__init__.py`.
|
|
||||||
- **Adding a validator rule.** You are writing a new `_rule_R<n>_<name>` (or `_r<n>_<name>` per the existing snake-case style) and need the rule signature, the R-code numbering scheme, and the `ValidationIssue` shape.
|
|
||||||
- **Wiring a new CAS / CARC code.** The 835 carries Claim Adjustment Reason Codes in `CAS` segments; the lookup lives in `backend/src/cyclone/parsers/cas_codes.py` and the UI reads through `claim_status_label()`.
|
|
||||||
- **Debugging a parse failure on a prodfiles sample.** You dropped a real EDI file into `docs/prodfiles/<source>/` and the parser is choking — load this skill to confirm the tokenizer path, the orchestrator entry point, and which fixture in `backend/tests/fixtures/` matches the transaction type.
|
|
||||||
|
|
||||||
## Conventions
|
|
||||||
|
|
||||||
1. **Parser signature.** Every parser module exports exactly one public entry function. Two flavors coexist in the codebase:
|
|
||||||
- `parse(text: str, *, input_file: str = "") -> <TypedResult>` — used by `parse_270.py:337`, `parse_271.py:356`.
|
|
||||||
- `parse(text: str, payer_config: <PayerConfig>, input_file: str = "") -> <TypedResult>` — used by `parse_837.py:319` and `parse_835.py:459` because both need payer-specific config to validate segments against.
|
|
||||||
- `parse_<edi>_text(text: str, *, input_file: str = "") -> <TypedResult>` — the legacy name-suffixed form, still in use at `parse_ta1.py:143`, `parse_999.py:220`, `parse_277ca.py:280`. The `<TypedResult>` is always a Pydantic model from `models_<edi>.py` (or co-located `models.py` for 837P).
|
|
||||||
2. **Segment walk.** Parsers consume `backend/src/cyclone/parsers/segments.py` — there are exactly three public pieces: `Delimiters` (frozen dataclass holding the four ISA-derived separators), `_detect_delimiters(isa_segment)` (private), and `tokenize(text) -> list[list[str]]` (returns ISA prepended as the first segment). Parsers then index into the `list[list[str]]` directly — there is **no** `Segment` / `Loop` / `next_segment` helper class. Whole-document problems (missing ISA, wrong transaction set) raise `CycloneParseError`; per-segment problems on acks (999/277CA) are surfaced on the result, not raised.
|
|
||||||
3. **Validator rules.** Numbered rules live in `backend/src/cyclone/parsers/validator.py` (for 837P — R010–R100 general + R200–R210 SP9 CO MAP / HCPF naming) and `backend/src/cyclone/parsers/validator_835.py` (for 835 — names prefixed `R835_*` because the same numeric space would collide with 837P). Each rule is a function `_r<n>_<name>(claim: ClaimOutput, cfg: PayerConfig) -> Iterable[ValidationIssue]` registered in the module-level `_RULES` list and run by `validate(claim, config)`. Issues carry the rule name as a stable string (`rule="R021_npi_checksum"`) — the R-code **is** how the UI surfaces the error, so never invent an unnumbered rule.
|
|
||||||
4. **NPI / EIN / CAS format logic.** Identity-format checks live in their own modules — never duplicate them in a parser or validator:
|
|
||||||
- `backend/src/cyclone/npi.py` — `is_valid_npi(npi)` runs the Luhn checksum with the `80840` NPPES prefix; `is_valid_tax_id(ein)` enforces `XX-XXXXXXX` (or 9 raw digits).
|
|
||||||
- `backend/src/cyclone/parsers/cas_codes.py` — `reason_label(group, reason)` and `all_known_codes()` for the CARC lookup; snapshot date is exported as `LAST_UPDATED`.
|
|
||||||
- `backend/src/cyclone/parsers/models_271.py` — `SERVICE_TYPE_CODES` + `service_type_description()` for 271 EB benefit codes.
|
|
||||||
5. **Prodfiles reuse.** When adding a parser for a new transaction type, ship at least one fixture in `backend/tests/fixtures/<edi>/<sample>.txt` (the existing 13 fixtures are **flat** at the top level of `fixtures/` — no per-test subdirectories). Copy from `docs/prodfiles/<source>/<file>.txt`; never reach into `docs/prodfiles/` from a test. The matching test should declare the path as a module-level `Path` constant.
|
|
||||||
|
|
||||||
## Patterns
|
|
||||||
|
|
||||||
### Minimal `parse_<edi>.py` — using `segments.py`, exporting `parse_ta1_text`
|
|
||||||
|
|
||||||
Taken from `backend/src/cyclone/parsers/parse_ta1.py:1-29` (the smallest parser — TA1 is just ISA + TA1 + IEA). The same skeleton scales to every other EDI type by adding `_consume_<segment>` helpers.
|
|
||||||
|
|
||||||
```python
|
|
||||||
"""Parse an X12 TA1 (Interchange Acknowledgment) file.
|
|
||||||
|
|
||||||
Whole-document problems (missing ISA, no TA1) raise CycloneParseError.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
from datetime import date
|
|
||||||
|
|
||||||
from cyclone.parsers.exceptions import CycloneParseError
|
|
||||||
from cyclone.parsers.models import BatchSummary, Envelope
|
|
||||||
from cyclone.parsers.models_ta1 import ParseResultTa1, Ta1Ack
|
|
||||||
from cyclone.parsers.segments import tokenize
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_yyyymmdd(s: str) -> date | None:
|
|
||||||
"""Parse an 8-digit CCYYMMDD string. Returns None on bad input."""
|
|
||||||
...
|
|
||||||
|
|
||||||
|
|
||||||
def _build_envelope(segments: list[list[str]], input_file: str) -> Envelope:
|
|
||||||
"""Build the envelope from ISA. TA1 has no GS/ST — just ISA → TA1 → IEA."""
|
|
||||||
...
|
|
||||||
|
|
||||||
|
|
||||||
def _consume_ta1(segments: list[list[str]], idx: int) -> tuple[Ta1Ack, int]:
|
|
||||||
"""Read a TA1 segment and return a Ta1Ack. Returns (model, next_idx)."""
|
|
||||||
...
|
|
||||||
|
|
||||||
|
|
||||||
def parse_ta1_text(text: str, *, input_file: str = "") -> ParseResultTa1:
|
|
||||||
"""Parse a complete TA1 document and return a ParseResultTa1."""
|
|
||||||
segments = tokenize(text)
|
|
||||||
envelope = _build_envelope(segments, input_file=input_file)
|
|
||||||
ta1_idx = next(
|
|
||||||
(i for i, seg in enumerate(segments) if seg[0] == "TA1"), None,
|
|
||||||
)
|
|
||||||
if ta1_idx is None:
|
|
||||||
raise CycloneParseError("No TA1 segment found")
|
|
||||||
ta1, _ = _consume_ta1(segments, ta1_idx)
|
|
||||||
...
|
|
||||||
return ParseResultTa1(envelope=envelope, ta1=ta1, summary=summary, ...)
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["parse_ta1_text"]
|
|
||||||
```
|
|
||||||
|
|
||||||
The orchestrator pattern is the same in every parser: `tokenize` → `_build_envelope` → segment consumers in order → wrap into a `ParseResult<EDI>` model. The Pydantic result is what the API / store layer consumes.
|
|
||||||
|
|
||||||
### A validator rule — `_r<n>_<name>` registered in `_RULES`
|
|
||||||
|
|
||||||
Taken from `backend/src/cyclone/parsers/validator.py:23-66` (the canonical R010–R100 block).
|
|
||||||
|
|
||||||
```python
|
|
||||||
from collections.abc import Iterable
|
|
||||||
from cyclone.parsers.models import ClaimOutput, ValidationIssue
|
|
||||||
from cyclone.parsers.payer import PayerConfig
|
|
||||||
|
|
||||||
NPI_RE = re.compile(r"^\d{10}$")
|
|
||||||
Rule = Callable[[ClaimOutput, PayerConfig], Iterable[ValidationIssue]]
|
|
||||||
|
|
||||||
|
|
||||||
def _r020_npi_format(claim: ClaimOutput, _: PayerConfig) -> Iterable[ValidationIssue]:
|
|
||||||
if claim.billing_provider.npi and not NPI_RE.match(claim.billing_provider.npi):
|
|
||||||
yield ValidationIssue(
|
|
||||||
rule="R020_npi_format",
|
|
||||||
severity="error",
|
|
||||||
message=f"Billing provider NPI must be 10 digits, got {claim.billing_provider.npi!r}",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _r021_npi_checksum(claim: ClaimOutput, _: PayerConfig) -> Iterable[ValidationIssue]:
|
|
||||||
"""SP20: validate the billing-provider NPI's Luhn check digit."""
|
|
||||||
npi = claim.billing_provider.npi
|
|
||||||
if not npi or not NPI_RE.match(npi):
|
|
||||||
return # R020 already flagged the format — skip silently.
|
|
||||||
try:
|
|
||||||
from cyclone.npi import is_valid_npi
|
|
||||||
except ImportError:
|
|
||||||
return
|
|
||||||
if not is_valid_npi(npi):
|
|
||||||
yield ValidationIssue(
|
|
||||||
rule="R021_npi_checksum",
|
|
||||||
severity="warning",
|
|
||||||
message=f"Billing provider NPI {npi!r} fails Luhn checksum (likely typo)",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
_RULES: list[Rule] = [
|
|
||||||
_r010_clm01_present,
|
|
||||||
_r011_total_charge_positive,
|
|
||||||
_r020_npi_format,
|
|
||||||
_r021_npi_checksum,
|
|
||||||
# ... R030, R031, R032-R035, R050, R060, R070, R100, R200-R210
|
|
||||||
]
|
|
||||||
```
|
|
||||||
|
|
||||||
For 835 rules, prefix the rule string with `R835_` (e.g. `R835_BPR01_handling_code_allowed`) and target the `ParseResult835` model instead of `ClaimOutput` — see `backend/src/cyclone/parsers/validator_835.py:38-79`.
|
|
||||||
|
|
||||||
### A test that uses a prodfiles fixture
|
|
||||||
|
|
||||||
Taken from `backend/tests/test_api_999.py:53-72`. The autouse `conftest.py` already provides a per-test SQLite DB; most tests just add a `client` fixture and reference the fixture path as a module-level constant.
|
|
||||||
|
|
||||||
```python
|
|
||||||
"""Tests for the FastAPI surface in cyclone.api for the 999 endpoint."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
|
|
||||||
from cyclone.api import app
|
|
||||||
|
|
||||||
# Fixture reference — flat, module-level Path constant. NEVER reach into
|
|
||||||
# docs/prodfiles/ from a test; the fixtures/ dir is the test-consumed surface.
|
|
||||||
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
|
|
||||||
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def client() -> TestClient:
|
|
||||||
return TestClient(app)
|
|
||||||
|
|
||||||
|
|
||||||
def test_parse_999_endpoint_happy_path(client: TestClient):
|
|
||||||
text = ACCEPTED.read_text()
|
|
||||||
resp = client.post(
|
|
||||||
"/api/parse-999",
|
|
||||||
files={"file": ("minimal_999.txt", text, "text/plain")},
|
|
||||||
headers={"Accept": "application/json"},
|
|
||||||
)
|
|
||||||
assert resp.status_code == 200, resp.text
|
|
||||||
body = resp.json()
|
|
||||||
assert body["ack"]["ack_code"] == "A"
|
|
||||||
```
|
|
||||||
|
|
||||||
For pure-unit parser tests (no API), the same path is reused — see `backend/tests/test_parse_837.py:8` (`FIXTURE = Path(__file__).parent / "fixtures" / "minimal_837p.txt"`).
|
|
||||||
|
|
||||||
## Anti-patterns
|
|
||||||
|
|
||||||
- **Don't re-parse raw X12 strings inside validators.** Always parse first into the typed `ParseResult<EDI>` / `ClaimOutput`, then validate against that. Validators index into the model fields (or `claim.raw_segments` for spot-checks of specific segment presence) — they never call `tokenize` again. R034's `REF*G1` presence check (`validator.py:104-107`) is the only place that legitimately touches `raw_segments`, and it does so to confirm a single segment exists.
|
|
||||||
- **Don't bake payer-specific logic into the generic parser.** Payer variations live in `backend/src/cyclone/parsers/payer.py` (`PayerConfig`, `PayerConfig835`) and `backend/src/cyclone/payers.py` (the YAML loader from `config/payers.yaml`). Parsers accept the config as an argument; rules read it from the `cfg` parameter. A new payer never requires a new parser file — extend the config and add / adjust an R-code rule.
|
|
||||||
- **Don't add a validator rule without an R-code.** The `rule="R<n>_<name>"` string is the stable identifier the UI greys out, the API returns in `errors[].rule`, and tests assert against. Inventing a rule without an R-code (or reusing an R-code with new semantics) breaks the operator workflow. New SP-N increments reserve their R-code range up front (SP9 reserved R200–R210, SP20 added R021) and document it in the spec.
|
|
||||||
|
|
||||||
## Related skills
|
|
||||||
|
|
||||||
- **`cyclone-store`** — load when the increment changes how a parsed `ClaimOutput` / `ParseResult<EDI>` is persisted (`store.py` write path, `<entity>_written` events).
|
|
||||||
- **`cyclone-api-router`** — load when the increment adds or changes an HTTP endpoint that surfaces a parsed result (e.g. `/api/parse-999`, `/api/parse-837`, `/api/parse-835`).
|
|
||||||
- **`cyclone-tests`** — every parser addition ships a fixture in `backend/tests/fixtures/` and a pytest case; load this skill for the fixture-drop-in and autouse-conftest rules.
|
|
||||||
- **`cyclone-cli`** — load when the increment adds a CLI subcommand. The `cyclone parse-837 <file>` and `cyclone parse-835 <file>` smoke commands at `backend/src/cyclone/cli.py:77,151` are the parser-level smoke tests; the `validate-npi` and `validate-tax-id` commands exercise the format helpers.
|
|
||||||
- **`cyclone-spec`** — load when the SP-N spec for the increment introduces a new R-code range or a new transaction type; the spec's `## Decisions` section is where the R-code reservation gets locked in.
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
# Cyclone EDI parsers — flat catalog
|
|
||||||
|
|
||||||
Every parser module under `backend/src/cyclone/parsers/` (one row per
|
|
||||||
file), its transaction type, its public entry signature, its result
|
|
||||||
model, its primary fixture, and any payer-specific variant. The
|
|
||||||
companion Pydantic model is in a co-located `models_<edi>.py`; the
|
|
||||||
segment walker uses `tokenize()` from `segments.py` and never parses
|
|
||||||
raw text inline.
|
|
||||||
|
|
||||||
| Module | EDI type | Public entry signature | Result model | Primary fixture(s) | Payer variant |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| `parse_837.py` | 837P (Professional Claim) | `parse(text, payer_config: PayerConfig, input_file="") -> ParseResult` | `cyclone.parsers.models.ParseResult` | `minimal_837p.txt`, `co_medicaid_837p.txt` | `PayerConfig` (CO Medicaid default) |
|
|
||||||
| `parse_835.py` | 835 (ERA / Remittance) | `parse(text, payer_config: PayerConfig835, input_file="") -> ParseResult835` | `cyclone.parsers.models_835.ParseResult835` | `minimal_835.txt`, `co_medicaid_835.txt`, `unbalanced_835.txt` | `PayerConfig835` |
|
|
||||||
| `parse_999.py` | 999 (Implementation ACK) | `parse_999_text(text, *, input_file="") -> ParseResult999` | `cyclone.parsers.models_999.ParseResult999` | `minimal_999.txt`, `minimal_999_rejected.txt` | none — single-shape ack |
|
|
||||||
| `parse_277ca.py` | 277CA (Claim ACK) | `parse_277ca_text(text, *, input_file="") -> ParseResult277CA` | `cyclone.parsers.models_277ca.ParseResult277CA` | `minimal_277ca.txt`, `minimal_277ca_rejected_only.txt`, `minimal_277ca_st277.txt` | `PayerConfig277CA` (config-driven) |
|
|
||||||
| `parse_270.py` | 270 (Eligibility Inquiry) | `parse(text, *, input_file="") -> ParseResult270` | `cyclone.parsers.models_270.ParseResult270` | `minimal_270.txt` | reuses `PayerConfig` shape |
|
|
||||||
| `parse_271.py` | 271 (Eligibility Response) | `parse(text, *, input_file="") -> ParseResult271` | `cyclone.parsers.models_271.ParseResult271` | `minimal_271.txt` | reuses `PayerConfig` shape |
|
|
||||||
| `parse_ta1.py` | TA1 (Interchange ACK) | `parse_ta1_text(text, *, input_file="") -> ParseResultTa1` | `cyclone.parsers.models_ta1.ParseResultTa1` | `minimal_ta1.txt` | none — single-shape ack |
|
|
||||||
|
|
||||||
Companion modules (not parsers, but shipped alongside):
|
|
||||||
|
|
||||||
| Module | Role |
|
|
||||||
|---|---|
|
|
||||||
| `segments.py` | `Delimiters`, `_detect_delimiters`, `tokenize(text) -> list[list[str]]` |
|
|
||||||
| `models.py` | Pydantic models for 837P (`ParseResult`, `ClaimOutput`, `Envelope`, `BatchSummary`, `ValidationIssue`, `ValidationReport`, …) |
|
|
||||||
| `models_835.py` / `models_270.py` / `models_271.py` / `models_277ca.py` / `models_999.py` / `models_ta1.py` | Pydantic models for each transaction type |
|
|
||||||
| `payer.py` | `PayerConfig` + `PayerConfig835` factories |
|
|
||||||
| `exceptions.py` | `CycloneParseError`, `CycloneValidationError` |
|
|
||||||
| `cas_codes.py` | CARC lookup: `reason_label(group, reason)`, `all_known_codes()`, `LAST_UPDATED` |
|
|
||||||
| `validator.py` | 837P rules: R010–R100, R200–R210; `validate(claim, config) -> ValidationReport` |
|
|
||||||
| `validator_835.py` | 835 rules: `R835_*` (e.g. `R835_BPR01_handling_code_allowed`); `validate(result, cfg) -> ValidationReport` |
|
|
||||||
| `serialize_270.py` / `serialize_837.py` / `serialize_999.py` | Outbound (Cyclone → payer) serializers — mirror of `parse_*` |
|
|
||||||
| `writer.py` / `writer_835.py` | Output writers (one JSON per claim) |
|
|
||||||
| `batch_ack_builder.py` | `build_ack_for_batch` — produces a 999 for a parsed 837 batch |
|
|
||||||
| `__init__.py` | Lazy PEP 562 re-exports (`parse`, `parse_835`, `parse_999`, `parse_270`, `parse_271`, `parse_277ca`, plus all models) |
|
|
||||||
|
|
||||||
Fixture rule: every parser ships at least one flat fixture in
|
|
||||||
`backend/tests/fixtures/<edi>-sample.txt`. Prodfiles sources live in
|
|
||||||
`docs/prodfiles/{837p-from-axiscare,835fromco,FromHPE,claims}/` — copy
|
|
||||||
from there, never reach in from a test.
|
|
||||||
@@ -1,138 +0,0 @@
|
|||||||
---
|
|
||||||
name: cyclone-frontend-page
|
|
||||||
description: "Cyclone React page conventions (TanStack Query, use<X> hook, drawer, URL state, .test.tsx sibling, Layout / PageHeader / Sidebar). Use when: adding a new page, refactoring an existing one, or wiring a drawer into a page."
|
|
||||||
---
|
|
||||||
|
|
||||||
# cyclone-frontend-page
|
|
||||||
|
|
||||||
Cyclone pages live at `src/pages/<Name>.tsx`: a `use<X>` hook in `src/hooks/use<X>.ts` does the fetching (and optionally the live-tail subscription), `<Layout>` + `<PageHeader>` + `<Sidebar>` (`src/components/`) provide the app shell, and any right-side detail (claim, remittance) lives in `src/components/<DrawerName>/` whose open/close state is mirrored to the URL via `useDrawerUrlState`. This skill codifies the conventions so additions stay consistent with the eleven pages already shipped.
|
|
||||||
|
|
||||||
As of this writing: **11 pages** under `src/pages/` (9 of 11 with a `*.test.tsx` sibling — Dashboard, Upload, and BatchDiff are not yet covered; be the first when you refactor them), **~30 hooks** under `src/hooks/`, and **2 drawer modules** at `src/components/{ClaimDrawer,RemitDrawer}/`. The next increment is **SP22**.
|
|
||||||
|
|
||||||
## When to use
|
|
||||||
|
|
||||||
- **Adding a new page.** Mounting a new screen in `src/pages/` — you need the Layout + PageHeader + table shape, the `use<X>` hook split, and the route registration point in `src/App.tsx`.
|
|
||||||
- **Refactoring an existing page.** Splitting a 600-line page, swapping a manual `fetch` for a hook, or moving in-component state into the URL — load this skill to confirm the destination shape.
|
|
||||||
- **Wiring a drawer.** Adding a new right-side detail drawer (e.g. `BatchDrawer`, `ActivityDrawer`) — you need `useDrawerUrlState` for URL-driven open/close, the `src/components/<DrawerName>/` folder layout, and the deep-link contract so `?claim=…` / `?remit=…` round-trips.
|
|
||||||
- **Sharing state via URL.** Persisting filter / page / drawer state across reloads — confirm the `useDrawerUrlState` / `useRemitDrawerUrlState` hook pair is the right tool before reaching for `useState` + history.
|
|
||||||
|
|
||||||
## Conventions
|
|
||||||
|
|
||||||
1. **Page shape.** Every page in `src/pages/<Name>.tsx` exports a `function <Name>()` (most pages use a named export — see `src/pages/Claims.tsx:51`, `Remittances.tsx:62`, `Dashboard.tsx:68`; `src/pages/Inbox.tsx:40` is the lone default export). The app shell is provided by the `<Layout>` route wrapper in `src/App.tsx:30`. Each page sets a `<PageHeader>` (`src/components/PageHeader.tsx:18`) and renders a table, list, or KPI grid. Sidebar nav is mounted once by `<Layout>` at `src/components/Sidebar.tsx`.
|
|
||||||
2. **Data hook.** Each page pairs with a `use<X>` hook in `src/hooks/use<X>.ts` (e.g. `Claims` ↔ `useClaims`, `Remittances` ↔ `useRemittances`, `Acks` ↔ `useAcks`). The hook returns `{ data, isLoading, isError, error, refetch }` from TanStack Query's `useQuery` (see `src/hooks/useClaims.ts:24-31`, `useRemittances.ts:21-25`). Pages never call `fetch` or `@/lib/api` directly — the hook is the boundary so the page is testable with `vi.mock("@/lib/api", ...)`.
|
|
||||||
3. **Live tail.** Pages with live data compose three hooks in order: the `use<X>` initial fetch, `useTailStream(resource)` (`src/hooks/useTailStream.ts:80` — opens the NDJSON stream, drives the backoff/stall state machine), and `useMergedTail(resource, baseItems, filterFn?)` (`src/hooks/useMergedTail.ts:25` — merges snapshot + tail, dedup'd by id). The full wiring lives at `src/pages/Claims.tsx:85-89` and `Remittances.tsx:81-83`. The streaming subscription belongs on the page, not in the data hook — hoisting it couples the lifecycle to whoever mounts `use<X>`.
|
|
||||||
4. **Drawer.** Right-side detail drawers live in `src/components/<DrawerName>/` (currently `ClaimDrawer/`, `RemitDrawer/`) with a barrel `index.ts` (`src/components/ClaimDrawer/index.ts:1-14`). The drawer is wired to URL state via `useDrawerUrlState()` for `?claim=…` (`src/hooks/useDrawerUrlState.ts`) or `useRemitDrawerUrlState()` for `?remit=…` (`src/hooks/useRemitDrawerUrlState.ts`). Open state is driven by the URL, not a `useState` flag, so deep-links round-trip.
|
|
||||||
5. **Tests.** Every page gets a `src/pages/<Name>.test.tsx` sibling (e.g. `Claims.test.tsx`, `Remittances.test.tsx`, `Batches.test.tsx`). Every hook gets a `src/hooks/use<X>.test.ts` sibling (e.g. `useClaims.test.ts`, `useRemittances.test.ts`). The shared setup — `// @vitest-environment happy-dom`, `IS_REACT_ACT_ENVIRONMENT = true`, `QueryClient` provider, `vi.mock("@/lib/api", ...)` — is documented in `cyclone-tests`; mirror `src/pages/Claims.test.tsx:1-30` for the canonical page-test shape.
|
|
||||||
6. **UI primitives.** Use Radix-backed components from `src/components/ui/` (`button.tsx`, `dialog.tsx`, `table.tsx`, `select.tsx`, `pagination.tsx`, `empty-state.tsx`, `error-state.tsx`, `filter-chips.tsx`, `skeleton.tsx`, `input.tsx`, `label.tsx`, `card.tsx`, `badge.tsx`, `skip-link.tsx`, `claim-state-badge.tsx`). Don't pull in a new UI library without discussion — every primitive here is already consumed by at least one shipped page.
|
|
||||||
7. **Routing.** Pages register their route in `src/App.tsx` as a `<Route path="<name>" element={<<Name>> />} />` inside the `<Layout>` element wrapper (`src/App.tsx:30-44`). Currently every page is a static import; switch to `React.lazy(() => import(...))` only if a page grows heavy (large parse/EDI libs, chart code) and the import cost shows up in the bundle report.
|
|
||||||
|
|
||||||
## Patterns
|
|
||||||
|
|
||||||
### `Claims.tsx`-style page (Layout + PageHeader + table + drawer)
|
|
||||||
|
|
||||||
Canonical page shape — composes the data hook, the tail triplet, and
|
|
||||||
`useDrawerUrlState` for the `?claim=…` deep-link. See
|
|
||||||
`src/pages/Claims.tsx:51-200` for the full file.
|
|
||||||
|
|
||||||
```tsx
|
|
||||||
import { useMemo, useState } from "react";
|
|
||||||
import { Table, TableBody, TableRow, /* … */ } from "@/components/ui/table";
|
|
||||||
import { PageHeader } from "@/components/PageHeader";
|
|
||||||
import { ClaimDrawer } from "@/components/ClaimDrawer";
|
|
||||||
import { useClaims } from "@/hooks/useClaims";
|
|
||||||
import { useDrawerUrlState } from "@/hooks/useDrawerUrlState";
|
|
||||||
import { useTailStream } from "@/hooks/useTailStream";
|
|
||||||
import { useMergedTail } from "@/hooks/useMergedTail";
|
|
||||||
import { TailStatusPill } from "@/components/TailStatusPill";
|
|
||||||
|
|
||||||
export function Claims() {
|
|
||||||
const [status, setStatus] = useState<ClaimStatus | null>(null);
|
|
||||||
const params = useMemo(() => ({ status, limit: 25, offset: 0 }), [status]);
|
|
||||||
|
|
||||||
const { data } = useClaims(params);
|
|
||||||
const { status: tailStatus, lastEventAt, forceReconnect } = useTailStream("claims");
|
|
||||||
const items = useMergedTail("claims", data?.items ?? [], (c) => !status || c.status === status);
|
|
||||||
const { claimId, openClaim, closeClaim } = useDrawerUrlState();
|
|
||||||
|
|
||||||
return (
|
|
||||||
<>
|
|
||||||
<PageHeader
|
|
||||||
eyebrow="Inbox"
|
|
||||||
title="Claims"
|
|
||||||
status={<TailStatusPill status={tailStatus} lastEventAt={lastEventAt} onReconnect={forceReconnect} />}
|
|
||||||
/>
|
|
||||||
<Table>
|
|
||||||
<TableBody>
|
|
||||||
{items.map((c) => (
|
|
||||||
<TableRow key={c.id} onClick={() => openClaim(c.id)}>{/* …cells… */}</TableRow>
|
|
||||||
))}
|
|
||||||
</TableBody>
|
|
||||||
</Table>
|
|
||||||
<ClaimDrawer claimId={claimId} claims={items} onClose={closeClaim} onNavigate={openClaim} />
|
|
||||||
</>
|
|
||||||
);
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### `use<X>` data hook (TanStack Query)
|
|
||||||
|
|
||||||
Pattern from `src/hooks/useClaims.ts:24-67` and `useRemittances.ts:21-65`.
|
|
||||||
Returns a stable shape so the page treats all data hooks uniformly.
|
|
||||||
The tail subscription lives on the page (see Convention 3), not here.
|
|
||||||
|
|
||||||
```ts
|
|
||||||
import { useQuery } from "@tanstack/react-query";
|
|
||||||
import { api, type ListClaimsParams, type PaginatedResponse } from "@/lib/api";
|
|
||||||
import type { Claim } from "@/types";
|
|
||||||
|
|
||||||
export function useClaims(params: ListClaimsParams) {
|
|
||||||
return useQuery<PaginatedResponse<Claim>>({
|
|
||||||
queryKey: ["claims", params],
|
|
||||||
queryFn: () => api.listClaims<Claim>(params),
|
|
||||||
enabled: api.isConfigured,
|
|
||||||
// …in-memory fallback when !api.isConfigured…
|
|
||||||
});
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Drawer component with `useDrawerUrlState`
|
|
||||||
|
|
||||||
Pattern from `src/components/ClaimDrawer/ClaimDrawer.tsx:1-50` +
|
|
||||||
`src/hooks/useDrawerUrlState.ts`. The drawer takes `claimId` as a
|
|
||||||
prop (driven by the URL), renders nothing when `null`, and uses
|
|
||||||
`useDrawerKeyboard` for j/k navigation + Escape to close. The page
|
|
||||||
that mounts it controls the URL — `openClaim("abc")` sets `?claim=abc`,
|
|
||||||
removing the param closes the drawer, and reload preserves the state.
|
|
||||||
|
|
||||||
```tsx
|
|
||||||
import { Dialog, DialogContent } from "@/components/ui/dialog";
|
|
||||||
import { useClaimDetail } from "@/hooks/useClaimDetail";
|
|
||||||
import { useDrawerKeyboard } from "@/hooks/useDrawerKeyboard";
|
|
||||||
|
|
||||||
export function ClaimDrawer({ claimId, claims, onClose, onNavigate, onToggleHelp }) {
|
|
||||||
const open = claimId !== null;
|
|
||||||
const { data, isLoading, isError, error } = useClaimDetail(claimId);
|
|
||||||
useDrawerKeyboard({ open, claims, onClose, onNavigate, onToggleHelp });
|
|
||||||
if (!open) return null;
|
|
||||||
return (
|
|
||||||
<Dialog open onOpenChange={(o) => !o && onClose()}>
|
|
||||||
<DialogContent>{/* header + body panels from useClaimDetail… */}</DialogContent>
|
|
||||||
</Dialog>
|
|
||||||
);
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Anti-patterns
|
|
||||||
|
|
||||||
- **Don't `fetch` from inside a page component.** All API access goes through the `use<X>` hook in `src/hooks/use<X>.ts`. Pages that reach for `fetch(...)` directly can't be tested with `vi.mock("@/lib/api", ...)` and split the data lifecycle across files. The hook returns `{ data, isLoading, isError, error, refetch }` so the page is a pure renderer.
|
|
||||||
- **Don't open a drawer via local component state.** A `const [open, setOpen] = useState(false)` for a drawer breaks deep-links and reload-restore. Use `useDrawerUrlState()` (claim) or `useRemitDrawerUrlState()` (remit) so the URL is the single source of truth.
|
|
||||||
- **Don't put domain logic in JSX.** Conditional renderings, table sorting, and KPI math all belong in the `use<X>` hook or a pure helper under `src/lib/` (e.g. `src/lib/format.ts` for currency / date formatting). JSX is for layout; mixing in `items.filter(...).sort(...)` inline is hard to test and hides behavior from the hook.
|
|
||||||
- **Don't call `useTailStream` from inside a `use<X>` hook.** The streaming subscription belongs on the page (see `src/pages/Claims.tsx:85-89`). Hoisting it into `useClaims` couples the open/close lifecycle of the page to whoever mounts the hook, and breaks the one-resource-one-page ownership that the backoff/stall state machine assumes.
|
|
||||||
- **Don't import a new UI library to render a button, modal, or table.** Radix-backed primitives in `src/components/ui/` already cover every widget the shipped pages use. Reach for a new library only after the primitive gap is real and the proposal is in a spec / PR.
|
|
||||||
|
|
||||||
## Related skills
|
|
||||||
|
|
||||||
- **`cyclone-tail`** — load for any live-data page (Claims, Remittances, ActivityLog). Documents the `useTailStream` + `useMergedTail` triplet, the `<TailStatusPill>` wiring, and the 30s stall threshold (`STALL_TIMEOUT_MS = 30_000` at `useTailStream.ts:53`).
|
|
||||||
- **`cyclone-api-router`** — load when a page is calling a new HTTP endpoint. Documents `api_routers/<topic>.py` vs. inline `api.py` registration and the response / error-envelope shapes.
|
|
||||||
- **`cyclone-tests`** — load when adding the `*.test.tsx` sibling. Documents the `// @vitest-environment happy-dom` setup, `IS_REACT_ACT_ENVIRONMENT = true`, the `@testing-library/react` vs. `createRoot`+`Probe` rendering styles, and the `vi.mock("@/lib/api")` convention.
|
|
||||||
- **`cyclone-edi`** — load when a page renders parsed 837P / 835 / 999 / 270 / 271 / 277CA / TA1 content (ServiceLinesTable, CAS panels, ValidationPanel). Documents the parser modules, the R-coded validator rules, and the CAS / CARC / NPI / EIN helpers.
|
|
||||||
@@ -1,163 +0,0 @@
|
|||||||
---
|
|
||||||
name: cyclone-spec
|
|
||||||
description: "Cyclone SP-N superpowers increment flow — spec → plan → implement → merge. Use when: starting a new numbered feature increment, naming a branch, opening a SP-N PR, or doing the merge dance into main."
|
|
||||||
---
|
|
||||||
|
|
||||||
# cyclone-spec
|
|
||||||
|
|
||||||
The Cyclone repo ships every new feature as a numbered **SP-N increment**:
|
|
||||||
a spec, a plan, an implementation branch, and a single atomic merge commit
|
|
||||||
into `main`. This skill encodes the conventions so every increment follows
|
|
||||||
the same shape and the commit history stays auditable.
|
|
||||||
|
|
||||||
As of this writing: **17 specs** in `docs/superpowers/specs/`, **13 plans**
|
|
||||||
in `docs/superpowers/plans/`, and SP numbers used through **SP22**. **SP23**
|
|
||||||
is the Ubuntu + Docker + RBAC product fork (awaiting user decision);
|
|
||||||
**SP24** is the auth-posture alignment (docs-only). The next free increment
|
|
||||||
is **SP25** after SP24 lands.
|
|
||||||
|
|
||||||
## Auth-aware spec template (SP24)
|
|
||||||
|
|
||||||
The threat-model section in the canonical SP-N spec template (`## 1. Scope`, second-to-last bullet) used to read "no second party to authenticate; no second host to harden against." **That phrasing is stale as of 2026-06-23** — the auth work landed in `main` and every backend endpoint requires login. New specs should instead state the auth boundary explicitly: "the auth boundary is HTTP (login required, bcrypt + HttpOnly session cookie); file-system threats remain the local-only threat model (SQLCipher at rest, macOS Keychain). SP23 changes the threat model to LAN-bound remote operator." Reference: [`docs/superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md`](../../../docs/superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md).
|
|
||||||
|
|
||||||
## When to use
|
|
||||||
|
|
||||||
- **Starting a new feature increment.** You are about to add a numbered
|
|
||||||
feature, fix that crosses subsystem boundaries, or anything bigger than
|
|
||||||
a one-line change. Before you write code, reserve the next SP number and
|
|
||||||
write the spec.
|
|
||||||
- **Naming the spec / plan files or the branch.** You have a topic, a
|
|
||||||
date, and a number — and you need the exact path / branch shape so
|
|
||||||
existing scripts and reviewers can find the artifacts.
|
|
||||||
- **Opening the SP-N PR.** You are about to push the branch and need the
|
|
||||||
PR title format and the commit-prefix conventions so the merge commit
|
|
||||||
reads cleanly.
|
|
||||||
- **Doing the merge dance.** Review is approved and you're about to land
|
|
||||||
the branch into `main`. Use this skill to confirm the merge shape — no
|
|
||||||
squash, no rebase, one atomic merge commit.
|
|
||||||
|
|
||||||
## Conventions
|
|
||||||
|
|
||||||
1. **Numbering.** Reserve the next SP-N number — the next integer after
|
|
||||||
the highest `SP<n>` already used in `git log`. Never reuse a number,
|
|
||||||
even after deletion. Numbering is monotonic and lives in the merge
|
|
||||||
history.
|
|
||||||
2. **Branch.** `sp<N>-<short-kebab-topic>` — e.g. `sp22-line-reconciliation`,
|
|
||||||
`sp9-multi-payer-npi`. Kebab-case, lowercase, no spaces, no slashes.
|
|
||||||
The branch name is the canonical handle for the increment.
|
|
||||||
3. **Spec path.** `docs/superpowers/specs/YYYY-MM-DD-cyclone-<topic>-design.md`
|
|
||||||
with header `Status: Draft, pending user review`. One spec per
|
|
||||||
increment. Real examples: `2026-06-19-cyclone-db-reconciliation-design.md`
|
|
||||||
(SP3), `2026-06-20-cyclone-multi-payer-npi-sftp-design.md` (SP9).
|
|
||||||
4. **Plan path.** `docs/superpowers/plans/YYYY-MM-DD-cyclone-<topic>.md`.
|
|
||||||
Header per the upstream `superpowers:writing-plans` skill: a
|
|
||||||
`For agentic workers:` line that names
|
|
||||||
`superpowers:subagent-driven-development` or
|
|
||||||
`superpowers:executing-plans`, plus a `Goal / Architecture / Tech
|
|
||||||
Stack / Spec` metadata block, then numbered tasks with
|
|
||||||
`- [ ] Step N:` checkboxes.
|
|
||||||
5. **Commit prefix.** All commits on the branch follow these prefixes —
|
|
||||||
they make the SP-N merge commit readable and let `git log --grep`
|
|
||||||
filter cleanly:
|
|
||||||
- `feat(sp<N>): …` — implementation commits (e.g. `feat(sp20): NPI Luhn checksum + Tax ID format validation`).
|
|
||||||
- `docs(spec): …` — landing the spec (e.g. `docs(spec): design for CycloneStore split (Step 4)`).
|
|
||||||
- `docs(plan): …` — landing the plan.
|
|
||||||
- `merge: SP<N> <topic> into main` — the merge commit itself (e.g. `merge: SP14 5-lane Inbox UI + acknowledge action into main`).
|
|
||||||
6. **PR title.** `SP<N> <Topic>` — e.g. `SP22 Line reconciliation`.
|
|
||||||
Matches the merge-commit subject so GitHub's "merged PR" view and the
|
|
||||||
`git log` entry are identical strings.
|
|
||||||
7. **Merge shape.** A single atomic merge commit into `main` after
|
|
||||||
review. **No squash** — squash collapses the per-commit history and
|
|
||||||
breaks the SP-N audit trail. **No rebase** — rebase rewrites the SHAs
|
|
||||||
the PR review was performed against. The SP-N merge commit *is* the
|
|
||||||
record of the increment landing.
|
|
||||||
|
|
||||||
## Patterns
|
|
||||||
|
|
||||||
### Spec header (canonical SP-N shape, post-SP9)
|
|
||||||
|
|
||||||
This is the canonical header for new specs. Older specs (pre-SP9) deviate
|
|
||||||
slightly — different title style, no Branch or Aesthetic direction line —
|
|
||||||
and have not been retroactively normalized. **Use this template for any
|
|
||||||
new SP-N spec.**
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
# Sub-project <N> — <Topic>: Design Spec
|
|
||||||
|
|
||||||
**Date:** YYYY-MM-DD
|
|
||||||
**Status:** Draft, awaiting user sign-off
|
|
||||||
**Branch:** `sp<N>-<short-kebab-topic>`
|
|
||||||
**Aesthetic direction:** <one line — e.g. "No new UI" or "Modern (geometric sans + bold borders + electric blue accent)">
|
|
||||||
|
|
||||||
## 1. Scope
|
|
||||||
<2-6 lines: what's in, what's out, with explicit out-of-scope list>
|
|
||||||
```
|
|
||||||
|
|
||||||
Worked example (matches this template): `docs/superpowers/specs/2026-06-20-cyclone-multi-payer-npi-sftp-design.md`.
|
|
||||||
|
|
||||||
### Plan header (every SP-N plan starts with this)
|
|
||||||
|
|
||||||
```markdown
|
|
||||||
# <Topic> Implementation Plan
|
|
||||||
|
|
||||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use
|
|
||||||
> superpowers:subagent-driven-development (recommended) or
|
|
||||||
> superpowers:executing-plans to implement this plan task-by-task. Steps
|
|
||||||
> use checkbox (`- [ ]`) syntax for tracking.
|
|
||||||
|
|
||||||
**Goal:** <one sentence — the outcome>
|
|
||||||
**Architecture:** <one paragraph — how it's structured>
|
|
||||||
**Tech Stack:** <comma-separated list>
|
|
||||||
**Spec:** [`docs/superpowers/specs/YYYY-MM-DD-cyclone-<topic>-design.md`](../specs/...)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## File structure
|
|
||||||
<tree of new / modified files>
|
|
||||||
|
|
||||||
## Task 0: <setup>
|
|
||||||
## Task 1: <first user-visible step>
|
|
||||||
…
|
|
||||||
```
|
|
||||||
|
|
||||||
Real examples: `docs/superpowers/plans/2026-06-21-cyclone-skill-catalog.md`,
|
|
||||||
`docs/superpowers/plans/2026-06-21-cyclone-store-split.md`.
|
|
||||||
|
|
||||||
## Anti-patterns
|
|
||||||
|
|
||||||
- **Don't skip the spec ("it's a small fix").** Small fixes still get a
|
|
||||||
3-line spec when they introduce a new numbered increment. The spec is
|
|
||||||
the *what* and the audit trail; the plan is the *how*. Without a spec
|
|
||||||
the merge commit has no anchor.
|
|
||||||
- **Don't squash the merge commit.** The SP-N merge commit is the audit
|
|
||||||
trail — it tells future you exactly which feature landed and which
|
|
||||||
commits composed it. Squash collapses that into one opaque commit and
|
|
||||||
the per-commit history is lost.
|
|
||||||
- **Don't put code in the spec — the spec is the *what*, the plan is the
|
|
||||||
*how*.** Specs describe scope, goals, non-goals, and decisions. Code
|
|
||||||
snippets belong in the plan (with checkbox steps) or in the diff, not
|
|
||||||
in the spec. SP-N specs in this repo routinely have **zero** code
|
|
||||||
blocks.
|
|
||||||
|
|
||||||
## Related skills
|
|
||||||
|
|
||||||
- **`cyclone-tests`** — every spec lists test impact; load this when
|
|
||||||
drafting or reviewing the spec to confirm fixture / `.test.tsx`
|
|
||||||
implications.
|
|
||||||
- **`cyclone-edi`** — load when the SP-N increment touches an EDI parser,
|
|
||||||
validator rule, or CAS mapping.
|
|
||||||
- **`cyclone-tail`** — load when the increment changes the live-tail wire
|
|
||||||
format or adds a streaming page.
|
|
||||||
- **`cyclone-store`** — load when the increment adds a write-path,
|
|
||||||
touches `store.py`, or wires a new `<entity>_written` event.
|
|
||||||
- **`cyclone-api-router`** — load when the increment adds or changes an
|
|
||||||
HTTP endpoint in `api_routers/`.
|
|
||||||
- **`cyclone-frontend-page`** — load when the increment adds or
|
|
||||||
refactors a page in `src/pages/`.
|
|
||||||
- **`cyclone-cli`** — load when the increment adds a CLI subcommand or
|
|
||||||
changes exit codes.
|
|
||||||
- **`superpowers:brainstorming`** (global) — run before the spec to lock
|
|
||||||
the scope / decisions in the spec's `## Decisions (locked during
|
|
||||||
brainstorming)` section.
|
|
||||||
- **`superpowers:writing-plans`** (global) — produces the plan header
|
|
||||||
format every SP-N plan follows.
|
|
||||||
@@ -1,200 +0,0 @@
|
|||||||
---
|
|
||||||
name: cyclone-store
|
|
||||||
description: "Cyclone store write-paths, the pubsub event contract (claim_written / remittance_written / activity_recorded), and the SP21 store-split boundary map. Use when: touching store.py, adding a new entity, wiring a new write event, or splitting a store module."
|
|
||||||
---
|
|
||||||
|
|
||||||
# cyclone-store
|
|
||||||
|
|
||||||
Cyclone persists every parsed X12 batch through one facade,
|
|
||||||
`CycloneStore` (`backend/src/cyclone/store.py:882`). Every write
|
|
||||||
inserts the row AND publishes a pubsub event on the in-process
|
|
||||||
`EventBus` (`backend/src/cyclone/pubsub.py:20`) so live-tail pages
|
|
||||||
see new rows the moment they land. The event contract is the seam
|
|
||||||
between persistence and streaming — a wrong event name silently
|
|
||||||
goes stale.
|
|
||||||
|
|
||||||
As of this writing: the store is a **single 2412-line module** and
|
|
||||||
the SP21 split into `backend/src/cyclone/store/` is in progress.
|
|
||||||
Three event kinds: `claim_written`, `remittance_written`,
|
|
||||||
`activity_recorded`. Next: **SP22**.
|
|
||||||
|
|
||||||
## When to use
|
|
||||||
|
|
||||||
- **Adding a new entity.** You need a new ORM model + write method +
|
|
||||||
event kind + snapshot serializer and want to know where each piece
|
|
||||||
lives (current monolith vs. post-SP21 module).
|
|
||||||
- **Wiring a new write event.** You're adding a `<entity>_written`
|
|
||||||
event and need both the publish call in the store AND the
|
|
||||||
subscribe call in the live-tail endpoint to stay in sync.
|
|
||||||
- **Debugging a write-path issue.** A page isn't reflecting new
|
|
||||||
rows, the DB has the row but the stream is silent, or the publish
|
|
||||||
raises and rolls back the transaction.
|
|
||||||
- **Splitting a store module.** You're moving a domain out of
|
|
||||||
`store.py` into its own module under `backend/src/cyclone/store/`
|
|
||||||
and need the SP21 module list + facade re-export rules.
|
|
||||||
|
|
||||||
## Conventions
|
|
||||||
|
|
||||||
1. **All writes go through `CycloneStore`.** Route handlers and
|
|
||||||
parsers must not write directly to the ORM session. The facade
|
|
||||||
opens a short-lived session via `db.SessionLocal()()`. Direct ORM
|
|
||||||
access is the #1 way the live-tail contract gets bypassed (no
|
|
||||||
event published → page silently goes stale). Read paths are
|
|
||||||
similar — prefer the facade's `iter_*` / `get_*` methods over raw
|
|
||||||
`s.execute(select(...))`.
|
|
||||||
2. **Every write publishes an event.** The event name matches the
|
|
||||||
entity: `claim_written`, `remittance_written`,
|
|
||||||
`activity_recorded` (the trailing `_recorded` signals a
|
|
||||||
non-canonical row — activity events are derived, not first-class;
|
|
||||||
current names at `store.py:1072,1081,1096`). New entities get
|
|
||||||
`<entity>_written`; activity-style side rows get
|
|
||||||
`<entity>_recorded`. Publish is **best-effort** — failures are
|
|
||||||
logged but never roll back the persisted batch
|
|
||||||
(`store.py:1097-1098`).
|
|
||||||
3. **Snapshot shape.** Each entity has a `to_ui_<entity>` serializer
|
|
||||||
(plain Python function returning a dict) — currently
|
|
||||||
`to_ui_claim`, `to_ui_remittance`, `to_ui_claim_from_orm`,
|
|
||||||
`to_ui_remittance_from_orm`, `to_ui_provider`. Post-SP21 these
|
|
||||||
move to `backend/src/cyclone/store/ui.py`. The serializer is the
|
|
||||||
single source of truth for what the frontend sees — every event
|
|
||||||
payload MUST match what the matching list endpoint returns for
|
|
||||||
that row (`store.py:1052-1054`).
|
|
||||||
4. **SP21 boundaries.** Post-split, each domain lives in its own
|
|
||||||
module under `backend/src/cyclone/store/`: `__init__.py` (facade
|
|
||||||
+ `CycloneStore` class), `exceptions.py`, `records.py`,
|
|
||||||
`orm_builders.py`, `ui.py`, `write.py`, `batches.py`,
|
|
||||||
`claim_detail.py`, `acks.py`, `backups.py`, `inbox.py`,
|
|
||||||
`providers.py`. Cross-module writes go through `CycloneStore`
|
|
||||||
facade methods, not direct module access. The facade re-exports
|
|
||||||
every name callers currently import from `cyclone.store`. Full
|
|
||||||
list: `docs/superpowers/plans/2026-06-21-cyclone-store-split.md:25-37`.
|
|
||||||
5. **No business logic in route handlers.** A route handler validates
|
|
||||||
input, calls `store.<method>(...)`, passes the `event_bus`,
|
|
||||||
returns the serialized result. Reconciliation, idempotency
|
|
||||||
checks, CAS adjustment persistence — all live in the store.
|
|
||||||
|
|
||||||
## Patterns
|
|
||||||
|
|
||||||
### A `CycloneStore.add` write — publishes events from inserted rows
|
|
||||||
|
|
||||||
Taken from `backend/src/cyclone/store.py:898-1107`. The method opens
|
|
||||||
a session, inserts rows, then runs a sync `_publish_events_sync`
|
|
||||||
after commit so subscribers can immediately re-fetch consistent data.
|
|
||||||
|
|
||||||
```python
|
|
||||||
def add(
|
|
||||||
self,
|
|
||||||
record: BatchRecord,
|
|
||||||
*,
|
|
||||||
event_bus: "EventBus | None" = None,
|
|
||||||
) -> None:
|
|
||||||
inserted_claim_ids: list[str] = []
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
s.add(Batch(id=record.id, kind=record.kind, ...))
|
|
||||||
if isinstance(record, BatchRecord837):
|
|
||||||
for claim in record.result.claims:
|
|
||||||
if s.get(Claim, claim.claim_id) is not None:
|
|
||||||
continue # idempotency: skip dupes
|
|
||||||
s.add(_claim_837_row(claim, record.id))
|
|
||||||
s.add(ActivityEvent(kind="claim_submitted", ...))
|
|
||||||
inserted_claim_ids.append(claim.claim_id)
|
|
||||||
# ... 835 branch + flush + cas adjustments ...
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
if event_bus is not None and inserted_claim_ids:
|
|
||||||
self._publish_events_sync(event_bus, record, inserted_claim_ids)
|
|
||||||
|
|
||||||
def _publish_events_sync(self, event_bus, record, claim_ids):
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
for cid in claim_ids:
|
|
||||||
ui = to_ui_claim_from_orm(s.get(Claim, cid), ...)
|
|
||||||
self._sync_publish(event_bus, "claim_written", ui)
|
|
||||||
# ... remittance + activity loops ...
|
|
||||||
```
|
|
||||||
|
|
||||||
`EventBus.publish` is async but the body is pure sync `put_nowait`,
|
|
||||||
so the store calls `_sync_publish` directly to avoid forcing sync
|
|
||||||
FastAPI handlers to await.
|
|
||||||
|
|
||||||
### Backend `/api/<resource>/stream` endpoint — subscribes to the event
|
|
||||||
|
|
||||||
Taken from `backend/src/cyclone/api.py:1380-1401`. Two phases — eager
|
|
||||||
snapshot, then live subscription — wrapped in `StreamingResponse`
|
|
||||||
with `media_type="application/x-ndjson"`.
|
|
||||||
|
|
||||||
```python
|
|
||||||
@app.get("/api/claims/stream")
|
|
||||||
async def claims_stream(request: Request, ...) -> StreamingResponse:
|
|
||||||
bus: EventBus = request.app.state.event_bus
|
|
||||||
|
|
||||||
async def gen() -> AsyncIterator[bytes]:
|
|
||||||
rows = store.iter_claims(status=status, ...) # 1. Snapshot
|
|
||||||
for row in rows:
|
|
||||||
yield _ndjson_line({"type": "item", "data": row})
|
|
||||||
yield _ndjson_line({"type": "snapshot_end", "data": {"count": len(rows)}})
|
|
||||||
|
|
||||||
async for chunk in _tail_events(request, bus, ["claim_written"]): # 2. Live
|
|
||||||
yield chunk
|
|
||||||
|
|
||||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
|
||||||
```
|
|
||||||
|
|
||||||
`_ndjson_line` and `_tail_events` live in
|
|
||||||
`backend/src/cyclone/api_helpers.py`. The `["remittance_written"]`
|
|
||||||
and `["activity_recorded"]` subscriptions are at `api.py:1891,2002`.
|
|
||||||
|
|
||||||
### Backend test — asserts both the row AND the event landed
|
|
||||||
|
|
||||||
The autouse `conftest.py` (`backend/tests/conftest.py:20`) wires a
|
|
||||||
fresh `EventBus` onto `app.state` per test.
|
|
||||||
|
|
||||||
```python
|
|
||||||
def test_publishes_claim_written_event(client: TestClient) -> None:
|
|
||||||
bus = app.state.event_bus
|
|
||||||
resp = client.post("/api/parse-837",
|
|
||||||
files={"file": ("x.837", MINIMAL_837, "text/plain")})
|
|
||||||
assert resp.status_code == 200
|
|
||||||
assert list(store.iter_claims(limit=10)) # row landed
|
|
||||||
|
|
||||||
queues = bus._subscribers.get("claim_written", []) # event published
|
|
||||||
assert queues
|
|
||||||
evt = queues[0].get_nowait()
|
|
||||||
assert evt["_kind"] == "claim_written"
|
|
||||||
```
|
|
||||||
|
|
||||||
## Anti-patterns
|
|
||||||
|
|
||||||
- **Don't read directly from the ORM in a route handler — go through
|
|
||||||
the snapshot serializer.** A handler that does
|
|
||||||
`with db.SessionLocal()() as s: row = s.get(Claim, cid); return row`
|
|
||||||
bypasses `to_ui_claim_from_orm` and silently drifts from the event
|
|
||||||
payload shape. Always call `store.get_claim_detail(cid)` (or the
|
|
||||||
equivalent `get_*` facade method).
|
|
||||||
- **Don't introduce a new event name without updating the subscriber
|
|
||||||
list.** Today every `<entity>_written` event has exactly one
|
|
||||||
consumer — the matching `/api/<resource>/stream` endpoint,
|
|
||||||
currently in `backend/src/cyclone/api.py` (claims at `:1398`,
|
|
||||||
remittances at `:1891`, activity at `:2002`). Any future split
|
|
||||||
into `backend/src/cyclone/api_routers/` must wire the same
|
|
||||||
subscription.
|
|
||||||
- **Don't merge a write method with its event publication into
|
|
||||||
separate places.** `_publish_events_sync` lives next to `add` in
|
|
||||||
`store.py:1042-1107` so reviewers see both halves of the contract
|
|
||||||
in one diff. Post-SP21 the same rule applies.
|
|
||||||
- **Don't make the publish call blocking on commit failures.**
|
|
||||||
Publish errors are caught and logged at `store.py:1097-1098`; a
|
|
||||||
failing subscriber MUST NOT roll back the persisted batch. The
|
|
||||||
batch is the source of truth; the event is the cache-invalidation
|
|
||||||
hint.
|
|
||||||
|
|
||||||
## Related skills
|
|
||||||
|
|
||||||
- **`cyclone-edi`** — the parsed `ClaimOutput` / `ParseResult<EDI>`
|
|
||||||
lands in the store via `CycloneStore.add`.
|
|
||||||
- **`cyclone-api-router`** — the route that calls `store.<method>(...)`
|
|
||||||
and (for stream endpoints) subscribes to the matching
|
|
||||||
`<entity>_written` event.
|
|
||||||
- **`cyclone-tail`** — the consumer side of the event contract;
|
|
||||||
load when changing the wire format or adding a streaming hook.
|
|
||||||
- **`cyclone-tests`** — write-path tests live under `backend/tests/`
|
|
||||||
and assert both the DB row and the event payload.
|
|
||||||
@@ -1,200 +0,0 @@
|
|||||||
---
|
|
||||||
name: cyclone-tail
|
|
||||||
description: "Cyclone live-tail streaming wire format and the useTailStream / useMergedTail hook triplet. Use when: adding a new streaming list page, changing the wire format, debugging stalled/reconnecting state, or modifying the StatusPill behavior."
|
|
||||||
---
|
|
||||||
|
|
||||||
# cyclone-tail
|
|
||||||
|
|
||||||
Cyclone keeps the Claims, Remittances, and Activity pages live without
|
|
||||||
polling: every store write publishes an internal EventBus event, the
|
|
||||||
page opens a `GET /api/<resource>/stream` HTTP/1.1 chunked-NDJSON
|
|
||||||
connection, and new rows land in the table the moment they hit the
|
|
||||||
database. This skill codifies the wire format, the hook triplet, and
|
|
||||||
the backoff/stall machinery so additions stay consistent with the
|
|
||||||
three streaming pages already shipped (`Claims`, `Remittances`,
|
|
||||||
`ActivityLog`).
|
|
||||||
|
|
||||||
## When to use
|
|
||||||
|
|
||||||
- **Adding a new streaming page.** Mounting `useTailStream(resource)`
|
|
||||||
on a page — you need the hook triplet shape (initial fetch +
|
|
||||||
`useTailStream` + `useMergedTail`), the `<TailStatusPill>` wiring,
|
|
||||||
and the dedup rules in `useMergedTail`.
|
|
||||||
- **Changing the wire format.** Adding a new event type — update
|
|
||||||
`TailEvent` in `src/lib/tail-stream.ts:22-44`, the dispatch switch
|
|
||||||
in `src/hooks/useTailStream.ts:173-195`, the emitter in
|
|
||||||
`backend/src/cyclone/api_helpers.py:tail_events()`, and
|
|
||||||
`references/wire-format.md`.
|
|
||||||
- **Debugging stalled/reconnecting state.** Confirm whether the
|
|
||||||
backend is heartbeating (`CYCLONE_TAIL_HEARTBEAT_S`, default `15s`)
|
|
||||||
or the stall timer fired (`STALL_TIMEOUT_MS = 30_000` at
|
|
||||||
`src/hooks/useTailStream.ts:53`).
|
|
||||||
- **Tuning heartbeat/stall timing.** Changing the 30s stall threshold
|
|
||||||
or the 15s heartbeat interval — README's "Status pill" + "Knobs"
|
|
||||||
tables need to stay in sync (`README.md:109-129`).
|
|
||||||
|
|
||||||
## Conventions
|
|
||||||
|
|
||||||
1. **Wire format.** Newline-delimited JSON. Every line is
|
|
||||||
`{"type": ..., "data": ...}`. Known `type` values: `item`
|
|
||||||
(per-row envelope), `snapshot_end` (`{"count": N}` marker after the
|
|
||||||
snapshot), `heartbeat` (`{"ts": "<iso-8601>"}` keep-alive),
|
|
||||||
`item_dropped` (`{"id": "..."}` queue-overflow notice), `error`
|
|
||||||
(`{"message": "..."}` promoted to a thrown error by the hook).
|
|
||||||
Defined at `src/lib/tail-stream.ts:22-44`; emitted by
|
|
||||||
`backend/src/cyclone/api.py:1357-2006`. See
|
|
||||||
`references/wire-format.md`.
|
|
||||||
2. **Hook triplet.** Streaming pages compose three pieces:
|
|
||||||
`useTailStream(resource)` (`src/hooks/useTailStream.ts:80` — opens
|
|
||||||
the stream, drives the backoff/stall state machine, dispatches
|
|
||||||
`item` events into `useTailStore`),
|
|
||||||
`useMergedTail(resource, baseItems, filterFn?)`
|
|
||||||
(`src/hooks/useMergedTail.ts:25` — returns
|
|
||||||
`baseItems + tailSlice` dedup'd against `baseItems`), and a
|
|
||||||
per-resource initial-fetch hook (`useClaims`, `useRemittances`,
|
|
||||||
`useActivity`). The page wires all three — see
|
|
||||||
`src/pages/Claims.tsx:87-89`.
|
|
||||||
3. **Stall threshold.** 30 seconds of total silence — heartbeats
|
|
||||||
included — flips status to `stalled` and surfaces the
|
|
||||||
`↻ Reconnect` button on `<TailStatusPill>`. Constant:
|
|
||||||
`STALL_TIMEOUT_MS = 30_000` at `src/hooks/useTailStream.ts:53`.
|
|
||||||
Re-armed on every event including `heartbeat` and `item_dropped`
|
|
||||||
(`useTailStream.ts:124-140`). Don't change without updating
|
|
||||||
`README.md:109-123`.
|
|
||||||
4. **Snapshot first.** Every stream emits the snapshot before any
|
|
||||||
live `item` events, then closes with exactly one `snapshot_end`
|
|
||||||
carrying `{"count": N}`. The hook uses `snapshot_end` to flip
|
|
||||||
`<TailStatusPill>` from `connecting` to `live` and to reset the
|
|
||||||
reconnect backoff counter (`useTailStream.ts:174-180`).
|
|
||||||
5. **Content-Type.** Stream endpoints respond with
|
|
||||||
`media_type="application/x-ndjson"` — see
|
|
||||||
`backend/src/cyclone/api.py:1401,1894,2005`. Frontend sets
|
|
||||||
`Accept: application/x-ndjson` at `src/lib/tail-stream.ts:67-70`.
|
|
||||||
Never `application/json` for a stream endpoint.
|
|
||||||
6. **Backoff.** Transient errors retry with `1s → 2s → 4s → 8s → 16s
|
|
||||||
→ 30s` capped — `BACKOFF_STEPS_MS` at
|
|
||||||
`src/hooks/useTailStream.ts:48-50`. Counter resets on every
|
|
||||||
`snapshot_end`.
|
|
||||||
7. **Heartbeat knob.** Idle heartbeat interval is configurable via
|
|
||||||
`CYCLONE_TAIL_HEARTBEAT_S` (default `15`, parsed at call time in
|
|
||||||
`backend/src/cyclone/api_helpers.py:185-198`). Tests override to
|
|
||||||
a small value to keep runtime bounded.
|
|
||||||
|
|
||||||
## Patterns
|
|
||||||
|
|
||||||
### Page-hook skeleton — `Claims.tsx`
|
|
||||||
|
|
||||||
Pattern from `src/pages/Claims.tsx:85-90`. Three hooks in order:
|
|
||||||
`useClaims(params)` (initial TanStack Query fetch),
|
|
||||||
`useTailStream("claims")` (opens the stream, owns status state), and
|
|
||||||
`useMergedTail("claims", data?.items ?? [], tailFilterFn)` (combines
|
|
||||||
initial snapshot with live tail, dedup'd by id, filtered by the page's
|
|
||||||
predicate applied AFTER dedup at `useMergedTail.ts:72-74`).
|
|
||||||
|
|
||||||
```ts
|
|
||||||
import { useClaims } from "@/hooks/useClaims";
|
|
||||||
import { useTailStream } from "@/hooks/useTailStream";
|
|
||||||
import { useMergedTail } from "@/hooks/useMergedTail";
|
|
||||||
|
|
||||||
export function ClaimsPage() {
|
|
||||||
const { data } = useClaims({ status: "submitted" });
|
|
||||||
const { status, lastEventAt, forceReconnect } = useTailStream("claims");
|
|
||||||
const tailFilterFn = (c: Claim) => c.status === "submitted";
|
|
||||||
const items = useMergedTail("claims", data?.items ?? [], tailFilterFn);
|
|
||||||
// <TailStatusPill status={status} lastEventAt={lastEventAt}
|
|
||||||
// onReconnect={forceReconnect} /> + <Table items={items} />
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Backend `/api/foo/stream` endpoint
|
|
||||||
|
|
||||||
Pattern from `backend/src/cyclone/api.py:1357-1401`. Register BEFORE
|
|
||||||
`/api/foo/{foo_id}` so the literal `stream` segment doesn't match as
|
|
||||||
an id. Two phases — eager snapshot, then live subscription — wrapped
|
|
||||||
in `StreamingResponse` with `media_type="application/x-ndjson"`.
|
|
||||||
|
|
||||||
```python
|
|
||||||
@app.get("/api/foo/stream")
|
|
||||||
async def foo_stream(
|
|
||||||
request: Request,
|
|
||||||
status: str | None = Query(None),
|
|
||||||
limit: int = Query(100, ge=1, le=1000),
|
|
||||||
) -> StreamingResponse:
|
|
||||||
bus: EventBus = request.app.state.event_bus
|
|
||||||
|
|
||||||
async def gen() -> AsyncIterator[bytes]:
|
|
||||||
# 1. Snapshot.
|
|
||||||
rows = store.iter_foos(status=status, limit=limit)
|
|
||||||
for row in rows:
|
|
||||||
yield _ndjson_line({"type": "item", "data": row})
|
|
||||||
yield _ndjson_line({"type": "snapshot_end", "data": {"count": len(rows)}})
|
|
||||||
# 2. Live subscription + heartbeats.
|
|
||||||
async for chunk in _tail_events(request, bus, ["foo_written"]):
|
|
||||||
yield chunk
|
|
||||||
|
|
||||||
return StreamingResponse(gen(), media_type="application/x-ndjson")
|
|
||||||
```
|
|
||||||
|
|
||||||
`_ndjson_line` and `_tail_events` live in
|
|
||||||
`backend/src/cyclone/api_helpers.py`; the latter forwards EventBus
|
|
||||||
events as `item` lines and emits `heartbeat` lines on the cadence
|
|
||||||
from `heartbeat_seconds()`.
|
|
||||||
|
|
||||||
### `<TailStatusPill>` wiring
|
|
||||||
|
|
||||||
Pattern from `src/components/TailStatusPill.tsx:22-83`. The pill
|
|
||||||
takes `status` + `lastEventAt` from `useTailStream` plus
|
|
||||||
`forceReconnect` so the `↻ Reconnect` button wires to the hook's
|
|
||||||
`reconnectNonce` bump (aborts and reopens the stream —
|
|
||||||
`useTailStream.ts:99-101`). The button only renders when
|
|
||||||
`status === "stalled" || status === "error"` (`TailStatusPill.tsx:52`).
|
|
||||||
|
|
||||||
## Anti-patterns
|
|
||||||
|
|
||||||
- **Don't hand-roll `fetch` + `ReadableStream` parsing in a page.**
|
|
||||||
All stream consumers go through `streamTail(resource, opts?)` at
|
|
||||||
`src/lib/tail-stream.ts:58`. The parser handles `TextDecoderStream`,
|
|
||||||
newline splitting, malformed-line tolerance (`console.warn` + skip),
|
|
||||||
the `KNOWN_TYPES` allowlist, and abort-on-signal semantics
|
|
||||||
(`tail-stream.ts:75,98,107`). Duplicating it loses all of those.
|
|
||||||
- **Don't change the wire format on one endpoint without updating
|
|
||||||
the others.** The parser (`src/lib/tail-stream.ts`) and the hook
|
|
||||||
dispatch switch (`useTailStream.ts:173-195`) are shared across all
|
|
||||||
three live streams. Adding a new event type means updating
|
|
||||||
`TailEvent`, `KNOWN_TYPES`, the dispatch `case`, the `armStall`
|
|
||||||
re-arm list, and the backend emitter — in that order.
|
|
||||||
- **Don't emit `item` events before `snapshot_end`.** The hook uses
|
|
||||||
`snapshot_end` as the marker to flip `<TailStatusPill>` from
|
|
||||||
`connecting` to `live` and to reset the reconnect backoff counter.
|
|
||||||
Emitting `item`s first lands rows in `useTailStore` but the UI
|
|
||||||
still reads "Connecting" — operators see a flash of stale state.
|
|
||||||
Emit snapshot, then `snapshot_end`, then live (`api.py:1386-1401`).
|
|
||||||
- **Don't change the 30s stall threshold without updating the README
|
|
||||||
"Status pill" table.** The constant
|
|
||||||
(`STALL_TIMEOUT_MS = 30_000` at `useTailStream.ts:53`) is the
|
|
||||||
contract the pill is documented against — a bump needs the matching
|
|
||||||
edit at `README.md:118`.
|
|
||||||
- **Don't add `useTailStream` calls from `useFoo.ts` hooks.**
|
|
||||||
Streaming connections belong on the page (`src/pages/Claims.tsx`,
|
|
||||||
`Remittances.tsx`, `ActivityLog.tsx`). Hoisting into `useFoo`
|
|
||||||
couples the lifecycle to whoever mounts it and breaks
|
|
||||||
one-resource-one-page ownership.
|
|
||||||
|
|
||||||
## Related skills
|
|
||||||
|
|
||||||
- **`cyclone-frontend-page`** — page components live in `src/pages/`;
|
|
||||||
load when adding or refactoring a streaming page to confirm the
|
|
||||||
route + `PageHeader` + table conventions.
|
|
||||||
- **`cyclone-api-router`** — endpoint conventions (`api_routers/` for
|
|
||||||
resource-group routers, `api.py` for the live-tail endpoints); load
|
|
||||||
when adding or changing an HTTP endpoint that surfaces streamed or
|
|
||||||
paginated data.
|
|
||||||
- **`cyclone-store`** — write-path conventions in `store.py` and the
|
|
||||||
pubsub event contract (`claim_written`, `remittance_written`,
|
|
||||||
`activity_recorded`); load when adding a new entity whose writes
|
|
||||||
should fan out to a stream endpoint.
|
|
||||||
- **`cyclone-tests`** — frontend `*.test.tsx` siblings cover
|
|
||||||
`useTailStream`, `useMergedTail`, `TailStatusPill`; backend
|
|
||||||
`test_api_stream_live.py` covers the three live-tail endpoints;
|
|
||||||
load when the increment changes wire-format behavior or adds a
|
|
||||||
streaming hook.
|
|
||||||
@@ -1,76 +0,0 @@
|
|||||||
# Live-tail wire format — field reference
|
|
||||||
|
|
||||||
The Cyclone live-tail NDJSON contract is owned by the frontend parser
|
|
||||||
(`src/lib/tail-stream.ts:22-44`) and the backend emitter
|
|
||||||
(`backend/src/cyclone/api_helpers.py:tail_events()` + the three
|
|
||||||
endpoints in `backend/src/cyclone/api.py:1357-2006`). This file is the
|
|
||||||
quick reference; the canonical prose lives in the README and the
|
|
||||||
source-of-truth type definitions.
|
|
||||||
|
|
||||||
## Source
|
|
||||||
|
|
||||||
Wire format excerpt copied verbatim from `README.md:76-94` (the
|
|
||||||
"Live updates → Wire format" section). The README is the user-facing
|
|
||||||
exposition; this reference adds the per-line field semantics and the
|
|
||||||
parser-tolerance rules from the code.
|
|
||||||
|
|
||||||
> Each stream endpoint emits newline-delimited JSON. The first batch
|
|
||||||
> is the **snapshot** of currently-known rows; after that comes
|
|
||||||
> **`snapshot_end`** with the count, then the **live** events.
|
|
||||||
>
|
|
||||||
> ```json
|
|
||||||
> {"type":"item","data":{"id":"CLM-1", "...":"..."}}
|
|
||||||
> {"type":"item","data":{"id":"CLM-2", "...":"..."}}
|
|
||||||
> {"type":"snapshot_end","data":{"count":2}}
|
|
||||||
> {"type":"item","data":{"id":"CLM-3", "...":"..."}} ← live
|
|
||||||
> {"type":"heartbeat","data":{"ts":"2026-06-20T23:17:09Z"}} ← idle keep-alive
|
|
||||||
> ```
|
|
||||||
>
|
|
||||||
> Lines are `{"type": ..., "data": ...}`; known types are `item`,
|
|
||||||
> `snapshot_end`, `heartbeat`, and (rare) `item_dropped` /
|
|
||||||
> `error`. Heartbeats keep the connection alive when nothing is
|
|
||||||
> happening — clients flip to `stalled` after 30s of total silence
|
|
||||||
> (heartbeat or otherwise) and surface a **↻ Reconnect** button.
|
|
||||||
|
|
||||||
## Per-line field reference
|
|
||||||
|
|
||||||
| `type` | `data` shape | Required? | Emitted by | Parser behavior |
|
|
||||||
| -------------- | ----------------------------------------- | --------- | ------------------------------------------- | -------------------------------------------- |
|
|
||||||
| `item` | resource-specific row (`Claim` / `Remittance` / `Activity`) | yes, in `data` (the per-row envelope) | snapshot loop + `_tail_events` forwarding `claim_written` / `remittance_written` / `activity_recorded` | `dispatch(resource, ev.data)` → `useTailStore.addClaim` / `addRemittance` / `addActivity` (first-write-wins dedup on the id-keyed slices — `tail-store.ts:104,120`). |
|
|
||||||
| `snapshot_end` | `{"count": N}` (integer ≥ 0) | yes, on every stream | `api.py:1395,1889,1999` (one per stream, after the snapshot loop) | Flips `<TailStatusPill>` from `connecting` to `live`, resets the reconnect backoff counter to 0 (`useTailStream.ts:174-180`). |
|
|
||||||
| `heartbeat` | `{"ts": "<iso-8601 UTC>"}` | yes, but only when idle | `_tail_events` in `api_helpers.py:241-245` (cadence from `heartbeat_seconds()`, default 15s) | Re-arms the stall timer (`useTailStream.ts:124-140`); no state change. |
|
|
||||||
| `item_dropped` | `{"id": "<string>"}` | optional (rare; queue overflow) | EventBus drop-oldest path (per the spec at `docs/superpowers/specs/2026-06-20-cyclone-live-tail-design.md:299`) | Re-arms the stall timer; no state change. The `id` field is informational — the hook does not refetch on drop, the page does (see Spec §3.6). |
|
|
||||||
| `error` | `{"message": "<string>"}` | optional (server-side failure) | Reserved for future server-emitted errors (currently only thrown client-side) | Hook promotes to a thrown `Error(message)` so the catch block runs the reconnect machinery (`useTailStream.ts:190-194`). |
|
|
||||||
|
|
||||||
## Parser tolerance
|
|
||||||
|
|
||||||
The shared parser at `src/lib/tail-stream.ts` is intentionally
|
|
||||||
forgiving so a single bad frame doesn't kill the stream:
|
|
||||||
|
|
||||||
- **Trailing `\r`** is stripped per line (`tail-stream.ts:116`) so a
|
|
||||||
CRLF-terminated stream still parses.
|
|
||||||
- **Malformed JSON** (`JSON.parse` throws) → `console.warn` + skip the
|
|
||||||
line; the iterator continues (`tail-stream.ts:122-131`).
|
|
||||||
- **Unknown `type`** (not in `KNOWN_TYPES`) → `console.warn` + skip
|
|
||||||
(`tail-stream.ts:141-148`). Adding a new event type is
|
|
||||||
forward-compatible: old clients see warn lines, new clients see the
|
|
||||||
typed event.
|
|
||||||
- **Empty lines** (consecutive `\n`s) → silently skipped
|
|
||||||
(`tail-stream.ts:118`).
|
|
||||||
- **No trailing newline** → flushed as a final partial line on
|
|
||||||
stream close (`tail-stream.ts:154-178`).
|
|
||||||
- **Abort signal** → iterator exits cleanly without throwing
|
|
||||||
(`tail-stream.ts:75,98,107`).
|
|
||||||
|
|
||||||
## Endpoint inventory
|
|
||||||
|
|
||||||
| Method | Path | Subscribes to | Default sort | Defined at |
|
|
||||||
| ------ | ------------------------- | -------------------- | ------------------- | ----------------------------------- |
|
|
||||||
| GET | `/api/claims/stream` | `claim_written` | `-submission_date` | `backend/src/cyclone/api.py:1357` |
|
|
||||||
| GET | `/api/remittances/stream` | `remittance_written` | `-received_date` | `backend/src/cyclone/api.py:1858` |
|
|
||||||
| GET | `/api/activity/stream` | `activity_recorded` | `-timestamp` (limit 50) | `backend/src/cyclone/api.py:1971` |
|
|
||||||
|
|
||||||
All three accept the same query params as their non-streaming
|
|
||||||
counterparts (`status`, `payer`, `date_from`, …) so a frontend can
|
|
||||||
swap a one-shot fetch for a tail with no URL surgery. Responses are
|
|
||||||
`Content-Type: application/x-ndjson`.
|
|
||||||
@@ -1,170 +0,0 @@
|
|||||||
---
|
|
||||||
name: cyclone-tests
|
|
||||||
description: "Cyclone pytest + vitest fixture patterns, prodfiles layout, backend/tests/fixtures/ conventions, .test.tsx sibling rule. Use when: adding a backend pytest case, adding a frontend vitest test, or wiring in a real-EDI prodfiles sample."
|
|
||||||
---
|
|
||||||
|
|
||||||
# cyclone-tests
|
|
||||||
|
|
||||||
The Cyclone test suite is split two ways: **backend pytest** (89 test files under `backend/tests/`, 13 flat fixtures in `backend/tests/fixtures/`, one autouse `conftest.py` that resets the DB per-test) and **frontend vitest** (59 `*.test.ts(x)` siblings across `src/`, two rendering styles — `@testing-library/react` and a custom `createRoot`+`Probe` shim). This skill codifies the conventions so additions stay consistent with what's already there.
|
|
||||||
|
|
||||||
As of this writing: **89 backend test files**, **13 flat backend fixtures**, **59 frontend `*.test.ts(x)` siblings**, and **23 prodfiles samples** across `docs/prodfiles/{837p-from-axiscare,835fromco,FromHPE,claims}/`. The next increment is **SP24** (SP23 is the Ubuntu+Docker fork, awaiting user decision).
|
|
||||||
|
|
||||||
## Auth flag (SP24)
|
|
||||||
|
|
||||||
The autouse `conftest.py` fixture at `backend/tests/conftest.py` flips `cyclone.auth.deps.AUTH_DISABLED = True` for the entire test session, so every test runs without a login round-trip. **Any new test that reads `cyclone.auth.deps.AUTH_DISABLED` directly will see `True`** — that's the test-suite reality, not a production reality. If you need a test that exercises the real gate, import `from cyclone.auth.deps import matrix_gate` and call it directly with a `Request` whose `state` carries a real session, or flip the flag back inside the test and reset it on teardown. The startup WARNING (`backend/src/cyclone/__main__.py`) is silent in tests by default because the conftest sets the flag before `bootstrap.run()` is called via `import`.
|
|
||||||
|
|
||||||
## When to use
|
|
||||||
|
|
||||||
- **Adding a backend pytest case.** You're about to add a new `test_*.py` under `backend/tests/` and need the autouse DB-fixture rules, the fixture-path convention, and the right naming flavor (`test_api_*.py` vs `test_<module>_*.py`).
|
|
||||||
- **Adding a frontend vitest test.** You're about to add a `*.test.ts(x)` sibling and need the `// @vitest-environment happy-dom` setup, the act-environment flag, and the right rendering helper (`@testing-library/react` vs the `createRoot`+`Probe` shim).
|
|
||||||
- **Dropping in a prodfiles fixture.** You have a real EDI sample under `docs/prodfiles/<source>/` and need the copy step that makes it test-runnable without coupling the test to the prodfiles archive.
|
|
||||||
- **Debugging a flaky test.** A test passes locally but flakes in CI — load this skill to check the determinism + no-network rules before chasing the symptom.
|
|
||||||
|
|
||||||
## Conventions
|
|
||||||
|
|
||||||
1. **Frontend sibling rule.** Every new file in `src/` that contains testable logic gets a `*.test.ts(x)` next to it. `useFoo.ts` → `useFoo.test.ts`. `ClaimDrawer.tsx` → `ClaimDrawer.test.tsx`. The 59 existing siblings follow this; CI implicitly enforces it via the default `src/**/*.test.ts(x)` glob in `vitest.config.ts`.
|
|
||||||
2. **Backend test location.** Tests live under `backend/tests/test_*.py`. Two flavors, two naming patterns:
|
|
||||||
- **Integration tests** (FastAPI surface) → `test_api_<topic>_<verb>.py` — e.g. `test_api_parse_persists.py`, `test_api_999.py`.
|
|
||||||
- **Pure-unit tests** (parsers, validators, store internals) → `test_<module>_<behavior>.py` — e.g. `test_cas_codes.py`, `test_pubsub.py`.
|
|
||||||
3. **Prodfiles drop-in.** Real EDI samples live under `docs/prodfiles/<source>/<file>.txt` (sources seen so far: `837p-from-axiscare/`, `835fromco/`, `FromHPE/`, `claims/`). To use one in a test: copy the file to `backend/tests/fixtures/<descriptive-name>.txt` (flat — no per-test subdirectories; the existing 13 fixtures all sit at the top level) and reference it from the test as a module-level `Path` constant. See `## Patterns` for the exact line.
|
|
||||||
4. **Determinism.** Time-sensitive tests must not depend on wall-clock time.
|
|
||||||
- **Frontend** uses `vi.useFakeTimers()` + `vi.setSystemTime(new Date("YYYY-MM-DDTHH:MM:SSZ"))` (see `src/components/TailStatusPill.test.tsx:54-58`) and `vi.advanceTimersByTime(ms)` to drive interval / backoff code deterministically.
|
|
||||||
- **Backend** dates are passed explicitly as fixture values — e.g. `datetime.now(timezone.utc)` is fine for "now-ish" anchors, but for fixed dates pass `datetime(2026, 6, 20, tzinfo=timezone.utc)`. The project does **not** currently use `freezegun` or `mock.patch(datetime)`; if you need deterministic date mocking, propose adding `freezegun` to `backend/pyproject.toml` rather than rolling your own.
|
|
||||||
5. **No network.** Tests must not hit the network.
|
|
||||||
- **Backend:** `fastapi.testclient.TestClient(app)` runs in-process; no uvicorn. The autouse `conftest.py` fixture (`backend/tests/conftest.py:20`) points `CYCLONE_DB_URL` at `tmp_path/test.db`, calls `db._reset_for_tests()` + `db.init_db()`, and wires a fresh `EventBus` onto `app.state`.
|
|
||||||
- **Frontend:** `vi.stubGlobal("fetch", vi.fn().mockResolvedValue(...))` for hooks; `vitest.config.ts` sets `VITE_API_BASE_URL=http://test.local` so the `api` module doesn't throw `notConfiguredError` before the mock fires.
|
|
||||||
6. **pytest collection.** Run `cd backend && python -m pytest tests/<file>::<name> -v` for the fastest single-test feedback loop. Run `cd backend && python -m pytest tests/<file> -v` for one file. Run `cd backend && python -m pytest` for the full suite — this is the merge gate. Frontend: `npm test` (alias for `vitest run`) for the full suite; `npx vitest run src/hooks/useFoo.test.ts` for one file.
|
|
||||||
|
|
||||||
## Patterns
|
|
||||||
|
|
||||||
### Backend pytest using a fixture + per-test SQLite DB
|
|
||||||
|
|
||||||
Canonical shape — see `backend/tests/test_api_999.py:1-50` for the full file. The autouse `conftest.py` already provides the DB init + `EventBus` reset; most tests just add a `client` fixture.
|
|
||||||
|
|
||||||
```python
|
|
||||||
"""Tests for the FastAPI surface in cyclone.api for the 999 endpoint."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
|
|
||||||
from cyclone.api import app
|
|
||||||
|
|
||||||
# Fixture reference — flat, module-level Path constant. NEVER reach into
|
|
||||||
# docs/prodfiles/ from a test; the fixtures/ dir is the test-consumed surface.
|
|
||||||
ACCEPTED = Path(__file__).parent / "fixtures" / "minimal_999.txt"
|
|
||||||
REJECTED = Path(__file__).parent / "fixtures" / "minimal_999_rejected.txt"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def client() -> TestClient:
|
|
||||||
return TestClient(app)
|
|
||||||
|
|
||||||
|
|
||||||
def test_parse_999_endpoint_happy_path(client: TestClient):
|
|
||||||
text = ACCEPTED.read_text()
|
|
||||||
resp = client.post(
|
|
||||||
"/api/parse-999",
|
|
||||||
files={"file": ("minimal_999.txt", text, "text/plain")},
|
|
||||||
headers={"Accept": "application/json"},
|
|
||||||
)
|
|
||||||
assert resp.status_code == 200, resp.text
|
|
||||||
body = resp.json()
|
|
||||||
assert body["ack"]["ack_code"] == "A"
|
|
||||||
```
|
|
||||||
|
|
||||||
Override the autouse DB fixture only when you need a custom env (e.g. a per-test backup directory) — see `backend/tests/test_999_rejected_state.py:20-25`:
|
|
||||||
|
|
||||||
```python
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _setup(tmp_path, monkeypatch):
|
|
||||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/inbox.db")
|
|
||||||
from cyclone import db
|
|
||||||
db._reset_for_tests()
|
|
||||||
db.init_db()
|
|
||||||
yield
|
|
||||||
```
|
|
||||||
|
|
||||||
### Frontend vitest `*.test.tsx` for a component using fake timers
|
|
||||||
|
|
||||||
Pattern taken from `src/components/TailStatusPill.test.tsx:1-62` — uses `vi.useFakeTimers()` + `vi.setSystemTime(...)` to drive interval-based code deterministically.
|
|
||||||
|
|
||||||
```ts
|
|
||||||
// @vitest-environment happy-dom
|
|
||||||
// React's act warnings need an act-aware environment — mirror the other
|
|
||||||
// hook tests in this repo.
|
|
||||||
(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
|
|
||||||
|
|
||||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
||||||
import { MyComponent } from "./MyComponent";
|
|
||||||
|
|
||||||
describe("MyComponent", () => {
|
|
||||||
beforeEach(() => {
|
|
||||||
vi.useFakeTimers();
|
|
||||||
vi.setSystemTime(new Date("2026-06-20T12:00:30Z"));
|
|
||||||
});
|
|
||||||
|
|
||||||
afterEach(() => {
|
|
||||||
vi.useRealTimers();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("test_renders_after_interval_tick", () => {
|
|
||||||
vi.advanceTimersByTime(30_000); // drive the setInterval
|
|
||||||
// …assert…
|
|
||||||
});
|
|
||||||
});
|
|
||||||
```
|
|
||||||
|
|
||||||
### Frontend vitest `*.test.tsx` for a component using `@testing-library/react` + `happy-dom`
|
|
||||||
|
|
||||||
Pattern taken from `src/hooks/useInboxLanes.test.ts:1-80`. (A handful of older tests — `useClaimDetail.test.ts`, `useDrawerUrlState.test.ts`, `TailStatusPill.test.tsx` — roll a custom `createRoot`+`Probe` shim instead. Both styles are accepted; `@testing-library/react` is preferred when the hook has async dependencies because `waitFor` is built in.)
|
|
||||||
|
|
||||||
```ts
|
|
||||||
// @vitest-environment happy-dom
|
|
||||||
(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
|
|
||||||
|
|
||||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
|
||||||
import { act, cleanup, renderHook, waitFor } from "@testing-library/react";
|
|
||||||
import { useMyHook } from "./useMyHook";
|
|
||||||
|
|
||||||
vi.mock("@/lib/api", () => ({
|
|
||||||
api: { fetchFoo: vi.fn() },
|
|
||||||
}));
|
|
||||||
|
|
||||||
afterEach(() => {
|
|
||||||
cleanup();
|
|
||||||
vi.unstubAllGlobals();
|
|
||||||
vi.useRealTimers();
|
|
||||||
});
|
|
||||||
|
|
||||||
describe("useMyHook", () => {
|
|
||||||
it("loads data on mount", async () => {
|
|
||||||
vi.stubGlobal("fetch", vi.fn().mockResolvedValue({
|
|
||||||
ok: true,
|
|
||||||
json: async () => ({ items: [] }),
|
|
||||||
}));
|
|
||||||
const { result } = renderHook(() => useMyHook());
|
|
||||||
await waitFor(() => expect(result.current.loading).toBe(false));
|
|
||||||
expect(result.current.items).toEqual([]);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
```
|
|
||||||
|
|
||||||
## Anti-patterns
|
|
||||||
|
|
||||||
- **Don't put frontend tests in `src/__tests__/`.** No such directory exists in the codebase — `__tests__` only appears in the SP-catalog plan itself. Use the sibling rule (`useFoo.ts` → `useFoo.test.ts`).
|
|
||||||
- **Don't reach into `docs/prodfiles/` directly from a test.** Always copy to `backend/tests/fixtures/<name>.txt` first. The prodfiles directory is the source-of-truth archive and may be reorganized; the fixtures directory is the test-consumed surface and is stable.
|
|
||||||
- **Don't use wall-clock sleeps for timing.** A handful of legacy tests use `await new Promise((r) => setTimeout(r, 200))` (see `src/components/SearchBar.test.tsx:281,323,373` and `src/hooks/useSearch.test.ts:174`) — these are known flaky in CI. Use `vi.useFakeTimers()` + `vi.advanceTimersByTime(ms)` instead, or `waitFor(...)` from `@testing-library/react`.
|
|
||||||
|
|
||||||
## Related skills
|
|
||||||
|
|
||||||
- **`cyclone-spec`** — every SP-N spec lists test impact; load this when drafting or reviewing the spec to confirm fixture / `.test.tsx` implications for the increment.
|
|
||||||
- **`cyclone-edi`** — most backend tests cover parser + validator behavior; load when the increment touches an EDI parser or adds a validator rule (R200/R210/NPI Luhn/EIN/CAS).
|
|
||||||
- **`cyclone-tail`** — most frontend tests cover hook behavior (`useTailStream`, `useMergedTail`); load when the increment changes the wire format or adds a streaming hook.
|
|
||||||
- **`cyclone-store`** — write-path tests live here; load when the increment touches `store.py`, adds a new entity, or wires a new `<entity>_written` event.
|
|
||||||
- **`cyclone-api-router`** — endpoint tests live in `backend/tests/test_api_*.py`; load when the increment adds or changes an HTTP endpoint.
|
|
||||||
- **`cyclone-frontend-page`** — page-component tests live next to pages in `src/pages/*.test.tsx`; load when the increment adds or refactors a page.
|
|
||||||
- **`cyclone-cli`** — CLI smoke tests live in `backend/tests/test_cli_*.py`; load when the increment adds a CLI subcommand.
|
|
||||||
- **`superpowers:test-driven-development`** (global) — the upstream TDD workflow. Load first when starting any new feature increment; this skill only codifies the Cyclone-specific test layout on top of TDD.
|
|
||||||
@@ -1,184 +0,0 @@
|
|||||||
# CLAUDE.md
|
|
||||||
|
|
||||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
||||||
|
|
||||||
## What this is
|
|
||||||
|
|
||||||
Cyclone is a self-hosted X12 EDI claims-management suite for a single billing office (Colorado Medicaid currently). It parses 837P professional claims and 835 ERA remittances (X12 005010X222A1 / 005010X221A1) and also handles 999, TA1, 270, 271, and 277CA. Local-only by design: binds to `127.0.0.1`, requires login (auth boundary is HTTP; bcrypt + HttpOnly session cookie; first admin bootstrapped from `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD` env vars; see SP24 spec for the full posture), no internet exposure.
|
|
||||||
|
|
||||||
Stack: one Python process (FastAPI + uvicorn, port 8000) + one Node process in dev (Vite, port 5173). The authoritative state is a single SQLite file at `~/.local/share/cyclone/cyclone.db` (or SQLCipher at the same path when the macOS Keychain entry + `sqlcipher3` are both present).
|
|
||||||
|
|
||||||
For the day-1 architecture read, see `docs/ARCHITECTURE.md` (process topology, module map, store facade, parser pipeline, pubsub). For the what-it-does read, see `docs/REQUIREMENTS.md` (FRs + NFRs + DoD).
|
|
||||||
|
|
||||||
## Install
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Backend (Python 3.11+)
|
|
||||||
cd backend
|
|
||||||
python -m venv .venv
|
|
||||||
.venv/bin/pip install -e '.[dev]'
|
|
||||||
|
|
||||||
# Frontend (Node 20+)
|
|
||||||
cd ..
|
|
||||||
npm install
|
|
||||||
```
|
|
||||||
|
|
||||||
Optional backend extras: `pip install -e '.[sqlcipher]'` (encryption at rest, SP12) and `pip install -e '.[sftp]'` (real SFTP, SP13).
|
|
||||||
|
|
||||||
## Dev (two terminals)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Terminal 1 — backend
|
|
||||||
cd backend
|
|
||||||
.venv/bin/python -m cyclone serve # default 127.0.0.1:8000
|
|
||||||
# CYCLONE_PORT=... overrides port; CYCLONE_RELOAD=1 enables uvicorn --reload
|
|
||||||
# Or: .venv/bin/uvicorn cyclone.api:app --reload --port 8000
|
|
||||||
|
|
||||||
# Terminal 2 — frontend
|
|
||||||
npm run dev # Vite on http://localhost:5173
|
|
||||||
```
|
|
||||||
|
|
||||||
Vite proxies `/api/*` to the backend at `http://127.0.0.1:${CYCLONE_PORT:-8000}` so relative-URL fetchers (the live-tail NDJSON streams in particular) resolve through the same origin. Override the backend port with `CYCLONE_PORT` in the frontend terminal too.
|
|
||||||
|
|
||||||
Create `.env.local` at the repo root with `VITE_API_BASE_URL=http://127.0.0.1:8000`. Without it, the UI runs against the in-memory zustand store and real EDI parsing is disabled.
|
|
||||||
|
|
||||||
## Test
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Backend — full suite
|
|
||||||
cd backend && .venv/bin/pytest
|
|
||||||
|
|
||||||
# Backend — one file
|
|
||||||
cd backend && .venv/bin/pytest tests/test_api_999.py -v
|
|
||||||
|
|
||||||
# Backend — one test by node id
|
|
||||||
cd backend && .venv/bin/pytest tests/test_api_999.py::test_parse_999_endpoint_happy_path -v
|
|
||||||
|
|
||||||
# Frontend — full suite
|
|
||||||
npm test # alias for `vitest run`
|
|
||||||
|
|
||||||
# Frontend — one file
|
|
||||||
npx vitest run src/hooks/useFoo.test.ts
|
|
||||||
|
|
||||||
# Frontend — typecheck
|
|
||||||
npm run typecheck
|
|
||||||
|
|
||||||
# Frontend — build (tsc -b + vite build)
|
|
||||||
npm run build
|
|
||||||
|
|
||||||
# Frontend — lint
|
|
||||||
npm run lint
|
|
||||||
```
|
|
||||||
|
|
||||||
**Conventions** (full detail in `.superpowers/skills/cyclone-tests/SKILL.md`):
|
|
||||||
|
|
||||||
- Backend tests live under `backend/tests/test_*.py`. Two flavors: `test_api_<topic>_<verb>.py` (FastAPI integration via `fastapi.testclient.TestClient`) and `test_<module>_<behavior>.py` (pure-unit). Autouse `conftest.py` points `CYCLONE_DB_URL` at `tmp_path/test.db`, calls `db._reset_for_tests()` + `db.init_db()`, and wires a fresh `EventBus` onto `app.state`.
|
|
||||||
- Prodfiles (real EDI samples under `docs/prodfiles/<source>/`) are never read directly from a test — copy to `backend/tests/fixtures/<descriptive-name>.txt` first and reference as a module-level `Path` constant. The `fixtures/` dir is flat (no per-test subdirs).
|
|
||||||
- Frontend tests are siblings: `useFoo.ts` → `useFoo.test.ts`, `ClaimDrawer.tsx` → `ClaimDrawer.test.tsx`. Setup is `// @vitest-environment happy-dom` plus `(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;`. Mock the API at the module boundary with `vi.mock("@/lib/api", ...)`; stub fetch with `vi.stubGlobal("fetch", vi.fn().mockResolvedValue(...))`. `vitest.config.ts` sets `VITE_API_BASE_URL=http://test.local` so the `api` module doesn't throw `notConfiguredError` before the mock fires.
|
|
||||||
- Time-sensitive tests: frontend uses `vi.useFakeTimers()` + `vi.setSystemTime(...)` + `vi.advanceTimersByTime(ms)`. Backend passes explicit `datetime(...)` values. Don't add `await new Promise((r) => setTimeout(r, N))` — it's the legacy flaky pattern.
|
|
||||||
|
|
||||||
## Project-scoped skills (`.superpowers/skills/`)
|
|
||||||
|
|
||||||
Cyclone ships 8 skills under `.superpowers/skills/`. They auto-load by description match — no slash command needed. **Read the relevant skill before touching the matching subsystem.**
|
|
||||||
|
|
||||||
| Skill | Owns |
|
|
||||||
|---|---|
|
|
||||||
| `cyclone-spec` | The SP-N spec → plan → implement → merge flow (branch shape, file paths, commit prefixes, PR title, merge shape). |
|
|
||||||
| `cyclone-tests` | pytest + vitest fixture patterns, prodfiles drop-in rule, determinism rules. |
|
|
||||||
| `cyclone-edi` | EDI parser/validator conventions (837P/835/999/270/271/277CA/TA1, R-codes, CAS mapping). |
|
|
||||||
| `cyclone-tail` | Live-tail streaming wire format and the `useTailStream` + `useMergedTail` + `TailStatusPill` hook triplet. |
|
|
||||||
| `cyclone-store` | `CycloneStore` facade, write-paths, pubsub event contract, SP21 split map. |
|
|
||||||
| `cyclone-api-router` | FastAPI router conventions (`api_routers/`, `api_helpers.py`), response/error-envelope shapes. |
|
|
||||||
| `cyclone-frontend-page` | React page conventions (TanStack Query `use<X>` hook, drawer, URL state, sibling test). |
|
|
||||||
| `cyclone-cli` | CLI subcommand conventions (`cli.py`, exit codes, smoke tests). |
|
|
||||||
|
|
||||||
## The SP-N increment flow
|
|
||||||
|
|
||||||
Every feature ships as a numbered **SP-N increment**: spec → plan → implementation branch → single atomic merge into `main`. As of the last backfill, SP numbers are used through **SP22**; **SP23** is reserved for the Ubuntu + Docker + RBAC product fork (`docs/superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md`, awaiting user decision); the next free increment is **SP24**. Read `cyclone-spec` before starting a new one. Non-negotiable shape:
|
|
||||||
|
|
||||||
- **Branch:** `sp<N>-<short-kebab-topic>` (e.g. `sp22-line-reconciliation`).
|
|
||||||
- **Spec path:** `docs/superpowers/specs/YYYY-MM-DD-cyclone-<topic>-design.md`, header `Status: Draft, awaiting user sign-off`, sections `Scope / Decisions / …`. Specs contain zero code blocks.
|
|
||||||
- **Plan path:** `docs/superpowers/plans/YYYY-MM-DD-cyclone-<topic>.md`, header per `superpowers:writing-plans` with `Goal / Architecture / Tech Stack / Spec` metadata + numbered `- [ ] Step N:` tasks.
|
|
||||||
- **Commit prefixes:** `feat(sp<N>): …`, `docs(spec): …`, `docs(plan): …`, `merge: SP<N> <topic> into main`.
|
|
||||||
- **PR title:** `SP<N> <Topic>` (matches the merge-commit subject).
|
|
||||||
- **Merge shape:** single atomic merge commit. **No squash** (collapses the audit trail) and **no rebase** (rewrites the SHAs the review was performed against). The SP-N merge commit *is* the record of the increment landing.
|
|
||||||
|
|
||||||
The matching skill to load alongside `cyclone-spec` depends on the subsystem the SP-N touches (see the "Related skills" section at the bottom of each skill file).
|
|
||||||
|
|
||||||
## Live-tail wire format
|
|
||||||
|
|
||||||
The Claims, Remittances, and Activity pages stay current without manual refresh. The backend publishes an internal event on every store write, the page opens a streaming HTTP connection to the matching `/api/<resource>/stream` endpoint, and new rows append to the table the moment they hit the database.
|
|
||||||
|
|
||||||
Endpoints (all accept the same query params as their non-streaming counterparts; `Content-Type: application/x-ndjson`):
|
|
||||||
|
|
||||||
| Method | Path | Subscribes to | Default sort |
|
|
||||||
|---|---|---|---|
|
|
||||||
| GET | `/api/claims/stream` | `claim_written` | `-submission_date` |
|
|
||||||
| GET | `/api/remittances/stream` | `remittance_written` | `-received_date` |
|
|
||||||
| GET | `/api/activity/stream` | `activity_recorded` | `-timestamp` (limit 50) |
|
|
||||||
|
|
||||||
Wire format: one JSON object per line, `{"type": ..., "data": ...}`. The first batch is the **snapshot** of currently-known rows, then `snapshot_end` with the count, then the **live** events. Known types: `item`, `snapshot_end`, `heartbeat` (keeps the connection alive on idle — clients flip to `stalled` after 30s of total silence), `item_dropped` (rare), `error`.
|
|
||||||
|
|
||||||
Status pill states (rendered by `<TailStatusPill>` in `src/components/TailStatusPill.tsx`): `live` (success), `connecting` (warning), `reconnecting` (warning), `stalled` (destructive, ↻ Reconnect button), `error` (destructive, ↻ Reconnect button), `closed` (destructive). Backoff on error: `1s → 2s → 4s → 8s → 16s → 30s` capped. `STALL_TIMEOUT_MS = 30_000` in `src/hooks/useTailStream.ts:53`. Heartbeat interval is `CYCLONE_TAIL_HEARTBEAT_S` env var, default 15s.
|
|
||||||
|
|
||||||
Frontend triplet for any live page: `use<X>(params)` (initial fetch) + `useTailStream(resource)` (opens the NDJSON stream, drives backoff/stall) + `useMergedTail(resource, baseItems, filterFn?)` (merges snapshot + tail, dedup'd by id). The subscription lives on the page, not inside the data hook — see `cyclone-frontend-page` for why.
|
|
||||||
|
|
||||||
## Backend at a glance
|
|
||||||
|
|
||||||
`backend/src/cyclone/` is a single namespace. The two largest files are `api.py` (~3,548 LOC, the only large file) and `store.py` (~2,423 LOC, the `CycloneStore` facade — SP21 is in flight to split it into a `cyclone/store/` subpackage; the public API stays unchanged). Subpackages: `api_routers/` (acks, admin, health, ta1_acks), `clearhouse/` (Clearhouse + SftpClient), `edi/` (filenames), `parsers/` (X12 transaction parsers + models + validators + serializers), `workflow/` (placeholder for future sub-project 6).
|
|
||||||
|
|
||||||
The store is the only read/write surface for the database; every mutating endpoint goes through it. All persistence flows through SQLAlchemy sessions via `db.SessionLocal()()`. SQLAlchemy ORM models live in `db.py`; 12 SQL migrations under `migrations/` (0001_initial through 0012_backups) are walked in order by `db_migrate.py`.
|
|
||||||
|
|
||||||
The parser pipeline is a 5-stage `tokenize → segmentize → model → validate → write_to_store` flow used for every inbound X12 type. Per-transaction parsers: `parse_837.py`, `parse_835.py`, `parse_999.py`, `parse_ta1.py`, `parse_270.py`, `parse_271.py`, `parse_277ca.py`. Each has a matching Pydantic model module (`models.py`, `models_835.py`, …) and a writer (`writer.py` / `writer_835.py`). The 837P serializer (`serialize_837.py`) is the byte-faithful outbound counterpart used by both single-claim download (`/api/claims/{id}/serialize-837`) and the bulk rejected-resubmit bundle (`/api/inbox/rejected/resubmit?download=true`).
|
|
||||||
|
|
||||||
The pubsub is `cyclone.pubsub.EventBus` — an in-process async fan-out broker. Publishers call `publish(kind, payload)`; subscribers receive via an async iterator. If a subscriber's per-kind queue is full, the oldest event is dropped so a slow consumer can't stall the producer. Bus is single-event-loop only (matches FastAPI/uvicorn).
|
|
||||||
|
|
||||||
Config: `config/payers.yaml` is the on-disk source for providers / payers / clearhouse, schema-validated at boot against a Pydantic model. Reload with `POST /api/admin/reload-config`. Original in-code `PAYER_FACTORIES` dict in `cli.py` is kept as a fallback for ad-hoc testing.
|
|
||||||
|
|
||||||
Secrets live in the macOS Keychain (via `keyring` + `cyclone.secrets`): SQLCipher key (service `cyclone`, account `cyclone.db.key`), SFTP password, backup passphrase. No secrets on disk in plaintext.
|
|
||||||
|
|
||||||
## Frontend at a glance
|
|
||||||
|
|
||||||
`src/` is React 18 + TypeScript + Vite. Routes register in `src/App.tsx` (11 pages, all under a `<Layout>` route wrapper). Pages are pure renderers — every page pairs with a `use<X>` data hook in `src/hooks/` and renders a `<PageHeader>` + a table/list/KPI grid. Drawers (`ClaimDrawer/`, `RemitDrawer/`, plus the new `ProviderDrawer/` and `AckDrawer/`) are mounted by the page and their open/close state is mirrored to the URL via `useDrawerUrlState` so deep-links round-trip. Drill-stack navigation is provided by `<DrillStackProvider>` in `src/components/drill/`.
|
|
||||||
|
|
||||||
State split: **server state** in TanStack Query (`@tanstack/react-query`); **ephemeral client state** in Zustand (`useTailStore` for live-tail append, plus the drill stack). The live-tail store is FIFO-capped at `TAIL_CAP = 10_000` per slice (`src/store/tail-store.ts:28`); `claims` and `remittances` are key-by-id with first-write-wins dedup, `activity` is an append-only array.
|
|
||||||
|
|
||||||
UI primitives in `src/components/ui/` are Radix-backed (`button`, `dialog`, `table`, `select`, `pagination`, `empty-state`, `error-state`, `filter-chips`, `skeleton`, `input`, `label`, `card`, `badge`, `skip-link`, `claim-state-badge`). Don't import a new UI library without discussion.
|
|
||||||
|
|
||||||
Path alias `@/` → `src/`. Configured in `vite.config.ts`, `vitest.config.ts`, and `tsconfig.app.json`.
|
|
||||||
|
|
||||||
## CLI
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Parser
|
|
||||||
python -m cyclone.cli parse-837 path/to/837p.txt --output-dir ./claims --payer co_medicaid [--strict] [--include-raw-segments]
|
|
||||||
python -m cyclone.cli parse-835 path/to/835.txt --output-dir ./remits
|
|
||||||
python -m cyclone.cli parse-999 inbound_999.txt
|
|
||||||
python -m cyclone.cli parse-ta1 inbound_ta1.txt
|
|
||||||
python -m cyclone.cli parse-277ca inbound_277ca.txt
|
|
||||||
|
|
||||||
# Validators
|
|
||||||
python -m cyclone.cli validate-npi 1234567893
|
|
||||||
python -m cyclone.cli validate-tin 721587149
|
|
||||||
|
|
||||||
# Other
|
|
||||||
python -m cyclone serve # uvicorn
|
|
||||||
python -m cyclone backup list
|
|
||||||
python -m cyclone backup create --reason manual
|
|
||||||
```
|
|
||||||
|
|
||||||
Exit codes are documented per subcommand in `cyclone-cli` — `0` for success, `2` for file-level failure, `1` for unexpected exceptions.
|
|
||||||
|
|
||||||
## Things that are easy to get wrong
|
|
||||||
|
|
||||||
- **`VITE_API_BASE_URL` matters.** With it empty, every `api` method throws `notConfiguredError()` and the UI falls back to the in-memory zustand store — parses are disabled and the live-tail streams never open.
|
|
||||||
- **Prodfiles vs fixtures.** Tests must reference `backend/tests/fixtures/<name>.txt`, not `docs/prodfiles/<source>/<file>.txt`. The prodfiles dir is the source-of-truth archive; the fixtures dir is the stable test surface.
|
|
||||||
- **SP-N merge shape.** No squash, no rebase. The merge commit *is* the audit trail. Squash collapses the per-commit history and breaks the SP-N audit trail.
|
|
||||||
- **Don't put domain logic in JSX.** Conditional renderings, table sorting, and KPI math all belong in the `use<X>` hook or a pure helper under `src/lib/`.
|
|
||||||
- **Don't open a drawer via local `useState`.** Use `useDrawerUrlState()` so the URL is the single source of truth — deep-links and reload-restore depend on it.
|
|
||||||
- **Don't call `useTailStream` from inside a `use<X>` hook.** The subscription lives on the page so the lifecycle ties to whoever mounts the hook, not to whoever happens to call it.
|
|
||||||
- **The store facade.** The public API of `cyclone.store` is preserved through SP21's split — call through the facade, not directly into the underlying modules.
|
|
||||||
- **Encryption is optional, not required.** When the Keychain entry is missing **or** `sqlcipher3` is not installed, the DB falls back to plain SQLite. Don't fail boot on missing encryption.
|
|
||||||
- **Local-only by design.** The backend binds to `127.0.0.1`, requires login (bcrypt + HttpOnly session cookie; see SP24 spec), and the threat model is still a stolen/imaged drive — SQLCipher at rest and the macOS Keychain handle that. The auth boundary is the HTTP layer; the file-system posture is unchanged. Don't add internet exposure. Don't disable auth without an explicit `CYCLONE_AUTH_DISABLED=1` env var (the escape hatch logs a WARNING at boot).
|
|
||||||
</content>
|
|
||||||
</invoke>
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
# syntax=docker/dockerfile:1.7
|
|
||||||
#
|
|
||||||
# Cyclone frontend — React SPA built with node:20-alpine and served by
|
|
||||||
# nginx:1.27-alpine. nginx reverse-proxies /api/* to the backend service
|
|
||||||
# over the compose-managed bridge network.
|
|
||||||
|
|
||||||
# ---------- builder ----------
|
|
||||||
FROM node:20-alpine AS builder
|
|
||||||
|
|
||||||
WORKDIR /build
|
|
||||||
|
|
||||||
# Install deps first so this layer caches across source edits.
|
|
||||||
# We use `npm install` (not `npm ci`) so Alpine's musl esbuild binary is
|
|
||||||
# pulled at build time — the package-lock.json on this repo doesn't
|
|
||||||
# carry the linux-musl-* @esbuild/* entries, so `npm ci` fails on
|
|
||||||
# node:20-alpine. `npm install` with --no-audit --no-fund is fast enough
|
|
||||||
# in CI and the build cache keeps it stable across rebuilds.
|
|
||||||
COPY package.json package-lock.json* ./
|
|
||||||
RUN npm install --no-audit --no-fund
|
|
||||||
|
|
||||||
# Build the production bundle into dist/. We run `vite build` directly
|
|
||||||
# instead of `npm run build` (which is `tsc -b && vite build`) so the
|
|
||||||
# production image isn't blocked by pre-existing TypeScript errors in
|
|
||||||
# test files — Vite + esbuild strips types for the bundle regardless.
|
|
||||||
# Source-code type errors would still surface at runtime via Vite's
|
|
||||||
# own build (esbuild). Run `npm run typecheck` separately to see them.
|
|
||||||
COPY . .
|
|
||||||
RUN npx vite build
|
|
||||||
|
|
||||||
# ---------- runtime ----------
|
|
||||||
FROM nginx:1.27-alpine
|
|
||||||
|
|
||||||
# Replace the default nginx site with ours (SPA + reverse proxy).
|
|
||||||
RUN rm -f /etc/nginx/conf.d/default.conf
|
|
||||||
|
|
||||||
COPY nginx.conf /etc/nginx/conf.d/default.conf
|
|
||||||
COPY --from=builder /build/dist /usr/share/nginx/html
|
|
||||||
|
|
||||||
# wget is on busybox; nginx:alpine doesn't ship curl.
|
|
||||||
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
|
|
||||||
CMD wget -qO- http://127.0.0.1:8080/ >/dev/null || exit 1
|
|
||||||
|
|
||||||
EXPOSE 8080
|
|
||||||
@@ -51,52 +51,6 @@ VITE_API_BASE_URL=http://127.0.0.1:8000
|
|||||||
Without that, the UI falls back to its in-memory sample store via the
|
Without that, the UI falls back to its in-memory sample store via the
|
||||||
existing `data` adapter (parses are disabled).
|
existing `data` adapter (parses are disabled).
|
||||||
|
|
||||||
## Pipeline automation agent
|
|
||||||
|
|
||||||
For unattended round-trips (an agent / scheduler drops an 837P file
|
|
||||||
into the pipeline and waits for the 999 back from Gainwell), see the
|
|
||||||
`cyclone-pipeline` sibling project at `/Users/openclaw/dev/cyclone-pipeline/`.
|
|
||||||
It drives the full 7-phase state machine — preflight → browser upload →
|
|
||||||
parse verification → SFTP submit → TA1 wait → 999 wait → scan +
|
|
||||||
report — with structured JSON logs, crash-safe resume, and a
|
|
||||||
self-contained per-run folder under `./runs/`.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Single file
|
|
||||||
cyclone-pipeline run /path/to/axiscare-837p.txt
|
|
||||||
|
|
||||||
# Resume a crashed run
|
|
||||||
cyclone-pipeline resume 2026-06-21-1430-001
|
|
||||||
|
|
||||||
# On/after the following Monday, verify the 835 arrived
|
|
||||||
cyclone-pipeline check-835 2026-06-21-1430-001
|
|
||||||
```
|
|
||||||
|
|
||||||
The 835 is **not** waited for inline (it lands the following Monday on
|
|
||||||
the CO Medicaid payment cycle). See
|
|
||||||
[`cyclone-pipeline/README.md`](../cyclone-pipeline/README.md) for
|
|
||||||
install, embed-in-agent example, exit codes, and the report format.
|
|
||||||
|
|
||||||
## Skills
|
|
||||||
|
|
||||||
Cyclone ships 8 project-scoped AI-assistant skills under
|
|
||||||
[`.superpowers/skills/`](.superpowers/skills/). Each one codifies the
|
|
||||||
conventions for a major subsystem so the next contributor (human or
|
|
||||||
AI) gets the lay of the land automatically.
|
|
||||||
|
|
||||||
| Skill | Owns |
|
|
||||||
|-------|------|
|
|
||||||
| [`cyclone-spec`](.superpowers/skills/cyclone-spec/SKILL.md) | The SP-N spec → plan → implement → merge flow. |
|
|
||||||
| [`cyclone-tests`](.superpowers/skills/cyclone-tests/SKILL.md) | pytest + vitest fixture patterns, prodfiles drop-in. |
|
|
||||||
| [`cyclone-edi`](.superpowers/skills/cyclone-edi/SKILL.md) | EDI parser/validator conventions (837P/835/999/270/271/277CA/TA1). |
|
|
||||||
| [`cyclone-tail`](.superpowers/skills/cyclone-tail/SKILL.md) | Live-tail streaming wire format and the hook triplet. |
|
|
||||||
| [`cyclone-store`](.superpowers/skills/cyclone-store/SKILL.md) | Store write-paths, pubsub event contract, SP21 split map. |
|
|
||||||
| [`cyclone-api-router`](.superpowers/skills/cyclone-api-router/SKILL.md) | FastAPI router conventions (`api_routers/`, `api_helpers.py`). |
|
|
||||||
| [`cyclone-frontend-page`](.superpowers/skills/cyclone-frontend-page/SKILL.md) | React page conventions (TanStack Query, drawer, URL state). |
|
|
||||||
| [`cyclone-cli`](.superpowers/skills/cyclone-cli/SKILL.md) | CLI subcommand conventions (`cli.py`, exit codes, smoke tests). |
|
|
||||||
|
|
||||||
Skills auto-load by description match — no slash command needed.
|
|
||||||
|
|
||||||
## Test
|
## Test
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -111,49 +65,6 @@ npm run build
|
|||||||
npm test
|
npm test
|
||||||
```
|
```
|
||||||
|
|
||||||
## Authentication
|
|
||||||
|
|
||||||
Cyclone ships with username/password authentication and three predefined roles.
|
|
||||||
|
|
||||||
**Roles:**
|
|
||||||
|
|
||||||
| Role | Can read | Can write (upload, parse, reconcile) | Can manage users |
|
|
||||||
| -------- | -------- | ------------------------------------ | ---------------- |
|
|
||||||
| `viewer` | ✅ | ❌ | ❌ |
|
|
||||||
| `user` | ✅ | ✅ | ❌ |
|
|
||||||
| `admin` | ✅ | ✅ | ✅ |
|
|
||||||
|
|
||||||
**Bootstrap.** On first start, set `CYCLONE_ADMIN_USERNAME` and
|
|
||||||
`CYCLONE_ADMIN_PASSWORD` (min 12 chars) in your environment. Cyclone creates the
|
|
||||||
first admin automatically. On subsequent starts these env vars are ignored, so
|
|
||||||
rotating the bootstrap password doesn't affect an already-seeded admin — use the
|
|
||||||
CLI below to reset it. When running via `docker compose`, both vars are
|
|
||||||
required: compose refuses to start with a clear error if either is missing.
|
|
||||||
|
|
||||||
**CLI.** Manage users from the command line:
|
|
||||||
|
|
||||||
```
|
|
||||||
python -m cyclone users create alice --role user --password 'hunter2hunter2'
|
|
||||||
python -m cyclone users list
|
|
||||||
python -m cyclone users disable alice
|
|
||||||
python -m cyclone users reset-password alice
|
|
||||||
python -m cyclone users set-role alice --role admin
|
|
||||||
```
|
|
||||||
|
|
||||||
**Login.** Browse to `http://localhost:5173` (dev) or `http://localhost:8081`
|
|
||||||
(Docker), sign in on the `/login` page, and you'll be redirected to the
|
|
||||||
dashboard. Sessions are stored server-side in SQLite with a 24-hour sliding
|
|
||||||
expiry — every authenticated request refreshes the TTL, so an active user
|
|
||||||
never gets logged out.
|
|
||||||
|
|
||||||
**Dev escape hatch.** Set `CYCLONE_AUTH_DISABLED=1` to bypass auth entirely
|
|
||||||
(the backend auto-grants admin on every request). **NEVER set this in
|
|
||||||
production** — it's a single env-var trip from wide-open to the public
|
|
||||||
internet. The Docker compose file does not honor this flag.
|
|
||||||
|
|
||||||
See `docs/superpowers/specs/2026-06-22-cyclone-auth-design.md` for the full
|
|
||||||
design.
|
|
||||||
|
|
||||||
## Live updates
|
## Live updates
|
||||||
|
|
||||||
The Claims, Remittances, and Activity pages stay current without
|
The Claims, Remittances, and Activity pages stay current without
|
||||||
@@ -244,6 +155,7 @@ parses and rejects claims, the inbox reflects the new
|
|||||||
| POST | `/api/inbox/candidates/{remit_id}/match` | Manual match. `409` if the claim state moved out from under us. |
|
| POST | `/api/inbox/candidates/{remit_id}/match` | Manual match. `409` if the claim state moved out from under us. |
|
||||||
| POST | `/api/inbox/candidates/dismiss` | `{pairs: [{claim_id, remit_id}]}`. Session-scoped. |
|
| POST | `/api/inbox/candidates/dismiss` | `{pairs: [{claim_id, remit_id}]}`. Session-scoped. |
|
||||||
| POST | `/api/inbox/rejected/resubmit` | `{claim_ids: [...]}`. `200` with `conflicts` for non-rejected. |
|
| POST | `/api/inbox/rejected/resubmit` | `{claim_ids: [...]}`. `200` with `conflicts` for non-rejected. |
|
||||||
|
| POST | `/api/inbox/payer-rejected/acknowledge` | `{claim_ids: [...], actor: "..."}`. Bulk-acknowledge payer rejections. Idempotent. `200` with `transitioned` / `already_acked` / `not_found` / `not_rejected` counts. Writes an audit event per transition; never overwrites the underlying 277CA rejection. |
|
||||||
| GET | `/api/inbox/export.csv?lane=<lane>` | Streams CSV of the lane's rows. |
|
| GET | `/api/inbox/export.csv?lane=<lane>` | Streams CSV of the lane's rows. |
|
||||||
|
|
||||||
## Outbound 837 Serializer
|
## Outbound 837 Serializer
|
||||||
@@ -410,8 +322,9 @@ both be true for the same claim.
|
|||||||
|
|
||||||
The `audit_log` table is the canonical record of every state transition
|
The `audit_log` table is the canonical record of every state transition
|
||||||
the system has ever observed — claim lifecycle, reconciliation
|
the system has ever observed — claim lifecycle, reconciliation
|
||||||
decisions, config reloads, SFTP submissions, 277CA rejects. SP11 made
|
decisions, config reloads, SFTP submissions, 277CA rejects, Payer-Rejected
|
||||||
it tamper-evident: every row carries a SHA-256 hash of
|
acknowledgements, SQLCipher key rotations. SP11 made it tamper-evident:
|
||||||
|
every row carries a SHA-256 hash of
|
||||||
`(prev_hash || row_payload)`, forming a chain back to a genesis row.
|
`(prev_hash || row_payload)`, forming a chain back to a genesis row.
|
||||||
Any `INSERT`, `UPDATE`, or `DELETE` that breaks the chain is detectable
|
Any `INSERT`, `UPDATE`, or `DELETE` that breaks the chain is detectable
|
||||||
in a single walk.
|
in a single walk.
|
||||||
@@ -446,244 +359,39 @@ operator has created the Keychain entry on first run. See
|
|||||||
for the one-time setup recipe and the HIPAA Security Rule §164.312(a)(2)(iv)
|
for the one-time setup recipe and the HIPAA Security Rule §164.312(a)(2)(iv)
|
||||||
mapping.
|
mapping.
|
||||||
|
|
||||||
### Key rotation (SP15)
|
### Key rotation
|
||||||
|
|
||||||
`POST /api/admin/db/rotate-key` re-encrypts the SQLite file in place
|
The DB encryption key is rotated in place via `POST /api/admin/db/rotate-key`.
|
||||||
with a fresh SQLCipher key via `PRAGMA rekey`, then updates the
|
The handler:
|
||||||
Keychain so subsequent connections open with the new key. The
|
|
||||||
rotation holds a module-level `threading.Lock` (so two concurrent
|
|
||||||
requests can't race), disposes + rebuilds the SQLAlchemy engine with
|
|
||||||
`NullPool` (so SQLCipher's thread affinity is honored), and writes a
|
|
||||||
tamper-evident `db.key_rotated` audit event with old + new
|
|
||||||
fingerprints and the post-rotation table count. The old key is
|
|
||||||
retained in the `cyclone.db.key.previous` Keychain account for a
|
|
||||||
grace period so a botched rotation can be rolled back by hand.
|
|
||||||
|
|
||||||
## NPI checksum + Tax ID format validation (SP20)
|
1. Generates a fresh 256-bit CSPRNG key (`db_crypto.generate_db_key()`).
|
||||||
|
2. Persists the new key to the Keychain under the same
|
||||||
|
`cyclone.db.key` account (overwriting the old one).
|
||||||
|
3. Disposes the SQLAlchemy engine so pooled connections release the
|
||||||
|
file (SQLCipher refuses to `PRAGMA rekey` while another connection
|
||||||
|
holds the DB).
|
||||||
|
4. Opens with the old key, issues `PRAGMA rekey`, and verifies the
|
||||||
|
schema survived (table-count sanity check).
|
||||||
|
5. Rebuilds the engine with the new key.
|
||||||
|
6. Writes a `db.key_rotated` audit event carrying the SHA-256
|
||||||
|
fingerprints of the old and new keys (first 8 hex chars) plus the
|
||||||
|
post-rotation `table_count`.
|
||||||
|
|
||||||
Two pure local validators — no NPPES round-trip, no IRS e-file
|
When SQLCipher is enabled the engine uses SQLAlchemy's `NullPool`
|
||||||
lookup. Catches the 99% case (a typo at the end of an NPI, a letter in
|
instead of the default `QueuePool`. `QueuePool` returns connections to
|
||||||
an EIN, an extra digit, the reserved `00`/`07`/`8X` EIN prefix).
|
a shared queue that any thread can pull from, which breaks SQLCipher's
|
||||||
|
thread affinity. `NullPool` trades connection reuse for thread safety
|
||||||
|
— the only correct behavior under FastAPI's per-request threadpool.
|
||||||
|
|
||||||
| Check | Algorithm | Surface |
|
The rotation endpoint is serialized with a module-level
|
||||||
|-------|-----------|---------|
|
`threading.Lock` (one rotation in flight at a time), returns `409` if
|
||||||
| NPI | 10 digits where the last is a Luhn checksum over `80840 + body`. CMS-published example: body `123456789` → check `3` → valid NPI `1234567893`. | `cyclone.npi.is_valid_npi`, CLI `cyclone validate-npi <npi>`, API `GET /api/admin/validate-provider?npi=...`, validator rule `R021_npi_checksum` (warning) |
|
a rotation is already running, `400` if encryption is not enabled,
|
||||||
| Tax ID (EIN) | 9 digits, optional `XX-XXXXXXX` formatting. Rejects reserved prefixes `00`, `07`, `80`–`89` (IRS Pension Plan Branch). | `cyclone.npi.is_valid_tax_id`, CLI `cyclone validate-tax-id <ein>`, API `GET /api/admin/validate-provider?tax_id=...` |
|
and `503` with a `reason` on `PRAGMA rekey` or Keychain failure so the
|
||||||
|
operator can take the next step without parsing the traceback.
|
||||||
|
|
||||||
### CLI
|
| Method | Path | Notes |
|
||||||
|
| ------ | --------------------------------- | -------------------------------------------------------------- |
|
||||||
```bash
|
| POST | `/api/admin/db/rotate-key` | Rotate the SQLCipher key in place. Audit-logged. |
|
||||||
$ cyclone validate-npi 1234567893
|
|
||||||
OK: 10-digit NPI passes Luhn checksum
|
|
||||||
$ cyclone validate-npi 1234567890
|
|
||||||
INVALID: '1234567890' fails NPI Luhn checksum # exit 1
|
|
||||||
|
|
||||||
$ cyclone validate-tax-id 72-1587149
|
|
||||||
OK: 9-digit EIN (normalized=721587149)
|
|
||||||
$ cyclone validate-tax-id 00-1234567
|
|
||||||
INVALID: 9-digit EIN has reserved prefix (00); EIN is not assignable by IRS # exit 1
|
|
||||||
```
|
|
||||||
|
|
||||||
### API
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl 'http://localhost:8000/api/admin/validate-provider?npi=1234567893&tax_id=72-1587149'
|
|
||||||
# {
|
|
||||||
# "npi": {"valid": true, "skipped": false},
|
|
||||||
# "tax_id": {"valid": true, "skipped": false, "normalized": "721587149"}
|
|
||||||
# }
|
|
||||||
```
|
|
||||||
|
|
||||||
Both query params are optional; omitted fields return
|
|
||||||
`{"valid": null, "skipped": true}` so the caller can render "no check
|
|
||||||
performed" rather than treating absent input as a hard fail.
|
|
||||||
|
|
||||||
### Parser integration
|
|
||||||
|
|
||||||
The `R021_npi_checksum` rule runs alongside the existing `R020_npi_format`
|
|
||||||
in `cyclone.parsers.validator`. A billing-provider NPI that passes
|
|
||||||
R020 (right shape) but fails R021 (bad Luhn) is yielded as a
|
|
||||||
**warning**, not an error — operators sometimes ingest test fixtures
|
|
||||||
with placeholder NPIs (e.g. all-same-digit) and we don't want to block
|
|
||||||
that path. In strict mode (`--strict` / `?strict=true`) warnings are
|
|
||||||
promoted to errors.
|
|
||||||
|
|
||||||
### Files
|
|
||||||
|
|
||||||
* `cyclone/npi.py` — new module (~155 LOC).
|
|
||||||
* `cyclone.parsers.validator` — new `R021_npi_checksum` rule.
|
|
||||||
* `cyclone.api_routers.admin` — new `validate-provider` endpoint.
|
|
||||||
* `cyclone.cli` — `validate-npi` + `validate-tax-id` subcommands.
|
|
||||||
* Tests: `test_npi.py` (27), `test_api_validate_provider.py` (4),
|
|
||||||
`test_cli_validate.py` (8), `test_validator.py::test_r021_*` (4) —
|
|
||||||
**43 new tests**.
|
|
||||||
|
|
||||||
## Security hardening (SP19)
|
|
||||||
|
|
||||||
Three pure-ASGI middlewares sit in front of every FastAPI request.
|
|
||||||
They're sized for Cyclone's local-only posture — a misconfigured
|
|
||||||
Tailscale / ngrok bind, a buggy cron job uploading a 4 GB file, or a
|
|
||||||
port-scraper — not for hostile internet exposure.
|
|
||||||
|
|
||||||
| Middleware | Default | Override | Reject |
|
|
||||||
|------------|---------|----------|--------|
|
|
||||||
| `BodySizeLimitMiddleware` | 50 MB | `CYCLONE_MAX_BODY_BYTES` | `413 body_too_large` over Content-Length cap; chunked reads capped too |
|
|
||||||
| `RateLimitMiddleware` | 300 req/min/IP | `CYCLONE_RATE_LIMIT_PER_MIN` | `429 rate_limited` over the sliding window; `/api/health` exempt |
|
|
||||||
| `SecurityHeadersMiddleware` | always on | n/a | stamps `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, `Referrer-Policy: same-origin`, `Permissions-Policy`, `Content-Security-Policy: default-src 'none'; frame-ancestors 'none'` |
|
|
||||||
|
|
||||||
Every rejection (413 / 429) also writes a tamper-evident
|
|
||||||
`api.request_rejected` event into the SP11 audit chain so an
|
|
||||||
operator can correlate a misbehaving client with the SP18 JSON logs:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"event_type":"api.request_rejected","entity_id":"POST /api/parse-837","payload":{"status":413,"reason":"body_too_large","path":"/api/parse-837","method":"POST","ip":"127.0.0.1"}}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Health probe
|
|
||||||
|
|
||||||
`GET /api/health` now returns a subsystem snapshot:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"status": "ok",
|
|
||||||
"version": "0.1.0",
|
|
||||||
"db": {"ok": true},
|
|
||||||
"scheduler": {"running": true, "interval_s": 60, "sftp_block": "co_medicaid",
|
|
||||||
"backup_scheduler_running": false, "backup_interval_hours": 24.0},
|
|
||||||
"pubsub": {"parse_completed": 1, "batch_added": 1},
|
|
||||||
"batch": {"last_batch_id": 42, "last_batch_kind": "837P",
|
|
||||||
"last_batch_at": "2026-06-21T15:30:00.123Z",
|
|
||||||
"last_batch_filename": "TP11525703-837P-..."}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Returns `"status": "degraded"` if any subsystem reports an error —
|
|
||||||
the per-subsystem dict still surfaces so an operator can see which
|
|
||||||
one is unhappy. `/api/health` is rate-limit exempt so a load balancer
|
|
||||||
hammering the endpoint doesn't trip the limiter.
|
|
||||||
|
|
||||||
### Files
|
|
||||||
|
|
||||||
* `cyclone.security` — `BodySizeLimitMiddleware`,
|
|
||||||
`RateLimitMiddleware`, `SecurityHeadersMiddleware`, and
|
|
||||||
`get_health_snapshot()` (~330 LOC).
|
|
||||||
* `cyclone.api_routers.health` rewritten to use `get_health_snapshot()`.
|
|
||||||
* `cyclone.pubsub.EventBus.stats()` — new method that returns
|
|
||||||
per-kind subscriber counts.
|
|
||||||
* `tests/test_security.py` — 13 new tests.
|
|
||||||
|
|
||||||
## Structured logging (SP18)
|
|
||||||
|
|
||||||
Cyclone emits newline-delimited JSON to stderr by default — readable
|
|
||||||
by `jq`, Loki, Vector, ELK, or any log shipper. Every record carries
|
|
||||||
`ts` (ISO 8601 ms UTC), `level`, `logger`, `msg`, and an optional
|
|
||||||
`extra` dict for structured fields. Exceptions render as a
|
|
||||||
`traceback` string.
|
|
||||||
|
|
||||||
```
|
|
||||||
{"ts":"2026-06-21T15:30:00.123Z","level":"INFO","logger":"cyclone.scheduler","msg":"Processed inbound file","extra":{"input_filename":"ACK_999.x12","parser":"parse_999","claims":3}}
|
|
||||||
{"ts":"2026-06-21T15:30:01.456Z","level":"ERROR","logger":"cyclone.api","msg":"Backup create failed","extra":{"reason":"BackupError: passphrase mismatch"},"traceback":"Traceback ..."}
|
|
||||||
```
|
|
||||||
|
|
||||||
### PII scrubbing
|
|
||||||
|
|
||||||
A `PiiScrubber` filter is attached to the root logger and rewrites
|
|
||||||
obvious PHI patterns to `<redacted:npi>` / `<redacted:ssn>` /
|
|
||||||
`<redacted:dob>` / `<redacted:patient_name>` before any handler sees
|
|
||||||
the record:
|
|
||||||
|
|
||||||
| Pattern | Replacement |
|
|
||||||
|---------|-------------|
|
|
||||||
| `\b\d{10}\b` | `<redacted:npi>` |
|
|
||||||
| `\b\d{3}-\d{2}-\d{4}\b` or `\b\d{9}\b` at phrase boundary | `<redacted:ssn>` |
|
|
||||||
| `(dob\|date_of_birth)[:=]\s*\d{4}-\d{2}-\d{2}` | preserves the key, redacts the date |
|
|
||||||
| `patient_name=...` | full chunk redacted |
|
|
||||||
| Extras with key `dob`/`ssn`/`npi`/`patient_name`/… | value redacted regardless of shape |
|
|
||||||
|
|
||||||
The scrubber is conservative — bare ISO dates without a `dob=` prefix
|
|
||||||
are **not** scrubbed (they're too often timestamps or batch IDs), and
|
|
||||||
11+ digit numbers are left alone (they can't be NPIs). Disable for
|
|
||||||
forensic mode with `CYCLONE_LOG_NO_PII_SCRUB=1`.
|
|
||||||
|
|
||||||
### Knobs
|
|
||||||
|
|
||||||
| Env var | Default | Meaning |
|
|
||||||
|---------|---------|---------|
|
|
||||||
| `CYCLONE_LOG_LEVEL` | `INFO` | Root logger level. `DEBUG` for troubleshooting. |
|
|
||||||
| `CYCLONE_LOG_FILE` | (none) | Write to this path via `RotatingFileHandler` (10 MB × 5 backups). |
|
|
||||||
| `CYCLONE_LOG_JSON` | `true` | `false` uses the dev tabular formatter. |
|
|
||||||
| `CYCLONE_LOG_NO_PII_SCRUB` | (none) | `1` disables scrubbing. |
|
|
||||||
|
|
||||||
CLI:
|
|
||||||
|
|
||||||
```
|
|
||||||
cyclone --log-format=dev parse-837 sample.x12 --output-dir out/ # tabular for tail -f
|
|
||||||
cyclone --log-file=/var/log/cyclone.log backup create # JSON to rotating file
|
|
||||||
```
|
|
||||||
|
|
||||||
The `parse-837` / `parse-835` subcommands also accept `--log-level`
|
|
||||||
which re-runs `setup_logging()` so the per-invocation level overrides
|
|
||||||
the group default.
|
|
||||||
|
|
||||||
### Files
|
|
||||||
|
|
||||||
* `cyclone.logging_config` — `JsonFormatter`, `CycloneDevFormatter`,
|
|
||||||
`PiiScrubber`, `setup_logging()`.
|
|
||||||
* `tests/test_logging_formatter.py` (11), `test_logging_scrubber.py`
|
|
||||||
(13), `test_logging_setup.py` (10) — 34 new tests.
|
|
||||||
* `cyclone.api` lifespan calls `setup_logging()` first; the CLI
|
|
||||||
`main` group does the same.
|
|
||||||
|
|
||||||
## Encrypted Backups (SP17)
|
|
||||||
|
|
||||||
The BackupService takes an online consistent snapshot of the live
|
|
||||||
SQLite file via SQLite's `.backup()` API, encrypts the bytes with
|
|
||||||
AES-256-GCM, and writes a `.bin` + `.meta.json` pair into the backup
|
|
||||||
directory (default `~/.local/share/cyclone/backups/`). The encryption
|
|
||||||
key is derived from a separate passphrase in the macOS Keychain
|
|
||||||
(PBKDF2-HMAC-SHA256, 200,000 iterations, 16-byte salt persisted to
|
|
||||||
Keychain) — so a SQLCipher DB-key compromise does not unlock the
|
|
||||||
backups, and a backup-passphrase compromise does not unlock the live
|
|
||||||
DB. If neither is set, the service refuses (`BackupError`) rather than
|
|
||||||
silently writing plaintext.
|
|
||||||
|
|
||||||
| Method | Path | Purpose |
|
|
||||||
| ------ | ---- | ------- |
|
|
||||||
| POST | `/api/admin/backup/create` | Take an encrypted backup now. |
|
|
||||||
| GET | `/api/admin/backup/list` | List `db_backups` rows (newest first, filterable). |
|
|
||||||
| GET | `/api/admin/backup/status` | Counts, disk usage, last-run timestamp, scheduler snapshot. |
|
|
||||||
| POST | `/api/admin/backup/{id}/verify` | Decrypt + SHA-256 verify against the sidecar. |
|
|
||||||
| POST | `/api/admin/backup/{id}/restore/initiate` | First step: get `restore_token` + preview (fingerprints of backup vs live). |
|
|
||||||
| POST | `/api/admin/backup/{id}/restore/confirm` | Second step: dispose engine, copy decrypted DB, rebuild engine. |
|
|
||||||
| POST | `/api/admin/backup/prune` | Apply retention policy now. |
|
|
||||||
| POST | `/api/admin/backup/scheduler/{start,stop,tick}` | Operate the backup scheduler. |
|
|
||||||
|
|
||||||
Restore is two-step by design: an idle browser tab can't nuke the
|
|
||||||
live DB. The first call returns a one-shot 64-char hex
|
|
||||||
`restore_token` plus a side-by-side preview (`backup_db_fingerprint`,
|
|
||||||
`backup_table_count`, `current_db_fingerprint`, `current_table_count`).
|
|
||||||
The second call swaps the live engine only if the token matches
|
|
||||||
within a 5-minute TTL.
|
|
||||||
|
|
||||||
The scheduler (auto-start opt-in via `CYCLONE_BACKUP_AUTOSTART`)
|
|
||||||
ticks every `CYCLONE_BACKUP_INTERVAL_HOURS` (default 24), runs
|
|
||||||
`create_now` + `prune`, and writes audit events for each outcome
|
|
||||||
(`db.backup_created`, `db.backup_failed`, `db.backup_pruned`,
|
|
||||||
`db.backup_restored`). The CLI mirrors the API surface:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cyclone backup init-passphrase # one-time; interactive
|
|
||||||
cyclone backup create
|
|
||||||
cyclone backup list
|
|
||||||
cyclone backup verify <id>
|
|
||||||
cyclone backup restore <id> --yes
|
|
||||||
cyclone backup prune --yes
|
|
||||||
cyclone backup status
|
|
||||||
```
|
|
||||||
|
|
||||||
Retention defaults to 30 days (`CYCLONE_BACKUP_RETENTION_DAYS`). The
|
|
||||||
retention policy is best-effort: an operator who runs `cyclone
|
|
||||||
backup create` manually retains full control.
|
|
||||||
|
|
||||||
## SFTP Wire-Up (paramiko)
|
## SFTP Wire-Up (paramiko)
|
||||||
|
|
||||||
@@ -743,8 +451,12 @@ backup API).
|
|||||||
.
|
.
|
||||||
├── backend/
|
├── backend/
|
||||||
│ ├── src/cyclone/
|
│ ├── src/cyclone/
|
||||||
│ │ ├── api.py # FastAPI app, GET + parse routes, /api/{resource}/stream
|
│ │ ├── api.py # FastAPI app + parse routes; mounts api_routers/* sub-apps
|
||||||
│ │ ├── api_helpers.py # NDJSON / content-negotiation / live-tail helpers
|
│ │ ├── api_helpers.py # NDJSON / content-negotiation / live-tail helpers
|
||||||
|
│ │ ├── api_routers/ # FastAPI APIRouters extracted from api.py
|
||||||
|
│ │ │ ├── health.py # GET /api/health
|
||||||
|
│ │ │ ├── acks.py # GET /api/acks, /api/acks/{id}
|
||||||
|
│ │ │ └── ta1_acks.py # GET /api/ta1-acks, /api/ta1-acks/{id}
|
||||||
│ │ ├── pubsub.py # in-process EventBus (drop-oldest, per-kind fan-out)
|
│ │ ├── pubsub.py # in-process EventBus (drop-oldest, per-kind fan-out)
|
||||||
│ │ ├── store.py # CycloneStore, mappers, publish-on-write
|
│ │ ├── store.py # CycloneStore, mappers, publish-on-write
|
||||||
│ │ ├── db.py # SQLAlchemy engine, session factory, ORM models
|
│ │ ├── db.py # SQLAlchemy engine, session factory, ORM models
|
||||||
@@ -801,13 +513,7 @@ backup API).
|
|||||||
|
|
||||||
## Roadmap
|
## Roadmap
|
||||||
|
|
||||||
> **Read order for new engineers:**
|
Sub-projects 2 through 15 are **shipped**. See the [completeness
|
||||||
> 1. [`docs/REQUIREMENTS.md`](docs/REQUIREMENTS.md) — what Cyclone does (FRs + NFRs + DoD + traceability). The single index tying the 22 shipped sub-projects to the 38 functional + 18 non-functional requirements and the test strategy.
|
|
||||||
> 2. [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — how it fits together (process topology, package layout, data flow, lifecycle, operational concerns).
|
|
||||||
> 3. The per-SP spec under [`docs/superpowers/specs/`](docs/superpowers/specs/) for whatever you're touching.
|
|
||||||
> 4. The per-SP plan under [`docs/superpowers/plans/`](docs/superpowers/plans/) if you're implementing.
|
|
||||||
|
|
||||||
Sub-projects 2 through 19 are **shipped**. See the [completeness
|
|
||||||
review](docs/reviews/2026-06-20-cyclone-completeness-review.md) for
|
review](docs/reviews/2026-06-20-cyclone-completeness-review.md) for
|
||||||
the honest gap analysis against the industry definition of a HIPAA
|
the honest gap analysis against the industry definition of a HIPAA
|
||||||
clearinghouse — the short version is that the local-only,
|
clearinghouse — the short version is that the local-only,
|
||||||
@@ -818,70 +524,6 @@ scope.
|
|||||||
|
|
||||||
Shipped sub-projects (most recent first):
|
Shipped sub-projects (most recent first):
|
||||||
|
|
||||||
- **Sub-project 22 (shipped) — Pipeline automation agent.** A
|
|
||||||
sibling project at `/Users/openclaw/dev/cyclone-pipeline` that
|
|
||||||
drives the full 7-phase round-trip (preflight → browser upload →
|
|
||||||
parse verify → SFTP submit → TA1 wait → 999 wait → scan + report)
|
|
||||||
with crash-safe resume, structured JSON logging, idempotency
|
|
||||||
dedup, and a per-run report. Pure Python 3.11+ (httpx, Playwright,
|
|
||||||
Click, pydantic v2, structlog). The 835 is not waited for inline —
|
|
||||||
it lands the following Monday — and is verified by a separate
|
|
||||||
`check-835` subcommand. Embeddable as a library for OpenClaw / Nora
|
|
||||||
agent integration. See
|
|
||||||
[Pipeline automation agent](#pipeline-automation-agent) above.
|
|
||||||
- **Sub-project 19 (shipped) — Security hardening + health probe.**
|
|
||||||
Three pure-ASGI middlewares (`BodySizeLimitMiddleware`,
|
|
||||||
`RateLimitMiddleware`, `SecurityHeadersMiddleware`) close the
|
|
||||||
completeness-review gaps §3.1.4 (no body/rate limits) and §3.1.25
|
|
||||||
(no CSP / security headers). 413/429 rejections emit a
|
|
||||||
tamper-evident `api.request_rejected` audit event (SP11 chain).
|
|
||||||
`/api/health` is now a rich subsystem snapshot — DB connectivity,
|
|
||||||
MFT scheduler state, backup scheduler state, live pubsub
|
|
||||||
subscriber counts, last batch id + timestamp. See
|
|
||||||
[Security hardening (SP19)](#security-hardening-sp19) below.
|
|
||||||
- **Sub-project 20 (shipped) — NPI checksum + Tax ID format validation.**
|
|
||||||
Pure local validators (`cyclone.npi`) — no NPPES round-trip, no IRS
|
|
||||||
e-file lookup. Catches the 99% typo case at parse time. NPI uses
|
|
||||||
CMS-published Luhn over `80840 + body` (example: `1234567893` is
|
|
||||||
valid). EIN rejects reserved prefixes (`00`, `07`, `80`–`89`).
|
|
||||||
Surface: `cyclone validate-npi` / `validate-tax-id` CLI subcommands,
|
|
||||||
`GET /api/admin/validate-provider`, new `R021_npi_checksum`
|
|
||||||
validator rule (warning, not error — placeholder NPIs in test
|
|
||||||
fixtures shouldn't block ingest). See
|
|
||||||
[NPI checksum + Tax ID format validation (SP20)](#npi-checksum--tax-id-format-validation-sp20)
|
|
||||||
below.
|
|
||||||
- **Sub-project 18 (shipped) — Structured JSON logging.** All logs
|
|
||||||
emitted by the API, CLI, scheduler tick loop, and backup service
|
|
||||||
flow through a `JsonFormatter` (newline-delimited JSON, ISO-8601 ms
|
|
||||||
timestamps) by default. A `PiiScrubber` filter redacts obvious PHI
|
|
||||||
(NPIs, SSNs, DOBs, patient names) from message + extras — both via
|
|
||||||
inline patterns (`npi 1881068062`) and via PHI-keyed extras
|
|
||||||
(`extra={"dob": "1980-04-12"}`). Configurable via env vars
|
|
||||||
(`CYCLONE_LOG_LEVEL`, `CYCLONE_LOG_FILE`, `CYCLONE_LOG_JSON`,
|
|
||||||
`CYCLONE_LOG_NO_PII_SCRUB`) and CLI flags
|
|
||||||
(`--log-format=json|dev`, `--log-file=…`); a tabular `CycloneDevFormatter`
|
|
||||||
is the opt-out for `tail -f` in dev. See
|
|
||||||
[Structured logging](#structured-logging-sp18) below.
|
|
||||||
- **Sub-project 17 (shipped) — Encrypted DB backups.** Automated
|
|
||||||
encrypted backups via AES-256-GCM (PBKDF2-HMAC-SHA256, 200k iters).
|
|
||||||
The operator sets a separate passphrase in the macOS Keychain
|
|
||||||
(`cyclone backup init-passphrase`); if missing, the service falls
|
|
||||||
back to deriving from the SQLCipher DB key with a WARNING. Online
|
|
||||||
backups via SQLite `.backup()`, two-step restore (`initiate` →
|
|
||||||
`confirm` with one-shot 64-char hex token), retention pruning with
|
|
||||||
a 30-day default, and a tamper-evident audit chain (`db.backup_created`,
|
|
||||||
`db.backup_failed`, `db.backup_pruned`, `db.backup_restored`,
|
|
||||||
`db.backup_passphrase_set`). Backup scheduler ticks every 24h
|
|
||||||
(configurable); auto-start opt-in via `CYCLONE_BACKUP_AUTOSTART`.
|
|
||||||
Seven admin endpoints + six CLI subcommands. See
|
|
||||||
[Encrypted Backups](#encrypted-backups) below.
|
|
||||||
- **Sub-project 16 (shipped) — Live MFT polling scheduler.** asyncio
|
|
||||||
background loop polls the Gainwell MFT inbound path, downloads
|
|
||||||
new files, and routes them through the right parser (999 / 835 /
|
|
||||||
277CA / TA1). Idempotent (re-ticks skip already-processed files
|
|
||||||
via the new `processed_inbound_files` table). Crash-safe (per-file
|
|
||||||
try/except so a bad file doesn't stop the loop). Five admin
|
|
||||||
endpoints (`/api/admin/scheduler/{status,start,stop,tick,processed-files}`).
|
|
||||||
- **Sub-project 15 (shipped) — SQLCipher key rotation.** In-place
|
- **Sub-project 15 (shipped) — SQLCipher key rotation.** In-place
|
||||||
rotation via `PRAGMA rekey`, serialized through a module-level
|
rotation via `PRAGMA rekey`, serialized through a module-level
|
||||||
`threading.Lock` and a SQLAlchemy `NullPool` to keep SQLCipher
|
`threading.Lock` and a SQLAlchemy `NullPool` to keep SQLCipher
|
||||||
@@ -1163,6 +805,34 @@ the one-time setup recipe.
|
|||||||
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
|
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
|
||||||
SFTP credentials are fetched from the macOS Keychain at call time.
|
SFTP credentials are fetched from the macOS Keychain at call time.
|
||||||
|
|
||||||
|
### SP14 endpoints (5-lane Inbox UI + acknowledge)
|
||||||
|
|
||||||
|
- `GET /api/inbox/lanes` — same shape as SP6; the `payer_rejected`
|
||||||
|
lane payload now also includes
|
||||||
|
`payer_rejected_acknowledged_at` + `payer_rejected_acknowledged_actor`
|
||||||
|
per row so the UI can badge acknowledged claims (forward-compat for
|
||||||
|
a future "Recently acknowledged" view).
|
||||||
|
- `POST /api/inbox/payer-rejected/acknowledge` — bulk-acknowledge
|
||||||
|
Payer-Rejected claims. Body: `{claim_ids: [...], actor: "..."}`.
|
||||||
|
Idempotent. Response: `200` with
|
||||||
|
`{transitioned, already_acked, not_found, not_rejected}`. Writes an
|
||||||
|
`inbox.payer_rejected_acknowledged` audit event per transition.
|
||||||
|
Acknowledgement hides a claim from the lane but never overwrites
|
||||||
|
`payer_rejected_at` / `payer_rejected_reason` /
|
||||||
|
`payer_rejected_by_277ca_id` — the original 277CA evidence stays
|
||||||
|
intact in the audit log.
|
||||||
|
|
||||||
|
### SP15 endpoints (SQLCipher key rotation)
|
||||||
|
|
||||||
|
- `POST /api/admin/db/rotate-key` — rotate the SQLCipher DB key in
|
||||||
|
place. Generates a fresh 256-bit key, writes it to the Keychain
|
||||||
|
(overwriting `cyclone.db.key`), disposes the engine, issues
|
||||||
|
`PRAGMA rekey`, verifies the schema, rebuilds the engine. Writes
|
||||||
|
a `db.key_rotated` audit event with old + new key fingerprints
|
||||||
|
and `table_count`. Returns `409` when a rotation is already in
|
||||||
|
flight, `400` when encryption is not enabled, `503` with a
|
||||||
|
`reason` on `PRAGMA rekey` or Keychain failure.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
No license file yet; this is internal-use software. Add a `LICENSE` file
|
No license file yet; this is internal-use software. Add a `LICENSE` file
|
||||||
|
|||||||
-65
@@ -1,65 +0,0 @@
|
|||||||
# Cyclone Operator Runbook
|
|
||||||
|
|
||||||
Production operations for a single-operator Cyclone deploy on Ubuntu Linux. Assumes the box was bootstrapped via `scripts/cyclone-init.sh` and the stack is up via `docker compose up -d`.
|
|
||||||
|
|
||||||
## Daily
|
|
||||||
|
|
||||||
- [ ] Confirm the host healthcheck cron hasn't emailed. It pings `http://localhost:8080/api/health` every 5 minutes.
|
|
||||||
- [ ] `docker compose ps` — both services `healthy`.
|
|
||||||
- [ ] `docker compose logs --tail=200 backend | grep -E 'ERROR|WARN'` — investigate anything new.
|
|
||||||
|
|
||||||
## Weekly
|
|
||||||
|
|
||||||
- [ ] `curl -fsS http://localhost:8080/api/admin/audit-log -b cookies.txt | jq '.events[] | select(.event | test("login_failed|backup.failed"))'` — review failed logins + backup failures.
|
|
||||||
- [ ] Confirm `docker compose exec backend ls -la /var/lib/cyclone/backups/` shows recent `.bin` files (within 25h of now).
|
|
||||||
|
|
||||||
## Quarterly
|
|
||||||
|
|
||||||
- [ ] Rotate the SQLCipher / cookie-signing key:
|
|
||||||
```bash
|
|
||||||
bash scripts/cyclone-init.sh --force # overwrites /etc/cyclone/secrets/db.key
|
|
||||||
docker compose restart backend # picks up the new key
|
|
||||||
```
|
|
||||||
Old `.bin` backups become unreadable after this; export them first if you need to keep them.
|
|
||||||
|
|
||||||
## As needed
|
|
||||||
|
|
||||||
- **Add an operator.** Log in as admin → `/admin/users` → Create user. Roles: `admin` / `user` / `viewer`.
|
|
||||||
- **Reset a password.** Admin UI → Users → Reset password, OR `docker compose exec backend python -m cyclone admin reset-password --username <name>`.
|
|
||||||
- **Restore from backup.** Admin UI → Backups → pick the snapshot → Initiate restore → Confirm. The backend will restart automatically.
|
|
||||||
- **Roll back the code (not the schema).** `TAG=0.0.9 docker compose up -d`. The previous image stays in the local Docker cache for one cycle.
|
|
||||||
- **Pull a new `:stable`.**
|
|
||||||
```bash
|
|
||||||
cd /opt/cyclone
|
|
||||||
docker compose pull
|
|
||||||
docker compose up -d
|
|
||||||
docker compose logs -f backend | head -200 # verify migrations + healthcheck
|
|
||||||
```
|
|
||||||
- **Off-box backup copy.** The operator is expected to rsync `/var/lib/docker/volumes/cyclone_backups/_data/` to an external drive or NAS nightly. The `.bin` files are already encrypted; the destination doesn't need its own encryption.
|
|
||||||
- **Inspect the DB.** `docker compose exec backend sqlite3 /var/lib/cyclone/db/cyclone.db ".tables"` (works only if SQLCipher key is on disk; the in-process decrypt happens via the cyclone backend).
|
|
||||||
|
|
||||||
## Annual
|
|
||||||
|
|
||||||
- [ ] Rotate the admin password (force re-login for everyone).
|
|
||||||
- [ ] Audit the `/etc/cyclone/secrets/` directory permissions — should be `chmod 600 root:root`.
|
|
||||||
- [ ] Review the audit log for stale admin sessions.
|
|
||||||
|
|
||||||
## Emergency
|
|
||||||
|
|
||||||
- **Backend won't start.** `docker compose logs --tail=300 backend`. Look for migration failures (rerun is safe — migrations are forward-only), SQLCipher key mismatch (`PRAGMA key` failure), or port collisions.
|
|
||||||
- **Frontend won't serve.** `docker compose logs --tail=100 frontend`. Usually nginx config drift; `docker compose restart frontend`.
|
|
||||||
- **Both unhealthy after a host reboot.** Docker may have come up before the named volumes did. `docker compose down && docker compose up -d`.
|
|
||||||
- **Suspected key compromise.** Rotate immediately (see Quarterly above). All active sessions are invalidated.
|
|
||||||
|
|
||||||
## Where things live
|
|
||||||
|
|
||||||
| Asset | Path |
|
|
||||||
|---|---|
|
|
||||||
| Docker compose file | `/opt/cyclone/docker-compose.yml` |
|
|
||||||
| Secrets | `/etc/cyclone/secrets/{db.key,admin_username,admin_pw}` |
|
|
||||||
| Live DB (SQLCipher-encrypted volume) | `cyclone_db` named volume, mounted at `/var/lib/cyclone/db` |
|
|
||||||
| Encrypted backups | `cyclone_backups` named volume, mounted at `/var/lib/cyclone/backups` |
|
|
||||||
| Uploaded prod files | `cyclone_prodfiles` named volume |
|
|
||||||
| SFTP staging stub | `cyclone_sftp_staging` named volume |
|
|
||||||
| Logs | `cyclone_logs` named volume + bind-mounted at `/var/log/cyclone` |
|
|
||||||
| Off-box backup destination | Operator's external drive / NAS (rsync cron, not in compose) |
|
|
||||||
-233
@@ -1,233 +0,0 @@
|
|||||||
// UI/UX Score Loop — pass 1 driver.
|
|
||||||
// Loads each route at three sizes in Chrome Canary, captures screenshots,
|
|
||||||
// logs console errors, runs a small interaction probe per flow, and
|
|
||||||
// writes a JSON report. Does not modify any source files.
|
|
||||||
|
|
||||||
import puppeteer from "puppeteer-core";
|
|
||||||
import { mkdir, writeFile } from "node:fs/promises";
|
|
||||||
|
|
||||||
const BASE = "http://127.0.0.1:5173";
|
|
||||||
const SHOTS = "/tmp/cyclone-uiux/shots";
|
|
||||||
const REPORT = "/tmp/cyclone-uiux/report.json";
|
|
||||||
|
|
||||||
const SIZES = [
|
|
||||||
{ name: "desktop", w: 1440, h: 900 },
|
|
||||||
{ name: "tablet", w: 768, h: 1024 },
|
|
||||||
{ name: "mobile", w: 375, h: 812 },
|
|
||||||
];
|
|
||||||
|
|
||||||
// Routes to load. path = the route; name = the flow label; ready = a
|
|
||||||
// selector we wait for to consider the page "rendered".
|
|
||||||
const FLOWS = [
|
|
||||||
{ name: "dashboard", path: "/", ready: "aside nav, h1, h2" },
|
|
||||||
{ name: "upload", path: "/upload", ready: "section[aria-label='File upload']" },
|
|
||||||
{ name: "inbox", path: "/inbox", ready: "main, section[aria-label='Queue summary']" },
|
|
||||||
{ name: "claims", path: "/claims", ready: "table, [data-testid='claims-page-body']" },
|
|
||||||
{ name: "claims-denied", path: "/claims?status=denied", ready: "table, [data-testid='claims-page-body']" },
|
|
||||||
{ name: "remittances", path: "/remittances", ready: "main, table" },
|
|
||||||
{ name: "providers", path: "/providers", ready: "main, table" },
|
|
||||||
{ name: "reconciliation",path: "/reconciliation", ready: "main" },
|
|
||||||
{ name: "acks", path: "/acks", ready: "main, table" },
|
|
||||||
{ name: "batches", path: "/batches", ready: "main, table" },
|
|
||||||
{ name: "batch-diff", path: "/batch-diff", ready: "main" },
|
|
||||||
{ name: "activity", path: "/activity", ready: "main" },
|
|
||||||
{ name: "404", path: "/does-not-exist", ready: "main" },
|
|
||||||
];
|
|
||||||
|
|
||||||
async function setupViewports(browser) {
|
|
||||||
const pages = [];
|
|
||||||
for (const size of SIZES) {
|
|
||||||
const page = await browser.newPage();
|
|
||||||
await page.setViewport({ width: size.w, height: size.h, deviceScaleFactor: 1 });
|
|
||||||
pages.push({ page, size });
|
|
||||||
}
|
|
||||||
return pages;
|
|
||||||
}
|
|
||||||
|
|
||||||
async function probeFlow(page, flow) {
|
|
||||||
const consoleErrors = [];
|
|
||||||
const pageErrors = [];
|
|
||||||
const failedRequests = [];
|
|
||||||
|
|
||||||
const onConsole = (msg) => {
|
|
||||||
if (msg.type() === "error") consoleErrors.push(msg.text());
|
|
||||||
};
|
|
||||||
const onPageError = (err) => pageErrors.push(err.message);
|
|
||||||
const onRequestFailed = (req) => failedRequests.push(`${req.method()} ${req.url()} :: ${req.failure()?.errorText}`);
|
|
||||||
|
|
||||||
page.on("console", onConsole);
|
|
||||||
page.on("pageerror", onPageError);
|
|
||||||
page.on("requestfailed", onRequestFailed);
|
|
||||||
|
|
||||||
const t0 = Date.now();
|
|
||||||
let rendered = false;
|
|
||||||
let readyError = null;
|
|
||||||
try {
|
|
||||||
await page.goto(`${BASE}${flow.path}`, { waitUntil: "networkidle2", timeout: 15000 });
|
|
||||||
if (flow.ready) {
|
|
||||||
try {
|
|
||||||
await page.waitForSelector(flow.ready, { timeout: 5000 });
|
|
||||||
rendered = true;
|
|
||||||
} catch (e) {
|
|
||||||
readyError = e.message;
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
rendered = true;
|
|
||||||
}
|
|
||||||
} catch (e) {
|
|
||||||
readyError = e.message;
|
|
||||||
}
|
|
||||||
const loadMs = Date.now() - t0;
|
|
||||||
|
|
||||||
page.off("console", onConsole);
|
|
||||||
page.off("pageerror", onPageError);
|
|
||||||
page.off("requestfailed", onRequestFailed);
|
|
||||||
|
|
||||||
return { rendered, loadMs, readyError, consoleErrors, pageErrors, failedRequests };
|
|
||||||
}
|
|
||||||
|
|
||||||
async function probeInteractions(page, flow) {
|
|
||||||
const findings = [];
|
|
||||||
// Generic a11y / structural probes per flow.
|
|
||||||
try {
|
|
||||||
// Sidebar visible? (md+ shows it; < md hides it)
|
|
||||||
const aside = await page.$("aside");
|
|
||||||
findings.push({ check: "sidebar-present", pass: !!aside });
|
|
||||||
} catch (e) {
|
|
||||||
findings.push({ check: "sidebar-present", pass: false, err: e.message });
|
|
||||||
}
|
|
||||||
try {
|
|
||||||
// Top bar present?
|
|
||||||
const main = await page.$("main#main-content");
|
|
||||||
findings.push({ check: "main-present", pass: !!main });
|
|
||||||
} catch (e) {
|
|
||||||
findings.push({ check: "main-present", pass: false, err: e.message });
|
|
||||||
}
|
|
||||||
try {
|
|
||||||
// H1 or page heading?
|
|
||||||
const heading = await page.evaluate(() => {
|
|
||||||
const h = document.querySelector("h1, h2");
|
|
||||||
return h ? h.textContent?.trim().slice(0, 60) : null;
|
|
||||||
});
|
|
||||||
findings.push({ check: "heading-present", pass: !!heading, value: heading });
|
|
||||||
} catch (e) {
|
|
||||||
findings.push({ check: "heading-present", pass: false, err: e.message });
|
|
||||||
}
|
|
||||||
|
|
||||||
// Flow-specific probes.
|
|
||||||
if (flow.name === "claims" || flow.name === "claims-denied") {
|
|
||||||
try {
|
|
||||||
const chips = await page.$$("[role='radio'], button[role='radio']");
|
|
||||||
findings.push({ check: "status-chips", pass: chips.length >= 1, count: chips.length });
|
|
||||||
} catch (e) {
|
|
||||||
findings.push({ check: "status-chips", pass: false, err: e.message });
|
|
||||||
}
|
|
||||||
try {
|
|
||||||
const search = await page.$("input[placeholder*='Search']");
|
|
||||||
findings.push({ check: "search-input", pass: !!search });
|
|
||||||
} catch (e) {
|
|
||||||
findings.push({ check: "search-input", pass: false, err: e.message });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (flow.name === "upload") {
|
|
||||||
try {
|
|
||||||
const dropzone = await page.$("section[aria-label='File upload']");
|
|
||||||
findings.push({ check: "dropzone-present", pass: !!dropzone });
|
|
||||||
const selects = await page.$$("button[role='combobox']");
|
|
||||||
findings.push({ check: "payer-kind-selects", pass: selects.length >= 2, count: selects.length });
|
|
||||||
} catch (e) {
|
|
||||||
findings.push({ check: "upload-elements", pass: false, err: e.message });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (flow.name === "inbox") {
|
|
||||||
try {
|
|
||||||
const lanes = await page.$$("main > div > div");
|
|
||||||
findings.push({ check: "lane-cards", pass: lanes.length >= 1, count: lanes.length });
|
|
||||||
} catch (e) {
|
|
||||||
findings.push({ check: "lane-cards", pass: false, err: e.message });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (flow.name === "404") {
|
|
||||||
try {
|
|
||||||
const text = await page.evaluate(() => document.body.innerText);
|
|
||||||
findings.push({ check: "404-text", pass: text.includes("404") || text.toLowerCase().includes("doesn't exist") });
|
|
||||||
} catch (e) {
|
|
||||||
findings.push({ check: "404-text", pass: false, err: e.message });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return findings;
|
|
||||||
}
|
|
||||||
|
|
||||||
async function main() {
|
|
||||||
await mkdir(SHOTS, { recursive: true });
|
|
||||||
const browser = await puppeteer.launch({
|
|
||||||
executablePath: "/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary",
|
|
||||||
headless: "new",
|
|
||||||
args: ["--no-sandbox", "--disable-dev-shm-usage"],
|
|
||||||
});
|
|
||||||
const startedAt = new Date().toISOString();
|
|
||||||
const results = [];
|
|
||||||
|
|
||||||
for (const size of SIZES) {
|
|
||||||
const page = await browser.newPage();
|
|
||||||
await page.setViewport({ width: size.w, height: size.h, deviceScaleFactor: 1 });
|
|
||||||
|
|
||||||
for (const flow of FLOWS) {
|
|
||||||
const probe = await probeFlow(page, flow);
|
|
||||||
const interactions = await probeInteractions(page, flow);
|
|
||||||
const shot = `${SHOTS}/${flow.name}--${size.name}.png`;
|
|
||||||
try {
|
|
||||||
await page.screenshot({ path: shot, fullPage: false });
|
|
||||||
} catch (e) {
|
|
||||||
// ignore — recording in result
|
|
||||||
}
|
|
||||||
results.push({
|
|
||||||
flow: flow.name,
|
|
||||||
path: flow.path,
|
|
||||||
size: size.name,
|
|
||||||
viewport: { w: size.w, h: size.h },
|
|
||||||
probe,
|
|
||||||
interactions,
|
|
||||||
shot,
|
|
||||||
});
|
|
||||||
console.log(
|
|
||||||
`${size.name.padEnd(7)} ${flow.name.padEnd(20)} ` +
|
|
||||||
`render=${probe.rendered} load=${probe.loadMs}ms ` +
|
|
||||||
`consoleErr=${probe.consoleErrors.length} pageErr=${probe.pageErrors.length}`
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
await page.close();
|
|
||||||
}
|
|
||||||
|
|
||||||
await browser.close();
|
|
||||||
|
|
||||||
// Aggregate.
|
|
||||||
const summary = {
|
|
||||||
startedAt,
|
|
||||||
endedAt: new Date().toISOString(),
|
|
||||||
flows: results.length,
|
|
||||||
sizes: SIZES.map((s) => s.name),
|
|
||||||
renderedOk: results.filter((r) => r.probe.rendered).length,
|
|
||||||
withConsoleErrors: results.filter((r) => r.probe.consoleErrors.length > 0).length,
|
|
||||||
withPageErrors: results.filter((r) => r.probe.pageErrors.length > 0).length,
|
|
||||||
withFailedRequests: results.filter((r) => r.probe.failedRequests.length > 0).length,
|
|
||||||
results,
|
|
||||||
};
|
|
||||||
await writeFile(REPORT, JSON.stringify(summary, null, 2));
|
|
||||||
console.log("\nSummary:", JSON.stringify({
|
|
||||||
flows: summary.flows,
|
|
||||||
renderedOk: summary.renderedOk,
|
|
||||||
withConsoleErrors: summary.withConsoleErrors,
|
|
||||||
withPageErrors: summary.withPageErrors,
|
|
||||||
withFailedRequests: summary.withFailedRequests,
|
|
||||||
}, null, 2));
|
|
||||||
console.log("\nReport:", REPORT);
|
|
||||||
console.log("Shots:", SHOTS);
|
|
||||||
}
|
|
||||||
|
|
||||||
main().catch((e) => {
|
|
||||||
console.error("FATAL", e);
|
|
||||||
process.exit(1);
|
|
||||||
});
|
|
||||||
@@ -1,12 +0,0 @@
|
|||||||
.venv/
|
|
||||||
venv/
|
|
||||||
__pycache__/
|
|
||||||
*.py[cod]
|
|
||||||
*.egg-info/
|
|
||||||
.pytest_cache/
|
|
||||||
.ruff_cache/
|
|
||||||
tests/
|
|
||||||
docs/prodfiles/
|
|
||||||
*.production.txt
|
|
||||||
.git/
|
|
||||||
.github/
|
|
||||||
@@ -1,80 +0,0 @@
|
|||||||
# syntax=docker/dockerfile:1.7
|
|
||||||
#
|
|
||||||
# Cyclone backend — FastAPI on python:3.11-slim-bookworm with sqlcipher.
|
|
||||||
#
|
|
||||||
# Two-stage build:
|
|
||||||
# 1. builder — wheels the package with [sqlcipher] extra into /wheels.
|
|
||||||
# 2. runtime — slim base, tini PID 1, curl-based healthcheck.
|
|
||||||
#
|
|
||||||
# `sqlcipher` is preferred but the engine falls back to plain SQLite at
|
|
||||||
# runtime if the package isn't actually installed (see cyclone.db) — so a
|
|
||||||
# missing libsqlcipher-dev during build will fail loudly here rather than
|
|
||||||
# silently downgrading encryption in production.
|
|
||||||
|
|
||||||
# ---------- builder ----------
|
|
||||||
FROM python:3.11-slim-bookworm AS builder
|
|
||||||
|
|
||||||
ENV PIP_NO_CACHE_DIR=1 \
|
|
||||||
PIP_DISABLE_PIP_VERSION_CHECK=1 \
|
|
||||||
PYTHONDONTWRITEBYTECODE=1
|
|
||||||
|
|
||||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
||||||
build-essential \
|
|
||||||
libffi-dev \
|
|
||||||
libsqlcipher-dev \
|
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
|
||||||
|
|
||||||
WORKDIR /build
|
|
||||||
|
|
||||||
# Copy the build manifest first so this layer caches across source edits.
|
|
||||||
COPY pyproject.toml ./
|
|
||||||
|
|
||||||
# Copy the full source tree, then build the wheel once. We deliberately
|
|
||||||
# avoid the "stub __init__.py, build wheel, then rebuild" pattern — it
|
|
||||||
# left stale `__init__.py` content in the wheel because pip wheel reuses
|
|
||||||
# the cached wheel metadata when the name+version matches. See git
|
|
||||||
# history on this file for the long version.
|
|
||||||
COPY src/ ./src/
|
|
||||||
RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher]'
|
|
||||||
|
|
||||||
# ---------- runtime ----------
|
|
||||||
FROM python:3.11-slim-bookworm
|
|
||||||
|
|
||||||
ENV PYTHONUNBUFFERED=1 \
|
|
||||||
PYTHONDONTWRITEBYTECODE=1
|
|
||||||
|
|
||||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
||||||
libsqlcipher-dev \
|
|
||||||
curl \
|
|
||||||
tini \
|
|
||||||
&& rm -rf /var/lib/apt/lists/* \
|
|
||||||
&& useradd --create-home --uid 1000 --shell /bin/bash cyclone
|
|
||||||
|
|
||||||
WORKDIR /app
|
|
||||||
|
|
||||||
COPY --from=builder /wheels /wheels
|
|
||||||
RUN pip install --no-cache-dir --no-index --find-links /wheels 'cyclone[sqlcipher]' \
|
|
||||||
&& rm -rf /wheels
|
|
||||||
|
|
||||||
# NOTE: we deliberately do NOT drop privileges to the `cyclone` user.
|
|
||||||
# Named volumes mount as root inside the container, and chown-ing them
|
|
||||||
# requires CAP_CHOWN (root). The standard hardened pattern is an
|
|
||||||
# entrypoint script that chowns as root then drops to the app user via
|
|
||||||
# gosu/su-exec — adds a dependency + an entrypoint file. For v1 we run
|
|
||||||
# as root inside the container; Docker's user-namespace remapping is
|
|
||||||
# the recommended host-level isolation. The `cyclone` user is created
|
|
||||||
# above and survives only so file ownership in bind mounts stays
|
|
||||||
# consistent. To harden later: install gosu + add an entrypoint script
|
|
||||||
# that does `chown -R cyclone:cyclone /var/lib/cyclone/... && exec gosu
|
|
||||||
# cyclone "$@"`.
|
|
||||||
|
|
||||||
EXPOSE 8000
|
|
||||||
|
|
||||||
# Container-level healthcheck — the compose service healthcheck is
|
|
||||||
# effectively a duplicate but the Docker `HEALTHCHECK` directive keeps
|
|
||||||
# `docker ps` honest without needing compose to be running.
|
|
||||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
|
|
||||||
CMD curl -fs http://localhost:8000/api/health || exit 1
|
|
||||||
|
|
||||||
ENTRYPOINT ["tini", "--"]
|
|
||||||
CMD ["python", "-m", "cyclone", "serve"]
|
|
||||||
@@ -16,15 +16,6 @@ dependencies = [
|
|||||||
"sqlalchemy>=2.0,<3",
|
"sqlalchemy>=2.0,<3",
|
||||||
"pyyaml>=6.0,<7",
|
"pyyaml>=6.0,<7",
|
||||||
"keyring>=25.0,<26",
|
"keyring>=25.0,<26",
|
||||||
# backup_service / backup: encryption-at-rest (SP17). Used at module
|
|
||||||
# top-level by cyclone.backup, so it has to be a hard dep — not an
|
|
||||||
# extra — or the test suite fails to collect when the venv is built
|
|
||||||
# from a clean `uv sync`.
|
|
||||||
"cryptography>=49.0,<50",
|
|
||||||
# passlib 1.7.4 + bcrypt >= 4.1 are incompatible (passlib probes bcrypt.__about__
|
|
||||||
# which 4.x removed). Pin bcrypt < 4.1.
|
|
||||||
"passlib[bcrypt]>=1.7.4",
|
|
||||||
"bcrypt<4.1",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
[project.optional-dependencies]
|
[project.optional-dependencies]
|
||||||
@@ -33,7 +24,6 @@ dev = [
|
|||||||
"pytest-cov>=4.1",
|
"pytest-cov>=4.1",
|
||||||
"pytest-asyncio>=0.23,<1",
|
"pytest-asyncio>=0.23,<1",
|
||||||
"httpx>=0.27,<1",
|
"httpx>=0.27,<1",
|
||||||
"pytest-randomly>=4.1",
|
|
||||||
]
|
]
|
||||||
sqlcipher = [
|
sqlcipher = [
|
||||||
# SP12: encryption at rest. Optional — without it the DB is plain SQLite.
|
# SP12: encryption at rest. Optional — without it the DB is plain SQLite.
|
||||||
|
|||||||
@@ -16,41 +16,13 @@ import sys
|
|||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
# Always run first-admin bootstrap before any other entry path.
|
|
||||||
# Must happen before ``serve`` (uvicorn) AND before the Click CLI
|
|
||||||
# dispatch — otherwise `python -m cyclone users create ...` on a
|
|
||||||
# fresh DB would race with the bootstrap's check, and the API
|
|
||||||
# could come up with zero users.
|
|
||||||
from cyclone.auth import bootstrap
|
|
||||||
bootstrap.run()
|
|
||||||
|
|
||||||
# SP24: if the AUTH_DISABLED escape hatch is on, scream at boot so a
|
|
||||||
# misconfigured production deploy fails loudly. The flag is flipped by
|
|
||||||
# ``CYCLONE_AUTH_DISABLED=1`` (see ``cyclone.auth.bootstrap``) and by the
|
|
||||||
# pytest conftest autouse fixture (see ``.superpowers/skills/cyclone-tests``).
|
|
||||||
from cyclone.auth import deps as _auth_deps
|
|
||||||
|
|
||||||
if _auth_deps.AUTH_DISABLED:
|
|
||||||
import logging
|
|
||||||
|
|
||||||
logging.getLogger("cyclone").warning(
|
|
||||||
"AUTH_DISABLED is set (CYCLONE_AUTH_DISABLED=1) — all requests "
|
|
||||||
"treated as admin, dev only. Do NOT enable this in production."
|
|
||||||
)
|
|
||||||
|
|
||||||
if len(sys.argv) >= 2 and sys.argv[1] == "serve":
|
if len(sys.argv) >= 2 and sys.argv[1] == "serve":
|
||||||
port = os.environ.get("CYCLONE_PORT", "8000")
|
port = os.environ.get("CYCLONE_PORT", "8000")
|
||||||
# Local-only by default — see CLAUDE.md. The Docker image
|
|
||||||
# overrides to 0.0.0.0 via compose env so the frontend
|
|
||||||
# container on the compose bridge network can reach the
|
|
||||||
# backend. Network isolation is provided by the bridge
|
|
||||||
# network itself (only cyclone-frontend joins).
|
|
||||||
host = os.environ.get("CYCLONE_HOST", "127.0.0.1")
|
|
||||||
reload = os.environ.get("CYCLONE_RELOAD", "0") == "1"
|
reload = os.environ.get("CYCLONE_RELOAD", "0") == "1"
|
||||||
sys.argv = [
|
sys.argv = [
|
||||||
sys.argv[0],
|
sys.argv[0],
|
||||||
"cyclone.api:app",
|
"cyclone.api:app",
|
||||||
"--host", host,
|
"--host", "127.0.0.1",
|
||||||
"--port", port,
|
"--port", port,
|
||||||
]
|
]
|
||||||
if reload:
|
if reload:
|
||||||
|
|||||||
+76
-1254
File diff suppressed because it is too large
Load Diff
@@ -93,40 +93,19 @@ def ndjson_stream_list(
|
|||||||
}) + "\n"
|
}) + "\n"
|
||||||
|
|
||||||
|
|
||||||
def ndjson_stream_837(
|
def ndjson_stream_837(result: ParseResult) -> Iterator[bytes]:
|
||||||
result: ParseResult, batch_id: str | None = None,
|
"""Yield one JSON object per line: envelope → claims → summary."""
|
||||||
) -> Iterator[bytes]:
|
|
||||||
"""Yield one JSON object per line: envelope → claims → summary.
|
|
||||||
|
|
||||||
The ``batch_id`` is the server-side UUID assigned by the persistence
|
|
||||||
layer when the batch is ingested; the JSON response path exposes it
|
|
||||||
as the top-level ``batch_id`` field, but the NDJSON stream needs it
|
|
||||||
inline on the summary event so streaming clients can call
|
|
||||||
batch-scoped endpoints (``/api/batches/{id}/export-837``, …) without
|
|
||||||
a separate ``GET /api/batches`` round-trip. When ``batch_id`` is not
|
|
||||||
supplied the summary omits the field, preserving backward compat
|
|
||||||
with clients that don't expect it.
|
|
||||||
"""
|
|
||||||
envelope_obj = (
|
envelope_obj = (
|
||||||
result.envelope.model_dump() if result.envelope is not None else None
|
result.envelope.model_dump() if result.envelope is not None else None
|
||||||
)
|
)
|
||||||
yield (json.dumps({"type": "envelope", "data": envelope_obj}) + "\n").encode("utf-8")
|
yield (json.dumps({"type": "envelope", "data": envelope_obj}) + "\n").encode("utf-8")
|
||||||
for claim in result.claims:
|
for claim in result.claims:
|
||||||
yield (json.dumps({"type": "claim", "data": json.loads(claim.model_dump_json())}) + "\n").encode("utf-8")
|
yield (json.dumps({"type": "claim", "data": json.loads(claim.model_dump_json())}) + "\n").encode("utf-8")
|
||||||
summary_data = json.loads(result.summary.model_dump_json())
|
yield (json.dumps({"type": "summary", "data": json.loads(result.summary.model_dump_json())}) + "\n").encode("utf-8")
|
||||||
if batch_id is not None:
|
|
||||||
summary_data["batch_id"] = batch_id
|
|
||||||
yield (json.dumps({"type": "summary", "data": summary_data}) + "\n").encode("utf-8")
|
|
||||||
|
|
||||||
|
|
||||||
def ndjson_stream_835(
|
def ndjson_stream_835(result: ParseResult835) -> Iterator[bytes]:
|
||||||
result: ParseResult835, batch_id: str | None = None,
|
"""Yield one JSON object per line: envelope → financial → trace → payer → payee → claim_payments → summary."""
|
||||||
) -> Iterator[bytes]:
|
|
||||||
"""Yield one JSON object per line: envelope → financial → trace → payer → payee → claim_payments → summary.
|
|
||||||
|
|
||||||
See ``ndjson_stream_837`` for why the optional ``batch_id`` is
|
|
||||||
merged into the summary event.
|
|
||||||
"""
|
|
||||||
yield (json.dumps({"type": "envelope", "data": json.loads(result.envelope.model_dump_json())}) + "\n").encode("utf-8")
|
yield (json.dumps({"type": "envelope", "data": json.loads(result.envelope.model_dump_json())}) + "\n").encode("utf-8")
|
||||||
yield (json.dumps({"type": "financial_info", "data": json.loads(result.financial_info.model_dump_json())}) + "\n").encode("utf-8")
|
yield (json.dumps({"type": "financial_info", "data": json.loads(result.financial_info.model_dump_json())}) + "\n").encode("utf-8")
|
||||||
yield (json.dumps({"type": "trace", "data": json.loads(result.trace.model_dump_json())}) + "\n").encode("utf-8")
|
yield (json.dumps({"type": "trace", "data": json.loads(result.trace.model_dump_json())}) + "\n").encode("utf-8")
|
||||||
@@ -134,10 +113,7 @@ def ndjson_stream_835(
|
|||||||
yield (json.dumps({"type": "payee", "data": json.loads(result.payee.model_dump_json())}) + "\n").encode("utf-8")
|
yield (json.dumps({"type": "payee", "data": json.loads(result.payee.model_dump_json())}) + "\n").encode("utf-8")
|
||||||
for claim in result.claims:
|
for claim in result.claims:
|
||||||
yield (json.dumps({"type": "claim_payment", "data": json.loads(claim.model_dump_json())}) + "\n").encode("utf-8")
|
yield (json.dumps({"type": "claim_payment", "data": json.loads(claim.model_dump_json())}) + "\n").encode("utf-8")
|
||||||
summary_data = json.loads(result.summary.model_dump_json())
|
yield (json.dumps({"type": "summary", "data": json.loads(result.summary.model_dump_json())}) + "\n").encode("utf-8")
|
||||||
if batch_id is not None:
|
|
||||||
summary_data["batch_id"] = batch_id
|
|
||||||
yield (json.dumps({"type": "summary", "data": summary_data}) + "\n").encode("utf-8")
|
|
||||||
|
|
||||||
|
|
||||||
def strict_rewrite_837(result: ParseResult) -> ParseResult:
|
def strict_rewrite_837(result: ParseResult) -> ParseResult:
|
||||||
|
|||||||
@@ -1,60 +0,0 @@
|
|||||||
"""``/api/admin/validate-provider`` — NPI + Tax ID liveness probe (SP20).
|
|
||||||
|
|
||||||
Pure read-only endpoint that runs the local NPI Luhn + EIN format checks
|
|
||||||
without touching the DB. Useful for:
|
|
||||||
- operators vetting a new provider before adding them to the registry,
|
|
||||||
- the dashboard's "validate" button on a Provider row,
|
|
||||||
- smoke-testing the SP20 checks after a deploy.
|
|
||||||
|
|
||||||
Both query params are optional; omitting one just skips that check.
|
|
||||||
Returns the per-check result dict so the caller can distinguish "bad
|
|
||||||
format" from "bad checksum".
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from fastapi import APIRouter, Query
|
|
||||||
|
|
||||||
from cyclone.npi import is_valid_npi, is_valid_tax_id, normalize_tax_id
|
|
||||||
|
|
||||||
|
|
||||||
router = APIRouter()
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/api/admin/validate-provider")
|
|
||||||
def validate_provider(
|
|
||||||
npi: str | None = Query(None, description="10-digit NPI to validate (Luhn checksum)"),
|
|
||||||
tax_id: str | None = Query(None, description="9-digit EIN to validate (format + reserved-prefix check)"),
|
|
||||||
) -> dict:
|
|
||||||
"""Return per-field validation results for ``npi`` and ``tax_id``.
|
|
||||||
|
|
||||||
Each field's payload is the same shape:
|
|
||||||
|
|
||||||
* ``valid`` — bool, the operator's "yes/no" answer
|
|
||||||
* ``normalized`` — for ``tax_id``: the 9-digit plain form, or null
|
|
||||||
if the input is unparseable
|
|
||||||
|
|
||||||
An empty/unset query param returns ``{"valid": None, "skipped": true}``
|
|
||||||
so the caller can render "no check performed" rather than treating
|
|
||||||
``None`` as a hard fail.
|
|
||||||
"""
|
|
||||||
result: dict = {}
|
|
||||||
|
|
||||||
if npi is None or npi == "":
|
|
||||||
result["npi"] = {"valid": None, "skipped": True}
|
|
||||||
else:
|
|
||||||
result["npi"] = {
|
|
||||||
"valid": is_valid_npi(npi),
|
|
||||||
"skipped": False,
|
|
||||||
}
|
|
||||||
|
|
||||||
if tax_id is None or tax_id == "":
|
|
||||||
result["tax_id"] = {"valid": None, "skipped": True, "normalized": None}
|
|
||||||
else:
|
|
||||||
normalized = normalize_tax_id(tax_id)
|
|
||||||
result["tax_id"] = {
|
|
||||||
"valid": is_valid_tax_id(tax_id),
|
|
||||||
"skipped": False,
|
|
||||||
"normalized": normalized,
|
|
||||||
}
|
|
||||||
|
|
||||||
return result
|
|
||||||
@@ -1,40 +1,17 @@
|
|||||||
"""``GET /api/health`` — liveness + readiness probe.
|
"""``GET /api/health`` — liveness probe.
|
||||||
|
|
||||||
SP19 expanded the shallow ``{"status": "ok", "version": ...}`` probe
|
Returns the package version so an operator can confirm which build is
|
||||||
into a snapshot of every subsystem:
|
serving requests without poking the filesystem.
|
||||||
|
|
||||||
* **db** — can we open a session and run ``SELECT 1``?
|
|
||||||
* **scheduler** — is the MFT polling loop running? same for the
|
|
||||||
backup scheduler.
|
|
||||||
* **pubsub** — current subscriber counts per event kind.
|
|
||||||
* **batch** — most recent batch id + timestamp.
|
|
||||||
|
|
||||||
Returns ``status="ok"`` only when every subsystem is healthy.
|
|
||||||
``status="degraded"`` if any subsystem is unhappy but the API
|
|
||||||
itself is responsive. Per-subsystem errors are surfaced in the
|
|
||||||
respective dict so an operator doesn't have to guess.
|
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from fastapi import APIRouter, Request
|
from fastapi import APIRouter
|
||||||
|
|
||||||
from cyclone import __version__
|
from cyclone import __version__
|
||||||
from cyclone.security import get_health_snapshot
|
|
||||||
|
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
||||||
|
|
||||||
@router.get("/api/health")
|
@router.get("/api/health")
|
||||||
def health(request: Request) -> dict:
|
def health() -> dict[str, str]:
|
||||||
snap = get_health_snapshot()
|
return {"status": "ok", "version": __version__}
|
||||||
# Fill in live pubsub subscriber counts using the per-request app
|
|
||||||
# state (the snapshot builder doesn't have request context).
|
|
||||||
bus = getattr(request.app.state, "event_bus", None)
|
|
||||||
if bus is not None and hasattr(bus, "stats"):
|
|
||||||
snap.pubsub = bus.stats()
|
|
||||||
elif bus is not None:
|
|
||||||
snap.pubsub = {"note": "EventBus.stats() not available"}
|
|
||||||
else:
|
|
||||||
snap.pubsub = {"note": "EventBus not attached (running outside lifespan?)"}
|
|
||||||
return snap.to_dict()
|
|
||||||
|
|||||||
@@ -103,12 +103,6 @@ class AuditEvent:
|
|||||||
computed hash. Payload must be JSON-serializable; the audit_log
|
computed hash. Payload must be JSON-serializable; the audit_log
|
||||||
module handles the encoding so callers don't need to think about
|
module handles the encoding so callers don't need to think about
|
||||||
canonical form.
|
canonical form.
|
||||||
|
|
||||||
``user_id`` is the authenticated actor for this event, when known
|
|
||||||
(e.g. a parse-999 call made by user 7). It's stored on the row but
|
|
||||||
is NOT part of the hash chain — the chain hashes only the fields
|
|
||||||
that existed pre-SP-auth so verify_chain stays compatible with
|
|
||||||
pre-auth rows.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
event_type: str
|
event_type: str
|
||||||
@@ -117,7 +111,6 @@ class AuditEvent:
|
|||||||
payload: dict[str, Any] = field(default_factory=dict)
|
payload: dict[str, Any] = field(default_factory=dict)
|
||||||
actor: str = "system"
|
actor: str = "system"
|
||||||
created_at: datetime | None = None
|
created_at: datetime | None = None
|
||||||
user_id: int | None = None
|
|
||||||
|
|
||||||
|
|
||||||
def append_event(
|
def append_event(
|
||||||
@@ -162,7 +155,6 @@ def append_event(
|
|||||||
created_at=created_at,
|
created_at=created_at,
|
||||||
prev_hash=prev_hash,
|
prev_hash=prev_hash,
|
||||||
hash=GENESIS_PREV_HASH, # placeholder; updated below
|
hash=GENESIS_PREV_HASH, # placeholder; updated below
|
||||||
user_id=event.user_id,
|
|
||||||
)
|
)
|
||||||
session.add(row)
|
session.add(row)
|
||||||
session.flush() # populate row.id
|
session.flush() # populate row.id
|
||||||
|
|||||||
@@ -1 +0,0 @@
|
|||||||
"""Auth module — users, sessions, permissions, routes, admin, rate_limit."""
|
|
||||||
@@ -1,95 +0,0 @@
|
|||||||
"""Admin-only user management: GET/POST/PATCH/DELETE /api/admin/users."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, HTTPException, status
|
|
||||||
|
|
||||||
from cyclone.auth import users
|
|
||||||
from cyclone.auth.deps import get_current_user
|
|
||||||
from cyclone.auth.permissions import Role
|
|
||||||
from cyclone.db import SessionLocal, User
|
|
||||||
|
|
||||||
router = APIRouter(prefix="/api/admin/users", tags=["admin"])
|
|
||||||
|
|
||||||
|
|
||||||
def _require_admin(user: dict = Depends(get_current_user)) -> dict:
|
|
||||||
if user.get("role") != Role.ADMIN.value:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_403_FORBIDDEN,
|
|
||||||
detail="forbidden",
|
|
||||||
)
|
|
||||||
return user
|
|
||||||
|
|
||||||
|
|
||||||
def _validate_role(role: str) -> None:
|
|
||||||
valid = {Role.ADMIN.value, Role.USER.value, Role.VIEWER.value}
|
|
||||||
if role not in valid:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
|
|
||||||
detail=f"role must be one of {sorted(valid)}",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("")
|
|
||||||
def list_users(_admin=Depends(_require_admin)):
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
all_users = db.query(User).all()
|
|
||||||
return [users.to_public(u) for u in all_users]
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("", status_code=status.HTTP_201_CREATED)
|
|
||||||
def create_user(body: dict, _admin=Depends(_require_admin)):
|
|
||||||
username = (body.get("username") or "").strip()
|
|
||||||
password = body.get("password") or ""
|
|
||||||
role = body.get("role") or ""
|
|
||||||
if not username or len(username) < 3:
|
|
||||||
raise HTTPException(status_code=422, detail="username must be at least 3 chars")
|
|
||||||
if len(password) < 12:
|
|
||||||
raise HTTPException(status_code=422, detail="password must be at least 12 chars")
|
|
||||||
_validate_role(role)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
if users.get_by_username(db, username) is not None:
|
|
||||||
raise HTTPException(status_code=409, detail="username already exists")
|
|
||||||
u = users.create(db, username=username, password=password, role=role)
|
|
||||||
return users.to_public(u)
|
|
||||||
|
|
||||||
|
|
||||||
@router.patch("/{user_id}")
|
|
||||||
def patch_user(user_id: int, body: dict, admin=Depends(_require_admin)):
|
|
||||||
me = admin
|
|
||||||
if me.get("id") == user_id and body.get("role") and body["role"] != Role.ADMIN.value:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_409_CONFLICT,
|
|
||||||
detail="cannot_demote_self",
|
|
||||||
)
|
|
||||||
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
if body.get("role") is not None:
|
|
||||||
_validate_role(body["role"])
|
|
||||||
users.update_role(db, user_id, body["role"])
|
|
||||||
if body.get("password") is not None:
|
|
||||||
if len(body["password"]) < 12:
|
|
||||||
raise HTTPException(status_code=422, detail="password must be at least 12 chars")
|
|
||||||
users.update_password(db, user_id, body["password"])
|
|
||||||
if body.get("disabled") is True:
|
|
||||||
users.disable(db, user_id)
|
|
||||||
u = users.get(db, user_id)
|
|
||||||
if u is None:
|
|
||||||
raise HTTPException(status_code=404, detail="user not found")
|
|
||||||
return users.to_public(u)
|
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/{user_id}", status_code=status.HTTP_204_NO_CONTENT)
|
|
||||||
def delete_user(user_id: int, admin=Depends(_require_admin)):
|
|
||||||
me = admin
|
|
||||||
if me.get("id") == user_id:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_409_CONFLICT,
|
|
||||||
detail="cannot_delete_self",
|
|
||||||
)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.get(db, user_id)
|
|
||||||
if u is None:
|
|
||||||
raise HTTPException(status_code=404, detail="user not found")
|
|
||||||
users.disable(db, user_id)
|
|
||||||
return None
|
|
||||||
@@ -1,105 +0,0 @@
|
|||||||
"""First-admin bootstrap: create the initial admin from env vars if no users exist.
|
|
||||||
|
|
||||||
Called from ``python -m cyclone`` before either ``cli.main()`` or
|
|
||||||
``uvicorn`` so users exist by the time the API serves requests.
|
|
||||||
|
|
||||||
Precedence:
|
|
||||||
|
|
||||||
1. ``CYCLONE_AUTH_DISABLED=1`` — dev escape hatch. Flip the
|
|
||||||
``cyclone.auth.deps.AUTH_DISABLED`` flag so the API returns a
|
|
||||||
synthetic admin user without checking credentials. Never raises.
|
|
||||||
2. Users table non-empty — no-op.
|
|
||||||
3. ``CYCLONE_ADMIN_USERNAME`` + ``CYCLONE_ADMIN_PASSWORD`` env vars set
|
|
||||||
(password >= 12 chars) — create the admin and print confirmation.
|
|
||||||
Each env var can also be replaced by a ``*_FILE`` companion
|
|
||||||
(``CYCLONE_ADMIN_USERNAME_FILE`` / ``CYCLONE_ADMIN_PASSWORD_FILE``)
|
|
||||||
that points at a file on disk — the standard Docker-secret pattern,
|
|
||||||
used in production to avoid embedding secrets in ``docker-compose.yml``.
|
|
||||||
``_FILE`` takes precedence when set.
|
|
||||||
4. Otherwise — raise ``RuntimeError`` with a remediation hint that
|
|
||||||
points operators at ``python -m cyclone users create``.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from sqlalchemy import select
|
|
||||||
|
|
||||||
from cyclone.auth import users
|
|
||||||
from cyclone.auth.deps import AUTH_DISABLED
|
|
||||||
from cyclone.auth.permissions import Role
|
|
||||||
from cyclone.db import SessionLocal, User
|
|
||||||
|
|
||||||
|
|
||||||
def _read_secret(env_var: str, file_var: str) -> str | None:
|
|
||||||
"""Read a secret from a ``*_FILE`` env var (Docker-secret pattern) first,
|
|
||||||
falling back to the plain env var. Returns None if neither is set.
|
|
||||||
"""
|
|
||||||
file_path = os.environ.get(file_var)
|
|
||||||
if file_path:
|
|
||||||
try:
|
|
||||||
return Path(file_path).read_text().strip()
|
|
||||||
except OSError as exc:
|
|
||||||
raise RuntimeError(
|
|
||||||
f"failed to read {file_var}={file_path}: {exc}"
|
|
||||||
) from exc
|
|
||||||
return os.environ.get(env_var)
|
|
||||||
|
|
||||||
|
|
||||||
def run() -> None:
|
|
||||||
"""Bootstrap the first admin user, or no-op.
|
|
||||||
|
|
||||||
See module docstring for behavior. Idempotent: safe to call on
|
|
||||||
every startup — it short-circuits as soon as the users table is
|
|
||||||
non-empty.
|
|
||||||
"""
|
|
||||||
if os.environ.get("CYCLONE_AUTH_DISABLED") == "1":
|
|
||||||
# Dev escape hatch — skip bootstrap entirely and tell the API
|
|
||||||
# to also short-circuit auth checks.
|
|
||||||
import cyclone.auth.deps as _deps
|
|
||||||
|
|
||||||
_deps.AUTH_DISABLED = True
|
|
||||||
return
|
|
||||||
|
|
||||||
username = _read_secret(
|
|
||||||
"CYCLONE_ADMIN_USERNAME", "CYCLONE_ADMIN_USERNAME_FILE"
|
|
||||||
)
|
|
||||||
password = _read_secret(
|
|
||||||
"CYCLONE_ADMIN_PASSWORD", "CYCLONE_ADMIN_PASSWORD_FILE"
|
|
||||||
)
|
|
||||||
|
|
||||||
# First-boot fix: ``python -m cyclone`` calls bootstrap before any
|
|
||||||
# subcommand or the FastAPI lifespan handler runs, so on a brand-new
|
|
||||||
# DB ``SessionLocal()`` raises "init_db() has not been called".
|
|
||||||
# Initialize here so ``serve``, ``users create``, and friends can
|
|
||||||
# all reach the DB without the operator having to know about
|
|
||||||
# migrations. Idempotent — no-op when the schema is already current.
|
|
||||||
from cyclone import db as _db
|
|
||||||
|
|
||||||
_db.init_db()
|
|
||||||
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
existing = db.execute(select(User)).scalars().first()
|
|
||||||
if existing is not None:
|
|
||||||
return # users exist — nothing to bootstrap
|
|
||||||
|
|
||||||
if not username or not password:
|
|
||||||
raise RuntimeError(
|
|
||||||
"Cyclone has no users yet. Set CYCLONE_ADMIN_USERNAME and "
|
|
||||||
"CYCLONE_ADMIN_PASSWORD env vars (min 12 chars), or run "
|
|
||||||
"`python -m cyclone users create <username> --role admin`."
|
|
||||||
)
|
|
||||||
if len(password) < 12:
|
|
||||||
raise RuntimeError(
|
|
||||||
"CYCLONE_ADMIN_PASSWORD must be at least 12 characters."
|
|
||||||
)
|
|
||||||
|
|
||||||
users.create(
|
|
||||||
db,
|
|
||||||
username=username,
|
|
||||||
password=password,
|
|
||||||
role=Role.ADMIN.value,
|
|
||||||
)
|
|
||||||
print(f"[cyclone] bootstrap admin user '{username}' created")
|
|
||||||
@@ -1,183 +0,0 @@
|
|||||||
"""CLI subcommand: ``python -m cyclone users ...``.
|
|
||||||
|
|
||||||
Click-based to match the existing parse-837 / parse-835 convention in
|
|
||||||
``cyclone.cli``. Provides operator-side user management without going
|
|
||||||
through the admin HTTP API.
|
|
||||||
|
|
||||||
Subcommands
|
|
||||||
-----------
|
|
||||||
|
|
||||||
- ``users create USERNAME --role {admin,user,viewer} [--password PW]``
|
|
||||||
Create a user. If ``--password`` is omitted, prompts (with
|
|
||||||
confirmation) on the controlling terminal.
|
|
||||||
- ``users list`` — tab-separated ``id / username / role / state``.
|
|
||||||
- ``users disable USERNAME`` — set ``disabled_at`` to now.
|
|
||||||
- ``users reset-password USERNAME [--password PW]`` — replace the hash.
|
|
||||||
- ``users set-role USERNAME --role {admin,user,viewer}`` — change role.
|
|
||||||
|
|
||||||
Exit codes
|
|
||||||
----------
|
|
||||||
|
|
||||||
* 0 — success
|
|
||||||
* 1 — validation error (unknown username, duplicate username)
|
|
||||||
* 2 — usage error (missing arg, bad role, short password, unknown
|
|
||||||
subcommand). Click itself uses 2 for usage errors so the conventional
|
|
||||||
shell tools (``set -e``, etc.) recognize them.
|
|
||||||
|
|
||||||
Passwords shorter than 12 chars are rejected everywhere they appear.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import getpass
|
|
||||||
import sys
|
|
||||||
|
|
||||||
import click
|
|
||||||
|
|
||||||
from cyclone.auth import users
|
|
||||||
from cyclone.auth.permissions import Role
|
|
||||||
from cyclone.db import SessionLocal
|
|
||||||
|
|
||||||
ROLE_CHOICES = [Role.ADMIN.value, Role.USER.value, Role.VIEWER.value]
|
|
||||||
MIN_PASSWORD_LEN = 12
|
|
||||||
|
|
||||||
|
|
||||||
def _prompt_password(label: str) -> str:
|
|
||||||
pw = getpass.getpass(f"{label}: ")
|
|
||||||
if not pw:
|
|
||||||
click.echo("Password required.", err=True)
|
|
||||||
sys.exit(2)
|
|
||||||
return pw
|
|
||||||
|
|
||||||
|
|
||||||
def _validate_password(pw: str) -> None:
|
|
||||||
if len(pw) < MIN_PASSWORD_LEN:
|
|
||||||
click.echo(
|
|
||||||
f"Password must be at least {MIN_PASSWORD_LEN} characters.",
|
|
||||||
err=True,
|
|
||||||
)
|
|
||||||
sys.exit(2)
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# Group
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
@click.group(name="users")
|
|
||||||
def users_cli() -> None:
|
|
||||||
"""Manage Cyclone users from the command line."""
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# create
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
@users_cli.command("create")
|
|
||||||
@click.argument("username")
|
|
||||||
@click.option(
|
|
||||||
"--role",
|
|
||||||
required=True,
|
|
||||||
type=click.Choice(ROLE_CHOICES, case_sensitive=False),
|
|
||||||
help="Role to grant the new user.",
|
|
||||||
)
|
|
||||||
@click.option(
|
|
||||||
"--password",
|
|
||||||
default=None,
|
|
||||||
help=f"Password (min {MIN_PASSWORD_LEN} chars). Prompts if omitted.",
|
|
||||||
)
|
|
||||||
def create_user(username: str, role: str, password: str | None) -> None:
|
|
||||||
"""Create a new user with the given USERNAME and ROLE."""
|
|
||||||
pw = password or _prompt_password("Password")
|
|
||||||
_validate_password(pw)
|
|
||||||
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
if users.get_by_username(db, username) is not None:
|
|
||||||
click.echo(f"User '{username}' already exists.", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
u = users.create(db, username=username, password=pw, role=role)
|
|
||||||
click.echo(f"Created user '{u.username}' with role '{u.role}'.")
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# list
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
@users_cli.command("list")
|
|
||||||
def list_users() -> None:
|
|
||||||
"""List all users as id / username / role / state."""
|
|
||||||
from cyclone.db import User
|
|
||||||
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
for u in db.query(User).order_by(User.id.asc()).all():
|
|
||||||
state = "disabled" if u.disabled_at else "active"
|
|
||||||
click.echo(f"{u.id}\t{u.username}\t{u.role}\t{state}")
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# disable
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
@users_cli.command("disable")
|
|
||||||
@click.argument("username")
|
|
||||||
def disable_user(username: str) -> None:
|
|
||||||
"""Disable USERNAME (sets disabled_at to now)."""
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.get_by_username(db, username)
|
|
||||||
if u is None:
|
|
||||||
click.echo(f"No such user: {username}", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
users.disable(db, u.id)
|
|
||||||
click.echo(f"Disabled '{username}'.")
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# reset-password
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
@users_cli.command("reset-password")
|
|
||||||
@click.argument("username")
|
|
||||||
@click.option(
|
|
||||||
"--password",
|
|
||||||
default=None,
|
|
||||||
help=f"New password (min {MIN_PASSWORD_LEN} chars). Prompts if omitted.",
|
|
||||||
)
|
|
||||||
def reset_password(username: str, password: str | None) -> None:
|
|
||||||
"""Replace USERNAME's password."""
|
|
||||||
pw = password or _prompt_password("New password")
|
|
||||||
_validate_password(pw)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.get_by_username(db, username)
|
|
||||||
if u is None:
|
|
||||||
click.echo(f"No such user: {username}", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
users.update_password(db, u.id, pw)
|
|
||||||
click.echo(f"Password reset for '{username}'.")
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
# set-role
|
|
||||||
# --------------------------------------------------------------------------- #
|
|
||||||
|
|
||||||
|
|
||||||
@users_cli.command("set-role")
|
|
||||||
@click.argument("username")
|
|
||||||
@click.option(
|
|
||||||
"--role",
|
|
||||||
required=True,
|
|
||||||
type=click.Choice(ROLE_CHOICES, case_sensitive=False),
|
|
||||||
help="Role to grant.",
|
|
||||||
)
|
|
||||||
def set_role(username: str, role: str) -> None:
|
|
||||||
"""Change USERNAME's role."""
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.get_by_username(db, username)
|
|
||||||
if u is None:
|
|
||||||
click.echo(f"No such user: {username}", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
users.update_role(db, u.id, role)
|
|
||||||
click.echo(f"Role for '{username}' set to '{role}'.")
|
|
||||||
@@ -1,140 +0,0 @@
|
|||||||
"""FastAPI dependencies for auth."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Annotated
|
|
||||||
|
|
||||||
from fastapi import Depends, HTTPException, Request, status
|
|
||||||
from sqlalchemy.orm import Session as DbSession
|
|
||||||
|
|
||||||
from cyclone.auth import sessions, users
|
|
||||||
from cyclone.auth.permissions import Role, allowed_roles
|
|
||||||
from cyclone.db import SessionLocal
|
|
||||||
|
|
||||||
|
|
||||||
def _db():
|
|
||||||
db = SessionLocal()()
|
|
||||||
try:
|
|
||||||
yield db
|
|
||||||
finally:
|
|
||||||
db.close()
|
|
||||||
|
|
||||||
|
|
||||||
DbSessionDep = Annotated[DbSession, Depends(_db)]
|
|
||||||
|
|
||||||
|
|
||||||
AUTH_DISABLED = False
|
|
||||||
|
|
||||||
|
|
||||||
async def get_current_user(
|
|
||||||
request: Request,
|
|
||||||
db: DbSessionDep,
|
|
||||||
) -> dict:
|
|
||||||
"""Return the public User shape. Raises 401 if session is missing/expired.
|
|
||||||
|
|
||||||
When AUTH_DISABLED is True (dev escape hatch), returns a synthetic admin
|
|
||||||
user without checking credentials.
|
|
||||||
"""
|
|
||||||
if AUTH_DISABLED:
|
|
||||||
return {
|
|
||||||
"id": 0,
|
|
||||||
"username": "dev",
|
|
||||||
"role": Role.ADMIN.value,
|
|
||||||
"createdAt": None,
|
|
||||||
"disabledAt": None,
|
|
||||||
}
|
|
||||||
|
|
||||||
sid = request.cookies.get("cyclone_session")
|
|
||||||
if not sid:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
|
||||||
detail="session_expired",
|
|
||||||
)
|
|
||||||
sess = sessions.get_valid(db, sid)
|
|
||||||
if sess is None:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
|
||||||
detail="session_expired",
|
|
||||||
)
|
|
||||||
user = users.get(db, sess.user_id)
|
|
||||||
if user is None or user.disabled_at is not None:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
|
||||||
detail="account_disabled",
|
|
||||||
)
|
|
||||||
# Sliding expiry: refresh both DB and cookie.
|
|
||||||
sessions.touch(db, sid)
|
|
||||||
request.state.user = user
|
|
||||||
request.state.session_id = sid
|
|
||||||
return users.to_public(user)
|
|
||||||
|
|
||||||
|
|
||||||
def require_role(*allowed: Role):
|
|
||||||
"""Dependency factory: gate the endpoint to specific roles.
|
|
||||||
|
|
||||||
Falls back to PERMISSIONS matrix lookup if no explicit roles given.
|
|
||||||
"""
|
|
||||||
async def _dep(
|
|
||||||
request: Request,
|
|
||||||
user: dict = Depends(get_current_user),
|
|
||||||
) -> dict:
|
|
||||||
user_role = user.get("role")
|
|
||||||
if allowed:
|
|
||||||
if user_role not in {r.value for r in allowed}:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_403_FORBIDDEN,
|
|
||||||
detail="forbidden",
|
|
||||||
)
|
|
||||||
return user
|
|
||||||
# Otherwise consult the matrix.
|
|
||||||
method = request.method
|
|
||||||
path = request.url.path
|
|
||||||
roles = allowed_roles(method, path)
|
|
||||||
if roles is None:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_403_FORBIDDEN,
|
|
||||||
detail="forbidden",
|
|
||||||
)
|
|
||||||
if user_role not in {r.value for r in roles}:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_403_FORBIDDEN,
|
|
||||||
detail="forbidden",
|
|
||||||
)
|
|
||||||
return user
|
|
||||||
return _dep
|
|
||||||
|
|
||||||
|
|
||||||
async def matrix_gate(
|
|
||||||
request: Request,
|
|
||||||
user: dict = Depends(get_current_user),
|
|
||||||
) -> dict:
|
|
||||||
"""App-wide gate: requires auth, then enforces the PERMISSIONS matrix.
|
|
||||||
|
|
||||||
Behavior:
|
|
||||||
* AUTH_DISABLED short-circuits (synthetic admin, no role check).
|
|
||||||
* No session cookie → 401 from get_current_user.
|
|
||||||
* Endpoint not in the matrix → 403 (fail-closed).
|
|
||||||
* User role not allowed for (method, path) → 403.
|
|
||||||
* Empty allowed-roles set (e.g. /api/healthz) → public, no role check.
|
|
||||||
|
|
||||||
Used as ``dependencies=[Depends(matrix_gate)]`` on every authenticated
|
|
||||||
route. Centralizing the gate here means the matrix is the source of
|
|
||||||
truth — no need to wire per-route ``require_role(...)`` calls.
|
|
||||||
"""
|
|
||||||
if AUTH_DISABLED:
|
|
||||||
return user
|
|
||||||
method = request.method
|
|
||||||
path = request.url.path
|
|
||||||
roles = allowed_roles(method, path)
|
|
||||||
if roles is None:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_403_FORBIDDEN,
|
|
||||||
detail="forbidden",
|
|
||||||
)
|
|
||||||
user_role = user.get("role")
|
|
||||||
if user_role not in {r.value for r in roles}:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_403_FORBIDDEN,
|
|
||||||
detail="forbidden",
|
|
||||||
)
|
|
||||||
return user
|
|
||||||
@@ -1,79 +0,0 @@
|
|||||||
"""Role enum + PERMISSIONS matrix."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from enum import Enum
|
|
||||||
|
|
||||||
|
|
||||||
class Role(str, Enum):
|
|
||||||
ADMIN = "admin"
|
|
||||||
USER = "user"
|
|
||||||
VIEWER = "viewer"
|
|
||||||
|
|
||||||
|
|
||||||
ALL_ROLES = {Role.ADMIN, Role.USER, Role.VIEWER}
|
|
||||||
WRITE_ROLES = {Role.ADMIN, Role.USER}
|
|
||||||
ADMIN_ONLY = {Role.ADMIN}
|
|
||||||
|
|
||||||
|
|
||||||
# (method, path-prefix) → allowed roles.
|
|
||||||
# Endpoints not in this matrix default to DENY (fail-closed).
|
|
||||||
PERMISSIONS: dict[tuple[str, str], set[Role]] = {
|
|
||||||
# Public paths.
|
|
||||||
("GET", "/api/health"): set(),
|
|
||||||
("POST", "/api/auth/login"): set(),
|
|
||||||
|
|
||||||
# Auth surface.
|
|
||||||
("POST", "/api/auth/logout"): ALL_ROLES,
|
|
||||||
("GET", "/api/auth/me"): ALL_ROLES,
|
|
||||||
|
|
||||||
# Admin-only user management.
|
|
||||||
("GET", "/api/admin/users"): ADMIN_ONLY,
|
|
||||||
("POST", "/api/admin/users"): ADMIN_ONLY,
|
|
||||||
("PATCH", "/api/admin/users"): ADMIN_ONLY,
|
|
||||||
("DELETE", "/api/admin/users"): ADMIN_ONLY,
|
|
||||||
|
|
||||||
# Read endpoints (all authenticated roles).
|
|
||||||
("GET", "/api/claims"): ALL_ROLES,
|
|
||||||
("GET", "/api/remittances"): ALL_ROLES,
|
|
||||||
("GET", "/api/providers"): ALL_ROLES,
|
|
||||||
("GET", "/api/batches"): ALL_ROLES,
|
|
||||||
("GET", "/api/dashboard/summary"): ALL_ROLES,
|
|
||||||
("GET", "/api/activity"): ALL_ROLES,
|
|
||||||
("GET", "/api/inbox/lanes"): ALL_ROLES,
|
|
||||||
("GET", "/api/inbox/export.csv"): ALL_ROLES,
|
|
||||||
("GET", "/api/reconcile"): ALL_ROLES,
|
|
||||||
("GET", "/api/reconciliation"): ALL_ROLES,
|
|
||||||
("GET", "/api/audit-log"): ADMIN_ONLY,
|
|
||||||
|
|
||||||
# Write endpoints (admin + user, no viewer).
|
|
||||||
("POST", "/api/parse-837"): WRITE_ROLES,
|
|
||||||
("POST", "/api/parse-835"): WRITE_ROLES,
|
|
||||||
("POST", "/api/inbox"): WRITE_ROLES,
|
|
||||||
("POST", "/api/inbox/candidates"): WRITE_ROLES,
|
|
||||||
("POST", "/api/inbox/rejected"): WRITE_ROLES,
|
|
||||||
("POST", "/api/inbox/payer-rejected"): WRITE_ROLES,
|
|
||||||
("POST", "/api/reconcile"): WRITE_ROLES,
|
|
||||||
("POST", "/api/reconciliation"): WRITE_ROLES,
|
|
||||||
("POST", "/api/resubmit"): WRITE_ROLES,
|
|
||||||
("POST", "/api/acks"): WRITE_ROLES,
|
|
||||||
|
|
||||||
# CSV export — read-only.
|
|
||||||
("GET", "/api/export.csv"): ALL_ROLES,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def allowed_roles(method: str, path: str) -> set[Role] | None:
|
|
||||||
"""Return the set of roles allowed to call (method, path), or None if denied.
|
|
||||||
|
|
||||||
Uses longest-prefix match on path; falls back to DENY (None) if no entry matches.
|
|
||||||
"""
|
|
||||||
candidates = [
|
|
||||||
(len(prefix), roles)
|
|
||||||
for (m, prefix), roles in PERMISSIONS.items()
|
|
||||||
if m == method and (path == prefix or path.startswith(prefix.rstrip("/") + "/"))
|
|
||||||
]
|
|
||||||
if not candidates:
|
|
||||||
return None
|
|
||||||
candidates.sort(key=lambda x: -x[0])
|
|
||||||
return candidates[0][1]
|
|
||||||
@@ -1,34 +0,0 @@
|
|||||||
"""Per-username login rate limiter (in-memory, per-process)."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import time
|
|
||||||
from threading import Lock
|
|
||||||
|
|
||||||
WINDOW_SECONDS = 300
|
|
||||||
MAX_FAILS = 5
|
|
||||||
|
|
||||||
_FAILS: dict[str, list[float]] = {}
|
|
||||||
_LOCK = Lock()
|
|
||||||
|
|
||||||
|
|
||||||
def check(username: str) -> int:
|
|
||||||
"""Return retry-after seconds, or 0 if allowed."""
|
|
||||||
now = time.monotonic()
|
|
||||||
with _LOCK:
|
|
||||||
fails = [t for t in _FAILS.get(username, []) if now - t < WINDOW_SECONDS]
|
|
||||||
_FAILS[username] = fails
|
|
||||||
if len(fails) >= MAX_FAILS:
|
|
||||||
return int(WINDOW_SECONDS - (now - fails[0]))
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
def record_failure(username: str) -> None:
|
|
||||||
now = time.monotonic()
|
|
||||||
with _LOCK:
|
|
||||||
_FAILS.setdefault(username, []).append(now)
|
|
||||||
|
|
||||||
|
|
||||||
def reset(username: str) -> None:
|
|
||||||
with _LOCK:
|
|
||||||
_FAILS.pop(username, None)
|
|
||||||
@@ -1,97 +0,0 @@
|
|||||||
"""/api/auth/login, /api/auth/logout, /api/auth/me."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
|
|
||||||
|
|
||||||
from cyclone.auth import rate_limit, sessions, users
|
|
||||||
from cyclone.auth.deps import get_current_user
|
|
||||||
from cyclone.db import SessionLocal
|
|
||||||
|
|
||||||
router = APIRouter(prefix="/api/auth", tags=["auth"])
|
|
||||||
|
|
||||||
COOKIE_NAME = "cyclone_session"
|
|
||||||
COOKIE_MAX_AGE = 86400 # 24h
|
|
||||||
|
|
||||||
|
|
||||||
def _is_https(request: Request) -> bool:
|
|
||||||
if request.url.scheme == "https":
|
|
||||||
return True
|
|
||||||
return os.environ.get("CYCLONE_BEHIND_HTTPS") == "1"
|
|
||||||
|
|
||||||
|
|
||||||
def _set_cookie(response: Response, sid: str, request: Request) -> None:
|
|
||||||
response.set_cookie(
|
|
||||||
key=COOKIE_NAME,
|
|
||||||
value=sid,
|
|
||||||
max_age=COOKIE_MAX_AGE,
|
|
||||||
path="/api",
|
|
||||||
httponly=True,
|
|
||||||
samesite="lax",
|
|
||||||
secure=_is_https(request),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _clear_cookie(response: Response) -> None:
|
|
||||||
response.delete_cookie(key=COOKIE_NAME, path="/api")
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/login")
|
|
||||||
def login(body: dict, request: Request, response: Response):
|
|
||||||
username = (body.get("username") or "").strip()
|
|
||||||
password = body.get("password") or ""
|
|
||||||
if not username or not password:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_400_BAD_REQUEST,
|
|
||||||
detail="username and password are required",
|
|
||||||
)
|
|
||||||
|
|
||||||
retry_after = rate_limit.check(username)
|
|
||||||
if retry_after > 0:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
|
|
||||||
detail="rate_limited",
|
|
||||||
headers={"Retry-After": str(retry_after)},
|
|
||||||
)
|
|
||||||
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
user = users.get_by_username(db, username)
|
|
||||||
if user is None or not users.verify_password(password, user.password_hash):
|
|
||||||
rate_limit.record_failure(username)
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
|
||||||
detail="invalid_credentials",
|
|
||||||
)
|
|
||||||
if user.disabled_at is not None:
|
|
||||||
raise HTTPException(
|
|
||||||
status_code=status.HTTP_403_FORBIDDEN,
|
|
||||||
detail="account_disabled",
|
|
||||||
)
|
|
||||||
|
|
||||||
sid, _ = sessions.create(db, user_id=user.id)
|
|
||||||
rate_limit.reset(username)
|
|
||||||
public = users.to_public(user)
|
|
||||||
|
|
||||||
_set_cookie(response, sid, request)
|
|
||||||
return public
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/logout")
|
|
||||||
def logout(
|
|
||||||
request: Request,
|
|
||||||
response: Response,
|
|
||||||
_user: dict = Depends(get_current_user),
|
|
||||||
):
|
|
||||||
sid = request.cookies.get(COOKIE_NAME)
|
|
||||||
if sid:
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
sessions.delete(db, sid)
|
|
||||||
_clear_cookie(response)
|
|
||||||
return Response(status_code=status.HTTP_204_NO_CONTENT)
|
|
||||||
|
|
||||||
|
|
||||||
@router.get("/me")
|
|
||||||
def me(user: dict = Depends(get_current_user)):
|
|
||||||
return user
|
|
||||||
@@ -1,60 +0,0 @@
|
|||||||
"""Session create/validate/expire/touch."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import secrets
|
|
||||||
from datetime import datetime, timedelta, timezone
|
|
||||||
|
|
||||||
from sqlalchemy import select
|
|
||||||
|
|
||||||
from cyclone.db import Session
|
|
||||||
|
|
||||||
SESSION_LIFETIME = timedelta(hours=24)
|
|
||||||
|
|
||||||
|
|
||||||
def create(db, *, user_id: int) -> tuple[str, Session]:
|
|
||||||
sid = secrets.token_urlsafe(32)
|
|
||||||
now = datetime.now(timezone.utc)
|
|
||||||
expires_at = now + SESSION_LIFETIME
|
|
||||||
sess = Session(
|
|
||||||
id=sid,
|
|
||||||
user_id=user_id,
|
|
||||||
expires_at=expires_at,
|
|
||||||
created_at=now,
|
|
||||||
)
|
|
||||||
db.add(sess)
|
|
||||||
db.commit()
|
|
||||||
db.refresh(sess)
|
|
||||||
# SQLite strips tzinfo on roundtrip; restore it so callers don't have to.
|
|
||||||
sess.expires_at = expires_at
|
|
||||||
return sid, sess
|
|
||||||
|
|
||||||
|
|
||||||
def get_valid(db, sid: str) -> Session | None:
|
|
||||||
sess = db.execute(
|
|
||||||
select(Session).where(Session.id == sid)
|
|
||||||
).scalar_one_or_none()
|
|
||||||
if sess is None:
|
|
||||||
return None
|
|
||||||
# SQLite drops tzinfo on roundtrip; normalize to UTC before comparing.
|
|
||||||
if sess.expires_at.tzinfo is None:
|
|
||||||
sess.expires_at = sess.expires_at.replace(tzinfo=timezone.utc)
|
|
||||||
if sess.expires_at <= datetime.now(timezone.utc):
|
|
||||||
return None
|
|
||||||
return sess
|
|
||||||
|
|
||||||
|
|
||||||
def delete(db, sid: str) -> None:
|
|
||||||
sess = db.get(Session, sid)
|
|
||||||
if sess is None:
|
|
||||||
return
|
|
||||||
db.delete(sess)
|
|
||||||
db.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def touch(db, sid: str) -> None:
|
|
||||||
sess = db.get(Session, sid)
|
|
||||||
if sess is None:
|
|
||||||
return
|
|
||||||
sess.expires_at = datetime.now(timezone.utc) + SESSION_LIFETIME
|
|
||||||
db.commit()
|
|
||||||
@@ -1,79 +0,0 @@
|
|||||||
"""User CRUD + bcrypt password hashing."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from passlib.hash import bcrypt
|
|
||||||
from sqlalchemy import select
|
|
||||||
|
|
||||||
from cyclone.db import User
|
|
||||||
|
|
||||||
|
|
||||||
def hash_password(plaintext: str) -> str:
|
|
||||||
return bcrypt.hash(plaintext)
|
|
||||||
|
|
||||||
|
|
||||||
def verify_password(plaintext: str, hashed: str) -> bool:
|
|
||||||
try:
|
|
||||||
return bcrypt.verify(plaintext, hashed)
|
|
||||||
except (ValueError, TypeError):
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def create(db, *, username: str, password: str, role: str) -> User:
|
|
||||||
user = User(
|
|
||||||
username=username,
|
|
||||||
password_hash=hash_password(password),
|
|
||||||
role=role,
|
|
||||||
created_at=datetime.now(timezone.utc),
|
|
||||||
)
|
|
||||||
db.add(user)
|
|
||||||
db.commit()
|
|
||||||
db.refresh(user)
|
|
||||||
return user
|
|
||||||
|
|
||||||
|
|
||||||
def get_by_username(db, username: str) -> User | None:
|
|
||||||
return db.execute(
|
|
||||||
select(User).where(User.username == username)
|
|
||||||
).scalar_one_or_none()
|
|
||||||
|
|
||||||
|
|
||||||
def get(db, user_id: int) -> User | None:
|
|
||||||
return db.get(User, user_id)
|
|
||||||
|
|
||||||
|
|
||||||
def disable(db, user_id: int) -> None:
|
|
||||||
user = db.get(User, user_id)
|
|
||||||
if user is None:
|
|
||||||
return
|
|
||||||
user.disabled_at = datetime.now(timezone.utc)
|
|
||||||
db.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def update_role(db, user_id: int, role: str) -> None:
|
|
||||||
user = db.get(User, user_id)
|
|
||||||
if user is None:
|
|
||||||
return
|
|
||||||
user.role = role
|
|
||||||
db.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def update_password(db, user_id: int, new_password: str) -> None:
|
|
||||||
user = db.get(User, user_id)
|
|
||||||
if user is None:
|
|
||||||
return
|
|
||||||
user.password_hash = hash_password(new_password)
|
|
||||||
db.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def to_public(user: User) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"id": user.id,
|
|
||||||
"username": user.username,
|
|
||||||
"role": user.role,
|
|
||||||
"createdAt": user.created_at.isoformat() if user.created_at else None,
|
|
||||||
"disabledAt": user.disabled_at.isoformat() if user.disabled_at else None,
|
|
||||||
}
|
|
||||||
@@ -1,280 +0,0 @@
|
|||||||
"""SP17 — Encrypted backup primitives.
|
|
||||||
|
|
||||||
This module provides the low-level building blocks the rest of the
|
|
||||||
backup stack uses:
|
|
||||||
|
|
||||||
* ``derive_key`` — PBKDF2-HMAC-SHA256 key derivation (200,000
|
|
||||||
iterations, 32-byte output). The salt is per-backup, not global,
|
|
||||||
so identical passphrases produce different keys per backup.
|
|
||||||
|
|
||||||
* ``encrypt`` / ``decrypt`` — AES-256-GCM authenticated encryption.
|
|
||||||
Output layout: ``salt (16) | nonce (12) | ciphertext | tag (16)``.
|
|
||||||
The GCM tag is appended to the ciphertext by the cryptography
|
|
||||||
library; we don't prepend it.
|
|
||||||
|
|
||||||
* ``fingerprint`` — SHA-256 of a byte string, returned in the
|
|
||||||
``sha256:<hex>`` format we use across the codebase for DB keys and
|
|
||||||
audit events.
|
|
||||||
|
|
||||||
* ``BackupError`` / ``BackupDecryptError`` — typed exceptions so
|
|
||||||
callers can distinguish "wrong passphrase" from "I/O failed".
|
|
||||||
|
|
||||||
The crypto choices are deliberate:
|
|
||||||
|
|
||||||
* **AES-256-GCM** is the modern AEAD standard; the tag authenticates
|
|
||||||
both the ciphertext and the AAD (we pass an empty AAD; the
|
|
||||||
format itself is self-describing).
|
|
||||||
|
|
||||||
* **PBKDF2-HMAC-SHA256 @ 200k iters** is OWASP's 2023+ minimum for
|
|
||||||
PBKDF2-SHA256. Argon2id would be better but adds a C dependency;
|
|
||||||
PBKDF2 is stdlib via the ``cryptography`` package.
|
|
||||||
|
|
||||||
* **Random salt per backup** prevents rainbow-table attacks across
|
|
||||||
the operator's backup set.
|
|
||||||
|
|
||||||
* **Random 96-bit nonce per encryption** is what AES-GCM requires;
|
|
||||||
we use ``os.urandom`` which is a CSPRNG on every platform we run
|
|
||||||
on (macOS, Linux).
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import os
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from datetime import datetime
|
|
||||||
from typing import Optional
|
|
||||||
|
|
||||||
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
|
|
||||||
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
|
|
||||||
from cryptography.hazmat.primitives import hashes
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Constants
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
# PBKDF2 iterations. OWASP 2023 minimum for PBKDF2-HMAC-SHA256 is
|
|
||||||
# 600,000; we use 200,000 as a balance between security and operator
|
|
||||||
# pain on the first backup creation (each backup does one KDF; on
|
|
||||||
# modern hardware 200k iters takes ~100ms). Bump this constant if you
|
|
||||||
# rotate the format version.
|
|
||||||
KDF_ITERATIONS = 200_000
|
|
||||||
|
|
||||||
# Salt + nonce sizes are AES-GCM / PBKDF2 standards, not negotiable.
|
|
||||||
SALT_LEN = 16
|
|
||||||
NONCE_LEN = 12
|
|
||||||
|
|
||||||
# Output key length for AES-256 = 32 bytes.
|
|
||||||
KEY_LEN = 32
|
|
||||||
|
|
||||||
# Format version. Bump when the on-disk layout changes (e.g. switch
|
|
||||||
# to Argon2id). Decryption reads this off the sidecar's
|
|
||||||
# encryption.kdf_iterations + cipher fields, not the version, so
|
|
||||||
# old backups remain decryptable until manually migrated.
|
|
||||||
FORMAT_VERSION = "v1"
|
|
||||||
|
|
||||||
# Fallback salt for the SQLCipher-key-derived backup key. Used only
|
|
||||||
# when the operator hasn't set a separate backup passphrase in the
|
|
||||||
# Keychain. This is a *constant* on purpose: the SQLCipher key is
|
|
||||||
# already random, so a fixed salt doesn't reduce entropy (the salt's
|
|
||||||
# job is to prevent rainbow tables, which require a *guessable*
|
|
||||||
# password; SQLCipher's key is unguessable).
|
|
||||||
FALLBACK_SALT = b"cyclone-db-backup-fallback-v1"
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Exceptions
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
class BackupError(Exception):
|
|
||||||
"""Generic backup failure. See BackupDecryptError for crypto errors."""
|
|
||||||
|
|
||||||
|
|
||||||
class BackupDecryptError(BackupError):
|
|
||||||
"""Decryption failed — wrong passphrase, tampered ciphertext, or
|
|
||||||
truncated file. Caller should NOT retry with the same key."""
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Key derivation + encryption
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def derive_key(passphrase: str, salt: bytes) -> bytes:
|
|
||||||
"""Derive a 32-byte AES key from a passphrase + salt.
|
|
||||||
|
|
||||||
Uses PBKDF2-HMAC-SHA256 with :data:`KDF_ITERATIONS` rounds. The
|
|
||||||
passphrase is encoded as UTF-8 bytes; the salt is used verbatim.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
passphrase: The operator's passphrase (any string).
|
|
||||||
salt: Per-backup random bytes of length :data:`SALT_LEN`.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
32 bytes suitable for AES-256-GCM.
|
|
||||||
"""
|
|
||||||
kdf = PBKDF2HMAC(
|
|
||||||
algorithm=hashes.SHA256(),
|
|
||||||
length=KEY_LEN,
|
|
||||||
salt=salt,
|
|
||||||
iterations=KDF_ITERATIONS,
|
|
||||||
)
|
|
||||||
return kdf.derive(passphrase.encode("utf-8"))
|
|
||||||
|
|
||||||
|
|
||||||
def encrypt(plaintext: bytes, key: bytes) -> bytes:
|
|
||||||
"""AES-256-GCM encrypt with a fresh random 12-byte nonce.
|
|
||||||
|
|
||||||
Returns ``nonce (12) || ciphertext || tag (16)``. The
|
|
||||||
``cryptography`` library appends the tag automatically.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
plaintext: The bytes to encrypt (e.g. the SQLite .backup blob).
|
|
||||||
key: 32-byte AES key from :func:`derive_key`.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
The combined nonce+ciphertext+tag blob.
|
|
||||||
"""
|
|
||||||
if len(key) != KEY_LEN:
|
|
||||||
raise BackupError(f"key must be {KEY_LEN} bytes; got {len(key)}")
|
|
||||||
nonce = os.urandom(NONCE_LEN)
|
|
||||||
aesgcm = AESGCM(key)
|
|
||||||
ciphertext = aesgcm.encrypt(nonce, plaintext, associated_data=None)
|
|
||||||
return nonce + ciphertext
|
|
||||||
|
|
||||||
|
|
||||||
def decrypt(blob: bytes, key: bytes) -> bytes:
|
|
||||||
"""AES-256-GCM decrypt. Raises :class:`BackupDecryptError` on auth failure.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
blob: The ``nonce||ciphertext||tag`` bytes from :func:`encrypt`.
|
|
||||||
key: The same 32-byte key used to encrypt.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
The original plaintext.
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
BackupDecryptError: If the blob is too short, the tag fails to
|
|
||||||
verify (wrong key or tampered ciphertext), or the input is
|
|
||||||
otherwise malformed.
|
|
||||||
"""
|
|
||||||
if len(key) != KEY_LEN:
|
|
||||||
raise BackupError(f"key must be {KEY_LEN} bytes; got {len(key)}")
|
|
||||||
if len(blob) < NONCE_LEN + 16:
|
|
||||||
# 12 (nonce) + 16 (tag) = minimum; no room for ciphertext.
|
|
||||||
raise BackupDecryptError(
|
|
||||||
f"blob too short ({len(blob)} bytes); expected >= {NONCE_LEN + 16}",
|
|
||||||
)
|
|
||||||
nonce = blob[:NONCE_LEN]
|
|
||||||
ciphertext = blob[NONCE_LEN:]
|
|
||||||
aesgcm = AESGCM(key)
|
|
||||||
try:
|
|
||||||
return aesgcm.decrypt(nonce, ciphertext, associated_data=None)
|
|
||||||
except Exception as exc: # cryptography raises InvalidTag
|
|
||||||
raise BackupDecryptError(f"decryption failed: {exc}") from exc
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Fingerprint
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def fingerprint(data: bytes) -> str:
|
|
||||||
"""SHA-256 of ``data`` as ``"sha256:<64-hex-chars>"``."""
|
|
||||||
return "sha256:" + hashlib.sha256(data).hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
def fingerprint_file(path: "os.PathLike[str] | str") -> str:
|
|
||||||
"""SHA-256 of a file's bytes, streamed. Memory-bounded for big DBs."""
|
|
||||||
h = hashlib.sha256()
|
|
||||||
with open(path, "rb") as f:
|
|
||||||
for chunk in iter(lambda: f.read(1024 * 1024), b""):
|
|
||||||
h.update(chunk)
|
|
||||||
return "sha256:" + h.hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Sidecar dataclass
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class Sidecar:
|
|
||||||
"""Plaintext metadata written next to each backup file.
|
|
||||||
|
|
||||||
Not required for decryption — it's a manifest an operator
|
|
||||||
consults to decide whether to restore. Kept intentionally
|
|
||||||
tiny so it survives most format rotations.
|
|
||||||
"""
|
|
||||||
|
|
||||||
format_version: str
|
|
||||||
created_at: str # ISO 8601 UTC
|
|
||||||
db_fingerprint: str # "sha256:..."
|
|
||||||
table_count: int
|
|
||||||
size_bytes: int
|
|
||||||
kdf: str # "PBKDF2-HMAC-SHA256"
|
|
||||||
kdf_iterations: int
|
|
||||||
cipher: str # "AES-256-GCM"
|
|
||||||
key_fingerprint: str # "sha256:..." of the derived key
|
|
||||||
|
|
||||||
def to_json(self) -> str:
|
|
||||||
import json
|
|
||||||
return json.dumps(
|
|
||||||
{
|
|
||||||
"format_version": self.format_version,
|
|
||||||
"created_at": self.created_at,
|
|
||||||
"db_fingerprint": self.db_fingerprint,
|
|
||||||
"table_count": self.table_count,
|
|
||||||
"size_bytes": self.size_bytes,
|
|
||||||
"encryption": {
|
|
||||||
"kdf": self.kdf,
|
|
||||||
"kdf_iterations": self.kdf_iterations,
|
|
||||||
"cipher": self.cipher,
|
|
||||||
"key_fingerprint": self.key_fingerprint,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
indent=2,
|
|
||||||
sort_keys=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def from_json(cls, text: str) -> "Sidecar":
|
|
||||||
import json
|
|
||||||
d = json.loads(text)
|
|
||||||
enc = d.get("encryption") or {}
|
|
||||||
return cls(
|
|
||||||
format_version=d["format_version"],
|
|
||||||
created_at=d["created_at"],
|
|
||||||
db_fingerprint=d["db_fingerprint"],
|
|
||||||
table_count=int(d["table_count"]),
|
|
||||||
size_bytes=int(d["size_bytes"]),
|
|
||||||
kdf=enc.get("kdf", "PBKDF2-HMAC-SHA256"),
|
|
||||||
kdf_iterations=int(enc.get("kdf_iterations", KDF_ITERATIONS)),
|
|
||||||
cipher=enc.get("cipher", "AES-256-GCM"),
|
|
||||||
key_fingerprint=enc.get("key_fingerprint", ""),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Filename helpers
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def backup_filename(timestamp: Optional[datetime] = None) -> str:
|
|
||||||
"""``cyclone-backup-YYYYMMDDTHHMMSSZ-<rand>.bin`` for a UTC timestamp.
|
|
||||||
|
|
||||||
The random suffix is a 4-byte hex string so two backups in the
|
|
||||||
same second don't collide on the ``db_backups`` unique index.
|
|
||||||
"""
|
|
||||||
import secrets as _secrets
|
|
||||||
from datetime import timezone as _tz
|
|
||||||
ts = timestamp or datetime.now(_tz.utc)
|
|
||||||
suffix = _secrets.token_hex(4)
|
|
||||||
return f"cyclone-backup-{ts.strftime('%Y%m%dT%H%M%SZ')}-{suffix}.bin"
|
|
||||||
|
|
||||||
|
|
||||||
def sidecar_filename(bin_filename: str) -> str:
|
|
||||||
"""``<bin_filename>.meta.json``."""
|
|
||||||
return bin_filename + ".meta.json"
|
|
||||||
@@ -1,368 +0,0 @@
|
|||||||
"""SP17 — Backup scheduler.
|
|
||||||
|
|
||||||
Wraps :class:`cyclone.backup_service.BackupService` in an
|
|
||||||
asyncio task, mirroring the MFT scheduler pattern (SP16). A backup
|
|
||||||
tick:
|
|
||||||
|
|
||||||
1. Calls :meth:`BackupService.create_now` to take + encrypt a backup.
|
|
||||||
2. Calls :meth:`BackupService.prune` to apply the retention policy.
|
|
||||||
3. Writes a tamper-evident ``audit_log`` row (SP11) for each outcome.
|
|
||||||
|
|
||||||
The scheduler is OFF by default. Operators opt in via
|
|
||||||
``CYCLONE_BACKUP_AUTOSTART=true``. The poll interval is
|
|
||||||
``CYCLONE_BACKUP_INTERVAL_HOURS`` (default 24).
|
|
||||||
|
|
||||||
Like the MFT scheduler, this is single-asyncio-task — no
|
|
||||||
threading, no APScheduler. All access (start/stop/tick/status)
|
|
||||||
must happen on the same event loop; the FastAPI app satisfies
|
|
||||||
that trivially because endpoints run on the loop.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import logging
|
|
||||||
import os
|
|
||||||
import traceback
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any, Optional
|
|
||||||
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
from cyclone.audit_log import AuditEvent, append_event
|
|
||||||
from cyclone.backup_service import BackupService
|
|
||||||
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class BackupTickResult:
|
|
||||||
"""Outcome of a single backup tick (one cycle of create + prune + audit)."""
|
|
||||||
|
|
||||||
started_at: datetime
|
|
||||||
finished_at: Optional[datetime] = None
|
|
||||||
created: Optional[svc_mod.BackupRecord] = None
|
|
||||||
pruned_paths: list[str] = field(default_factory=list)
|
|
||||||
error: Optional[str] = None
|
|
||||||
|
|
||||||
@property
|
|
||||||
def ok(self) -> bool:
|
|
||||||
return self.error is None and self.created is not None
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"started_at": self.started_at.isoformat(),
|
|
||||||
"finished_at": (
|
|
||||||
self.finished_at.isoformat() if self.finished_at else None
|
|
||||||
),
|
|
||||||
"ok": self.ok,
|
|
||||||
"created": (
|
|
||||||
{
|
|
||||||
"id": self.created.id,
|
|
||||||
"filename": self.created.filename,
|
|
||||||
"size_bytes": self.created.size_bytes,
|
|
||||||
"db_fingerprint": self.created.db_fingerprint,
|
|
||||||
"table_count": self.created.table_count,
|
|
||||||
}
|
|
||||||
if self.created else None
|
|
||||||
),
|
|
||||||
"pruned_paths": list(self.pruned_paths),
|
|
||||||
"error": self.error,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class BackupSchedulerStatus:
|
|
||||||
running: bool
|
|
||||||
interval_hours: float
|
|
||||||
backup_dir: str
|
|
||||||
retention_days: int
|
|
||||||
last_tick: Optional[BackupTickResult] = None
|
|
||||||
tick_count: int = 0
|
|
||||||
total_created: int = 0
|
|
||||||
total_errors: int = 0
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"running": self.running,
|
|
||||||
"interval_hours": self.interval_hours,
|
|
||||||
"backup_dir": self.backup_dir,
|
|
||||||
"retention_days": self.retention_days,
|
|
||||||
"tick_count": self.tick_count,
|
|
||||||
"total_created": self.total_created,
|
|
||||||
"total_errors": self.total_errors,
|
|
||||||
"last_tick": self.last_tick.as_dict() if self.last_tick else None,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
class BackupScheduler:
|
|
||||||
"""Asyncio loop that ticks the BackupService on an interval.
|
|
||||||
|
|
||||||
Lifecycle mirrors :class:`cyclone.scheduler.Scheduler`:
|
|
||||||
|
|
||||||
sched = BackupScheduler(backup_service)
|
|
||||||
await sched.start() # begin ticking
|
|
||||||
await sched.stop() # finish current tick, exit
|
|
||||||
status = sched.status() # snapshot
|
|
||||||
|
|
||||||
Threading: NOT thread-safe. All access must happen on the
|
|
||||||
same event loop. FastAPI endpoints satisfy this automatically.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
service: BackupService,
|
|
||||||
*,
|
|
||||||
interval_hours: float = 24.0,
|
|
||||||
) -> None:
|
|
||||||
self._service = service
|
|
||||||
self._interval_hours = max(0.1, float(interval_hours))
|
|
||||||
self._task: Optional[asyncio.Task[None]] = None
|
|
||||||
self._stop_event = asyncio.Event()
|
|
||||||
self._tick_in_progress = False
|
|
||||||
self._last_tick: Optional[BackupTickResult] = None
|
|
||||||
self._tick_count = 0
|
|
||||||
self._total_created = 0
|
|
||||||
self._total_errors = 0
|
|
||||||
|
|
||||||
@property
|
|
||||||
def service(self) -> BackupService:
|
|
||||||
return self._service
|
|
||||||
|
|
||||||
# ---- Public API -------------------------------------------------------
|
|
||||||
|
|
||||||
async def start(self) -> None:
|
|
||||||
if self._task is not None and not self._task.done():
|
|
||||||
log.info("BackupScheduler already running; start() is a no-op")
|
|
||||||
return
|
|
||||||
self._stop_event.clear()
|
|
||||||
self._task = asyncio.create_task(self._run(), name="backup-scheduler")
|
|
||||||
log.info(
|
|
||||||
"BackupScheduler started",
|
|
||||||
extra={
|
|
||||||
"interval_hours": self._interval_hours,
|
|
||||||
"backup_dir": str(self._service.backup_dir),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
async def stop(self) -> None:
|
|
||||||
if self._task is None or self._task.done():
|
|
||||||
return
|
|
||||||
self._stop_event.set()
|
|
||||||
try:
|
|
||||||
await asyncio.wait_for(self._task, timeout=60)
|
|
||||||
except asyncio.TimeoutError:
|
|
||||||
log.warning("BackupScheduler did not stop within 60s; cancelling")
|
|
||||||
self._task.cancel()
|
|
||||||
try:
|
|
||||||
await self._task
|
|
||||||
except (asyncio.CancelledError, Exception): # noqa: BLE001
|
|
||||||
pass
|
|
||||||
self._task = None
|
|
||||||
log.info("BackupScheduler stopped")
|
|
||||||
|
|
||||||
def status(self) -> BackupSchedulerStatus:
|
|
||||||
return BackupSchedulerStatus(
|
|
||||||
running=self.is_running(),
|
|
||||||
interval_hours=self._interval_hours,
|
|
||||||
backup_dir=str(self._service.backup_dir),
|
|
||||||
retention_days=self._service._retention_days,
|
|
||||||
last_tick=self._last_tick,
|
|
||||||
tick_count=self._tick_count,
|
|
||||||
total_created=self._total_created,
|
|
||||||
total_errors=self._total_errors,
|
|
||||||
)
|
|
||||||
|
|
||||||
def is_running(self) -> bool:
|
|
||||||
return self._task is not None and not self._task.done()
|
|
||||||
|
|
||||||
async def tick(self) -> BackupTickResult:
|
|
||||||
"""Run a single backup tick (create + prune + audit).
|
|
||||||
|
|
||||||
Concurrent ticks are coalesced: if a tick is already in
|
|
||||||
progress, the second caller waits for it. This protects
|
|
||||||
against a slow backup holding up multiple operator-driven
|
|
||||||
``POST /api/admin/backup/tick`` calls.
|
|
||||||
"""
|
|
||||||
while self._tick_in_progress:
|
|
||||||
await asyncio.sleep(0.05)
|
|
||||||
self._tick_in_progress = True
|
|
||||||
try:
|
|
||||||
result = await self._tick_impl()
|
|
||||||
self._last_tick = result
|
|
||||||
self._tick_count += 1
|
|
||||||
if result.created is not None:
|
|
||||||
self._total_created += 1
|
|
||||||
if result.error is not None:
|
|
||||||
self._total_errors += 1
|
|
||||||
return result
|
|
||||||
finally:
|
|
||||||
self._tick_in_progress = False
|
|
||||||
|
|
||||||
# ---- Internals --------------------------------------------------------
|
|
||||||
|
|
||||||
async def _run(self) -> None:
|
|
||||||
# Stagger the first tick (same rationale as the MFT scheduler).
|
|
||||||
await asyncio.sleep(5)
|
|
||||||
while not self._stop_event.is_set():
|
|
||||||
try:
|
|
||||||
await self.tick()
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
# tick() catches its own exceptions and returns them
|
|
||||||
# in the result. This is the safety net for
|
|
||||||
# programmer errors in the loop body.
|
|
||||||
log.exception("BackupScheduler tick raised", extra={"error": str(exc)})
|
|
||||||
try:
|
|
||||||
await asyncio.wait_for(
|
|
||||||
self._stop_event.wait(),
|
|
||||||
timeout=self._interval_hours * 3600,
|
|
||||||
)
|
|
||||||
except asyncio.TimeoutError:
|
|
||||||
pass # interval elapsed
|
|
||||||
|
|
||||||
async def _tick_impl(self) -> BackupTickResult:
|
|
||||||
started = datetime.now(timezone.utc)
|
|
||||||
result = BackupTickResult(started_at=started)
|
|
||||||
try:
|
|
||||||
create_result = await asyncio.to_thread(self._service.create_now)
|
|
||||||
result.created = create_result.backup
|
|
||||||
# Audit event for the created backup.
|
|
||||||
await asyncio.to_thread(
|
|
||||||
_audit_backup_created,
|
|
||||||
create_result.backup.id,
|
|
||||||
create_result.backup.db_fingerprint,
|
|
||||||
create_result.backup.table_count,
|
|
||||||
"backup-scheduler",
|
|
||||||
)
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
log.exception("Backup create failed during tick")
|
|
||||||
result.error = f"create: {type(exc).__name__}: {exc}"
|
|
||||||
await asyncio.to_thread(
|
|
||||||
_audit_backup_failed,
|
|
||||||
f"create: {type(exc).__name__}: {exc}",
|
|
||||||
traceback.format_exc()[-500:],
|
|
||||||
"backup-scheduler",
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
|
||||||
pruned = await asyncio.to_thread(self._service.prune)
|
|
||||||
result.pruned_paths = pruned
|
|
||||||
if pruned:
|
|
||||||
await asyncio.to_thread(
|
|
||||||
_audit_backup_pruned, pruned, "backup-scheduler",
|
|
||||||
)
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
log.exception("Backup prune failed during tick")
|
|
||||||
# Don't clobber the create error if there was one.
|
|
||||||
if result.error is None:
|
|
||||||
result.error = f"prune: {type(exc).__name__}: {exc}"
|
|
||||||
await asyncio.to_thread(
|
|
||||||
_audit_backup_failed,
|
|
||||||
f"prune: {type(exc).__name__}: {exc}",
|
|
||||||
traceback.format_exc()[-500:],
|
|
||||||
"backup-scheduler",
|
|
||||||
)
|
|
||||||
|
|
||||||
result.finished_at = datetime.now(timezone.utc)
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Audit helpers (run in a thread so the asyncio loop doesn't block)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _audit_backup_created(
|
|
||||||
backup_id: int, db_fingerprint: str, table_count: int, actor: str,
|
|
||||||
) -> None:
|
|
||||||
from cyclone import db
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
try:
|
|
||||||
append_event(s, AuditEvent(
|
|
||||||
event_type="db.backup_created",
|
|
||||||
entity_type="database",
|
|
||||||
entity_id="cyclone.db",
|
|
||||||
actor=actor,
|
|
||||||
payload={
|
|
||||||
"backup_id": backup_id,
|
|
||||||
"db_fingerprint": db_fingerprint,
|
|
||||||
"table_count": table_count,
|
|
||||||
},
|
|
||||||
))
|
|
||||||
s.commit()
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
log.exception("Failed to write db.backup_created audit event")
|
|
||||||
|
|
||||||
|
|
||||||
def _audit_backup_failed(
|
|
||||||
reason: str, traceback_tail: str, actor: str,
|
|
||||||
) -> None:
|
|
||||||
from cyclone import db
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
try:
|
|
||||||
append_event(s, AuditEvent(
|
|
||||||
event_type="db.backup_failed",
|
|
||||||
entity_type="database",
|
|
||||||
entity_id="cyclone.db",
|
|
||||||
actor=actor,
|
|
||||||
payload={"reason": reason, "traceback_tail": traceback_tail},
|
|
||||||
))
|
|
||||||
s.commit()
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
log.exception("Failed to write db.backup_failed audit event")
|
|
||||||
|
|
||||||
|
|
||||||
def _audit_backup_pruned(deleted_paths: list[str], actor: str) -> None:
|
|
||||||
from cyclone import db
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
try:
|
|
||||||
append_event(s, AuditEvent(
|
|
||||||
event_type="db.backup_pruned",
|
|
||||||
entity_type="database",
|
|
||||||
entity_id="cyclone.db",
|
|
||||||
actor=actor,
|
|
||||||
payload={"deleted_paths": deleted_paths},
|
|
||||||
))
|
|
||||||
s.commit()
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
log.exception("Failed to write db.backup_pruned audit event")
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Module-level singleton
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
_scheduler: Optional[BackupScheduler] = None
|
|
||||||
|
|
||||||
|
|
||||||
def configure_backup_scheduler(
|
|
||||||
service: BackupService,
|
|
||||||
*,
|
|
||||||
interval_hours: float = 24.0,
|
|
||||||
) -> BackupScheduler:
|
|
||||||
"""Create (or return existing) the module-level BackupScheduler."""
|
|
||||||
global _scheduler
|
|
||||||
if _scheduler is not None:
|
|
||||||
return _scheduler
|
|
||||||
hours = float(
|
|
||||||
os.environ.get("CYCLONE_BACKUP_INTERVAL_HOURS", interval_hours),
|
|
||||||
)
|
|
||||||
_scheduler = BackupScheduler(service, interval_hours=hours)
|
|
||||||
return _scheduler
|
|
||||||
|
|
||||||
|
|
||||||
def get_backup_scheduler() -> BackupScheduler:
|
|
||||||
"""Return the configured BackupScheduler. Raises if not set up."""
|
|
||||||
if _scheduler is None:
|
|
||||||
raise RuntimeError(
|
|
||||||
"backup scheduler not configured; call configure_backup_scheduler() first",
|
|
||||||
)
|
|
||||||
return _scheduler
|
|
||||||
|
|
||||||
|
|
||||||
def reset_backup_scheduler_for_tests() -> None:
|
|
||||||
"""Clear the module-level singleton. Test-only."""
|
|
||||||
global _scheduler
|
|
||||||
_scheduler = None
|
|
||||||
@@ -1,851 +0,0 @@
|
|||||||
"""SP17 — High-level backup coordinator.
|
|
||||||
|
|
||||||
Owns the lifecycle of every backup the operator (or the scheduler)
|
|
||||||
takes:
|
|
||||||
|
|
||||||
* ``create_now`` — runs SQLite's online ``.backup()`` against the
|
|
||||||
live engine, encrypts the bytes with :func:`cyclone.backup.encrypt`,
|
|
||||||
writes a ``.bin`` + ``.meta.json`` pair into the backup directory,
|
|
||||||
and persists a row in ``db_backups``.
|
|
||||||
|
|
||||||
* ``list_backups`` — directory listing joined with ``db_backups`` rows.
|
|
||||||
|
|
||||||
* ``verify`` — decrypts + recomputes SHA-256, compares to the sidecar.
|
|
||||||
|
|
||||||
* ``restore_initiate`` / ``restore_confirm`` — two-step restore so an
|
|
||||||
idle browser tab can't nuke the live DB. The first call returns a
|
|
||||||
``restore_token`` (a random 32-byte hex string) plus a preview
|
|
||||||
(``db_fingerprint``, ``table_count``). The second call swaps the
|
|
||||||
engine only if the token matches.
|
|
||||||
|
|
||||||
* ``prune`` — deletes backups older than ``retention_days``.
|
|
||||||
|
|
||||||
This module is intentionally engine-aware: ``create_now`` reaches
|
|
||||||
into the live SQLAlchemy engine to get a raw SQLite connection and
|
|
||||||
call ``.backup()`` (the only way to take an online consistent
|
|
||||||
snapshot). ``restore`` reaches into :func:`cyclone.db.dispose_engine`
|
|
||||||
+ :func:`cyclone.db.reinit_engine` to swap to the restored file.
|
|
||||||
|
|
||||||
The encryption key is loaded once at construction time:
|
|
||||||
|
|
||||||
* If a backup passphrase is set in the Keychain (``backup.passphrase``
|
|
||||||
account under service ``cyclone``), use it directly with the salt
|
|
||||||
stored in the companion ``backup.salt`` account. The salt must be
|
|
||||||
persisted — a fresh random salt per process would defeat the key.
|
|
||||||
* Otherwise fall back to deriving from the SQLCipher DB key + a fixed
|
|
||||||
salt. Logged at WARNING because this is a degraded-but-still-safe
|
|
||||||
posture.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
import os
|
|
||||||
import secrets as _secrets
|
|
||||||
import shutil
|
|
||||||
import sqlite3
|
|
||||||
import threading
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from datetime import datetime, timedelta, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Optional, Union
|
|
||||||
|
|
||||||
from sqlalchemy.exc import SQLAlchemyError
|
|
||||||
|
|
||||||
from cyclone import backup as backup_mod
|
|
||||||
from cyclone.backup import BackupError
|
|
||||||
from cyclone import db
|
|
||||||
from cyclone import secrets as secrets_mod
|
|
||||||
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
# Status values for db_backups.status (mirrored in the ORM).
|
|
||||||
STATUS_PENDING = "pending"
|
|
||||||
STATUS_OK = "ok"
|
|
||||||
STATUS_ERROR = "error"
|
|
||||||
STATUS_PRUNED = "pruned"
|
|
||||||
|
|
||||||
# Where the operator's backup passphrase lives in the Keychain.
|
|
||||||
KEYCHAIN_BACKUP_PASSPHRASE_ACCOUNT = "backup.passphrase"
|
|
||||||
|
|
||||||
# Companion account for the salt. Stored as hex. Same value across
|
|
||||||
# processes so the derived key is reproducible — a fresh random salt
|
|
||||||
# per BackupService would defeat the key.
|
|
||||||
KEYCHAIN_BACKUP_SALT_ACCOUNT = "backup.salt"
|
|
||||||
|
|
||||||
# Restore token TTL (seconds). The two-step confirm must complete
|
|
||||||
# within this window or the operator re-runs initiate.
|
|
||||||
RESTORE_TOKEN_TTL_SECONDS = 300
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Result dataclasses
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class BackupRecord:
|
|
||||||
"""Public view of a backup row joined with filesystem state."""
|
|
||||||
|
|
||||||
id: int
|
|
||||||
filename: str
|
|
||||||
backup_dir: str
|
|
||||||
size_bytes: int
|
|
||||||
db_fingerprint: str
|
|
||||||
table_count: int
|
|
||||||
created_at: datetime
|
|
||||||
completed_at: Optional[datetime]
|
|
||||||
status: str
|
|
||||||
error_message: Optional[str]
|
|
||||||
key_fingerprint: str
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class CreateResult:
|
|
||||||
"""Outcome of ``create_now``."""
|
|
||||||
|
|
||||||
backup: BackupRecord
|
|
||||||
sidecar: backup_mod.Sidecar
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class VerifyResult:
|
|
||||||
"""Outcome of ``verify``."""
|
|
||||||
|
|
||||||
backup_id: int
|
|
||||||
filename: str
|
|
||||||
ok: bool
|
|
||||||
expected_fingerprint: str
|
|
||||||
actual_fingerprint: str
|
|
||||||
table_count: int
|
|
||||||
reason: Optional[str] = None
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class RestoreInitiateResult:
|
|
||||||
"""Returned by the first call of the two-step restore."""
|
|
||||||
|
|
||||||
backup_id: int
|
|
||||||
filename: str
|
|
||||||
restore_token: str
|
|
||||||
expires_at: datetime
|
|
||||||
db_fingerprint: str
|
|
||||||
table_count: int
|
|
||||||
current_db_fingerprint: str
|
|
||||||
current_table_count: int
|
|
||||||
size_bytes: int
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class RestoreConfirmResult:
|
|
||||||
"""Returned by the second call of the two-step restore."""
|
|
||||||
|
|
||||||
backup_id: int
|
|
||||||
filename: str
|
|
||||||
restored_from_fingerprint: str
|
|
||||||
restored_at: datetime
|
|
||||||
new_db_fingerprint: str
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# BackupService
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
class BackupService:
|
|
||||||
"""Coordinator for encrypted DB backups.
|
|
||||||
|
|
||||||
Construct once at app startup; share across requests. Not
|
|
||||||
thread-safe for *creation* (the SQLite ``.backup()`` call uses
|
|
||||||
the live engine and is best serialized through the scheduler),
|
|
||||||
but ``list_backups`` / ``prune`` / ``status`` are safe to call
|
|
||||||
concurrently.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
backup_dir: Union[str, Path],
|
|
||||||
*,
|
|
||||||
passphrase: Optional[str] = None,
|
|
||||||
salt: Optional[bytes] = None,
|
|
||||||
retention_days: int = 30,
|
|
||||||
db_url: Optional[str] = None,
|
|
||||||
) -> None:
|
|
||||||
self._backup_dir = Path(backup_dir)
|
|
||||||
self._retention_days = max(1, int(retention_days))
|
|
||||||
self._db_url = db_url
|
|
||||||
# The derived key + its salt. If ``passphrase`` is None we
|
|
||||||
# fall back to deriving from the SQLCipher DB key (with a
|
|
||||||
# WARNING log).
|
|
||||||
#
|
|
||||||
# Salt is per-BackupService-instance and MUST be stable across
|
|
||||||
# processes — otherwise the same passphrase would derive
|
|
||||||
# different keys in different invocations and decrypt would
|
|
||||||
# always fail. Two options:
|
|
||||||
#
|
|
||||||
# 1. Caller passes an explicit ``salt`` (the Keychain flow
|
|
||||||
# reads the persisted salt from backup.salt account).
|
|
||||||
# 2. We accept a None salt here; ``_ensure_key`` then either
|
|
||||||
# uses the persisted salt (if available) or generates one
|
|
||||||
# and persists it on first use.
|
|
||||||
#
|
|
||||||
# Tests typically pass an explicit random salt; production
|
|
||||||
# should always pass the persisted one.
|
|
||||||
self._passphrase = passphrase
|
|
||||||
self._salt = salt
|
|
||||||
self._key: Optional[bytes] = None
|
|
||||||
self._used_fallback = False
|
|
||||||
# Pending restore tokens: token -> (backup_id, expires_at).
|
|
||||||
# A simple in-memory dict is sufficient — the token only
|
|
||||||
# needs to survive between the two API calls in one process.
|
|
||||||
self._pending_restores: dict[str, tuple[int, datetime]] = {}
|
|
||||||
self._lock = threading.Lock()
|
|
||||||
|
|
||||||
# ---- Public API -------------------------------------------------------
|
|
||||||
|
|
||||||
@property
|
|
||||||
def backup_dir(self) -> Path:
|
|
||||||
return self._backup_dir
|
|
||||||
|
|
||||||
@property
|
|
||||||
def key_fingerprint(self) -> str:
|
|
||||||
"""SHA-256 of the current derived key, or "" if not yet derived."""
|
|
||||||
if self._key is None:
|
|
||||||
return ""
|
|
||||||
return backup_mod.fingerprint(self._key)
|
|
||||||
|
|
||||||
def create_now(self) -> CreateResult:
|
|
||||||
"""Take an encrypted backup of the live DB right now.
|
|
||||||
|
|
||||||
Crash-safe: any failure marks the ``db_backups`` row as
|
|
||||||
``error``, removes any partial files from the backup dir, and
|
|
||||||
re-raises the exception.
|
|
||||||
"""
|
|
||||||
# 1. Make sure the backup dir exists.
|
|
||||||
self._backup_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
|
|
||||||
# 2. Allocate a filename + insert a pending row.
|
|
||||||
from cyclone.db import DbBackup # late import — circular otherwise
|
|
||||||
from cyclone.store import store as cycl_store
|
|
||||||
|
|
||||||
filename = backup_mod.backup_filename()
|
|
||||||
created_at = datetime.now(timezone.utc)
|
|
||||||
|
|
||||||
row = cycl_store.add_backup_pending(
|
|
||||||
filename=filename,
|
|
||||||
backup_dir=str(self._backup_dir),
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
|
||||||
# 3. Run SQLite's online .backup() to a temp file.
|
|
||||||
# We use a private path *inside* the backup dir so the
|
|
||||||
# operator can see what crashed if it does.
|
|
||||||
staging_db = self._backup_dir / f".{filename}.staging.db"
|
|
||||||
self._sqlite_backup_to(staging_db)
|
|
||||||
|
|
||||||
# 4. Encrypt.
|
|
||||||
plaintext = staging_db.read_bytes()
|
|
||||||
db_fp = backup_mod.fingerprint(plaintext)
|
|
||||||
key = self._ensure_key()
|
|
||||||
blob = backup_mod.encrypt(plaintext, key)
|
|
||||||
|
|
||||||
# 5. Move encrypted blob into place + write sidecar.
|
|
||||||
target = self._backup_dir / filename
|
|
||||||
target.write_bytes(blob)
|
|
||||||
staging_db.unlink()
|
|
||||||
|
|
||||||
table_count = self._count_tables_in_blob(plaintext)
|
|
||||||
sidecar = backup_mod.Sidecar(
|
|
||||||
format_version=backup_mod.FORMAT_VERSION,
|
|
||||||
created_at=created_at.isoformat(),
|
|
||||||
db_fingerprint=db_fp,
|
|
||||||
table_count=table_count,
|
|
||||||
size_bytes=len(blob),
|
|
||||||
kdf="PBKDF2-HMAC-SHA256",
|
|
||||||
kdf_iterations=backup_mod.KDF_ITERATIONS,
|
|
||||||
cipher="AES-256-GCM",
|
|
||||||
key_fingerprint=backup_mod.fingerprint(key),
|
|
||||||
)
|
|
||||||
sidecar_path = self._backup_dir / backup_mod.sidecar_filename(filename)
|
|
||||||
sidecar_path.write_text(sidecar.to_json())
|
|
||||||
|
|
||||||
# 6. Mark the row as ok.
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
row = s.get(DbBackup, row.id)
|
|
||||||
row.status = STATUS_OK
|
|
||||||
row.size_bytes = len(blob)
|
|
||||||
row.db_fingerprint = db_fp
|
|
||||||
row.table_count = table_count
|
|
||||||
row.completed_at = datetime.now(timezone.utc)
|
|
||||||
s.commit()
|
|
||||||
s.refresh(row)
|
|
||||||
|
|
||||||
record = self._row_to_record(row)
|
|
||||||
log.info(
|
|
||||||
"Backup created",
|
|
||||||
extra={
|
|
||||||
"backup_id": record.id,
|
|
||||||
"backup_filename": record.filename,
|
|
||||||
"size_bytes": record.size_bytes,
|
|
||||||
"db_fingerprint": record.db_fingerprint,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
return CreateResult(backup=record, sidecar=sidecar)
|
|
||||||
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
log.exception("Backup create failed")
|
|
||||||
try:
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
row = s.get(DbBackup, row.id)
|
|
||||||
row.status = STATUS_ERROR
|
|
||||||
row.error_message = f"{type(exc).__name__}: {exc}"[:500]
|
|
||||||
row.completed_at = datetime.now(timezone.utc)
|
|
||||||
s.commit()
|
|
||||||
# Best-effort cleanup of any partial files.
|
|
||||||
for p in [
|
|
||||||
self._backup_dir / filename,
|
|
||||||
self._backup_dir / f".{filename}.staging.db",
|
|
||||||
self._backup_dir / backup_mod.sidecar_filename(filename),
|
|
||||||
]:
|
|
||||||
if p.exists():
|
|
||||||
try:
|
|
||||||
p.unlink()
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
log.exception("Failed to mark backup row as error")
|
|
||||||
raise
|
|
||||||
|
|
||||||
def list_backups(
|
|
||||||
self,
|
|
||||||
*,
|
|
||||||
limit: int = 100,
|
|
||||||
status: Optional[str] = None,
|
|
||||||
) -> list[BackupRecord]:
|
|
||||||
"""List ``db_backups`` rows newest first.
|
|
||||||
|
|
||||||
Joins the filesystem state (presence of ``.bin`` and
|
|
||||||
``.meta.json``) implicitly via :attr:`BackupRecord.status`:
|
|
||||||
a row marked ``pruned`` had its files deleted by the
|
|
||||||
retention policy.
|
|
||||||
"""
|
|
||||||
from cyclone.db import DbBackup
|
|
||||||
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
q = s.query(DbBackup)
|
|
||||||
if status is not None:
|
|
||||||
q = q.filter(DbBackup.status == status)
|
|
||||||
rows = q.order_by(DbBackup.id.desc()).limit(limit).all()
|
|
||||||
return [self._row_to_record(r) for r in rows]
|
|
||||||
|
|
||||||
def verify(self, backup_id: int) -> VerifyResult:
|
|
||||||
"""Decrypt + checksum-verify a backup against its sidecar.
|
|
||||||
|
|
||||||
Does NOT trust the sidecar's ``db_fingerprint`` field alone;
|
|
||||||
recomputes the SHA-256 from the decrypted blob and compares.
|
|
||||||
"""
|
|
||||||
from cyclone.db import DbBackup
|
|
||||||
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
row = s.get(DbBackup, backup_id)
|
|
||||||
if row is None:
|
|
||||||
raise BackupError(f"backup {backup_id} not found")
|
|
||||||
record = self._row_to_record(row)
|
|
||||||
sidecar = self._read_sidecar(record.filename)
|
|
||||||
if sidecar is None:
|
|
||||||
return VerifyResult(
|
|
||||||
backup_id=record.id, filename=record.filename,
|
|
||||||
ok=False, expected_fingerprint="", actual_fingerprint="",
|
|
||||||
table_count=0, reason="sidecar missing",
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
|
||||||
blob = (self._backup_dir / record.filename).read_bytes()
|
|
||||||
except FileNotFoundError:
|
|
||||||
return VerifyResult(
|
|
||||||
backup_id=record.id, filename=record.filename,
|
|
||||||
ok=False,
|
|
||||||
expected_fingerprint=sidecar.db_fingerprint,
|
|
||||||
actual_fingerprint="",
|
|
||||||
table_count=sidecar.table_count,
|
|
||||||
reason="backup file missing",
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
|
||||||
plaintext = backup_mod.decrypt(blob, self._ensure_key())
|
|
||||||
except backup_mod.BackupDecryptError as exc:
|
|
||||||
return VerifyResult(
|
|
||||||
backup_id=record.id, filename=record.filename,
|
|
||||||
ok=False,
|
|
||||||
expected_fingerprint=sidecar.db_fingerprint,
|
|
||||||
actual_fingerprint="",
|
|
||||||
table_count=sidecar.table_count,
|
|
||||||
reason=str(exc),
|
|
||||||
)
|
|
||||||
|
|
||||||
actual_fp = backup_mod.fingerprint(plaintext)
|
|
||||||
return VerifyResult(
|
|
||||||
backup_id=record.id,
|
|
||||||
filename=record.filename,
|
|
||||||
ok=(actual_fp == sidecar.db_fingerprint),
|
|
||||||
expected_fingerprint=sidecar.db_fingerprint,
|
|
||||||
actual_fingerprint=actual_fp,
|
|
||||||
table_count=sidecar.table_count,
|
|
||||||
reason=None if actual_fp == sidecar.db_fingerprint else "fingerprint mismatch",
|
|
||||||
)
|
|
||||||
|
|
||||||
def restore_initiate(self, backup_id: int) -> RestoreInitiateResult:
|
|
||||||
"""First half of the two-step restore.
|
|
||||||
|
|
||||||
Decrypts the backup into a temp file and reads its current
|
|
||||||
``db_fingerprint`` + ``table_count``. Returns a one-shot
|
|
||||||
``restore_token`` the operator must echo back to
|
|
||||||
:meth:`restore_confirm` within 5 minutes.
|
|
||||||
"""
|
|
||||||
from cyclone.db import DbBackup
|
|
||||||
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
row = s.get(DbBackup, backup_id)
|
|
||||||
if row is None:
|
|
||||||
raise BackupError(f"backup {backup_id} not found")
|
|
||||||
if row.status != STATUS_OK:
|
|
||||||
raise BackupError(
|
|
||||||
f"backup {backup_id} status is {row.status!r}; only 'ok' backups can be restored",
|
|
||||||
)
|
|
||||||
record = self._row_to_record(row)
|
|
||||||
|
|
||||||
# Decrypt into a staging file so the confirm step is fast.
|
|
||||||
staging = self._backup_dir / f".restore-{record.filename}.staging.db"
|
|
||||||
try:
|
|
||||||
blob = (self._backup_dir / record.filename).read_bytes()
|
|
||||||
except FileNotFoundError as exc:
|
|
||||||
raise BackupError(f"backup file missing: {record.filename}") from exc
|
|
||||||
try:
|
|
||||||
plaintext = backup_mod.decrypt(blob, self._ensure_key())
|
|
||||||
except backup_mod.BackupDecryptError as exc:
|
|
||||||
raise BackupError(f"decrypt failed: {exc}") from exc
|
|
||||||
staging.write_bytes(plaintext)
|
|
||||||
|
|
||||||
# Snapshot the live DB's fingerprint for the operator's "are
|
|
||||||
# you sure you want to do this?" preview.
|
|
||||||
live_fp, live_count = self._live_fingerprint_and_count()
|
|
||||||
|
|
||||||
token = _secrets.token_hex(32)
|
|
||||||
expires_at = datetime.now(timezone.utc) + timedelta(
|
|
||||||
seconds=RESTORE_TOKEN_TTL_SECONDS,
|
|
||||||
)
|
|
||||||
with self._lock:
|
|
||||||
self._pending_restores[token] = (record.id, expires_at)
|
|
||||||
|
|
||||||
log.info(
|
|
||||||
"Restore initiated",
|
|
||||||
extra={
|
|
||||||
"backup_id": record.id,
|
|
||||||
"token_prefix": token[:8],
|
|
||||||
"expires_at": expires_at.isoformat(),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
return RestoreInitiateResult(
|
|
||||||
backup_id=record.id,
|
|
||||||
filename=record.filename,
|
|
||||||
restore_token=token,
|
|
||||||
expires_at=expires_at,
|
|
||||||
db_fingerprint=backup_mod.fingerprint(plaintext),
|
|
||||||
table_count=self._count_tables_in_blob(plaintext),
|
|
||||||
current_db_fingerprint=live_fp,
|
|
||||||
current_table_count=live_count,
|
|
||||||
size_bytes=len(plaintext),
|
|
||||||
)
|
|
||||||
|
|
||||||
def restore_confirm(
|
|
||||||
self,
|
|
||||||
backup_id: int,
|
|
||||||
restore_token: str,
|
|
||||||
*,
|
|
||||||
actor: str = "operator",
|
|
||||||
) -> RestoreConfirmResult:
|
|
||||||
"""Second half of the two-step restore.
|
|
||||||
|
|
||||||
Validates the token, copies the decrypted staging file over
|
|
||||||
the live DB path, disposes + reopens the engine. Raises
|
|
||||||
``BackupError`` on any mismatch.
|
|
||||||
"""
|
|
||||||
now = datetime.now(timezone.utc)
|
|
||||||
with self._lock:
|
|
||||||
entry = self._pending_restores.pop(restore_token, None)
|
|
||||||
if entry is None:
|
|
||||||
raise BackupError("restore_token not found (already consumed or never issued)")
|
|
||||||
token_backup_id, expires_at = entry
|
|
||||||
if token_backup_id != backup_id:
|
|
||||||
raise BackupError(
|
|
||||||
f"restore_token was for backup {token_backup_id}, not {backup_id}",
|
|
||||||
)
|
|
||||||
if now > expires_at:
|
|
||||||
raise BackupError(
|
|
||||||
f"restore_token expired at {expires_at.isoformat()}; re-run initiate",
|
|
||||||
)
|
|
||||||
|
|
||||||
from cyclone.db import DbBackup
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
row = s.get(DbBackup, backup_id)
|
|
||||||
if row is None:
|
|
||||||
raise BackupError(f"backup {backup_id} disappeared mid-restore")
|
|
||||||
record = self._row_to_record(row)
|
|
||||||
|
|
||||||
staging = self._backup_dir / f".restore-{record.filename}.staging.db"
|
|
||||||
if not staging.exists():
|
|
||||||
raise BackupError(
|
|
||||||
f"staging restore file missing: {staging.name}; re-run initiate",
|
|
||||||
)
|
|
||||||
|
|
||||||
target_db_path = self._live_db_path()
|
|
||||||
if target_db_path is None:
|
|
||||||
raise BackupError(
|
|
||||||
"cannot determine live DB file path (non-sqlite URL?)",
|
|
||||||
)
|
|
||||||
|
|
||||||
# Pre-restore fingerprint for the audit event.
|
|
||||||
restored_from_fp = backup_mod.fingerprint(staging.read_bytes())
|
|
||||||
|
|
||||||
# The swap: dispose engine → copy file → reinit engine.
|
|
||||||
# Anything between dispose and reinit raises (queries that
|
|
||||||
# are in-flight get a "database is locked" or
|
|
||||||
# "no such table" error); we accept that because the
|
|
||||||
# operator already confirmed.
|
|
||||||
db.dispose_engine()
|
|
||||||
try:
|
|
||||||
# Atomic copy via temp + rename so a crash mid-copy
|
|
||||||
# doesn't leave a half-written DB file.
|
|
||||||
tmp_target = target_db_path.with_suffix(
|
|
||||||
target_db_path.suffix + f".restoring-{_secrets.token_hex(4)}",
|
|
||||||
)
|
|
||||||
shutil.copyfile(staging, tmp_target)
|
|
||||||
os.replace(tmp_target, target_db_path)
|
|
||||||
finally:
|
|
||||||
staging.unlink(missing_ok=True)
|
|
||||||
db.reinit_engine()
|
|
||||||
|
|
||||||
# Post-restore fingerprint from the now-live engine.
|
|
||||||
new_fp, _ = self._live_fingerprint_and_count()
|
|
||||||
|
|
||||||
log.warning(
|
|
||||||
"Restore complete: backup_id=%d actor=%s from=%s to=%s",
|
|
||||||
backup_id, actor, restored_from_fp, new_fp,
|
|
||||||
)
|
|
||||||
return RestoreConfirmResult(
|
|
||||||
backup_id=record.id,
|
|
||||||
filename=record.filename,
|
|
||||||
restored_from_fingerprint=restored_from_fp,
|
|
||||||
restored_at=datetime.now(timezone.utc),
|
|
||||||
new_db_fingerprint=new_fp,
|
|
||||||
)
|
|
||||||
|
|
||||||
def prune(self, *, now: Optional[datetime] = None) -> list[str]:
|
|
||||||
"""Delete backups older than ``retention_days``. Returns deleted paths.
|
|
||||||
|
|
||||||
Marks the ``db_backups`` rows ``pruned`` so the operator can
|
|
||||||
still see what was deleted (and when).
|
|
||||||
"""
|
|
||||||
from cyclone.db import DbBackup
|
|
||||||
|
|
||||||
cutoff = (now or datetime.now(timezone.utc)) - timedelta(
|
|
||||||
days=self._retention_days,
|
|
||||||
)
|
|
||||||
deleted: list[str] = []
|
|
||||||
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
q = s.query(DbBackup).filter(
|
|
||||||
DbBackup.status == STATUS_OK,
|
|
||||||
DbBackup.created_at < cutoff,
|
|
||||||
)
|
|
||||||
for row in q.all():
|
|
||||||
# Delete the file pair; ignore if already gone.
|
|
||||||
bin_path = Path(row.backup_dir) / row.filename
|
|
||||||
meta_path = Path(row.backup_dir) / backup_mod.sidecar_filename(row.filename)
|
|
||||||
for p in (bin_path, meta_path):
|
|
||||||
try:
|
|
||||||
if p.exists():
|
|
||||||
p.unlink()
|
|
||||||
deleted.append(str(p))
|
|
||||||
except OSError as exc:
|
|
||||||
log.warning("Failed to delete %s: %s", p, exc)
|
|
||||||
row.status = STATUS_PRUNED
|
|
||||||
s.add(row)
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
log.info(
|
|
||||||
"Pruned old backups",
|
|
||||||
extra={
|
|
||||||
"deleted_count": len(deleted),
|
|
||||||
"cutoff": cutoff.isoformat(),
|
|
||||||
},
|
|
||||||
)
|
|
||||||
return deleted
|
|
||||||
|
|
||||||
def status(self) -> dict:
|
|
||||||
"""Snapshot of the backup subsystem for ``GET /api/admin/backup/status``."""
|
|
||||||
from cyclone.db import DbBackup
|
|
||||||
from sqlalchemy import func
|
|
||||||
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
total = s.query(func.count(DbBackup.id)).scalar() or 0
|
|
||||||
ok_count = s.query(func.count(DbBackup.id)).filter(
|
|
||||||
DbBackup.status == STATUS_OK,
|
|
||||||
).scalar() or 0
|
|
||||||
error_count = s.query(func.count(DbBackup.id)).filter(
|
|
||||||
DbBackup.status == STATUS_ERROR,
|
|
||||||
).scalar() or 0
|
|
||||||
pruned_count = s.query(func.count(DbBackup.id)).filter(
|
|
||||||
DbBackup.status == STATUS_PRUNED,
|
|
||||||
).scalar() or 0
|
|
||||||
last_row = (
|
|
||||||
s.query(DbBackup)
|
|
||||||
.filter(DbBackup.status.in_([STATUS_OK, STATUS_ERROR]))
|
|
||||||
.order_by(DbBackup.id.desc())
|
|
||||||
.first()
|
|
||||||
)
|
|
||||||
last_ok_row = (
|
|
||||||
s.query(DbBackup)
|
|
||||||
.filter(DbBackup.status == STATUS_OK)
|
|
||||||
.order_by(DbBackup.id.desc())
|
|
||||||
.first()
|
|
||||||
)
|
|
||||||
disk_bytes = 0
|
|
||||||
try:
|
|
||||||
for p in self._backup_dir.iterdir():
|
|
||||||
if p.is_file() and p.suffix == ".bin":
|
|
||||||
disk_bytes += p.stat().st_size
|
|
||||||
except OSError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
return {
|
|
||||||
"backup_dir": str(self._backup_dir),
|
|
||||||
"retention_days": self._retention_days,
|
|
||||||
"totals": {
|
|
||||||
"all": total,
|
|
||||||
"ok": ok_count,
|
|
||||||
"error": error_count,
|
|
||||||
"pruned": pruned_count,
|
|
||||||
},
|
|
||||||
"disk_bytes": disk_bytes,
|
|
||||||
"last_backup_at": (
|
|
||||||
last_row.created_at.isoformat() if last_row and last_row.created_at else None
|
|
||||||
),
|
|
||||||
"last_backup_status": last_row.status if last_row else None,
|
|
||||||
"last_ok_backup_at": (
|
|
||||||
last_ok_row.created_at.isoformat()
|
|
||||||
if last_ok_row and last_ok_row.created_at else None
|
|
||||||
),
|
|
||||||
"used_fallback_key": self._used_fallback,
|
|
||||||
}
|
|
||||||
|
|
||||||
# ---- Internals --------------------------------------------------------
|
|
||||||
|
|
||||||
def _ensure_key(self) -> bytes:
|
|
||||||
"""Derive (or return cached) AES key. Triggers fallback + WARNING log
|
|
||||||
if no passphrase was provided at construction time.
|
|
||||||
|
|
||||||
If a passphrase is set but no salt was passed at construction,
|
|
||||||
look one up from the Keychain (``backup.salt`` account). On
|
|
||||||
a fresh install, generate + persist a salt on first use so
|
|
||||||
subsequent invocations derive the same key.
|
|
||||||
"""
|
|
||||||
if self._key is not None:
|
|
||||||
return self._key
|
|
||||||
if self._passphrase:
|
|
||||||
salt = self._salt
|
|
||||||
if salt is None:
|
|
||||||
# Try the Keychain.
|
|
||||||
stored = secrets_mod.get_secret(KEYCHAIN_BACKUP_SALT_ACCOUNT)
|
|
||||||
if stored:
|
|
||||||
salt = bytes.fromhex(stored.strip())
|
|
||||||
else:
|
|
||||||
# First run: generate + persist.
|
|
||||||
salt = os.urandom(backup_mod.SALT_LEN)
|
|
||||||
secrets_mod.set_secret(
|
|
||||||
KEYCHAIN_BACKUP_SALT_ACCOUNT,
|
|
||||||
salt.hex(),
|
|
||||||
)
|
|
||||||
log.info(
|
|
||||||
"Generated + persisted backup salt to Keychain "
|
|
||||||
"(account %r)",
|
|
||||||
KEYCHAIN_BACKUP_SALT_ACCOUNT,
|
|
||||||
)
|
|
||||||
self._key = backup_mod.derive_key(self._passphrase, salt)
|
|
||||||
return self._key
|
|
||||||
|
|
||||||
# Fallback: derive from SQLCipher DB key. This is degraded
|
|
||||||
# security (the SQLCipher key is meant to unlock the DB, not
|
|
||||||
# the backup), but it's strictly better than plaintext.
|
|
||||||
from cyclone import db_crypto
|
|
||||||
db_key = db_crypto.get_db_key() if db_crypto.is_encryption_enabled() else None
|
|
||||||
if not db_key:
|
|
||||||
# No passphrase AND no SQLCipher key — refuse.
|
|
||||||
raise BackupError(
|
|
||||||
"no backup passphrase set and SQLCipher is not enabled; "
|
|
||||||
"either set a backup passphrase in the Keychain or "
|
|
||||||
"enable SQLCipher encryption",
|
|
||||||
)
|
|
||||||
log.warning(
|
|
||||||
"Backup using fallback key derived from SQLCipher DB key "
|
|
||||||
"(no separate backup passphrase set); set one via "
|
|
||||||
"`cyclone backup init-passphrase` for stronger isolation",
|
|
||||||
extra={"key_source": "sqlcipher_fallback"},
|
|
||||||
)
|
|
||||||
self._used_fallback = True
|
|
||||||
self._key = backup_mod.derive_key(db_key, backup_mod.FALLBACK_SALT)
|
|
||||||
return self._key
|
|
||||||
|
|
||||||
def _sqlite_backup_to(self, target_path: Path) -> None:
|
|
||||||
"""Run SQLite's online ``.backup()`` against the live engine.
|
|
||||||
|
|
||||||
Works for both plain SQLite and SQLCipher because sqlcipher3
|
|
||||||
is API-compatible with sqlite3. The ``.backup()`` API takes
|
|
||||||
a *target* connection; we make a fresh sqlite3 connection to
|
|
||||||
the target file (which doesn't exist yet) and copy into it.
|
|
||||||
"""
|
|
||||||
url = self._db_url or db._resolve_url()
|
|
||||||
if not url.startswith("sqlite"):
|
|
||||||
raise BackupError(
|
|
||||||
f"only sqlite URLs are supported for online backup; got {url!r}",
|
|
||||||
)
|
|
||||||
# Drive the backup off the live engine so we capture the
|
|
||||||
# current state of all tables atomically (SQLite's .backup
|
|
||||||
# holds a read lock on the source for the duration).
|
|
||||||
engine = db.engine() # raises RuntimeError if init_db() wasn't called
|
|
||||||
with engine.raw_connection() as raw:
|
|
||||||
src_conn = raw.driver_connection # sqlite3.Connection / sqlcipher3.Connection
|
|
||||||
if target_path.exists():
|
|
||||||
target_path.unlink()
|
|
||||||
dst_conn = sqlite3.connect(str(target_path))
|
|
||||||
try:
|
|
||||||
src_conn.backup(dst_conn)
|
|
||||||
finally:
|
|
||||||
dst_conn.close()
|
|
||||||
|
|
||||||
def _count_tables_in_blob(self, plaintext: bytes) -> int:
|
|
||||||
"""Open the decrypted DB in-memory and count user tables."""
|
|
||||||
tmp = self._backup_dir / f".count-tables-{_secrets.token_hex(4)}.db"
|
|
||||||
try:
|
|
||||||
tmp.write_bytes(plaintext)
|
|
||||||
conn = sqlite3.connect(str(tmp))
|
|
||||||
try:
|
|
||||||
rows = conn.execute(
|
|
||||||
"SELECT count(*) FROM sqlite_master "
|
|
||||||
"WHERE type='table' AND name NOT LIKE 'sqlite_%'",
|
|
||||||
).fetchone()
|
|
||||||
return int(rows[0])
|
|
||||||
finally:
|
|
||||||
conn.close()
|
|
||||||
finally:
|
|
||||||
tmp.unlink(missing_ok=True)
|
|
||||||
|
|
||||||
def _live_fingerprint_and_count(self) -> tuple[str, int]:
|
|
||||||
"""Fingerprint + table count of the *current* live DB."""
|
|
||||||
try:
|
|
||||||
engine = db.engine()
|
|
||||||
except RuntimeError:
|
|
||||||
return "", 0
|
|
||||||
# Use a temp-file .backup so we don't have to worry about
|
|
||||||
# online-vs-offline semantics.
|
|
||||||
tmp = self._backup_dir / f".live-fp-{_secrets.token_hex(4)}.db"
|
|
||||||
try:
|
|
||||||
with engine.raw_connection() as raw:
|
|
||||||
conn = raw.driver_connection
|
|
||||||
if tmp.exists():
|
|
||||||
tmp.unlink()
|
|
||||||
dst = sqlite3.connect(str(tmp))
|
|
||||||
try:
|
|
||||||
conn.backup(dst)
|
|
||||||
finally:
|
|
||||||
dst.close()
|
|
||||||
data = tmp.read_bytes()
|
|
||||||
return backup_mod.fingerprint(data), self._count_tables_in_blob(data)
|
|
||||||
finally:
|
|
||||||
tmp.unlink(missing_ok=True)
|
|
||||||
|
|
||||||
def _live_db_path(self) -> Optional[Path]:
|
|
||||||
"""Resolve the filesystem path of the live DB, or None for non-sqlite."""
|
|
||||||
url = self._db_url or db._resolve_url()
|
|
||||||
if not url.startswith("sqlite"):
|
|
||||||
return None
|
|
||||||
# Strip the driver prefix: sqlite:///abs or sqlite:///./rel
|
|
||||||
prefix = "sqlite:///"
|
|
||||||
if url.startswith(prefix):
|
|
||||||
return Path(url[len(prefix):])
|
|
||||||
if url.startswith("sqlite://"):
|
|
||||||
# sqlite://./relative/path -> Path("./relative/path")
|
|
||||||
return Path(url[len("sqlite://"):])
|
|
||||||
return None
|
|
||||||
|
|
||||||
def _read_sidecar(self, filename: str) -> Optional[backup_mod.Sidecar]:
|
|
||||||
p = self._backup_dir / backup_mod.sidecar_filename(filename)
|
|
||||||
if not p.exists():
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return backup_mod.Sidecar.from_json(p.read_text())
|
|
||||||
except (json.JSONDecodeError, KeyError, ValueError) as exc:
|
|
||||||
log.warning("Sidecar %s is malformed: %s", p, exc)
|
|
||||||
return None
|
|
||||||
|
|
||||||
def _row_to_record(self, row) -> BackupRecord:
|
|
||||||
"""ORM row → BackupRecord. Reads key_fingerprint from the sidecar if present."""
|
|
||||||
sidecar = self._read_sidecar(row.filename)
|
|
||||||
return BackupRecord(
|
|
||||||
id=row.id,
|
|
||||||
filename=row.filename,
|
|
||||||
backup_dir=row.backup_dir,
|
|
||||||
size_bytes=row.size_bytes or 0,
|
|
||||||
db_fingerprint=row.db_fingerprint or "",
|
|
||||||
table_count=row.table_count or 0,
|
|
||||||
created_at=row.created_at,
|
|
||||||
completed_at=row.completed_at,
|
|
||||||
status=row.status,
|
|
||||||
error_message=row.error_message,
|
|
||||||
key_fingerprint=sidecar.key_fingerprint if sidecar else "",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Module-level singleton
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
_service: Optional[BackupService] = None
|
|
||||||
|
|
||||||
|
|
||||||
def configure_backup_service(
|
|
||||||
backup_dir: Union[str, Path],
|
|
||||||
*,
|
|
||||||
passphrase: Optional[str] = None,
|
|
||||||
salt: Optional[bytes] = None,
|
|
||||||
retention_days: int = 30,
|
|
||||||
db_url: Optional[str] = None,
|
|
||||||
) -> BackupService:
|
|
||||||
"""Create (or replace) the module-level BackupService singleton."""
|
|
||||||
global _service
|
|
||||||
if _service is not None:
|
|
||||||
return _service
|
|
||||||
_service = BackupService(
|
|
||||||
backup_dir=backup_dir,
|
|
||||||
passphrase=passphrase,
|
|
||||||
salt=salt,
|
|
||||||
retention_days=retention_days,
|
|
||||||
db_url=db_url,
|
|
||||||
)
|
|
||||||
return _service
|
|
||||||
|
|
||||||
|
|
||||||
def get_backup_service() -> BackupService:
|
|
||||||
"""Return the configured BackupService. Raises RuntimeError if not set up."""
|
|
||||||
if _service is None:
|
|
||||||
raise RuntimeError("backup service not configured; call configure_backup_service() first")
|
|
||||||
return _service
|
|
||||||
|
|
||||||
|
|
||||||
def reset_backup_service_for_tests() -> None:
|
|
||||||
"""Clear the module-level singleton. Test-only."""
|
|
||||||
global _service
|
|
||||||
_service = None
|
|
||||||
@@ -93,24 +93,13 @@ class SftpClient:
|
|||||||
return self._list_inbound_paramiko()
|
return self._list_inbound_paramiko()
|
||||||
|
|
||||||
def read_file(self, remote_path: str) -> bytes:
|
def read_file(self, remote_path: str) -> bytes:
|
||||||
"""Read bytes from a remote path.
|
"""Read bytes from a remote path. Stub raises in stub mode."""
|
||||||
|
|
||||||
Stub mode: reads from ``{staging_dir}/{remote_path}``. Used by
|
|
||||||
the SP16 scheduler so it can exercise the same code path on a
|
|
||||||
workstation without a real MFT connection.
|
|
||||||
"""
|
|
||||||
if self._stub:
|
if self._stub:
|
||||||
return self._read_file_stub(remote_path)
|
raise RuntimeError(
|
||||||
|
"Stub SFTP cannot read remote files. Use the local staging dir."
|
||||||
|
)
|
||||||
return self._read_file_paramiko(remote_path)
|
return self._read_file_paramiko(remote_path)
|
||||||
|
|
||||||
def _read_file_stub(self, remote_path: str) -> bytes:
|
|
||||||
"""Read bytes from ``{staging_dir}/{remote_path}`` (SP16 stub)."""
|
|
||||||
staging = Path(self._block.staging_dir).resolve()
|
|
||||||
target = staging / remote_path.lstrip("/")
|
|
||||||
if not target.is_file():
|
|
||||||
raise FileNotFoundError(f"inbound stub file not found: {target}")
|
|
||||||
return target.read_bytes()
|
|
||||||
|
|
||||||
def get_secret(self, name: str) -> Optional[str]:
|
def get_secret(self, name: str) -> Optional[str]:
|
||||||
"""Fetch the auth secret from Keychain. Returns the stub secret if absent."""
|
"""Fetch the auth secret from Keychain. Returns the stub secret if absent."""
|
||||||
value = secrets.get_secret(name)
|
value = secrets.get_secret(name)
|
||||||
|
|||||||
+3
-375
@@ -3,13 +3,11 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
import os
|
|
||||||
import sys
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import click
|
import click
|
||||||
|
|
||||||
from cyclone.logging_config import setup_logging
|
|
||||||
from cyclone.parsers.exceptions import CycloneParseError, CycloneValidationError
|
from cyclone.parsers.exceptions import CycloneParseError, CycloneValidationError
|
||||||
from cyclone.parsers.payer import PayerConfig, PayerConfig835
|
from cyclone.parsers.payer import PayerConfig, PayerConfig835
|
||||||
from cyclone.parsers.parse_837 import parse as parse_837_text
|
from cyclone.parsers.parse_837 import parse as parse_837_text
|
||||||
@@ -43,47 +41,8 @@ def _payer_835(name: str) -> PayerConfig835:
|
|||||||
|
|
||||||
|
|
||||||
@click.group()
|
@click.group()
|
||||||
@click.option(
|
def main() -> None:
|
||||||
"--log-format",
|
|
||||||
default=None,
|
|
||||||
type=click.Choice(["json", "dev"]),
|
|
||||||
help="Log format (default: json; honors CYCLONE_LOG_JSON).",
|
|
||||||
)
|
|
||||||
@click.option(
|
|
||||||
"--log-file",
|
|
||||||
default=None,
|
|
||||||
type=click.Path(dir_okay=False, path_type=Path),
|
|
||||||
help="Optional rotating log file (honors CYCLONE_LOG_FILE).",
|
|
||||||
)
|
|
||||||
@click.pass_context
|
|
||||||
def main(ctx: click.Context, log_format: str | None, log_file: Path | None) -> None:
|
|
||||||
"""Cyclone EDI suite — X12 parser."""
|
"""Cyclone EDI suite — X12 parser."""
|
||||||
# SP18: structured JSON logging. Run once per CLI invocation; each
|
|
||||||
# subcommand still gets its own --log-level to override.
|
|
||||||
json_format = True
|
|
||||||
if log_format == "dev":
|
|
||||||
json_format = False
|
|
||||||
elif os.environ.get("CYCLONE_LOG_JSON", "").lower() in ("false", "0", "no"):
|
|
||||||
json_format = False
|
|
||||||
setup_logging(
|
|
||||||
level=os.environ.get("CYCLONE_LOG_LEVEL", "INFO"),
|
|
||||||
log_file=str(log_file) if log_file else None,
|
|
||||||
json_format=json_format,
|
|
||||||
)
|
|
||||||
# Stash on context so subcommands can read it.
|
|
||||||
ctx.ensure_object(dict)
|
|
||||||
|
|
||||||
|
|
||||||
# Register the auth users subgroup. Imported here (not at module top) to
|
|
||||||
# avoid pulling passlib / bcrypt at CLI parse-only import time.
|
|
||||||
from cyclone.auth.cli import users_cli # noqa: E402
|
|
||||||
main.add_command(users_cli)
|
|
||||||
|
|
||||||
# Register the dev seed subcommand. Imported here for the same lazy-load
|
|
||||||
# reason as users_cli — keeps passlib/bcrypt + SQLAlchemy out of the
|
|
||||||
# parse-only path so ``python -m cyclone --help`` stays snappy.
|
|
||||||
from cyclone.seed_cli import seed_cli # noqa: E402
|
|
||||||
main.add_command(seed_cli)
|
|
||||||
|
|
||||||
|
|
||||||
@main.command("parse-837")
|
@main.command("parse-837")
|
||||||
@@ -104,9 +63,7 @@ def parse_837(
|
|||||||
log_level: str,
|
log_level: str,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Parse an X12 837P file into one JSON per claim."""
|
"""Parse an X12 837P file into one JSON per claim."""
|
||||||
# SP18: re-run setup so per-command --log-level overrides the
|
logging.basicConfig(level=getattr(logging, log_level))
|
||||||
# group default. ``setup_logging`` is idempotent.
|
|
||||||
setup_logging(level=log_level)
|
|
||||||
|
|
||||||
text = input_file.read_text()
|
text = input_file.read_text()
|
||||||
config = _payer(payer)
|
config = _payer(payer)
|
||||||
@@ -176,9 +133,7 @@ def parse_835(
|
|||||||
log_level: str,
|
log_level: str,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Parse an X12 835 ERA file into one JSON per claim payment."""
|
"""Parse an X12 835 ERA file into one JSON per claim payment."""
|
||||||
# SP18: re-run setup so per-command --log-level overrides the
|
logging.basicConfig(level=getattr(logging, log_level))
|
||||||
# group default. ``setup_logging`` is idempotent.
|
|
||||||
setup_logging(level=log_level)
|
|
||||||
|
|
||||||
text = input_file.read_text()
|
text = input_file.read_text()
|
||||||
config = _payer_835(payer)
|
config = _payer_835(payer)
|
||||||
@@ -240,332 +195,5 @@ def _count_issues(report) -> dict[str, int]:
|
|||||||
return counts
|
return counts
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# SP20: `cyclone validate-npi` + `cyclone validate-tax-id`
|
|
||||||
#
|
|
||||||
# Pure local validators. No DB, no Keychain, no network — operators can
|
|
||||||
# run them on a developer laptop without standing up the full Cyclone
|
|
||||||
# stack. Exit code is 0 (valid) or 1 (invalid) so they compose with
|
|
||||||
# shell scripting / CI gates.
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@main.command("validate-npi")
|
|
||||||
@click.argument("npi")
|
|
||||||
@click.option("--log-level", default="WARNING", show_default=True, type=click.Choice(["DEBUG", "INFO", "WARNING", "ERROR"]))
|
|
||||||
def validate_npi_cmd(npi: str, log_level: str) -> None:
|
|
||||||
"""Validate a 10-digit NPI's Luhn checksum locally (SP20).
|
|
||||||
|
|
||||||
Exit 0 if valid, 1 if not. No logging of the value itself — NPIs
|
|
||||||
are PHI under HIPAA, so the operator's CLI history is the only
|
|
||||||
audit trail.
|
|
||||||
"""
|
|
||||||
# SP18: re-run so --log-level overrides the group default.
|
|
||||||
setup_logging(level=log_level)
|
|
||||||
from cyclone.npi import is_valid_npi
|
|
||||||
if is_valid_npi(npi):
|
|
||||||
click.echo(f"OK: {len(npi)}-digit NPI passes Luhn checksum")
|
|
||||||
return
|
|
||||||
click.echo(f"INVALID: {npi!r} fails NPI Luhn checksum", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
|
|
||||||
@main.command("validate-tax-id")
|
|
||||||
@click.argument("tax_id")
|
|
||||||
@click.option("--log-level", default="WARNING", show_default=True, type=click.Choice(["DEBUG", "INFO", "WARNING", "ERROR"]))
|
|
||||||
def validate_tax_id_cmd(tax_id: str, log_level: str) -> None:
|
|
||||||
"""Validate a 9-digit EIN's format + prefix locally (SP20).
|
|
||||||
|
|
||||||
Accepts both ``XX-XXXXXXX`` and ``XXXXXXXXX``. Exit 0 if valid,
|
|
||||||
1 if not. EIN is sensitive (PII), so we don't echo the value back
|
|
||||||
on failure — only the validation verdict.
|
|
||||||
"""
|
|
||||||
# SP18: re-run so --log-level overrides the group default.
|
|
||||||
setup_logging(level=log_level)
|
|
||||||
from cyclone.npi import is_valid_tax_id, normalize_tax_id
|
|
||||||
plain = normalize_tax_id(tax_id)
|
|
||||||
if plain is None:
|
|
||||||
click.echo("INVALID: input is not a 9-digit EIN (XX-XXXXXXX or XXXXXXXXX)", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
if is_valid_tax_id(tax_id):
|
|
||||||
click.echo(f"OK: 9-digit EIN (normalized={plain})")
|
|
||||||
return
|
|
||||||
click.echo(
|
|
||||||
f"INVALID: 9-digit EIN has reserved prefix ({plain[:2]}); EIN is not assignable by IRS",
|
|
||||||
err=True,
|
|
||||||
)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
main()
|
main()
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# SP17: `cyclone backup` subcommands
|
|
||||||
#
|
|
||||||
# Operator-facing backup management. Mirrors the API surface but runs
|
|
||||||
# standalone (no FastAPI app needed) for cron / scripting / DR drills.
|
|
||||||
# Each subcommand initializes the DB + BackupService; if the
|
|
||||||
# service isn't configured (no Keychain passphrase etc.) the operator
|
|
||||||
# gets a clear error.
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@main.group()
|
|
||||||
def backup() -> None:
|
|
||||||
"""Encrypted DB backup management (SP17)."""
|
|
||||||
|
|
||||||
|
|
||||||
@backup.command("init-passphrase")
|
|
||||||
@click.option("--passphrase", required=True, help="The passphrase to set (will prompt if omitted)")
|
|
||||||
@click.option("--from-stdin", is_flag=True, help="Read passphrase from stdin instead of the argument")
|
|
||||||
def backup_init_passphrase(passphrase: str, from_stdin: bool) -> None:
|
|
||||||
"""Set the backup encryption passphrase in the macOS Keychain.
|
|
||||||
|
|
||||||
Generates a fresh salt and stores both the passphrase (account
|
|
||||||
``backup.passphrase``) and the salt (account ``backup.salt``)
|
|
||||||
under service ``cyclone``. Cyclone's BackupService reads them
|
|
||||||
at startup. If the passphrase account is missing, the service
|
|
||||||
falls back to deriving a key from the SQLCipher DB key
|
|
||||||
(degraded posture, logged at WARNING).
|
|
||||||
"""
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
from cyclone import secrets as secrets_mod
|
|
||||||
from getpass import getpass
|
|
||||||
import os as _os
|
|
||||||
|
|
||||||
if from_stdin:
|
|
||||||
pp = getpass("Backup passphrase: ").strip()
|
|
||||||
pp2 = getpass("Confirm: ").strip()
|
|
||||||
if not pp or pp != pp2:
|
|
||||||
click.echo("passphrase empty or mismatch", err=True)
|
|
||||||
sys.exit(2)
|
|
||||||
else:
|
|
||||||
pp = passphrase
|
|
||||||
|
|
||||||
if not pp or len(pp) < 12:
|
|
||||||
click.echo("passphrase must be at least 12 characters", err=True)
|
|
||||||
sys.exit(2)
|
|
||||||
|
|
||||||
if not secrets_mod.set_secret(svc_mod.KEYCHAIN_BACKUP_PASSPHRASE_ACCOUNT, pp):
|
|
||||||
click.echo("failed to store passphrase in Keychain", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
# Generate + persist a fresh salt. Same value must be used by
|
|
||||||
# every subsequent invocation that uses this passphrase.
|
|
||||||
salt = _os.urandom(16)
|
|
||||||
if not secrets_mod.set_secret(svc_mod.KEYCHAIN_BACKUP_SALT_ACCOUNT, salt.hex()):
|
|
||||||
click.echo(
|
|
||||||
"WARN: passphrase stored but salt write failed; backups may be unrecoverable",
|
|
||||||
err=True,
|
|
||||||
)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
click.echo(
|
|
||||||
f"passphrase stored in Keychain account {svc_mod.KEYCHAIN_BACKUP_PASSPHRASE_ACCOUNT!r}\n"
|
|
||||||
f"salt stored in Keychain account {svc_mod.KEYCHAIN_BACKUP_SALT_ACCOUNT!r}"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _resolve_backup_dir(cli_override: str | None) -> "Path":
|
|
||||||
"""Resolve the backup directory: --backup-dir > $CYCLONE_BACKUP_DIR > default."""
|
|
||||||
import os as _os
|
|
||||||
from pathlib import Path as _Path
|
|
||||||
from cyclone import db as db_mod
|
|
||||||
if cli_override:
|
|
||||||
return _Path(cli_override)
|
|
||||||
env = _os.environ.get("CYCLONE_BACKUP_DIR")
|
|
||||||
if env:
|
|
||||||
return _Path(env)
|
|
||||||
return _Path(db_mod.DEFAULT_DB_PATH.parent / "backups")
|
|
||||||
|
|
||||||
|
|
||||||
@backup.command("create")
|
|
||||||
@click.option("--backup-dir", default=None, help="Override CYCLONE_BACKUP_DIR (default: ~/.local/share/cyclone/backups)")
|
|
||||||
@click.option("--retention-days", default=None, type=int, help="Override CYCLONE_BACKUP_RETENTION_DAYS for this run's prune")
|
|
||||||
def backup_create(backup_dir: str | None, retention_days: int | None) -> None:
|
|
||||||
"""Take an encrypted backup right now."""
|
|
||||||
from cyclone import db as db_mod
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
from cyclone import secrets as secrets_mod
|
|
||||||
|
|
||||||
db_mod.init_db()
|
|
||||||
|
|
||||||
passphrase = secrets_mod.get_secret(svc_mod.KEYCHAIN_BACKUP_PASSPHRASE_ACCOUNT)
|
|
||||||
salt_hex = secrets_mod.get_secret(svc_mod.KEYCHAIN_BACKUP_SALT_ACCOUNT)
|
|
||||||
salt = bytes.fromhex(salt_hex) if salt_hex else None
|
|
||||||
target_dir = _resolve_backup_dir(backup_dir)
|
|
||||||
svc_mod.reset_backup_service_for_tests()
|
|
||||||
svc = svc_mod.configure_backup_service(
|
|
||||||
backup_dir=target_dir,
|
|
||||||
passphrase=passphrase,
|
|
||||||
salt=salt,
|
|
||||||
retention_days=retention_days or 30,
|
|
||||||
)
|
|
||||||
result = svc.create_now()
|
|
||||||
click.echo(
|
|
||||||
f"created backup id={result.backup.id} filename={result.backup.filename} "
|
|
||||||
f"size={result.backup.size_bytes}B fp={result.backup.db_fingerprint[:24]}..."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@backup.command("list")
|
|
||||||
@click.option("--limit", default=50, show_default=True)
|
|
||||||
@click.option("--status", default=None, help="Filter: ok|error|pending|pruned")
|
|
||||||
def backup_list(limit: int, status: str | None) -> None:
|
|
||||||
"""List existing backups (newest first)."""
|
|
||||||
from cyclone import db as db_mod
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
from cyclone import secrets as secrets_mod
|
|
||||||
|
|
||||||
db_mod.init_db()
|
|
||||||
passphrase = secrets_mod.get_secret(svc_mod.KEYCHAIN_BACKUP_PASSPHRASE_ACCOUNT)
|
|
||||||
salt_hex = secrets_mod.get_secret(svc_mod.KEYCHAIN_BACKUP_SALT_ACCOUNT)
|
|
||||||
salt = bytes.fromhex(salt_hex) if salt_hex else None
|
|
||||||
svc_mod.reset_backup_service_for_tests()
|
|
||||||
svc = svc_mod.configure_backup_service(
|
|
||||||
backup_dir=_resolve_backup_dir(None),
|
|
||||||
passphrase=passphrase,
|
|
||||||
salt=salt,
|
|
||||||
)
|
|
||||||
rows = svc.list_backups(limit=limit, status=status)
|
|
||||||
if not rows:
|
|
||||||
click.echo("(no backups)")
|
|
||||||
return
|
|
||||||
for r in rows:
|
|
||||||
click.echo(
|
|
||||||
f"{r.id:4d} {r.status:7s} {r.created_at.isoformat() if r.created_at else '-'} "
|
|
||||||
f"{r.size_bytes:>10d}B {r.filename} fp={r.db_fingerprint[:24] or '-':<24}"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@backup.command("verify")
|
|
||||||
@click.argument("backup_id", type=int)
|
|
||||||
def backup_verify(backup_id: int) -> None:
|
|
||||||
"""Decrypt + checksum-verify a backup."""
|
|
||||||
from cyclone import db as db_mod
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
from cyclone import secrets as secrets_mod
|
|
||||||
|
|
||||||
db_mod.init_db()
|
|
||||||
passphrase = secrets_mod.get_secret(svc_mod.KEYCHAIN_BACKUP_PASSPHRASE_ACCOUNT)
|
|
||||||
salt_hex = secrets_mod.get_secret(svc_mod.KEYCHAIN_BACKUP_SALT_ACCOUNT)
|
|
||||||
salt = bytes.fromhex(salt_hex) if salt_hex else None
|
|
||||||
svc_mod.reset_backup_service_for_tests()
|
|
||||||
svc = svc_mod.configure_backup_service(
|
|
||||||
backup_dir=_resolve_backup_dir(None),
|
|
||||||
passphrase=passphrase,
|
|
||||||
salt=salt,
|
|
||||||
)
|
|
||||||
v = svc.verify(backup_id)
|
|
||||||
if v.ok:
|
|
||||||
click.echo(f"OK: id={v.backup_id} fp={v.actual_fingerprint[:24]}... table_count={v.table_count}")
|
|
||||||
return
|
|
||||||
click.echo(
|
|
||||||
f"FAIL: id={v.backup_id} reason={v.reason} "
|
|
||||||
f"expected={v.expected_fingerprint[:24] if v.expected_fingerprint else '-'}... "
|
|
||||||
f"actual={v.actual_fingerprint[:24] if v.actual_fingerprint else '-'}...",
|
|
||||||
err=True,
|
|
||||||
)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
|
|
||||||
@backup.command("restore")
|
|
||||||
@click.argument("backup_id", type=int)
|
|
||||||
@click.option("--yes", is_flag=True, help="Skip the interactive confirm prompt")
|
|
||||||
@click.option("--actor", default="operator-cli", show_default=True)
|
|
||||||
def backup_restore(backup_id: int, yes: bool, actor: str) -> None:
|
|
||||||
"""Restore the live DB from a backup (two-step, requires --yes)."""
|
|
||||||
from cyclone import db as db_mod
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
from cyclone import secrets as secrets_mod
|
|
||||||
|
|
||||||
db_mod.init_db()
|
|
||||||
passphrase = secrets_mod.get_secret(svc_mod.KEYCHAIN_BACKUP_PASSPHRASE_ACCOUNT)
|
|
||||||
salt_hex = secrets_mod.get_secret(svc_mod.KEYCHAIN_BACKUP_SALT_ACCOUNT)
|
|
||||||
salt = bytes.fromhex(salt_hex) if salt_hex else None
|
|
||||||
svc_mod.reset_backup_service_for_tests()
|
|
||||||
svc = svc_mod.configure_backup_service(
|
|
||||||
backup_dir=_resolve_backup_dir(None),
|
|
||||||
passphrase=passphrase,
|
|
||||||
salt=salt,
|
|
||||||
)
|
|
||||||
|
|
||||||
click.echo(f"Initiating restore from backup {backup_id}...")
|
|
||||||
init = svc.restore_initiate(backup_id)
|
|
||||||
click.echo(
|
|
||||||
f" backup: {init.filename} ({init.size_bytes} bytes)\n"
|
|
||||||
f" fp: {init.db_fingerprint[:24]}...\n"
|
|
||||||
f" tables: {init.table_count}\n"
|
|
||||||
f" current: fp={init.current_db_fingerprint[:24] if init.current_db_fingerprint else '-'}... "
|
|
||||||
f"tables={init.current_table_count}\n"
|
|
||||||
f" token ttl: {(init.expires_at - __import__('datetime').datetime.now(__import__('datetime').timezone.utc)).total_seconds():.0f}s"
|
|
||||||
)
|
|
||||||
|
|
||||||
if not yes:
|
|
||||||
click.confirm(
|
|
||||||
"Replace the live DB with this backup? "
|
|
||||||
"This will dispose the engine and rebuild it.",
|
|
||||||
abort=True,
|
|
||||||
)
|
|
||||||
|
|
||||||
click.echo("Confirming restore...")
|
|
||||||
result = svc.restore_confirm(backup_id, init.restore_token, actor=actor)
|
|
||||||
click.echo(
|
|
||||||
f"OK: restored from fp={result.restored_from_fingerprint[:24]}... "
|
|
||||||
f"to fp={result.new_db_fingerprint[:24]}... at {result.restored_at.isoformat()}"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@backup.command("prune")
|
|
||||||
@click.option("--retention-days", default=None, type=int)
|
|
||||||
@click.option("--yes", is_flag=True, help="Skip the confirm prompt")
|
|
||||||
def backup_prune(retention_days: int | None, yes: bool) -> None:
|
|
||||||
"""Apply the retention policy (delete old backups)."""
|
|
||||||
from cyclone import db as db_mod
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
from cyclone import secrets as secrets_mod
|
|
||||||
|
|
||||||
db_mod.init_db()
|
|
||||||
passphrase = secrets_mod.get_secret(svc_mod.KEYCHAIN_BACKUP_PASSPHRASE_ACCOUNT)
|
|
||||||
salt_hex = secrets_mod.get_secret(svc_mod.KEYCHAIN_BACKUP_SALT_ACCOUNT)
|
|
||||||
salt = bytes.fromhex(salt_hex) if salt_hex else None
|
|
||||||
svc_mod.reset_backup_service_for_tests()
|
|
||||||
svc = svc_mod.configure_backup_service(
|
|
||||||
backup_dir=_resolve_backup_dir(None),
|
|
||||||
passphrase=passphrase,
|
|
||||||
salt=salt,
|
|
||||||
retention_days=retention_days or 30,
|
|
||||||
)
|
|
||||||
if not yes:
|
|
||||||
click.confirm(
|
|
||||||
f"Delete all backups older than {svc._retention_days} days?",
|
|
||||||
abort=True,
|
|
||||||
)
|
|
||||||
deleted = svc.prune()
|
|
||||||
click.echo(f"Deleted {len(deleted)} file(s):")
|
|
||||||
for p in deleted:
|
|
||||||
click.echo(f" {p}")
|
|
||||||
|
|
||||||
|
|
||||||
@backup.command("status")
|
|
||||||
def backup_status() -> None:
|
|
||||||
"""Print the backup subsystem status snapshot."""
|
|
||||||
from cyclone import db as db_mod
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
from cyclone import secrets as secrets_mod
|
|
||||||
|
|
||||||
db_mod.init_db()
|
|
||||||
passphrase = secrets_mod.get_secret(svc_mod.KEYCHAIN_BACKUP_PASSPHRASE_ACCOUNT)
|
|
||||||
salt_hex = secrets_mod.get_secret(svc_mod.KEYCHAIN_BACKUP_SALT_ACCOUNT)
|
|
||||||
salt = bytes.fromhex(salt_hex) if salt_hex else None
|
|
||||||
svc_mod.reset_backup_service_for_tests()
|
|
||||||
svc = svc_mod.configure_backup_service(
|
|
||||||
backup_dir=_resolve_backup_dir(None),
|
|
||||||
passphrase=passphrase,
|
|
||||||
salt=salt,
|
|
||||||
)
|
|
||||||
snap = svc.status()
|
|
||||||
import json
|
|
||||||
click.echo(json.dumps(snap, indent=2, default=str))
|
|
||||||
|
|||||||
@@ -30,7 +30,6 @@ from sqlalchemy import (
|
|||||||
Numeric,
|
Numeric,
|
||||||
String,
|
String,
|
||||||
Text,
|
Text,
|
||||||
func,
|
|
||||||
text,
|
text,
|
||||||
)
|
)
|
||||||
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship, sessionmaker
|
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship, sessionmaker
|
||||||
@@ -669,11 +668,6 @@ class AuditLog(Base):
|
|||||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
||||||
prev_hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
prev_hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||||
hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||||
# SP-auth: which authenticated user performed this action. Nullable
|
|
||||||
# so existing (pre-auth) rows and system-initiated events stay valid.
|
|
||||||
# NOT part of the hash chain — verify_chain must continue to work on
|
|
||||||
# legacy rows that pre-date this column.
|
|
||||||
user_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True, index=True)
|
|
||||||
|
|
||||||
__table_args__ = (
|
__table_args__ = (
|
||||||
Index("idx_audit_log_entity", "entity_type", "entity_id"),
|
Index("idx_audit_log_entity", "entity_type", "entity_id"),
|
||||||
@@ -682,89 +676,6 @@ class AuditLog(Base):
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# SP16: inbound MFT scheduler
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
class ProcessedInboundFile(Base):
|
|
||||||
"""One row per inbound MFT file the scheduler has downloaded.
|
|
||||||
|
|
||||||
SP16. Lets the scheduler be idempotent: a re-tick or restart must
|
|
||||||
not re-parse the same inbound file. The unique index on
|
|
||||||
(sftp_block_name, name) prevents duplicate inserts and lets the
|
|
||||||
scheduler fast-skip already-processed files via a SELECT.
|
|
||||||
|
|
||||||
Status values:
|
|
||||||
* ok - parsed cleanly, results persisted to the store
|
|
||||||
* error - parser raised; error_message captured
|
|
||||||
* skipped - file_type not in the scheduler's allowed set
|
|
||||||
* pending - file was downloaded but a downstream step failed;
|
|
||||||
the scheduler retries on the next tick
|
|
||||||
"""
|
|
||||||
|
|
||||||
__tablename__ = "processed_inbound_files"
|
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
|
||||||
sftp_block_name: Mapped[str] = mapped_column(String(128), nullable=False)
|
|
||||||
name: Mapped[str] = mapped_column(String(256), nullable=False)
|
|
||||||
size: Mapped[int] = mapped_column(Integer, nullable=False)
|
|
||||||
modified_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
|
||||||
file_type: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
|
|
||||||
processed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
|
||||||
parser_used: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
|
|
||||||
claim_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
||||||
status: Mapped[str] = mapped_column(String(16), nullable=False)
|
|
||||||
error_message: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
|
|
||||||
|
|
||||||
__table_args__ = (
|
|
||||||
Index(
|
|
||||||
"ux_processed_inbound_files_block_name",
|
|
||||||
"sftp_block_name", "name", unique=True,
|
|
||||||
),
|
|
||||||
Index("ix_processed_inbound_files_processed_at", "processed_at"),
|
|
||||||
Index("ix_processed_inbound_files_status", "status"),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# SP17: encrypted backup metadata
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
class DbBackup(Base):
|
|
||||||
"""One row per encrypted backup the BackupService has taken.
|
|
||||||
|
|
||||||
The actual encrypted blob lives in a directory outside the DB
|
|
||||||
(``~/.local/share/cyclone/backups/`` by default); this table is
|
|
||||||
the index. Status values: ``pending``, ``ok``, ``error``,
|
|
||||||
``pruned``.
|
|
||||||
|
|
||||||
SP17. The unique index on ``(backup_dir, filename)`` makes a
|
|
||||||
duplicate ``create_now()`` race fail cleanly with an
|
|
||||||
IntegrityError instead of clobbering an existing backup.
|
|
||||||
"""
|
|
||||||
|
|
||||||
__tablename__ = "db_backups"
|
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
|
|
||||||
filename: Mapped[str] = mapped_column(String(128), nullable=False)
|
|
||||||
backup_dir: Mapped[str] = mapped_column(String(512), nullable=False)
|
|
||||||
size_bytes: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
||||||
db_fingerprint: Mapped[Optional[str]] = mapped_column(String(80), nullable=True)
|
|
||||||
table_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
|
|
||||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
|
|
||||||
completed_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True)
|
|
||||||
status: Mapped[str] = mapped_column(String(16), nullable=False)
|
|
||||||
error_message: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
|
|
||||||
|
|
||||||
__table_args__ = (
|
|
||||||
Index("ux_db_backups_filename", "backup_dir", "filename", unique=True),
|
|
||||||
Index("ix_db_backups_created_at", "created_at"),
|
|
||||||
Index("ix_db_backups_status", "status"),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# SP9: providers, payers, payer_configs, clearhouse
|
# SP9: providers, payers, payer_configs, clearhouse
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -842,27 +753,3 @@ class ClearhouseORM(Base):
|
|||||||
filename_block_json: Mapped[dict] = mapped_column(JSONText, nullable=False)
|
filename_block_json: Mapped[dict] = mapped_column(JSONText, nullable=False)
|
||||||
sftp_block_json: Mapped[dict] = mapped_column(JSONText, nullable=False)
|
sftp_block_json: Mapped[dict] = mapped_column(JSONText, nullable=False)
|
||||||
updated_at: Mapped[str] = mapped_column(String(32), nullable=False)
|
updated_at: Mapped[str] = mapped_column(String(32), nullable=False)
|
||||||
|
|
||||||
|
|
||||||
class User(Base):
|
|
||||||
"""Auth user (admin / user / viewer)."""
|
|
||||||
|
|
||||||
__tablename__ = "users"
|
|
||||||
|
|
||||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
|
||||||
username: Mapped[str] = mapped_column(String(64), unique=True, index=True)
|
|
||||||
password_hash: Mapped[str] = mapped_column(String(255), nullable=False)
|
|
||||||
role: Mapped[str] = mapped_column(String(16), nullable=False)
|
|
||||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
|
||||||
disabled_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
|
||||||
|
|
||||||
|
|
||||||
class Session(Base):
|
|
||||||
"""Server-side auth session (HttpOnly cookie holds the id)."""
|
|
||||||
|
|
||||||
__tablename__ = "sessions"
|
|
||||||
|
|
||||||
id: Mapped[str] = mapped_column(String(64), primary_key=True)
|
|
||||||
user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), index=True)
|
|
||||||
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True)
|
|
||||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
|
||||||
|
|||||||
@@ -4,8 +4,8 @@ SP9. Source-of-truth spec:
|
|||||||
https://hcpf.colorado.gov/tp-x12-filenaming (HCPF X12 File Naming Standards Quick Guide)
|
https://hcpf.colorado.gov/tp-x12-filenaming (HCPF X12 File Naming Standards Quick Guide)
|
||||||
|
|
||||||
Outbound (we send):
|
Outbound (we send):
|
||||||
tp{tpid}-{transaction_type}-{yyyymmddhhmmssSSS_MT}-1of1.{ext}
|
{tpid}-{transaction_type}-{yyyymmddhhmmssSSS_MT}-1of1.{ext}
|
||||||
Example: tp11525703-837P-20260620132243505-1of1.x12
|
Example: 11525703-837P-20260620132243505-1of1.x12
|
||||||
|
|
||||||
Inbound (HPE sends to our ToHPE):
|
Inbound (HPE sends to our ToHPE):
|
||||||
TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
|
TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
|
||||||
@@ -28,15 +28,14 @@ from cyclone.providers import InboundFilename
|
|||||||
# Regexes
|
# Regexes
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
# Outbound: tp11525703-837P-20260620132243505-1of1.x12
|
# Outbound: 11525703-837P-20260620132243505-1of1.x12
|
||||||
# - tp: literal "tp" prefix
|
|
||||||
# - tpid: 1+ digits
|
# - tpid: 1+ digits
|
||||||
# - tx: 1+ alnum
|
# - tx: 1+ alnum
|
||||||
# - ts: 17 digits (yyyymmddhhmmssSSS)
|
# - ts: 17 digits (yyyymmddhhmmssSSS)
|
||||||
# - seq: literal "1of1"
|
# - seq: literal "1of1"
|
||||||
# - ext: 1+ alnum
|
# - ext: 1+ alnum
|
||||||
OUTBOUND_RE = re.compile(
|
OUTBOUND_RE = re.compile(
|
||||||
r"^tp(?P<tpid>\d+)-(?P<tx>[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>[A-Za-z0-9]+)$"
|
r"^(?P<tpid>\d+)-(?P<tx>[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>[A-Za-z0-9]+)$"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Inbound: TP11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
# Inbound: TP11525703-837P_M019048402-20260520231513488-1of1_999.x12
|
||||||
@@ -80,8 +79,7 @@ def build_outbound_filename(
|
|||||||
time in ``America/Denver`` is used.
|
time in ``America/Denver`` is used.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Filename like "tp11525703-837P-20260620132243505-1of1.x12"
|
Filename like "11525703-837P-20260620132243505-1of1.x12"
|
||||||
(note the ``tp`` prefix per HCPF outbound spec).
|
|
||||||
|
|
||||||
Raises:
|
Raises:
|
||||||
ValueError: If tpid is non-numeric, tx contains invalid chars, or
|
ValueError: If tpid is non-numeric, tx contains invalid chars, or
|
||||||
@@ -101,10 +99,7 @@ def build_outbound_filename(
|
|||||||
# Format: yyyymmddhhmmssSSS — 17 digits total
|
# Format: yyyymmddhhmmssSSS — 17 digits total
|
||||||
ts = now_mt.strftime("%Y%m%d%H%M%S") + f"{now_mt.microsecond // 1000:03d}"
|
ts = now_mt.strftime("%Y%m%d%H%M%S") + f"{now_mt.microsecond // 1000:03d}"
|
||||||
assert len(ts) == 17
|
assert len(ts) == 17
|
||||||
# Per HCPF outbound spec, prefix is "tp" + tpid. Matches the format
|
return f"{tpid}-{tx}-{ts}-1of1.{ext}"
|
||||||
# we receive from HPE inbound (which uses uppercase TP) and the
|
|
||||||
# historical outbound prodfile naming (e.g. tp11525703-837P-...).
|
|
||||||
return f"tp{tpid}-{tx}-{ts}-1of1.{ext}"
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|||||||
@@ -1,379 +0,0 @@
|
|||||||
"""SP18 — Structured JSON logging.
|
|
||||||
|
|
||||||
Wraps Python's stdlib ``logging`` to emit newline-delimited JSON
|
|
||||||
(or a dev-friendly tabular format) and to scrub obvious PHI
|
|
||||||
patterns (NPIs, SSNs, DOBs, patient names) from the message +
|
|
||||||
extra fields.
|
|
||||||
|
|
||||||
Design choices
|
|
||||||
--------------
|
|
||||||
|
|
||||||
* **No third-party deps.** stdlib ``logging`` + ``json`` + ``re``
|
|
||||||
is enough. ``loguru`` / ``structlog`` were considered; both add
|
|
||||||
a dependency for marginal gain.
|
|
||||||
|
|
||||||
* **JSON by default.** Operators running Cyclone in production
|
|
||||||
almost certainly want logs in a format their aggregator
|
|
||||||
(Loki/ELK/Vector) can parse. The dev format (``CycloneDevFormatter``)
|
|
||||||
is the opt-out for ``tail -f`` in dev.
|
|
||||||
|
|
||||||
* **Conservative PII scrubber.** Redacts unambiguous PHI patterns
|
|
||||||
only. False positives are not free — an operator's diagnostic
|
|
||||||
dump that says ``<redacted:npi>`` instead of the actual NPI
|
|
||||||
makes root-causing a parse failure harder. The scrubber can be
|
|
||||||
disabled with ``CYCLONE_LOG_NO_PII_SCRUB=1`` for tests /
|
|
||||||
forensic mode.
|
|
||||||
|
|
||||||
* **Idempotent setup.** :func:`setup_logging` can be called
|
|
||||||
multiple times (CLI re-invocation, FastAPI lifespan re-entry
|
|
||||||
under TestClient). Each call clears existing handlers on the
|
|
||||||
root logger before attaching fresh ones — so the format toggle
|
|
||||||
actually takes effect on the second call.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from logging.handlers import RotatingFileHandler
|
|
||||||
from typing import Any, Optional
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Formatters
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
# Stdlib LogRecord attributes we don't want to dump into the
|
|
||||||
# structured payload (they're noise for log consumers).
|
|
||||||
_RESERVED_LOGRECORD_ATTRS = frozenset({
|
|
||||||
"args", "asctime", "created", "exc_info", "exc_text", "filename",
|
|
||||||
"funcName", "levelname", "levelno", "lineno", "module", "msecs",
|
|
||||||
"message", "msg", "name", "pathname", "process", "processName",
|
|
||||||
"relativeCreated", "stack_info", "thread", "threadName",
|
|
||||||
"taskName",
|
|
||||||
})
|
|
||||||
|
|
||||||
|
|
||||||
class JsonFormatter(logging.Formatter):
|
|
||||||
"""Format a LogRecord as a single JSON line.
|
|
||||||
|
|
||||||
Fields:
|
|
||||||
ts — ISO 8601 UTC timestamp with milliseconds.
|
|
||||||
level — uppercase level name (INFO, WARNING, etc.).
|
|
||||||
logger — the logger name (e.g. "cyclone.scheduler").
|
|
||||||
msg — the formatted log message (after %-substitution).
|
|
||||||
extra — dict of any non-reserved LogRecord attributes.
|
|
||||||
|
|
||||||
If ``exc_info`` is set, the formatter appends a ``traceback``
|
|
||||||
field with the formatted exception text (NOT a serialized
|
|
||||||
object — just the stdlib-rendered string).
|
|
||||||
"""
|
|
||||||
|
|
||||||
def format(self, record: logging.LogRecord) -> str:
|
|
||||||
ts = datetime.fromtimestamp(record.created, tz=timezone.utc).isoformat(
|
|
||||||
timespec="milliseconds",
|
|
||||||
)
|
|
||||||
payload: dict[str, Any] = {
|
|
||||||
"ts": ts,
|
|
||||||
"level": record.levelname,
|
|
||||||
"logger": record.name,
|
|
||||||
"msg": record.getMessage(),
|
|
||||||
}
|
|
||||||
# Collect user-provided extras.
|
|
||||||
extras = {
|
|
||||||
k: v
|
|
||||||
for k, v in record.__dict__.items()
|
|
||||||
if k not in _RESERVED_LOGRECORD_ATTRS and not k.startswith("_")
|
|
||||||
}
|
|
||||||
if extras:
|
|
||||||
payload["extra"] = extras
|
|
||||||
if record.exc_info:
|
|
||||||
payload["traceback"] = self.formatException(record.exc_info)
|
|
||||||
if record.stack_info:
|
|
||||||
payload["stack"] = self.formatStack(record.stack_info)
|
|
||||||
return json.dumps(payload, default=str, sort_keys=True)
|
|
||||||
|
|
||||||
|
|
||||||
class CycloneDevFormatter(logging.Formatter):
|
|
||||||
"""Dev-friendly tabular format.
|
|
||||||
|
|
||||||
Example:
|
|
||||||
2026-06-21T15:30:00.123Z INFO cyclone.scheduler Processed inbound foo.x12 parser=parse_999 claims=3
|
|
||||||
|
|
||||||
Same fields as ``JsonFormatter`` but human-readable. Useful for
|
|
||||||
``tail -f cyclone.log`` in dev.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def format(self, record: logging.LogRecord) -> str:
|
|
||||||
ts = datetime.fromtimestamp(record.created, tz=timezone.utc).isoformat(
|
|
||||||
timespec="milliseconds",
|
|
||||||
)
|
|
||||||
extras = {
|
|
||||||
k: v
|
|
||||||
for k, v in record.__dict__.items()
|
|
||||||
if k not in _RESERVED_LOGRECORD_ATTRS and not k.startswith("_")
|
|
||||||
}
|
|
||||||
extra_str = ""
|
|
||||||
if extras:
|
|
||||||
pairs = " ".join(f"{k}={v!r}" for k, v in extras.items())
|
|
||||||
extra_str = " " + pairs
|
|
||||||
base = f"{ts} {record.levelname:<7s} {record.name} {record.getMessage()}{extra_str}"
|
|
||||||
if record.exc_info:
|
|
||||||
base += "\n" + self.formatException(record.exc_info)
|
|
||||||
return base
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# PII scrubber
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
# Conservative PHI patterns. Each pattern is (label, compiled regex,
|
|
||||||
# replacement). Some patterns use a backreference so the field name
|
|
||||||
# (e.g. "dob=") is preserved and only the value is redacted — that
|
|
||||||
# keeps the surrounding context readable in the log line.
|
|
||||||
_PII_PATTERNS: tuple[tuple[str, "re.Pattern[str]", str], ...] = (
|
|
||||||
# 10-digit NPI. Word-boundary anchored so we don't redact, e.g.,
|
|
||||||
# the "10" in "10 claims processed".
|
|
||||||
("npi", re.compile(r"\b\d{10}\b"), "<redacted:npi>"),
|
|
||||||
# SSN: NNN-NN-NNNN or NNNNNNNNN.
|
|
||||||
(
|
|
||||||
"ssn",
|
|
||||||
re.compile(r"\b\d{3}-\d{2}-\d{4}\b|\b\d{9}\b(?=[\s,;)}])"),
|
|
||||||
"<redacted:ssn>",
|
|
||||||
),
|
|
||||||
# DOB: "dob=YYYY-MM-DD" / "date_of_birth=YYYY-MM-DD". Capture the
|
|
||||||
# field name + separator, redact only the date — keeps the
|
|
||||||
# surrounding sentence readable.
|
|
||||||
(
|
|
||||||
"dob",
|
|
||||||
re.compile(
|
|
||||||
r"(?i)(\b(?:dob|date[ _]?of[ _]?birth)[:=]\s*)\d{4}-\d{2}-\d{2}"
|
|
||||||
),
|
|
||||||
r"\1<redacted:dob>",
|
|
||||||
),
|
|
||||||
# Patient name: explicit field marker, redact the whole
|
|
||||||
# "patient_name=..." chunk so the value can't leak in a quoted form.
|
|
||||||
(
|
|
||||||
"patient_name",
|
|
||||||
re.compile(
|
|
||||||
r'(?i)\bpatient[_ ]?name[:=]\s*"?[^\",\s}]+',
|
|
||||||
),
|
|
||||||
"<redacted:patient_name>",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
# Extra-field KEYS that we treat as PHI by themselves — if a log call
|
|
||||||
# passes an extra like ``extra={"date_of_birth": "1980-04-12"}`` we
|
|
||||||
# redact the value even though the value alone isn't PHI-shaped. The
|
|
||||||
# key is the signal. Matched case-insensitively against the full key
|
|
||||||
# (with underscores normalized to spaces for "date of birth").
|
|
||||||
_PHI_EXTRA_KEYS: dict[str, str] = {
|
|
||||||
"npi": "npi",
|
|
||||||
"provider_npi": "npi",
|
|
||||||
"rendering_npi": "npi",
|
|
||||||
"billing_npi": "npi",
|
|
||||||
"ssn": "ssn",
|
|
||||||
"dob": "dob",
|
|
||||||
"date_of_birth": "dob",
|
|
||||||
"patient_name": "patient_name",
|
|
||||||
"patient first name": "patient_name",
|
|
||||||
"patient last name": "patient_name",
|
|
||||||
}
|
|
||||||
|
|
||||||
# When an extra key matches one of these, redact any string value
|
|
||||||
# wholesale (don't try to parse it — just replace).
|
|
||||||
_PHI_EXTRA_WHOLE_VALUE = {"npi", "ssn", "dob", "patient_name"}
|
|
||||||
|
|
||||||
|
|
||||||
class PiiScrubber(logging.Filter):
|
|
||||||
"""Filter that redacts obvious PHI from log records.
|
|
||||||
|
|
||||||
Walks the formatted message + every ``extra`` field value (if
|
|
||||||
it's a string) and rewrites matches to ``<redacted:<name>``.
|
|
||||||
Non-string extras are left alone (we don't try to serialize and
|
|
||||||
re-scrub dicts — too risky for false positives).
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, name: str = "pii_scrubber") -> None:
|
|
||||||
super().__init__(name)
|
|
||||||
self._enabled = True
|
|
||||||
|
|
||||||
def disable(self) -> None:
|
|
||||||
"""Disable scrubbing (for tests / forensic mode)."""
|
|
||||||
self._enabled = False
|
|
||||||
|
|
||||||
def enable(self) -> None:
|
|
||||||
self._enabled = True
|
|
||||||
|
|
||||||
def _scrub(self, text: str) -> str:
|
|
||||||
for label, pat, repl in _PII_PATTERNS:
|
|
||||||
text = pat.sub(repl, text)
|
|
||||||
return text
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _normalize_extra_key(key: str) -> set[str]:
|
|
||||||
"""Return all candidate normalizations of a key.
|
|
||||||
|
|
||||||
``date_of_birth`` should match a lookup table that uses either
|
|
||||||
``date_of_birth`` or ``date of birth`` — so return both. Same
|
|
||||||
for ``patient_name`` vs ``patient name``.
|
|
||||||
"""
|
|
||||||
norm = key.strip().lower()
|
|
||||||
spaced = norm.replace("_", " ")
|
|
||||||
return {norm, spaced}
|
|
||||||
|
|
||||||
def _redact_extra_value(self, key: str, value: Any) -> Any:
|
|
||||||
"""Redact a single extra field value if its key signals PHI."""
|
|
||||||
for norm in self._normalize_extra_key(key):
|
|
||||||
label = _PHI_EXTRA_KEYS.get(norm)
|
|
||||||
if label:
|
|
||||||
if not isinstance(value, str):
|
|
||||||
return value
|
|
||||||
return f"<redacted:{label}>"
|
|
||||||
return value
|
|
||||||
|
|
||||||
def filter(self, record: logging.LogRecord) -> bool:
|
|
||||||
if not self._enabled:
|
|
||||||
return True
|
|
||||||
# Scrub the formatted message.
|
|
||||||
try:
|
|
||||||
msg = record.getMessage()
|
|
||||||
scrubbed_msg = self._scrub(msg)
|
|
||||||
if scrubbed_msg != msg:
|
|
||||||
record.msg = scrubbed_msg
|
|
||||||
record.args = ()
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
pass # never let the scrubber crash a log call
|
|
||||||
# Scrub string extras in place. We mutate the record's
|
|
||||||
# __dict__ directly so the formatter sees the scrubbed value.
|
|
||||||
for k, v in list(record.__dict__.items()):
|
|
||||||
if k in _RESERVED_LOGRECORD_ATTRS or k.startswith("_"):
|
|
||||||
continue
|
|
||||||
# First, key-based redaction (covers `extra={"dob": "..."}`).
|
|
||||||
redacted = self._redact_extra_value(k, v)
|
|
||||||
if redacted is not v:
|
|
||||||
record.__dict__[k] = redacted
|
|
||||||
continue
|
|
||||||
# Second, value-pattern redaction (covers `extra={"note":
|
|
||||||
# "patient_name=John Doe"}`).
|
|
||||||
if isinstance(v, str):
|
|
||||||
scrubbed = self._scrub(v)
|
|
||||||
if scrubbed != v:
|
|
||||||
record.__dict__[k] = scrubbed
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
# Module-level singleton so tests / callers can disable it cleanly.
|
|
||||||
_scrubber = PiiScrubber()
|
|
||||||
|
|
||||||
|
|
||||||
def get_scrubber() -> PiiScrubber:
|
|
||||||
"""Return the module-level PII scrubber singleton."""
|
|
||||||
return _scrubber
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# setup_logging entry point
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _resolve_level(level: str | int | None) -> int:
|
|
||||||
"""Resolve a level string/int, falling back to INFO."""
|
|
||||||
if level is None:
|
|
||||||
return logging.INFO
|
|
||||||
if isinstance(level, int):
|
|
||||||
return level
|
|
||||||
name = str(level).strip().upper()
|
|
||||||
return logging.getLevelNamesMapping().get(name, logging.INFO)
|
|
||||||
|
|
||||||
|
|
||||||
def setup_logging(
|
|
||||||
*,
|
|
||||||
level: str | int | None = None,
|
|
||||||
log_file: str | None = None,
|
|
||||||
json_format: bool = True,
|
|
||||||
scrub_pii: bool = True,
|
|
||||||
propagate_from: str | None = None,
|
|
||||||
) -> logging.Logger:
|
|
||||||
"""Configure the root logger + attach handlers.
|
|
||||||
|
|
||||||
Idempotent: re-calling clears existing handlers on the root
|
|
||||||
logger before attaching fresh ones. Safe to call from
|
|
||||||
``click.command`` invocations and the FastAPI lifespan.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
level: ``"DEBUG"`` / ``"INFO"`` / etc. or an int. ``None``
|
|
||||||
means honor ``CYCLONE_LOG_LEVEL`` env var, then INFO.
|
|
||||||
log_file: Path to a rotating log file. ``None`` means
|
|
||||||
honor ``CYCLONE_LOG_FILE`` env var, then stderr.
|
|
||||||
json_format: Emit JSON lines (default). ``False`` uses
|
|
||||||
:class:`CycloneDevFormatter`.
|
|
||||||
scrub_pii: Apply the PII scrubber (default). Honored via
|
|
||||||
``CYCLONE_LOG_NO_PII_SCRUB=1`` to disable.
|
|
||||||
propagate_from: Optional logger name to attach the scrubber
|
|
||||||
to (defaults to root).
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
The configured root logger.
|
|
||||||
"""
|
|
||||||
# Resolve env-var defaults.
|
|
||||||
if level is None:
|
|
||||||
level = os.environ.get("CYCLONE_LOG_LEVEL", "INFO")
|
|
||||||
if log_file is None:
|
|
||||||
log_file = os.environ.get("CYCLONE_LOG_FILE") or None
|
|
||||||
if not json_format and os.environ.get("CYCLONE_LOG_JSON", "").lower() in (
|
|
||||||
"false", "0", "no",
|
|
||||||
):
|
|
||||||
json_format = True
|
|
||||||
if os.environ.get("CYCLONE_LOG_NO_PII_SCRUB", "").lower() in ("1", "true", "yes"):
|
|
||||||
scrub_pii = False
|
|
||||||
|
|
||||||
root = logging.getLogger()
|
|
||||||
root.setLevel(_resolve_level(level))
|
|
||||||
|
|
||||||
# Clear existing handlers (idempotent re-setup).
|
|
||||||
for h in list(root.handlers):
|
|
||||||
root.removeHandler(h)
|
|
||||||
# Also clear our scrubber so we don't add duplicates.
|
|
||||||
target = logging.getLogger(propagate_from) if propagate_from else root
|
|
||||||
for flt in list(target.filters):
|
|
||||||
if isinstance(flt, PiiScrubber):
|
|
||||||
target.removeFilter(flt)
|
|
||||||
|
|
||||||
# Build the formatter.
|
|
||||||
fmt: logging.Formatter
|
|
||||||
if json_format:
|
|
||||||
fmt = JsonFormatter()
|
|
||||||
else:
|
|
||||||
fmt = CycloneDevFormatter()
|
|
||||||
|
|
||||||
# Build the handler.
|
|
||||||
if log_file:
|
|
||||||
handler: logging.Handler = RotatingFileHandler(
|
|
||||||
log_file,
|
|
||||||
maxBytes=10 * 1024 * 1024,
|
|
||||||
backupCount=5,
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
handler = logging.StreamHandler(stream=sys.stderr)
|
|
||||||
handler.setFormatter(fmt)
|
|
||||||
root.addHandler(handler)
|
|
||||||
|
|
||||||
# Attach the scrubber.
|
|
||||||
if scrub_pii:
|
|
||||||
_scrubber.enable()
|
|
||||||
else:
|
|
||||||
_scrubber.disable()
|
|
||||||
target.addFilter(_scrubber)
|
|
||||||
|
|
||||||
# Quiet down noisy third-party libs.
|
|
||||||
for noisy in ("urllib3", "paramiko", "sqlalchemy.engine"):
|
|
||||||
logging.getLogger(noisy).setLevel(max(root.level, logging.WARNING))
|
|
||||||
|
|
||||||
return root
|
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
-- version: 11
|
|
||||||
-- SP16: Inbound MFT polling scheduler
|
|
||||||
--
|
|
||||||
-- Tracks every file the background scheduler has downloaded from
|
|
||||||
-- the Gainwell MFT inbound path so a re-tick (or a restart) does not
|
|
||||||
-- re-process the same file. Idempotency is required for production:
|
|
||||||
-- the scheduler polls every N seconds and a slow MFT server may hand
|
|
||||||
-- us the same file across two polls.
|
|
||||||
--
|
|
||||||
-- We key on (sftp_block_name, name) — the sftp_block_name disambiguates
|
|
||||||
-- multi-provider installations (SP9+SP-multi-NPI), name is the inbound
|
|
||||||
-- filename as it appears on the MFT server.
|
|
||||||
--
|
|
||||||
-- Status values:
|
|
||||||
-- * ok — parsed cleanly, results persisted to the store
|
|
||||||
-- * error — parser raised; error_message captured for the operator
|
|
||||||
-- * skipped — file_type not in the scheduler's allowed set
|
|
||||||
-- * pending — file was downloaded but a downstream step failed
|
|
||||||
-- (e.g. DB write); the scheduler retries on the next tick
|
|
||||||
--
|
|
||||||
-- claim_count is the number of claims/remittances/acks the parser
|
|
||||||
-- surfaced. Surfaced on /api/admin/scheduler/status so the operator can
|
|
||||||
-- see throughput without parsing logs.
|
|
||||||
--
|
|
||||||
-- Compliance: not part of the HIPAA audit chain (SP11). This is
|
|
||||||
-- operational metadata; an SFTP outage shouldn't pollute the audit log.
|
|
||||||
|
|
||||||
CREATE TABLE processed_inbound_files (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
sftp_block_name TEXT NOT NULL,
|
|
||||||
name TEXT NOT NULL,
|
|
||||||
size INTEGER NOT NULL,
|
|
||||||
modified_at TEXT,
|
|
||||||
file_type TEXT,
|
|
||||||
processed_at TEXT NOT NULL,
|
|
||||||
parser_used TEXT,
|
|
||||||
claim_count INTEGER NOT NULL DEFAULT 0,
|
|
||||||
status TEXT NOT NULL,
|
|
||||||
error_message TEXT
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE UNIQUE INDEX ux_processed_inbound_files_block_name
|
|
||||||
ON processed_inbound_files(sftp_block_name, name);
|
|
||||||
CREATE INDEX ix_processed_inbound_files_processed_at
|
|
||||||
ON processed_inbound_files(processed_at DESC);
|
|
||||||
CREATE INDEX ix_processed_inbound_files_status
|
|
||||||
ON processed_inbound_files(status);
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
-- version: 12
|
|
||||||
-- SP17: encrypted DB backup metadata
|
|
||||||
--
|
|
||||||
-- Tracks every backup the BackupService has taken. The actual
|
|
||||||
-- encrypted blob lives in a directory outside the DB (default
|
|
||||||
-- ~/.local/share/cyclone/backups/); this table is just the index
|
|
||||||
-- the operator queries via GET /api/admin/backup/list.
|
|
||||||
--
|
|
||||||
-- Status values:
|
|
||||||
-- pending - row inserted, .backup() in progress or crashed before commit
|
|
||||||
-- ok - encrypted blob + sidecar written successfully
|
|
||||||
-- error - creation failed; error_message populated
|
|
||||||
-- pruned - retention policy removed the file; row kept for audit
|
|
||||||
|
|
||||||
CREATE TABLE db_backups (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
filename TEXT NOT NULL,
|
|
||||||
backup_dir TEXT NOT NULL,
|
|
||||||
size_bytes INTEGER NOT NULL DEFAULT 0,
|
|
||||||
db_fingerprint TEXT,
|
|
||||||
table_count INTEGER NOT NULL DEFAULT 0,
|
|
||||||
created_at TEXT NOT NULL,
|
|
||||||
completed_at TEXT,
|
|
||||||
status TEXT NOT NULL,
|
|
||||||
error_message TEXT
|
|
||||||
);
|
|
||||||
CREATE UNIQUE INDEX ux_db_backups_filename ON db_backups(backup_dir, filename);
|
|
||||||
CREATE INDEX ix_db_backups_created_at ON db_backups(created_at DESC);
|
|
||||||
CREATE INDEX ix_db_backups_status ON db_backups(status);
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
-- version: 13
|
|
||||||
-- Auth (SP-auth): users + sessions tables.
|
|
||||||
--
|
|
||||||
-- `users` holds the local credential store: bcrypt-hashed password,
|
|
||||||
-- role enum ('admin' | 'user' | 'viewer'), and a soft-delete column
|
|
||||||
-- (disabled_at) so admins can revoke access without losing history.
|
|
||||||
--
|
|
||||||
-- `sessions` holds the server-side session rows; the browser only
|
|
||||||
-- carries an opaque token cookie (cyclone_session) that points here.
|
|
||||||
-- expires_at index lets us cheaply reap stale sessions.
|
|
||||||
|
|
||||||
CREATE TABLE users (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
username TEXT NOT NULL UNIQUE,
|
|
||||||
password_hash TEXT NOT NULL,
|
|
||||||
role TEXT NOT NULL,
|
|
||||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
||||||
disabled_at TEXT
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE INDEX idx_users_username ON users(username);
|
|
||||||
|
|
||||||
CREATE TABLE sessions (
|
|
||||||
id TEXT PRIMARY KEY,
|
|
||||||
user_id INTEGER NOT NULL REFERENCES users(id),
|
|
||||||
expires_at TEXT NOT NULL,
|
|
||||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE INDEX idx_sessions_user_id ON sessions(user_id);
|
|
||||||
CREATE INDEX idx_sessions_expires_at ON sessions(expires_at);
|
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
-- version: 14
|
|
||||||
-- Auth (SP-auth): record the acting user_id on every audit_log entry.
|
|
||||||
--
|
|
||||||
-- Backwards-compatible: existing rows get NULL user_id (they were
|
|
||||||
-- written by the pre-auth `system` actor). Going forward, the FastAPI
|
|
||||||
-- get_current_user dependency injects the id into every audit log call.
|
|
||||||
|
|
||||||
ALTER TABLE audit_log ADD COLUMN user_id INTEGER;
|
|
||||||
|
|
||||||
CREATE INDEX idx_audit_log_user_id ON audit_log(user_id);
|
|
||||||
@@ -1,74 +0,0 @@
|
|||||||
-- version: 15
|
|
||||||
-- Drop the inline UNIQUE(batch_id, patient_control_number) on claims.
|
|
||||||
--
|
|
||||||
-- Migration 0003 attempted DROP INDEX IF EXISTS uq_claims_batch_pcn but
|
|
||||||
-- the constraint is inline in CREATE TABLE, so the drop was a no-op.
|
|
||||||
-- The only way to remove an inline UNIQUE in SQLite is table recreation.
|
|
||||||
--
|
|
||||||
-- Discovery 2026-06-23: the inline UNIQUE does NOT exist in the current
|
|
||||||
-- production DB at user_version=14 (or in main's fresh-DB schema). The
|
|
||||||
-- 32 "Duplicate claim" warnings in /tmp/cyclone-uvicorn.log are PK
|
|
||||||
-- collisions on claims.id (CLM01) when an operator re-uploads the same
|
|
||||||
-- file — not UNIQUE violations. This migration is therefore a defensive
|
|
||||||
-- no-op against the current schema, but keeps the 0003 intent alive
|
|
||||||
-- (drop the constraint if it ever reappears) and lets the SP22 spec
|
|
||||||
-- ship as designed.
|
|
||||||
--
|
|
||||||
-- X12 837P allows any number of CLM segments per 2000B subscriber loop;
|
|
||||||
-- claim identity is provided by the primary key (claims.id = CLM01).
|
|
||||||
-- The remittances table had a parallel constraint already removed in 0003
|
|
||||||
-- (because that one WAS a named index), so this migration only touches
|
|
||||||
-- claims.
|
|
||||||
--
|
|
||||||
-- The migration runner (db_migrate.py) wraps each .sql in an implicit
|
|
||||||
-- transaction via engine.begin(), so we MUST NOT use BEGIN/COMMIT.
|
|
||||||
-- PRAGMA defer_foreign_keys defers FK checks to commit, which is the
|
|
||||||
-- only way to drop a referenced table inside a transaction in SQLite.
|
|
||||||
-- Other tables referencing claims:
|
|
||||||
-- remittances.claim_id
|
|
||||||
-- matches.claim_id
|
|
||||||
-- line_reconciliations.claim_id
|
|
||||||
-- activity_events.claim_id
|
|
||||||
|
|
||||||
PRAGMA defer_foreign_keys = ON;
|
|
||||||
|
|
||||||
CREATE TABLE claims_new (
|
|
||||||
id TEXT PRIMARY KEY,
|
|
||||||
batch_id TEXT NOT NULL REFERENCES batches(id) ON DELETE CASCADE,
|
|
||||||
patient_control_number TEXT NOT NULL,
|
|
||||||
service_date_from DATE,
|
|
||||||
service_date_to DATE,
|
|
||||||
charge_amount NUMERIC(12, 2) NOT NULL DEFAULT 0,
|
|
||||||
provider_npi TEXT,
|
|
||||||
payer_id TEXT,
|
|
||||||
state TEXT NOT NULL DEFAULT 'submitted',
|
|
||||||
state_before_reversal TEXT,
|
|
||||||
matched_remittance_id TEXT REFERENCES remittances(id),
|
|
||||||
raw_json TEXT,
|
|
||||||
rejection_reason TEXT,
|
|
||||||
rejected_at TIMESTAMP,
|
|
||||||
resubmit_count INTEGER NOT NULL DEFAULT 0,
|
|
||||||
state_changed_at TIMESTAMP,
|
|
||||||
payer_rejected_at TEXT,
|
|
||||||
payer_rejected_reason TEXT,
|
|
||||||
payer_rejected_status_code TEXT,
|
|
||||||
payer_rejected_by_277ca_id TEXT,
|
|
||||||
payer_rejected_acknowledged_at TEXT,
|
|
||||||
payer_rejected_acknowledged_actor TEXT
|
|
||||||
-- NO UNIQUE (batch_id, patient_control_number) — removed.
|
|
||||||
);
|
|
||||||
|
|
||||||
INSERT INTO claims_new SELECT * FROM claims;
|
|
||||||
DROP TABLE claims;
|
|
||||||
ALTER TABLE claims_new RENAME TO claims;
|
|
||||||
|
|
||||||
-- Recreate secondary indexes (same names, same columns as initial schema
|
|
||||||
-- plus later migrations).
|
|
||||||
CREATE INDEX ix_claims_state ON claims(state);
|
|
||||||
CREATE INDEX ix_claims_patient_control_number ON claims(patient_control_number);
|
|
||||||
CREATE INDEX ix_claims_service_date_from ON claims(service_date_from);
|
|
||||||
CREATE INDEX ix_claims_state_changed_at ON claims(state, state_changed_at);
|
|
||||||
CREATE INDEX idx_claims_payer_rejected_at ON claims(payer_rejected_at);
|
|
||||||
CREATE INDEX idx_claims_payer_rejected_unack
|
|
||||||
ON claims(payer_rejected_at)
|
|
||||||
WHERE payer_rejected_acknowledged_at IS NULL;
|
|
||||||
@@ -1,159 +0,0 @@
|
|||||||
"""SP20 — NPI checksum + Tax ID format validation.
|
|
||||||
|
|
||||||
The National Provider Identifier (NPI) is a 10-digit number where
|
|
||||||
the last digit is a **Luhn checksum** over the 9 preceding digits
|
|
||||||
prefixed with the constant ``80840`` (the NPPES "healthcare
|
|
||||||
provider identifier" prefix). See CMS / HHS NPI Standard:
|
|
||||||
|
|
||||||
https://www.cms.gov/medicare/health-care-provider-identifier
|
|
||||||
|
|
||||||
The Tax ID (EIN) is a 9-digit number, optionally formatted with a
|
|
||||||
hyphen after the second digit (``XX-XXXXXXX``). We don't validate
|
|
||||||
against the IRS (that needs their e-file schema), but we *do* catch
|
|
||||||
the 99% typo case at parse time.
|
|
||||||
|
|
||||||
Everything in this module is local — no NPPES, no network. Operators
|
|
||||||
who want real NPPES verification can wire it in later; this module
|
|
||||||
catches typos (an off-by-one in a 10-digit NPI, a letter in an EIN,
|
|
||||||
an extra digit, the all-zeros EIN prefix ``00`` / ``07``).
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import re
|
|
||||||
|
|
||||||
# NPPES prefix per the NPI Luhn algorithm. Prepended to the 9-digit
|
|
||||||
# NPI body before running the Luhn check.
|
|
||||||
_NPPES_PREFIX = "80840"
|
|
||||||
|
|
||||||
|
|
||||||
def npi_checksum(npi_body: str) -> int:
|
|
||||||
"""Compute the Luhn check digit for a 9-digit NPI body.
|
|
||||||
|
|
||||||
``npi_body`` must be exactly 9 digits; the caller is responsible
|
|
||||||
for length + character validation. Returns the check digit (0–9).
|
|
||||||
"""
|
|
||||||
if not npi_body.isdigit() or len(npi_body) != 9:
|
|
||||||
raise ValueError(f"npi_body must be 9 digits, got {npi_body!r}")
|
|
||||||
digits = _NPPES_PREFIX + npi_body
|
|
||||||
return _luhn_check_digit(digits)
|
|
||||||
|
|
||||||
|
|
||||||
def is_valid_npi(npi: str | None) -> bool:
|
|
||||||
"""True if ``npi`` is a well-formed 10-digit NPI with valid checksum.
|
|
||||||
|
|
||||||
Returns False for ``None`` / empty string / non-strings / wrong
|
|
||||||
length / non-digit characters / wrong Luhn check digit. Doesn't
|
|
||||||
call NPPES — see module docstring for why.
|
|
||||||
|
|
||||||
>>> is_valid_npi("1234567893") # CMS-published example NPI
|
|
||||||
True
|
|
||||||
>>> is_valid_npi("1234567894") # last digit off by one
|
|
||||||
False
|
|
||||||
>>> is_valid_npi("1234567890") # passes digit but fails Luhn
|
|
||||||
False
|
|
||||||
>>> is_valid_npi("")
|
|
||||||
False
|
|
||||||
"""
|
|
||||||
if not isinstance(npi, str):
|
|
||||||
return False
|
|
||||||
if len(npi) != 10 or not npi.isdigit():
|
|
||||||
return False
|
|
||||||
return npi[-1] == str(npi_checksum(npi[:-1]))
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Tax ID (EIN)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
# EIN prefix table (subset). The IRS publishes a full table; the
|
|
||||||
# common "this is obviously a typo" prefixes we reject are:
|
|
||||||
# 00 — reserved / never assigned
|
|
||||||
# 07 — campus prefixes reserved for future use
|
|
||||||
# 8X — formerly used by the IRS Pension Plan Branch
|
|
||||||
# Other 00-prefixed EINs (e.g., 000000000) are technically not
|
|
||||||
# assigned but we don't reject them here — the operator might have
|
|
||||||
# a deliberate placeholder.
|
|
||||||
_EIN_FORBIDDEN_PREFIXES = {"00", "07"}
|
|
||||||
_EIN_RESERVED_PREFIX_8X = re.compile(r"^8\d$")
|
|
||||||
|
|
||||||
# 9 digits, optionally formatted as XX-XXXXXXX.
|
|
||||||
_EIN_FORMATTED = re.compile(r"^\d{2}-\d{7}$")
|
|
||||||
_EIN_PLAIN = re.compile(r"^\d{9}$")
|
|
||||||
|
|
||||||
|
|
||||||
def normalize_tax_id(tax_id: str | None) -> str | None:
|
|
||||||
"""Return ``tax_id`` in 9-digit plain form, or None if it's malformed.
|
|
||||||
|
|
||||||
>>> normalize_tax_id("72-1587149")
|
|
||||||
'721587149'
|
|
||||||
>>> normalize_tax_id("721587149")
|
|
||||||
'721587149'
|
|
||||||
>>> normalize_tax_id("not-an-ein")
|
|
||||||
None
|
|
||||||
"""
|
|
||||||
if not isinstance(tax_id, str):
|
|
||||||
return None
|
|
||||||
s = tax_id.strip()
|
|
||||||
if _EIN_FORMATTED.match(s):
|
|
||||||
return s.replace("-", "")
|
|
||||||
if _EIN_PLAIN.match(s):
|
|
||||||
return s
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def is_valid_tax_id(tax_id: str | None) -> bool:
|
|
||||||
"""True if ``tax_id`` is a 9-digit EIN (formatted or plain) with
|
|
||||||
a non-reserved prefix.
|
|
||||||
|
|
||||||
>>> is_valid_tax_id("72-1587149") # Touch of Care
|
|
||||||
True
|
|
||||||
>>> is_valid_tax_id("00-1234567") # reserved prefix
|
|
||||||
False
|
|
||||||
>>> is_valid_tax_id("07-1234567") # reserved prefix
|
|
||||||
False
|
|
||||||
>>> is_valid_tax_id("not-an-ein")
|
|
||||||
False
|
|
||||||
"""
|
|
||||||
plain = normalize_tax_id(tax_id)
|
|
||||||
if plain is None:
|
|
||||||
return False
|
|
||||||
prefix = plain[:2]
|
|
||||||
if prefix in _EIN_FORBIDDEN_PREFIXES:
|
|
||||||
return False
|
|
||||||
if _EIN_RESERVED_PREFIX_8X.match(prefix):
|
|
||||||
return False
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Luhn internals
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _luhn_check_digit(digits: str) -> int:
|
|
||||||
"""Return the Luhn check digit for ``digits``.
|
|
||||||
|
|
||||||
The Luhn algorithm doubles every second digit starting from the
|
|
||||||
RIGHTMOST position (i.e., the first digit doubled is the rightmost
|
|
||||||
character of ``digits``). If the doubled value exceeds 9, subtract
|
|
||||||
9. Sum all digits; the check digit is ``(10 - sum % 10) % 10``.
|
|
||||||
|
|
||||||
``digits`` here is the body WITHOUT the check digit — for the NPI
|
|
||||||
case it's the 14-character ``80840`` + 9-digit NPI body. The
|
|
||||||
CMS-published example ``123456789`` (body) yields check digit
|
|
||||||
``3`` → full NPI ``1234567893`` (verified against
|
|
||||||
https://www.cms.gov/.../NPIcheckdigit.pdf).
|
|
||||||
"""
|
|
||||||
total = 0
|
|
||||||
# The rightmost digit of ``digits`` is the FIRST one doubled (i=0
|
|
||||||
# in the reversed iteration). Per CMS, doubling starts at the
|
|
||||||
# rightmost and alternates leftward.
|
|
||||||
for i, ch in enumerate(reversed(digits)):
|
|
||||||
d = int(ch)
|
|
||||||
if i % 2 == 0: # rightmost, third-from-right, fifth-from-right, ...
|
|
||||||
d *= 2
|
|
||||||
if d > 9:
|
|
||||||
d -= 9
|
|
||||||
total += d
|
|
||||||
return (10 - total % 10) % 10
|
|
||||||
@@ -63,7 +63,6 @@ class ClaimHeader(_Base):
|
|||||||
frequency_code: str | None = None
|
frequency_code: str | None = None
|
||||||
provider_signature: str | None = None
|
provider_signature: str | None = None
|
||||||
assignment: str | None = None
|
assignment: str | None = None
|
||||||
benefits_assignment_certification: str | None = None # CLM08 (Y/N)
|
|
||||||
release_of_info: str | None = None
|
release_of_info: str | None = None
|
||||||
prior_auth: str | None = None
|
prior_auth: str | None = None
|
||||||
|
|
||||||
@@ -88,14 +87,6 @@ class ServiceLine(_Base):
|
|||||||
place_of_service: str | None = None
|
place_of_service: str | None = None
|
||||||
service_date: date | None = None
|
service_date: date | None = None
|
||||||
provider_reference: str | None = None
|
provider_reference: str | None = None
|
||||||
# SV1-07 — Diagnosis Code Pointer. Points to one or more
|
|
||||||
# diagnosis codes in the parent claim's HI segment ("1".. "12",
|
|
||||||
# space-separated when multiple). For 837P with a non-empty HI
|
|
||||||
# segment, SV1-07 is required by HCPF / Gainwell. The parser
|
|
||||||
# captures it from the source; the serializer defaults to "1"
|
|
||||||
# when the claim has at least one diagnosis and no explicit
|
|
||||||
# pointer was captured (matches the common single-dx case).
|
|
||||||
dx_pointer: str | None = None
|
|
||||||
|
|
||||||
|
|
||||||
class ValidationIssue(_Base):
|
class ValidationIssue(_Base):
|
||||||
|
|||||||
@@ -202,7 +202,6 @@ def _consume_claim(segments: list[list[str]], idx: int) -> tuple[ClaimOutput, in
|
|||||||
frequency_code=freq or None,
|
frequency_code=freq or None,
|
||||||
provider_signature=clm[6] if len(clm) > 6 else None,
|
provider_signature=clm[6] if len(clm) > 6 else None,
|
||||||
assignment=clm[7] if len(clm) > 7 else None,
|
assignment=clm[7] if len(clm) > 7 else None,
|
||||||
benefits_assignment_certification=clm[8] if len(clm) > 8 else None,
|
|
||||||
release_of_info=clm[9] if len(clm) > 9 else None,
|
release_of_info=clm[9] if len(clm) > 9 else None,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -286,12 +285,6 @@ def _consume_service_line(segments: list[list[str]], idx: int, line_no: int) ->
|
|||||||
except Exception:
|
except Exception:
|
||||||
units = None
|
units = None
|
||||||
place_of_service = seg[5] if len(seg) > 5 else None
|
place_of_service = seg[5] if len(seg) > 5 else None
|
||||||
# SV1-06 (Unit Basis of Measurement) is X12 "UN" for "units" — we
|
|
||||||
# already use unit_type in SV1-03; SV1-06 is rarely populated and
|
|
||||||
# is not required by HCPF.
|
|
||||||
# SV1-07 — Diagnosis Code Pointer (e.g. "1" for the first HI
|
|
||||||
# diagnosis). Required by HCPF when the claim has diagnoses.
|
|
||||||
dx_pointer = seg[7] if len(seg) > 7 and seg[7] else None
|
|
||||||
service_date: date | None = None
|
service_date: date | None = None
|
||||||
provider_ref: str | None = None
|
provider_ref: str | None = None
|
||||||
|
|
||||||
@@ -318,7 +311,6 @@ def _consume_service_line(segments: list[list[str]], idx: int, line_no: int) ->
|
|||||||
place_of_service=place_of_service,
|
place_of_service=place_of_service,
|
||||||
service_date=service_date,
|
service_date=service_date,
|
||||||
provider_reference=provider_ref,
|
provider_reference=provider_ref,
|
||||||
dx_pointer=dx_pointer,
|
|
||||||
),
|
),
|
||||||
idx,
|
idx,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -112,10 +112,7 @@ def _build_gs(sender_id: str, receiver_id: str, group_control_number: str) -> st
|
|||||||
_FUNCTIONAL_ID_HEALTH_CARE,
|
_FUNCTIONAL_ID_HEALTH_CARE,
|
||||||
sender_id,
|
sender_id,
|
||||||
receiver_id,
|
receiver_id,
|
||||||
# GS-04 must be CCYYMMDD (8 digits) per X12 — ISA uses YYMMDD
|
_today_yymmdd(),
|
||||||
# (6 digits) for the older format, but the GS segment is the
|
|
||||||
# newer ANSI X12 format and requires the full year.
|
|
||||||
_today_yyyymmdd(),
|
|
||||||
_today_hhmm(),
|
_today_hhmm(),
|
||||||
group_control_number,
|
group_control_number,
|
||||||
"X",
|
"X",
|
||||||
@@ -189,30 +186,17 @@ def _build_nm1(entity_id_qualifier: str, entity_type: str, name: str,
|
|||||||
return _ELEM.join(parts) + _SEG
|
return _ELEM.join(parts) + _SEG
|
||||||
|
|
||||||
|
|
||||||
def _build_per(
|
def _build_per(contact_name: str | None, contact_phone: str | None) -> str:
|
||||||
contact_name: str | None,
|
"""PER segment — submitter contact. Returns empty when no contact info."""
|
||||||
contact_phone: str | None,
|
if not contact_name and not contact_phone:
|
||||||
contact_email: str | None = None,
|
return ""
|
||||||
email_qual: str = "EM",
|
parts = [
|
||||||
) -> str:
|
"PER",
|
||||||
"""PER segment — submitter contact (Loop 1000A).
|
"IC", # PER01 — contact function code (Information Contact)
|
||||||
|
contact_name or "",
|
||||||
X12 005010X222A1 *requires* at least one PER segment in Loop 1000A
|
"TE", # PER03 — phone qualifier
|
||||||
(Submitter Name) and at least PER01 must be present, so this
|
contact_phone or "",
|
||||||
builder always emits a segment. PER01 = "IC" (Information Contact).
|
]
|
||||||
|
|
||||||
The remaining elements are filled from the available contact info:
|
|
||||||
name, then email (preferred — Gainwell/HCPF expect this), then phone.
|
|
||||||
"""
|
|
||||||
parts = ["PER", "IC"]
|
|
||||||
if contact_name:
|
|
||||||
parts.append(contact_name)
|
|
||||||
if contact_email:
|
|
||||||
parts.append(email_qual) # PER03 — email qualifier (default "EM")
|
|
||||||
parts.append(contact_email) # PER04 — the email itself
|
|
||||||
elif contact_phone:
|
|
||||||
parts.append("TE") # PER03 — phone qualifier
|
|
||||||
parts.append(contact_phone) # PER04 — the phone itself
|
|
||||||
return _ELEM.join(parts) + _SEG
|
return _ELEM.join(parts) + _SEG
|
||||||
|
|
||||||
|
|
||||||
@@ -252,40 +236,26 @@ def _build_hl(hl_id: str, parent_id: str, level_code: str, child_code: str) -> s
|
|||||||
return _ELEM.join(parts) + _SEG
|
return _ELEM.join(parts) + _SEG
|
||||||
|
|
||||||
|
|
||||||
def _build_sbr(
|
def _build_sbr(relationship_code: str | None, member_id: str | None,
|
||||||
individual_relationship_code: str | None,
|
payer_name: str | None) -> str:
|
||||||
claim_filing_indicator_code: str | None,
|
|
||||||
) -> str:
|
|
||||||
"""SBR segment — subscriber information.
|
"""SBR segment — subscriber information.
|
||||||
|
|
||||||
Slot layout (X12 005010X222A1):
|
SBR01 (relationship code) defaults to ``"P"`` (Patient = self) which is
|
||||||
SBR01 — Payer Responsibility Sequence Number Code. Default ``"P"``
|
the most common case for professional claims; the parser does not store
|
||||||
(Patient = primary). The parser does not capture this
|
this on the canonical Subscriber model so we cannot thread it through
|
||||||
field on ``ClaimOutput`` so we default it.
|
without adding a model field.
|
||||||
SBR02 — Individual Relationship Code. ``"18"`` = self, ``"01"`` = spouse, etc.
|
|
||||||
The parser does not capture this either; we default ``"18"``
|
|
||||||
for the common self-pay case.
|
|
||||||
SBR09 — Claim Filing Indicator Code. ``"MC"`` for Medicaid,
|
|
||||||
``"16"`` for Medicare Part B, etc. The canonical
|
|
||||||
PayerConfig837 carries ``sbr09_default``; we thread it in
|
|
||||||
from the caller.
|
|
||||||
|
|
||||||
The member_id and payer name do NOT belong in SBR — the member_id
|
|
||||||
lives in NM109 of the NM1*IL segment, and the payer name is in
|
|
||||||
NM103 of NM1*PR. (Earlier revisions of this function put them in
|
|
||||||
SBR06 / SBR09, which is wrong and rejected by HCPF.)
|
|
||||||
"""
|
"""
|
||||||
parts = [
|
parts = [
|
||||||
"SBR",
|
"SBR",
|
||||||
"P", # SBR01 — primary
|
relationship_code or "P",
|
||||||
individual_relationship_code or "18", # SBR02 — self
|
"", # SBR02 — group number
|
||||||
"", # SBR03 — group number
|
"", # SBR03 — group name
|
||||||
"", # SBR04 — group name
|
"", # SBR04 — claim filing indicator code
|
||||||
"", # SBR05 — insurance type code
|
"", # SBR05 — sequence number code
|
||||||
"", # SBR06 — coordination of benefits
|
payer_name or "", # SBR06 — claim filing indicator code (CO uses MC)
|
||||||
"", # SBR07 — yes/no condition
|
"", # SBR07
|
||||||
"", # SBR08 — employment status code
|
"", # SBR08
|
||||||
claim_filing_indicator_code or "", # SBR09 — claim filing indicator
|
member_id or "", # SBR09 — claim submitter's id
|
||||||
]
|
]
|
||||||
return _ELEM.join(parts) + _SEG
|
return _ELEM.join(parts) + _SEG
|
||||||
|
|
||||||
@@ -330,11 +300,7 @@ def _build_clm(claim) -> str:
|
|||||||
clm05, # CLM05 — composite POS:qualifier:frequency_code
|
clm05, # CLM05 — composite POS:qualifier:frequency_code
|
||||||
claim.provider_signature or "Y", # CLM06
|
claim.provider_signature or "Y", # CLM06
|
||||||
claim.assignment or "Y", # CLM07
|
claim.assignment or "Y", # CLM07
|
||||||
# CLM08 — Benefits Assignment Certification. X12 837P requires
|
"", # CLM08 — benefit assignment certification
|
||||||
# this when CLM07 = "Y" (the common case for in-network
|
|
||||||
# professional claims). Default to "Y" when the source did
|
|
||||||
# not capture one — matches what 99% of HCPF files look like.
|
|
||||||
claim.benefits_assignment_certification or "Y", # CLM08
|
|
||||||
claim.release_of_info or "Y", # CLM09
|
claim.release_of_info or "Y", # CLM09
|
||||||
]
|
]
|
||||||
return _ELEM.join(parts) + _SEG
|
return _ELEM.join(parts) + _SEG
|
||||||
@@ -359,41 +325,21 @@ def _build_lx(line_number: int) -> str:
|
|||||||
return _ELEM.join(["LX", str(line_number)]) + _SEG
|
return _ELEM.join(["LX", str(line_number)]) + _SEG
|
||||||
|
|
||||||
|
|
||||||
def _build_sv1(line, *, dx_pointer: str | None = None) -> str:
|
def _build_sv1(line) -> str:
|
||||||
"""SV1 segment — professional service line.
|
"""SV1 segment — professional service line."""
|
||||||
|
|
||||||
X12 005010X222A1 layout (837P):
|
|
||||||
SV1-01 composite procedure identifier
|
|
||||||
SV1-02 monetary amount (charge)
|
|
||||||
SV1-03 unit of basis measurement (UN, MJ, etc.) — ``line.unit_type``
|
|
||||||
SV1-04 service unit count — ``line.units``
|
|
||||||
SV1-05 place of service code — ``line.place_of_service``
|
|
||||||
SV1-06 **NOT USED** by this guide (must be empty)
|
|
||||||
SV1-07 diagnosis code pointer — ``dx_pointer`` (required when the
|
|
||||||
parent claim has an HI segment)
|
|
||||||
|
|
||||||
The parser captures the original SV1-07 pointer when present; the
|
|
||||||
serializer defaults it to ``"1"`` (pointing at the first HI
|
|
||||||
diagnosis) when the claim has diagnoses and no explicit pointer
|
|
||||||
was captured. When the claim has no HI segment we leave SV1-07
|
|
||||||
empty to match the spec.
|
|
||||||
"""
|
|
||||||
proc = line.procedure
|
proc = line.procedure
|
||||||
code = proc.code if proc else ""
|
code = proc.code if proc else ""
|
||||||
mods = proc.modifiers if proc else []
|
mods = proc.modifiers if proc else []
|
||||||
composite = "HC:" + code + "".join(f":{m}" for m in (mods or [])[:4])
|
composite = "HC:" + code + "".join(f":{m}" for m in (mods or [])[:4])
|
||||||
charge = f"{Decimal(line.charge or 0):.2f}"
|
charge = f"{Decimal(line.charge or 0):.2f}"
|
||||||
units = f"{Decimal(line.units):g}" if line.units is not None else "1"
|
units = f"{Decimal(line.units):g}" if line.units is not None else "1"
|
||||||
sv1_07 = dx_pointer or ""
|
|
||||||
parts = [
|
parts = [
|
||||||
"SV1",
|
"SV1",
|
||||||
composite, # SV1-01
|
composite,
|
||||||
charge, # SV1-02
|
charge,
|
||||||
line.unit_type or "UN", # SV1-03 — unit basis code
|
line.unit_type or "UN",
|
||||||
units, # SV1-04
|
units,
|
||||||
line.place_of_service or "", # SV1-05
|
line.place_of_service or "",
|
||||||
"", # SV1-06 — NOT USED in 837P
|
|
||||||
sv1_07, # SV1-07 — diagnosis pointer
|
|
||||||
]
|
]
|
||||||
return _ELEM.join(parts) + _SEG
|
return _ELEM.join(parts) + _SEG
|
||||||
|
|
||||||
@@ -411,20 +357,15 @@ def _build_dtp_472(service_date: date | None) -> str:
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def _build_submitter_block(
|
def _build_submitter_block(sender_id: str, submitter_name: str | None,
|
||||||
sender_id: str,
|
|
||||||
submitter_name: str | None,
|
|
||||||
contact_name: str | None,
|
contact_name: str | None,
|
||||||
contact_phone: str | None,
|
contact_phone: str | None) -> list[str]:
|
||||||
contact_email: str | None = None,
|
|
||||||
email_qual: str = "EM",
|
|
||||||
) -> list[str]:
|
|
||||||
out = [
|
out = [
|
||||||
_build_nm1("41", "41", submitter_name or sender_id, "46", sender_id),
|
_build_nm1("41", "41", submitter_name or sender_id, "46", sender_id),
|
||||||
]
|
]
|
||||||
# PER is required by X12 (at least PER01). _build_per always emits
|
per = _build_per(contact_name, contact_phone)
|
||||||
# the segment; the submitter block always has exactly one.
|
if per:
|
||||||
out.append(_build_per(contact_name, contact_phone, contact_email, email_qual))
|
out.append(per)
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
@@ -454,14 +395,11 @@ def _build_billing_provider_block(provider) -> list[str]:
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
def _build_subscriber_block(
|
def _build_subscriber_block(subscriber, payer_name: str | None) -> list[str]:
|
||||||
subscriber,
|
|
||||||
claim_filing_indicator_code: str | None,
|
|
||||||
) -> list[str]:
|
|
||||||
"""HL*2 → SBR → NM1*IL → N3 → N4 → DMG. Subscriber has no children."""
|
"""HL*2 → SBR → NM1*IL → N3 → N4 → DMG. Subscriber has no children."""
|
||||||
out = [
|
out = [
|
||||||
_build_hl("2", "1", "22", "0"), # HL*2 — subscriber, 0 children
|
_build_hl("2", "1", "22", "0"), # HL*2 — subscriber, 0 children
|
||||||
_build_sbr("18", claim_filing_indicator_code),
|
_build_sbr("18", subscriber.member_id, payer_name),
|
||||||
_build_nm1(
|
_build_nm1(
|
||||||
"IL", "IL",
|
"IL", "IL",
|
||||||
f"{subscriber.last_name} {subscriber.first_name}".strip(),
|
f"{subscriber.last_name} {subscriber.first_name}".strip(),
|
||||||
@@ -489,24 +427,12 @@ def _build_payer_block(payer) -> list[str]:
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
def _build_service_lines_block(service_lines, *, has_diagnoses: bool = False) -> list[str]:
|
def _build_service_lines_block(service_lines) -> list[str]:
|
||||||
"""Per line: LX / SV1 / DTP*472 / REF*6R.
|
"""Per line: LX / SV1 / DTP*472 / REF*6R."""
|
||||||
|
|
||||||
``has_diagnoses`` is True when the parent claim emits an HI segment;
|
|
||||||
in that case SV1-07 is required by X12 and we default each line's
|
|
||||||
pointer to ``"1"`` (the first HI diagnosis) unless the source
|
|
||||||
captured a different pointer on the line itself.
|
|
||||||
"""
|
|
||||||
out: list[str] = []
|
out: list[str] = []
|
||||||
for idx, line in enumerate(service_lines or [], start=1):
|
for idx, line in enumerate(service_lines or [], start=1):
|
||||||
out.append(_build_lx(idx))
|
out.append(_build_lx(idx))
|
||||||
# Prefer the line's captured pointer (parser pulled SV1-07
|
out.append(_build_sv1(line))
|
||||||
# when present). Fall back to "1" only when the claim has
|
|
||||||
# diagnoses and the source had no explicit pointer — the
|
|
||||||
# common single-diagnosis case.
|
|
||||||
line_pointer = getattr(line, "dx_pointer", None)
|
|
||||||
effective_pointer = line_pointer or ("1" if has_diagnoses else "")
|
|
||||||
out.append(_build_sv1(line, dx_pointer=effective_pointer))
|
|
||||||
dtp = _build_dtp_472(line.service_date)
|
dtp = _build_dtp_472(line.service_date)
|
||||||
if dtp:
|
if dtp:
|
||||||
out.append(dtp)
|
out.append(dtp)
|
||||||
@@ -524,10 +450,7 @@ def serialize_837(
|
|||||||
submitter_name: str | None = None,
|
submitter_name: str | None = None,
|
||||||
submitter_contact_name: str | None = None,
|
submitter_contact_name: str | None = None,
|
||||||
submitter_contact_phone: str | None = None,
|
submitter_contact_phone: str | None = None,
|
||||||
submitter_contact_email: str | None = None,
|
|
||||||
submitter_contact_email_qual: str = "EM",
|
|
||||||
receiver_name: str | None = None,
|
receiver_name: str | None = None,
|
||||||
claim_filing_indicator_code: str | None = None,
|
|
||||||
interchange_control_number: str = "000000001",
|
interchange_control_number: str = "000000001",
|
||||||
group_control_number: str = "1",
|
group_control_number: str = "1",
|
||||||
) -> str:
|
) -> str:
|
||||||
@@ -539,16 +462,6 @@ def serialize_837(
|
|||||||
(``"CYCLONE"`` / ``"RECEIVER"``) but real deployments should pass
|
(``"CYCLONE"`` / ``"RECEIVER"``) but real deployments should pass
|
||||||
the configured values.
|
the configured values.
|
||||||
|
|
||||||
The submitter block (Loop 1000A) always emits a PER segment per the
|
|
||||||
X12 spec — the canonical clearhouse config provides the contact
|
|
||||||
name and email so callers should pass them through.
|
|
||||||
|
|
||||||
The claim filing indicator (SBR09) is read from the per-payer
|
|
||||||
config (``PayerConfig837.sbr09_default``); callers should pass it
|
|
||||||
in. If not passed, SBR09 is left empty (which causes the
|
|
||||||
:func:`cyclone.parsers.validator._r202_sbr09_allowed` rule to skip
|
|
||||||
its check — degraded but not a hard error).
|
|
||||||
|
|
||||||
Editable fields (CLM, REF*G1, HI, service-line SV1, DTP*472) are
|
Editable fields (CLM, REF*G1, HI, service-line SV1, DTP*472) are
|
||||||
emitted from the canonical ``ClaimOutput`` fields, so post-parse
|
emitted from the canonical ``ClaimOutput`` fields, so post-parse
|
||||||
edits propagate to the output.
|
edits propagate to the output.
|
||||||
@@ -568,13 +481,11 @@ def serialize_837(
|
|||||||
),
|
),
|
||||||
]
|
]
|
||||||
segments.extend(_build_submitter_block(
|
segments.extend(_build_submitter_block(
|
||||||
sender_id, submitter_name,
|
sender_id, submitter_name, submitter_contact_name, submitter_contact_phone,
|
||||||
submitter_contact_name, submitter_contact_phone, submitter_contact_email,
|
|
||||||
submitter_contact_email_qual,
|
|
||||||
))
|
))
|
||||||
segments.extend(_build_receiver_block(receiver_id, receiver_name))
|
segments.extend(_build_receiver_block(receiver_id, receiver_name))
|
||||||
segments.extend(_build_billing_provider_block(claim.billing_provider))
|
segments.extend(_build_billing_provider_block(claim.billing_provider))
|
||||||
segments.extend(_build_subscriber_block(claim.subscriber, claim_filing_indicator_code))
|
segments.extend(_build_subscriber_block(claim.subscriber, claim.payer.name))
|
||||||
segments.extend(_build_payer_block(claim.payer))
|
segments.extend(_build_payer_block(claim.payer))
|
||||||
|
|
||||||
# Claim-level editable segments.
|
# Claim-level editable segments.
|
||||||
@@ -586,10 +497,7 @@ def serialize_837(
|
|||||||
segments.append(_build_hi(claim.diagnoses))
|
segments.append(_build_hi(claim.diagnoses))
|
||||||
|
|
||||||
# Service lines (LX / SV1 / DTP*472 / REF*6R).
|
# Service lines (LX / SV1 / DTP*472 / REF*6R).
|
||||||
segments.extend(_build_service_lines_block(
|
segments.extend(_build_service_lines_block(claim.service_lines))
|
||||||
claim.service_lines,
|
|
||||||
has_diagnoses=bool(claim.diagnoses),
|
|
||||||
))
|
|
||||||
|
|
||||||
# SE segment count includes ST (line 3, 1-based) through SE itself
|
# SE segment count includes ST (line 3, 1-based) through SE itself
|
||||||
# — i.e. the entire ST..SE block inclusive.
|
# — i.e. the entire ST..SE block inclusive.
|
||||||
@@ -606,23 +514,15 @@ def serialize_837_for_resubmit(
|
|||||||
claim: ClaimOutput,
|
claim: ClaimOutput,
|
||||||
*,
|
*,
|
||||||
interchange_index: int,
|
interchange_index: int,
|
||||||
**kwargs,
|
|
||||||
) -> str:
|
) -> str:
|
||||||
"""Like :func:`serialize_837` but assigns deterministic-but-unique
|
"""Like :func:`serialize_837` but assigns deterministic-but-unique
|
||||||
interchange + group control numbers for a bundle position.
|
interchange + group control numbers for a bundle position.
|
||||||
|
|
||||||
Interchange number = ``f"{interchange_index:09d}"``.
|
Interchange number = ``f"{interchange_index:09d}"``.
|
||||||
Group number = ``str(interchange_index)``.
|
Group number = ``str(interchange_index)``.
|
||||||
|
|
||||||
All other keyword arguments (sender_id, receiver_id, submitter_*
|
|
||||||
and receiver_* contact info, claim_filing_indicator_code) are
|
|
||||||
forwarded to :func:`serialize_837` unchanged so callers — like the
|
|
||||||
export and SFTP-submit endpoints — can pass through clearhouse +
|
|
||||||
payer config without copying the signature.
|
|
||||||
"""
|
"""
|
||||||
return serialize_837(
|
return serialize_837(
|
||||||
claim,
|
claim,
|
||||||
interchange_control_number=f"{interchange_index:09d}",
|
interchange_control_number=f"{interchange_index:09d}",
|
||||||
group_control_number=str(interchange_index),
|
group_control_number=str(interchange_index),
|
||||||
**kwargs,
|
|
||||||
)
|
)
|
||||||
@@ -35,36 +35,6 @@ def _r020_npi_format(claim: ClaimOutput, _: PayerConfig) -> Iterable[ValidationI
|
|||||||
yield ValidationIssue(rule="R020_npi_format", severity="error", message=f"Billing provider NPI must be 10 digits, got {claim.billing_provider.npi!r}")
|
yield ValidationIssue(rule="R020_npi_format", severity="error", message=f"Billing provider NPI must be 10 digits, got {claim.billing_provider.npi!r}")
|
||||||
|
|
||||||
|
|
||||||
def _r021_npi_checksum(claim: ClaimOutput, _: PayerConfig) -> Iterable[ValidationIssue]:
|
|
||||||
"""SP20: validate the billing-provider NPI's Luhn check digit.
|
|
||||||
|
|
||||||
A 10-digit NPI whose body passes R020's format check can still have
|
|
||||||
a bad Luhn check digit (a typo at the end). Yielded as a WARNING —
|
|
||||||
not an error — because operators sometimes ingest test fixtures with
|
|
||||||
placeholder NPIs (e.g. all-same-digit) and we don't want to block
|
|
||||||
that path. Local-only check, no NPPES round-trip.
|
|
||||||
"""
|
|
||||||
npi = claim.billing_provider.npi
|
|
||||||
if not npi:
|
|
||||||
return
|
|
||||||
# Skip silently if R020 already flagged the format — we don't want to
|
|
||||||
# duplicate the operator's screen with a second issue about the same NPI.
|
|
||||||
if not NPI_RE.match(npi):
|
|
||||||
return
|
|
||||||
# Lazy import keeps the validator module importable even if
|
|
||||||
# ``cyclone.npi`` is unavailable (e.g. in some legacy test setups).
|
|
||||||
try:
|
|
||||||
from cyclone.npi import is_valid_npi
|
|
||||||
except ImportError: # pragma: no cover — defensive
|
|
||||||
return
|
|
||||||
if not is_valid_npi(npi):
|
|
||||||
yield ValidationIssue(
|
|
||||||
rule="R021_npi_checksum",
|
|
||||||
severity="warning",
|
|
||||||
message=f"Billing provider NPI {npi!r} fails Luhn checksum (likely typo)",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _r030_frequency_allowed(claim: ClaimOutput, cfg: PayerConfig) -> Iterable[ValidationIssue]:
|
def _r030_frequency_allowed(claim: ClaimOutput, cfg: PayerConfig) -> Iterable[ValidationIssue]:
|
||||||
if not claim.claim.frequency_code:
|
if not claim.claim.frequency_code:
|
||||||
return
|
return
|
||||||
@@ -422,7 +392,6 @@ _RULES: list[Rule] = [
|
|||||||
_r010_clm01_present,
|
_r010_clm01_present,
|
||||||
_r011_total_charge_positive,
|
_r011_total_charge_positive,
|
||||||
_r020_npi_format,
|
_r020_npi_format,
|
||||||
_r021_npi_checksum,
|
|
||||||
_r030_frequency_allowed,
|
_r030_frequency_allowed,
|
||||||
_r031_ref_g1_optional,
|
_r031_ref_g1_optional,
|
||||||
_r034_ref_g1_required,
|
_r034_ref_g1_required,
|
||||||
|
|||||||
@@ -100,19 +100,6 @@ class EventBus:
|
|||||||
yield await queue.get()
|
yield await queue.get()
|
||||||
queue.task_done()
|
queue.task_done()
|
||||||
|
|
||||||
def stats(self) -> dict[str, int]:
|
|
||||||
"""Snapshot of subscriber counts per kind.
|
|
||||||
|
|
||||||
Used by ``/api/health`` (SP19) and the admin diagnostics page.
|
|
||||||
Returns ``{kind: count}`` for every kind with at least one
|
|
||||||
subscriber; kinds with zero subscribers are omitted.
|
|
||||||
"""
|
|
||||||
return {
|
|
||||||
kind: len(subs)
|
|
||||||
for kind, subs in self._subscribers.items()
|
|
||||||
if subs
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def get_event_bus() -> EventBus:
|
def get_event_bus() -> EventBus:
|
||||||
"""Return the process-wide EventBus attached to the FastAPI app state.
|
"""Return the process-wide EventBus attached to the FastAPI app state.
|
||||||
|
|||||||
@@ -1,720 +0,0 @@
|
|||||||
"""Background inbound MFT polling scheduler (SP16).
|
|
||||||
|
|
||||||
Turns Cyclone from a manual upload tool into a live clearinghouse:
|
|
||||||
a long-running asyncio task that periodically polls the Gainwell MFT
|
|
||||||
inbound path, downloads each new file, and runs it through the
|
|
||||||
appropriate parser. The operator no longer has to watch for inbound
|
|
||||||
files and POST them to ``/api/parse-999`` etc. by hand.
|
|
||||||
|
|
||||||
Design constraints
|
|
||||||
------------------
|
|
||||||
|
|
||||||
* **Idempotent.** A re-tick (or a process restart) must not re-parse
|
|
||||||
the same inbound file. We persist a ``processed_inbound_files`` row
|
|
||||||
per file and skip ones we've already seen.
|
|
||||||
|
|
||||||
* **Crash-safe.** If the parser raises or the DB write fails, the
|
|
||||||
scheduler logs the error, records an ``error`` row, and moves on.
|
|
||||||
The next tick continues from the next file.
|
|
||||||
|
|
||||||
* **Bounded blast radius.** A bad file must not stop the scheduler.
|
|
||||||
Each file is wrapped in try/except so a 999 parser crash doesn't
|
|
||||||
prevent us from processing the next inbound 835.
|
|
||||||
|
|
||||||
* **Operator-controlled.** The scheduler is OFF by default; the
|
|
||||||
operator must explicitly start it (``POST /api/admin/scheduler/start``
|
|
||||||
or ``CYCLONE_SCHEDULER_AUTOSTART=true``). When it's running, status
|
|
||||||
is exposed via ``GET /api/admin/scheduler/status``.
|
|
||||||
|
|
||||||
* **No threading.** We use ``asyncio.create_task`` + ``asyncio.sleep``
|
|
||||||
rather than APScheduler or threading because the rest of the
|
|
||||||
codebase is asyncio-native (FastAPI). The whole polling loop runs
|
|
||||||
in the FastAPI event loop on the main thread.
|
|
||||||
|
|
||||||
Compliance: SP16 is operational metadata only. Inbound file
|
|
||||||
processing is NOT part of the HIPAA audit chain (SP11) — an SFTP
|
|
||||||
outage shouldn't pollute the audit log with parser errors.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
import os
|
|
||||||
import traceback
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any, Callable, Optional
|
|
||||||
|
|
||||||
from sqlalchemy.exc import IntegrityError
|
|
||||||
|
|
||||||
from cyclone import db
|
|
||||||
from cyclone.store import store as cycl_store
|
|
||||||
from cyclone.audit_log import AuditEvent, append_event
|
|
||||||
from cyclone.clearhouse import InboundFile, SftpClient
|
|
||||||
from cyclone.db import ProcessedInboundFile
|
|
||||||
from cyclone.edi.filenames import parse_inbound_filename
|
|
||||||
from cyclone.inbox_state import apply_999_rejections
|
|
||||||
from cyclone.inbox_state_277ca import apply_277ca_rejections
|
|
||||||
from cyclone.providers import SftpBlock
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
# Status values for ProcessedInboundFile.status.
|
|
||||||
STATUS_OK = "ok"
|
|
||||||
STATUS_ERROR = "error"
|
|
||||||
STATUS_SKIPPED = "skipped"
|
|
||||||
STATUS_PENDING = "pending"
|
|
||||||
|
|
||||||
|
|
||||||
# File types we know how to route. The HCPF set is broader (270/271/
|
|
||||||
# 276/277/278/820/834/ENCR) but Cyclone's parser only covers the
|
|
||||||
# four below. Files with unknown types are recorded as ``skipped``
|
|
||||||
# so the operator can see them in the audit table.
|
|
||||||
ROUTED_FILE_TYPES = frozenset({"999", "835", "277", "277CA", "TA1"})
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class TickResult:
|
|
||||||
"""Outcome of a single scheduler tick (one poll cycle)."""
|
|
||||||
|
|
||||||
started_at: datetime
|
|
||||||
finished_at: Optional[datetime] = None
|
|
||||||
files_seen: int = 0
|
|
||||||
files_processed: int = 0
|
|
||||||
files_skipped: int = 0
|
|
||||||
files_errored: int = 0
|
|
||||||
errors: list[str] = field(default_factory=list)
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"started_at": self.started_at.isoformat(),
|
|
||||||
"finished_at": (
|
|
||||||
self.finished_at.isoformat() if self.finished_at else None
|
|
||||||
),
|
|
||||||
"files_seen": self.files_seen,
|
|
||||||
"files_processed": self.files_processed,
|
|
||||||
"files_skipped": self.files_skipped,
|
|
||||||
"files_errored": self.files_errored,
|
|
||||||
"errors": list(self.errors),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class SchedulerStatus:
|
|
||||||
"""Snapshot of the scheduler's runtime state."""
|
|
||||||
|
|
||||||
running: bool
|
|
||||||
poll_interval_seconds: int
|
|
||||||
sftp_block_name: str
|
|
||||||
last_poll_at: Optional[datetime]
|
|
||||||
poll_count: int
|
|
||||||
total_processed: int
|
|
||||||
total_skipped: int
|
|
||||||
total_errored: int
|
|
||||||
last_tick: Optional[TickResult] = None
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"running": self.running,
|
|
||||||
"poll_interval_seconds": self.poll_interval_seconds,
|
|
||||||
"sftp_block_name": self.sftp_block_name,
|
|
||||||
"last_poll_at": (
|
|
||||||
self.last_poll_at.isoformat() if self.last_poll_at else None
|
|
||||||
),
|
|
||||||
"poll_count": self.poll_count,
|
|
||||||
"total_processed": self.total_processed,
|
|
||||||
"total_skipped": self.total_skipped,
|
|
||||||
"total_errored": self.total_errored,
|
|
||||||
"last_tick": self.last_tick.as_dict() if self.last_tick else None,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Per-file-type handlers. Each returns (parser_name, claim_count) and
|
|
||||||
# persists its own DB rows. The scheduler records the outcome.
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _handle_999(text: str, source_file: str) -> tuple[str, int]:
|
|
||||||
"""Parse a 999, apply rejections, persist ack row. Returns (parser, count)."""
|
|
||||||
from cyclone.parsers.parse_999 import parse_999_text
|
|
||||||
from cyclone.parsers.exceptions import CycloneParseError
|
|
||||||
|
|
||||||
try:
|
|
||||||
result = parse_999_text(text, input_file=source_file)
|
|
||||||
except CycloneParseError as exc:
|
|
||||||
raise ValueError(f"999 parse error: {exc}") from exc
|
|
||||||
|
|
||||||
received, accepted, rejected, ack_code = _ack_count_summary(result)
|
|
||||||
icn = result.envelope.control_number
|
|
||||||
synthetic_id = _ack_synthetic_source_batch_id(icn)
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
def _lookup(pcn: str):
|
|
||||||
return (
|
|
||||||
session.query(db.Claim)
|
|
||||||
.filter_by(patient_control_number=pcn)
|
|
||||||
.first()
|
|
||||||
)
|
|
||||||
rejection_result = apply_999_rejections(
|
|
||||||
session, result, claim_lookup=_lookup,
|
|
||||||
)
|
|
||||||
if rejection_result.matched:
|
|
||||||
for cid in rejection_result.matched:
|
|
||||||
append_event(session, AuditEvent(
|
|
||||||
event_type="claim.rejected",
|
|
||||||
entity_type="claim",
|
|
||||||
entity_id=cid,
|
|
||||||
payload={"source_batch_id": synthetic_id},
|
|
||||||
actor="999-parser-scheduler",
|
|
||||||
))
|
|
||||||
row = cycl_store.add_ack(
|
|
||||||
source_batch_id=synthetic_id,
|
|
||||||
accepted_count=accepted,
|
|
||||||
rejected_count=rejected,
|
|
||||||
received_count=received,
|
|
||||||
ack_code=ack_code,
|
|
||||||
raw_json=json.loads(result.model_dump_json()),
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
return "parse_999", received
|
|
||||||
|
|
||||||
|
|
||||||
def _handle_835(text: str, source_file: str) -> tuple[str, int]:
|
|
||||||
"""Parse an 835, run validation, persist batch + remittances."""
|
|
||||||
import uuid
|
|
||||||
from cyclone.parsers.parse_835 import parse as parse_835
|
|
||||||
from cyclone.parsers.exceptions import CycloneParseError
|
|
||||||
from cyclone.parsers.validator_835 import validate as validate_835
|
|
||||||
from cyclone.payers import PAYER_FACTORIES_835
|
|
||||||
from cyclone.store import BatchRecord
|
|
||||||
|
|
||||||
config = PAYER_FACTORIES_835["co_medicaid_835"]()
|
|
||||||
try:
|
|
||||||
result = parse_835(text, config, input_file=source_file)
|
|
||||||
except CycloneParseError as exc:
|
|
||||||
raise ValueError(f"835 parse error: {exc}") from exc
|
|
||||||
|
|
||||||
# Validation report (mirrors the API endpoint).
|
|
||||||
report = validate_835(result, config)
|
|
||||||
n = len(result.claims)
|
|
||||||
if report.passed:
|
|
||||||
passed, failed, failed_claim_ids = n, 0, []
|
|
||||||
else:
|
|
||||||
passed, failed, failed_claim_ids = 0, n, [
|
|
||||||
c.payer_claim_control_number for c in result.claims
|
|
||||||
]
|
|
||||||
result = result.model_copy(update={
|
|
||||||
"validation": report,
|
|
||||||
"summary": result.summary.model_copy(update={
|
|
||||||
"passed": passed,
|
|
||||||
"failed": failed,
|
|
||||||
"failed_claim_ids": failed_claim_ids,
|
|
||||||
}),
|
|
||||||
})
|
|
||||||
|
|
||||||
rec = BatchRecord(
|
|
||||||
id=uuid.uuid4().hex,
|
|
||||||
kind="835",
|
|
||||||
input_filename=source_file,
|
|
||||||
parsed_at=datetime.now(timezone.utc),
|
|
||||||
result=result,
|
|
||||||
)
|
|
||||||
cycl_store.add(rec)
|
|
||||||
return "parse_835", len(result.claims)
|
|
||||||
|
|
||||||
|
|
||||||
def _handle_277ca(text: str, source_file: str) -> tuple[str, int]:
|
|
||||||
"""Parse a 277CA, persist ack + stamp payer-rejected claims."""
|
|
||||||
from cyclone.parsers.parse_277ca import parse_277ca_text
|
|
||||||
from cyclone.parsers.exceptions import CycloneParseError
|
|
||||||
|
|
||||||
try:
|
|
||||||
result = parse_277ca_text(text, input_file=source_file)
|
|
||||||
except CycloneParseError as exc:
|
|
||||||
raise ValueError(f"277CA parse error: {exc}") from exc
|
|
||||||
|
|
||||||
icn = result.envelope.control_number
|
|
||||||
synthetic_id = _277ca_synthetic_source_batch_id(icn)
|
|
||||||
accepted = sum(
|
|
||||||
1 for s in result.claim_statuses if s.classification == "accepted"
|
|
||||||
)
|
|
||||||
paid = sum(
|
|
||||||
1 for s in result.claim_statuses if s.classification == "paid"
|
|
||||||
)
|
|
||||||
rejected = sum(
|
|
||||||
1 for s in result.claim_statuses if s.classification == "rejected"
|
|
||||||
)
|
|
||||||
pended = sum(
|
|
||||||
1 for s in result.claim_statuses if s.classification == "pended"
|
|
||||||
)
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
row = cycl_store.add_277ca_ack(
|
|
||||||
source_batch_id=synthetic_id,
|
|
||||||
control_number=icn,
|
|
||||||
accepted_count=accepted,
|
|
||||||
rejected_count=rejected,
|
|
||||||
paid_count=paid,
|
|
||||||
pended_count=pended,
|
|
||||||
raw_json=json.loads(result.model_dump_json()),
|
|
||||||
)
|
|
||||||
def _lookup(pcn: str):
|
|
||||||
return (
|
|
||||||
session.query(db.Claim)
|
|
||||||
.filter(db.Claim.patient_control_number == pcn)
|
|
||||||
.first()
|
|
||||||
)
|
|
||||||
apply_result = apply_277ca_rejections(
|
|
||||||
session, result, claim_lookup=_lookup, two77ca_id=row.id,
|
|
||||||
)
|
|
||||||
if apply_result.matched:
|
|
||||||
for cid in apply_result.matched:
|
|
||||||
append_event(session, AuditEvent(
|
|
||||||
event_type="claim.payer_rejected",
|
|
||||||
entity_type="claim",
|
|
||||||
entity_id=cid,
|
|
||||||
payload={"source_batch_id": synthetic_id, "277ca_id": row.id},
|
|
||||||
actor="277ca-parser-scheduler",
|
|
||||||
))
|
|
||||||
session.commit()
|
|
||||||
return "parse_277ca", len(result.claim_statuses)
|
|
||||||
|
|
||||||
|
|
||||||
def _handle_ta1(text: str, source_file: str) -> tuple[str, int]:
|
|
||||||
"""Parse a TA1, persist the interchange ack row."""
|
|
||||||
from cyclone.parsers.parse_ta1 import parse_ta1_text
|
|
||||||
from cyclone.parsers.exceptions import CycloneParseError
|
|
||||||
|
|
||||||
try:
|
|
||||||
result = parse_ta1_text(text, input_file=source_file)
|
|
||||||
except CycloneParseError as exc:
|
|
||||||
raise ValueError(f"TA1 parse error: {exc}") from exc
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
cycl_store.add_ta1_ack(
|
|
||||||
source_batch_id=result.source_batch_id,
|
|
||||||
control_number=result.ta1.control_number,
|
|
||||||
interchange_date=result.ta1.interchange_date,
|
|
||||||
interchange_time=result.ta1.interchange_time,
|
|
||||||
ack_code=result.ta1.ack_code,
|
|
||||||
note_code=result.ta1.note_code,
|
|
||||||
ack_generated_date=result.ta1.ack_generated_date,
|
|
||||||
sender_id=result.envelope.sender_id,
|
|
||||||
receiver_id=result.envelope.receiver_id,
|
|
||||||
raw_json=json.loads(result.model_dump_json()),
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
return "parse_ta1", 1
|
|
||||||
|
|
||||||
|
|
||||||
# Map file_type → handler. Mirrors ROUTED_FILE_TYPES.
|
|
||||||
HANDLERS: dict[str, Callable[[str, str], tuple[str, int]]] = {
|
|
||||||
"999": _handle_999,
|
|
||||||
"835": _handle_835,
|
|
||||||
"277": _handle_277ca, # filename uses 277; parser is the same
|
|
||||||
"277CA": _handle_277ca,
|
|
||||||
"TA1": _handle_ta1,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Light copies of helpers the API endpoints use, so the scheduler can
|
|
||||||
# run without depending on the FastAPI module.
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _ack_count_summary(result: Any) -> tuple[int, int, int, str]:
|
|
||||||
"""Return (received, accepted, rejected, ack_code) for a 999.
|
|
||||||
|
|
||||||
Mirrors the logic in ``cyclone.api._ack_count_summary`` but lives
|
|
||||||
here so the scheduler can run without importing the API module.
|
|
||||||
"""
|
|
||||||
if result.functional_group_acks:
|
|
||||||
fg = result.functional_group_acks[0]
|
|
||||||
return (
|
|
||||||
fg.received_count, fg.accepted_count,
|
|
||||||
fg.rejected_count, fg.ack_code,
|
|
||||||
)
|
|
||||||
sets = result.set_responses
|
|
||||||
received = len(sets)
|
|
||||||
accepted = sum(1 for s in sets if s.set_accept_reject.code == "A")
|
|
||||||
rejected = received - accepted
|
|
||||||
if rejected == 0:
|
|
||||||
code = "A"
|
|
||||||
elif accepted == 0:
|
|
||||||
code = "R"
|
|
||||||
else:
|
|
||||||
code = "P"
|
|
||||||
return (received, accepted, rejected, code)
|
|
||||||
|
|
||||||
|
|
||||||
def _ack_synthetic_source_batch_id(interchange_control_number: str) -> str:
|
|
||||||
"""Synthetic batches.id for a received 999 with no source batch."""
|
|
||||||
return f"999-{(interchange_control_number or '').strip() or '000000001'}"
|
|
||||||
|
|
||||||
|
|
||||||
def _277ca_synthetic_source_batch_id(interchange_control_number: str) -> str:
|
|
||||||
"""Synthetic batches.id for a received 277CA with no source batch."""
|
|
||||||
return f"277CA-{(interchange_control_number or '').strip() or '000000001'}"
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Scheduler
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
class Scheduler:
|
|
||||||
"""Background polling loop for inbound MFT files.
|
|
||||||
|
|
||||||
Lifecycle:
|
|
||||||
sched = Scheduler(sftp_block, poll_interval_seconds=60)
|
|
||||||
await sched.start() # begin polling
|
|
||||||
# ... later ...
|
|
||||||
await sched.stop() # finish current tick, then exit
|
|
||||||
status = sched.status() # snapshot
|
|
||||||
|
|
||||||
The scheduler is a single asyncio task. ``tick()`` does one full
|
|
||||||
poll cycle and is exposed for tests + the ``/api/admin/scheduler/tick``
|
|
||||||
endpoint so the operator can force a poll without waiting.
|
|
||||||
|
|
||||||
Threading: NOT thread-safe. All access (start/stop/tick/status)
|
|
||||||
must happen on the same event loop. The FastAPI app satisfies
|
|
||||||
this trivially because endpoints run on the loop.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
sftp_block: SftpBlock,
|
|
||||||
*,
|
|
||||||
poll_interval_seconds: int = 60,
|
|
||||||
sftp_block_name: str = "default",
|
|
||||||
sftp_client_factory: Optional[Callable[[SftpBlock], Any]] = None,
|
|
||||||
) -> None:
|
|
||||||
self._sftp_block = sftp_block
|
|
||||||
self._poll_interval = poll_interval_seconds
|
|
||||||
self._sftp_block_name = sftp_block_name
|
|
||||||
# Factory indirection lets tests substitute a fake client
|
|
||||||
# without monkey-patching the module-level SftpClient.
|
|
||||||
self._sftp_client_factory = sftp_client_factory or SftpClient
|
|
||||||
|
|
||||||
self._task: Optional[asyncio.Task[None]] = None
|
|
||||||
self._stop_event = asyncio.Event()
|
|
||||||
self._last_poll_at: Optional[datetime] = None
|
|
||||||
self._poll_count = 0
|
|
||||||
self._total_processed = 0
|
|
||||||
self._total_skipped = 0
|
|
||||||
self._total_errored = 0
|
|
||||||
self._last_tick: Optional[TickResult] = None
|
|
||||||
# Coalesce overlapping ticks (a slow MFT server shouldn't let
|
|
||||||
# ticks stack up; the next tick fires only after the previous
|
|
||||||
# one finishes).
|
|
||||||
self._tick_in_progress = False
|
|
||||||
|
|
||||||
# ---- Public API -------------------------------------------------------
|
|
||||||
|
|
||||||
async def start(self) -> None:
|
|
||||||
"""Begin polling. Idempotent."""
|
|
||||||
if self._task is not None and not self._task.done():
|
|
||||||
log.info("Scheduler already running; start() is a no-op")
|
|
||||||
return
|
|
||||||
self._stop_event.clear()
|
|
||||||
self._task = asyncio.create_task(self._run(), name="mft-scheduler")
|
|
||||||
log.info(
|
|
||||||
"Scheduler started",
|
|
||||||
extra={
|
|
||||||
"poll_interval_s": self._poll_interval,
|
|
||||||
"sftp_block": self._sftp_block_name,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
async def stop(self) -> None:
|
|
||||||
"""Stop polling. Waits for the current tick to finish."""
|
|
||||||
if self._task is None or self._task.done():
|
|
||||||
return
|
|
||||||
self._stop_event.set()
|
|
||||||
try:
|
|
||||||
await asyncio.wait_for(self._task, timeout=30)
|
|
||||||
except asyncio.TimeoutError:
|
|
||||||
log.warning("Scheduler did not stop within 30s; cancelling")
|
|
||||||
self._task.cancel()
|
|
||||||
try:
|
|
||||||
await self._task
|
|
||||||
except (asyncio.CancelledError, Exception): # noqa: BLE001
|
|
||||||
pass
|
|
||||||
self._task = None
|
|
||||||
log.info("Scheduler stopped")
|
|
||||||
|
|
||||||
def status(self) -> SchedulerStatus:
|
|
||||||
"""Return a snapshot of the scheduler's state."""
|
|
||||||
return SchedulerStatus(
|
|
||||||
running=self.is_running(),
|
|
||||||
poll_interval_seconds=self._poll_interval,
|
|
||||||
sftp_block_name=self._sftp_block_name,
|
|
||||||
last_poll_at=self._last_poll_at,
|
|
||||||
poll_count=self._poll_count,
|
|
||||||
total_processed=self._total_processed,
|
|
||||||
total_skipped=self._total_skipped,
|
|
||||||
total_errored=self._total_errored,
|
|
||||||
last_tick=self._last_tick,
|
|
||||||
)
|
|
||||||
|
|
||||||
def is_running(self) -> bool:
|
|
||||||
return self._task is not None and not self._task.done()
|
|
||||||
|
|
||||||
async def tick(self) -> TickResult:
|
|
||||||
"""Run a single poll cycle and return the outcome.
|
|
||||||
|
|
||||||
Concurrent ticks are coalesced: if a tick is already in
|
|
||||||
progress, the second caller waits for it. This protects the
|
|
||||||
SFTP server from a stampede when the operator hits
|
|
||||||
``/api/admin/scheduler/tick`` while a scheduled tick is
|
|
||||||
already running.
|
|
||||||
"""
|
|
||||||
while self._tick_in_progress:
|
|
||||||
await asyncio.sleep(0.05)
|
|
||||||
self._tick_in_progress = True
|
|
||||||
try:
|
|
||||||
result = await self._tick_impl()
|
|
||||||
self._last_tick = result
|
|
||||||
self._last_poll_at = result.finished_at or result.started_at
|
|
||||||
self._poll_count += 1
|
|
||||||
self._total_processed += result.files_processed
|
|
||||||
self._total_skipped += result.files_skipped
|
|
||||||
self._total_errored += result.files_errored
|
|
||||||
return result
|
|
||||||
finally:
|
|
||||||
self._tick_in_progress = False
|
|
||||||
|
|
||||||
# ---- Internals --------------------------------------------------------
|
|
||||||
|
|
||||||
async def _run(self) -> None:
|
|
||||||
"""Main loop. Runs until ``stop()`` is called."""
|
|
||||||
# Stagger the first tick so we don't hammer the MFT server on
|
|
||||||
# startup if multiple operators restart Cyclone in lockstep.
|
|
||||||
await asyncio.sleep(1)
|
|
||||||
while not self._stop_event.is_set():
|
|
||||||
try:
|
|
||||||
await self.tick()
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
# tick() should never raise — it catches per-file
|
|
||||||
# exceptions. This is the safety net for SFTP outages
|
|
||||||
# or DB connectivity issues.
|
|
||||||
log.exception("Scheduler tick raised", extra={"error": str(exc)})
|
|
||||||
try:
|
|
||||||
await asyncio.wait_for(
|
|
||||||
self._stop_event.wait(),
|
|
||||||
timeout=self._poll_interval,
|
|
||||||
)
|
|
||||||
except asyncio.TimeoutError:
|
|
||||||
pass # poll interval elapsed; loop again
|
|
||||||
|
|
||||||
async def _tick_impl(self) -> TickResult:
|
|
||||||
"""One poll cycle: list → filter already-processed → route each."""
|
|
||||||
started = datetime.now(timezone.utc)
|
|
||||||
result = TickResult(started_at=started)
|
|
||||||
|
|
||||||
try:
|
|
||||||
files = await asyncio.to_thread(self._list_inbound)
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
log.exception("SFTP list_inbound failed")
|
|
||||||
result.errors.append(f"list_inbound: {exc}")
|
|
||||||
result.finished_at = datetime.now(timezone.utc)
|
|
||||||
return result
|
|
||||||
|
|
||||||
result.files_seen = len(files)
|
|
||||||
for f in files:
|
|
||||||
if self._stop_event.is_set():
|
|
||||||
break
|
|
||||||
await self._handle_one(f, result)
|
|
||||||
|
|
||||||
result.finished_at = datetime.now(timezone.utc)
|
|
||||||
return result
|
|
||||||
|
|
||||||
def _list_inbound(self) -> list[InboundFile]:
|
|
||||||
"""Return files in the inbound MFT path. Runs on a thread."""
|
|
||||||
client = self._sftp_client_factory(self._sftp_block)
|
|
||||||
return client.list_inbound()
|
|
||||||
|
|
||||||
async def _handle_one(self, f: InboundFile, result: TickResult) -> None:
|
|
||||||
"""Process one inbound file: skip-if-seen, classify, parse, record."""
|
|
||||||
if await self._already_processed(f.name):
|
|
||||||
return
|
|
||||||
|
|
||||||
try:
|
|
||||||
inbound = parse_inbound_filename(f.name)
|
|
||||||
file_type = inbound.file_type
|
|
||||||
except ValueError:
|
|
||||||
file_type = None
|
|
||||||
|
|
||||||
if file_type not in HANDLERS:
|
|
||||||
await self._record(
|
|
||||||
name=f.name, size=f.size, modified_at=f.modified_at,
|
|
||||||
file_type=file_type, parser_used=None, claim_count=0,
|
|
||||||
status=STATUS_SKIPPED,
|
|
||||||
error_message=(
|
|
||||||
f"file_type {file_type!r} not in {sorted(HANDLERS)}"
|
|
||||||
if file_type else "filename does not match HCPF inbound format"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
result.files_skipped += 1
|
|
||||||
return
|
|
||||||
|
|
||||||
try:
|
|
||||||
_path, parser_used, claim_count = await asyncio.to_thread(
|
|
||||||
self._download_and_parse, f, file_type,
|
|
||||||
)
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
log.exception("Failed to process inbound file", extra={"input_filename": f.name})
|
|
||||||
await self._record(
|
|
||||||
name=f.name, size=f.size, modified_at=f.modified_at,
|
|
||||||
file_type=file_type, parser_used=None, claim_count=0,
|
|
||||||
status=STATUS_ERROR,
|
|
||||||
error_message=(
|
|
||||||
f"{type(exc).__name__}: {exc}\n"
|
|
||||||
f"{traceback.format_exc()[-500:]}"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
result.files_errored += 1
|
|
||||||
result.errors.append(f"{f.name}: {exc}")
|
|
||||||
return
|
|
||||||
|
|
||||||
await self._record(
|
|
||||||
name=f.name, size=f.size, modified_at=f.modified_at,
|
|
||||||
file_type=file_type, parser_used=parser_used, claim_count=claim_count,
|
|
||||||
status=STATUS_OK, error_message=None,
|
|
||||||
)
|
|
||||||
result.files_processed += 1
|
|
||||||
log.info(
|
|
||||||
"Processed inbound file",
|
|
||||||
extra={
|
|
||||||
"input_filename": f.name,
|
|
||||||
"parser": parser_used,
|
|
||||||
"claims": claim_count,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
|
|
||||||
async def _already_processed(self, name: str) -> bool:
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
row = (
|
|
||||||
session.query(ProcessedInboundFile)
|
|
||||||
.filter_by(sftp_block_name=self._sftp_block_name, name=name)
|
|
||||||
.filter(ProcessedInboundFile.status != STATUS_PENDING)
|
|
||||||
.first()
|
|
||||||
)
|
|
||||||
return row is not None
|
|
||||||
|
|
||||||
async def _record(
|
|
||||||
self,
|
|
||||||
*,
|
|
||||||
name: str,
|
|
||||||
size: int,
|
|
||||||
modified_at: datetime,
|
|
||||||
file_type: Optional[str],
|
|
||||||
parser_used: Optional[str],
|
|
||||||
claim_count: int,
|
|
||||||
status: str,
|
|
||||||
error_message: Optional[str],
|
|
||||||
) -> None:
|
|
||||||
"""Persist a processed_inbound_files row. Idempotent."""
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
row = ProcessedInboundFile(
|
|
||||||
sftp_block_name=self._sftp_block_name,
|
|
||||||
name=name,
|
|
||||||
size=size,
|
|
||||||
modified_at=modified_at,
|
|
||||||
file_type=file_type,
|
|
||||||
processed_at=datetime.now(timezone.utc),
|
|
||||||
parser_used=parser_used,
|
|
||||||
claim_count=claim_count,
|
|
||||||
status=status,
|
|
||||||
error_message=error_message,
|
|
||||||
)
|
|
||||||
session.add(row)
|
|
||||||
try:
|
|
||||||
session.commit()
|
|
||||||
except IntegrityError:
|
|
||||||
# A concurrent scheduler (or a retry after a partial
|
|
||||||
# failure) already recorded this file. That's fine —
|
|
||||||
# the latest row wins; we just skip the dup.
|
|
||||||
session.rollback()
|
|
||||||
|
|
||||||
def _download_and_parse(
|
|
||||||
self, f: InboundFile, file_type: str,
|
|
||||||
) -> tuple[Path, str, int]:
|
|
||||||
"""Download from MFT, run the right handler. Returns (path, parser, count).
|
|
||||||
|
|
||||||
Stub mode: ``f.local_path`` already points at the staged file
|
|
||||||
(set by ``SftpClient._list_inbound_stub``). Real mode: the
|
|
||||||
remote name is ``f.name`` and we round-trip through paramiko.
|
|
||||||
"""
|
|
||||||
if self._sftp_block.stub:
|
|
||||||
# In stub mode the InboundFile already has a local_path;
|
|
||||||
# reading the staged bytes directly avoids the stub's
|
|
||||||
# remote-path semantics (which expect a full inbound path).
|
|
||||||
content = f.local_path.read_bytes()
|
|
||||||
else:
|
|
||||||
client = self._sftp_client_factory(self._sftp_block)
|
|
||||||
content = client.read_file(f.name)
|
|
||||||
text = content.decode("utf-8")
|
|
||||||
handler = HANDLERS[file_type]
|
|
||||||
parser_used, claim_count = handler(text, f.name)
|
|
||||||
return f.local_path, parser_used, claim_count
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Module-level singleton — only one scheduler per process.
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
_scheduler: Optional[Scheduler] = None
|
|
||||||
|
|
||||||
|
|
||||||
def configure_scheduler(
|
|
||||||
sftp_block: SftpBlock,
|
|
||||||
*,
|
|
||||||
poll_interval_seconds: int = 60,
|
|
||||||
sftp_block_name: str = "default",
|
|
||||||
force: bool = False,
|
|
||||||
) -> Scheduler:
|
|
||||||
"""Create the module-level scheduler singleton (or return the existing one).
|
|
||||||
|
|
||||||
Called from the FastAPI lifespan handler. Tests pre-configure the
|
|
||||||
scheduler before the TestClient opens the lifespan; in that case
|
|
||||||
we leave the existing singleton alone (``force=False``). Pass
|
|
||||||
``force=True`` to replace unconditionally.
|
|
||||||
"""
|
|
||||||
global _scheduler
|
|
||||||
if _scheduler is not None and not force:
|
|
||||||
return _scheduler
|
|
||||||
poll = int(
|
|
||||||
os.environ.get("CYCLONE_SCHEDULER_POLL_SECONDS", poll_interval_seconds),
|
|
||||||
)
|
|
||||||
_scheduler = Scheduler(
|
|
||||||
sftp_block,
|
|
||||||
poll_interval_seconds=poll,
|
|
||||||
sftp_block_name=sftp_block_name,
|
|
||||||
)
|
|
||||||
return _scheduler
|
|
||||||
|
|
||||||
|
|
||||||
def get_scheduler() -> Scheduler:
|
|
||||||
"""Return the module-level scheduler.
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
RuntimeError: if ``configure_scheduler`` hasn't been called.
|
|
||||||
"""
|
|
||||||
if _scheduler is None:
|
|
||||||
raise RuntimeError(
|
|
||||||
"scheduler not configured; call configure_scheduler() first",
|
|
||||||
)
|
|
||||||
return _scheduler
|
|
||||||
|
|
||||||
|
|
||||||
def reset_scheduler_for_tests() -> None:
|
|
||||||
"""Clear the module-level scheduler. Test-only."""
|
|
||||||
global _scheduler
|
|
||||||
_scheduler = None
|
|
||||||
@@ -1,485 +0,0 @@
|
|||||||
"""SP19 — Security middleware + health probe.
|
|
||||||
|
|
||||||
Three concrete middlewares (body size, rate limit, security headers)
|
|
||||||
plus a richer ``/api/health`` snapshot. Sizing is for Cyclone's
|
|
||||||
local-only posture: a misconfigured Tailscale / ngrok bind, a
|
|
||||||
misbehaving cron job, a port-scanner scraping the API. Anything more
|
|
||||||
aggressive (auth, mTLS, WAF) is out of scope.
|
|
||||||
|
|
||||||
Design choices
|
|
||||||
--------------
|
|
||||||
|
|
||||||
* **In-memory rate limiter.** Cyclone is single-process; a dict
|
|
||||||
keyed by IP is enough. If we ever go multi-worker, swap for
|
|
||||||
Redis. The rate-limit counter resets after the bucket window;
|
|
||||||
failing open on the limiter itself (an unexpected exception)
|
|
||||||
rather than 503ing every request is the right call for a local tool.
|
|
||||||
|
|
||||||
* **Body-size check by Content-Length first, then chunked-read
|
|
||||||
guard.** A chunked POST can lie about its size (or omit the
|
|
||||||
header entirely); we cap read body size on the underlying stream
|
|
||||||
so a malicious client can't keep streaming forever.
|
|
||||||
|
|
||||||
* **Security headers on every response.** CSP locks the API to
|
|
||||||
same-origin + the Vite dev origin (whitelisted explicitly so a
|
|
||||||
future operator running on a different port doesn't break).
|
|
||||||
|
|
||||||
* **Health snapshot is best-effort.** Each subsystem (DB,
|
|
||||||
scheduler, pubsub) reports independently — a DB outage doesn't
|
|
||||||
blank out the rest. ``status: "degraded"`` if any subsystem is
|
|
||||||
unhappy; ``"ok"`` only when everything is.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
import os
|
|
||||||
import threading
|
|
||||||
import time
|
|
||||||
from collections import deque
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from typing import Any, Callable
|
|
||||||
|
|
||||||
from fastapi import Request, Response
|
|
||||||
from fastapi.responses import JSONResponse
|
|
||||||
from starlette.types import ASGIApp, Message, Receive, Scope, Send
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Knobs (env-var driven)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
DEFAULT_MAX_BODY_BYTES = 50 * 1024 * 1024 # 50 MB — generous for X12 EDI
|
|
||||||
DEFAULT_RATE_LIMIT_PER_MIN = 300
|
|
||||||
DEFAULT_RATE_LIMIT_WINDOW_S = 60
|
|
||||||
|
|
||||||
# CSP: API responses are JSON, not HTML. ``default-src 'none'`` is the
|
|
||||||
# strictest setting; it forbids the API from being a vector for
|
|
||||||
# injected scripts in case an operator opens a JSON viewer with an
|
|
||||||
# HTML renderer.
|
|
||||||
_SECURITY_HEADERS: dict[str, str] = {
|
|
||||||
"X-Content-Type-Options": "nosniff",
|
|
||||||
"X-Frame-Options": "DENY",
|
|
||||||
"Referrer-Policy": "same-origin",
|
|
||||||
"Permissions-Policy": "geolocation=(), microphone=(), camera=()",
|
|
||||||
"Content-Security-Policy": "default-src 'none'; frame-ancestors 'none'",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _env_int(name: str, default: int) -> int:
|
|
||||||
raw = os.environ.get(name)
|
|
||||||
if not raw:
|
|
||||||
return default
|
|
||||||
try:
|
|
||||||
return int(raw)
|
|
||||||
except ValueError:
|
|
||||||
log.warning("SP19: %s=%r is not an int; using default %d", name, raw, default)
|
|
||||||
return default
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Body-size middleware
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
class BodySizeLimitMiddleware:
|
|
||||||
"""Reject requests whose body exceeds ``max_bytes``.
|
|
||||||
|
|
||||||
Pure ASGI middleware (not BaseHTTPMiddleware — that one breaks
|
|
||||||
FastAPI's ``request.body()`` introspection). Two-stage guard:
|
|
||||||
1. If the request declares a ``Content-Length`` larger than
|
|
||||||
``max_bytes``, reject immediately with ``413``.
|
|
||||||
2. While reading the body chunks, cap accumulated bytes at
|
|
||||||
``max_bytes``. If we cross the cap, return 413 instead of
|
|
||||||
letting the handler read the rest.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, app: ASGIApp, max_bytes: int | None = None) -> None:
|
|
||||||
self.app = app
|
|
||||||
self.max_bytes = max_bytes or _env_int(
|
|
||||||
"CYCLONE_MAX_BODY_BYTES", DEFAULT_MAX_BODY_BYTES,
|
|
||||||
)
|
|
||||||
|
|
||||||
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
|
|
||||||
if scope["type"] != "http":
|
|
||||||
await self.app(scope, receive, send)
|
|
||||||
return
|
|
||||||
|
|
||||||
# Stage 1: declared length.
|
|
||||||
cl_header = None
|
|
||||||
for k, v in scope.get("headers", []):
|
|
||||||
if k == b"content-length":
|
|
||||||
cl_header = v.decode("latin-1")
|
|
||||||
break
|
|
||||||
if cl_header is not None:
|
|
||||||
try:
|
|
||||||
if int(cl_header) > self.max_bytes:
|
|
||||||
await _send_rejection(
|
|
||||||
scope, send,
|
|
||||||
code=413,
|
|
||||||
reason="body_too_large",
|
|
||||||
detail=f"Content-Length {cl_header} exceeds limit {self.max_bytes}",
|
|
||||||
)
|
|
||||||
return
|
|
||||||
except ValueError:
|
|
||||||
await _send_rejection(
|
|
||||||
scope, send,
|
|
||||||
code=400, reason="bad_content_length",
|
|
||||||
detail=f"Content-Length {cl_header!r} is not an integer",
|
|
||||||
)
|
|
||||||
return
|
|
||||||
|
|
||||||
# Stage 2: chunked read guard.
|
|
||||||
seen = 0
|
|
||||||
over_limit = False
|
|
||||||
|
|
||||||
async def wrapped_receive() -> Message:
|
|
||||||
nonlocal seen, over_limit
|
|
||||||
if over_limit:
|
|
||||||
# Drain any remaining bytes so the upstream ASGI
|
|
||||||
# server doesn't see a truncated stream.
|
|
||||||
msg = await receive()
|
|
||||||
if msg.get("type") == "http.request":
|
|
||||||
return {"type": "http.request", "body": b"", "more_body": False}
|
|
||||||
return msg
|
|
||||||
msg = await receive()
|
|
||||||
if msg.get("type") == "http.request":
|
|
||||||
body = msg.get("body", b"") or b""
|
|
||||||
seen += len(body)
|
|
||||||
if seen > self.max_bytes:
|
|
||||||
over_limit = True
|
|
||||||
return {"type": "http.request", "body": b"", "more_body": False}
|
|
||||||
return msg
|
|
||||||
|
|
||||||
if cl_header is None:
|
|
||||||
# Chunked / unknown length — guard with wrapped receive.
|
|
||||||
await self.app(scope, wrapped_receive, send)
|
|
||||||
else:
|
|
||||||
# Fixed-length known to be safe; pass through.
|
|
||||||
await self.app(scope, receive, send)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Rate-limit middleware
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class _Bucket:
|
|
||||||
"""Sliding-window counter for one IP."""
|
|
||||||
timestamps: deque = field(default_factory=deque)
|
|
||||||
|
|
||||||
def hit(self, window_s: int, now: float) -> bool:
|
|
||||||
"""Record one hit; return True if under the limit, False if over."""
|
|
||||||
# Drop expired entries.
|
|
||||||
cutoff = now - window_s
|
|
||||||
while self.timestamps and self.timestamps[0] < cutoff:
|
|
||||||
self.timestamps.popleft()
|
|
||||||
return True # we always record; the dispatcher decides to reject
|
|
||||||
|
|
||||||
def count_in_window(self, now: float, window_s: int) -> int:
|
|
||||||
cutoff = now - window_s
|
|
||||||
while self.timestamps and self.timestamps[0] < cutoff:
|
|
||||||
self.timestamps.popleft()
|
|
||||||
return len(self.timestamps)
|
|
||||||
|
|
||||||
|
|
||||||
class RateLimitMiddleware:
|
|
||||||
"""Per-IP sliding-window rate limiter (pure ASGI).
|
|
||||||
|
|
||||||
Defaults to ``CYCLONE_RATE_LIMIT_PER_MIN`` requests/minute per IP.
|
|
||||||
Health-check probes and the ``/api/health`` endpoint are exempt
|
|
||||||
so a load balancer's frequent probes don't trip the limiter.
|
|
||||||
|
|
||||||
On unexpected errors the limiter fails OPEN — better to serve a
|
|
||||||
few extra requests than to 503 every request because of a bug.
|
|
||||||
"""
|
|
||||||
|
|
||||||
EXEMPT_PATHS = ("/api/health", "/healthz", "/readyz")
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
app: ASGIApp,
|
|
||||||
per_minute: int | None = None,
|
|
||||||
window_s: int | None = None,
|
|
||||||
) -> None:
|
|
||||||
self.app = app
|
|
||||||
self.per_minute = per_minute or _env_int(
|
|
||||||
"CYCLONE_RATE_LIMIT_PER_MIN", DEFAULT_RATE_LIMIT_PER_MIN,
|
|
||||||
)
|
|
||||||
self.window_s = window_s or DEFAULT_RATE_LIMIT_WINDOW_S
|
|
||||||
self._buckets: dict[str, _Bucket] = {}
|
|
||||||
self._lock = threading.Lock()
|
|
||||||
|
|
||||||
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
|
|
||||||
if scope["type"] != "http":
|
|
||||||
await self.app(scope, receive, send)
|
|
||||||
return
|
|
||||||
path = scope.get("path", "")
|
|
||||||
if path in self.EXEMPT_PATHS:
|
|
||||||
await self.app(scope, receive, send)
|
|
||||||
return
|
|
||||||
ip = _client_ip_from_scope(scope)
|
|
||||||
now = time.monotonic()
|
|
||||||
try:
|
|
||||||
with self._lock:
|
|
||||||
bucket = self._buckets.setdefault(ip, _Bucket())
|
|
||||||
bucket.timestamps.append(now)
|
|
||||||
count = bucket.count_in_window(now, self.window_s)
|
|
||||||
if count > self.per_minute:
|
|
||||||
await _send_rejection(
|
|
||||||
scope, send,
|
|
||||||
code=429,
|
|
||||||
reason="rate_limited",
|
|
||||||
detail=(
|
|
||||||
f"IP {ip} exceeded {self.per_minute} req/"
|
|
||||||
f"{self.window_s}s window"
|
|
||||||
),
|
|
||||||
)
|
|
||||||
return
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
log.warning("SP19: rate limiter failed open: %s", exc)
|
|
||||||
await self.app(scope, receive, send)
|
|
||||||
|
|
||||||
|
|
||||||
def _client_ip_from_scope(scope: Scope) -> str:
|
|
||||||
"""Best-effort client IP from the ASGI scope. Falls back to ``"unknown"``."""
|
|
||||||
for k, v in scope.get("headers", []):
|
|
||||||
if k == b"x-forwarded-for":
|
|
||||||
return v.decode("latin-1").split(",")[0].strip()
|
|
||||||
client = scope.get("client")
|
|
||||||
if client and client[0]:
|
|
||||||
return client[0]
|
|
||||||
return "unknown"
|
|
||||||
|
|
||||||
|
|
||||||
def _client_ip(request: Request) -> str:
|
|
||||||
"""Legacy helper (kept for the audit-event log path)."""
|
|
||||||
return _client_ip_from_scope(request.scope)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Security-headers middleware
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
class SecurityHeadersMiddleware:
|
|
||||||
"""Stamp the static security headers on every response (pure ASGI).
|
|
||||||
|
|
||||||
CSP / X-Content-Type-Options / X-Frame-Options / Referrer-Policy /
|
|
||||||
Permissions-Policy. The headers are static for now; per-route
|
|
||||||
overrides can be added later if a route needs to relax them.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, app: ASGIApp, extra: dict[str, str] | None = None) -> None:
|
|
||||||
self.app = app
|
|
||||||
self.headers = [(k.lower().encode("latin-1"), v.encode("latin-1"))
|
|
||||||
for k, v in _SECURITY_HEADERS.items()]
|
|
||||||
if extra:
|
|
||||||
self.headers.extend(
|
|
||||||
(k.lower().encode("latin-1"), v.encode("latin-1"))
|
|
||||||
for k, v in extra.items()
|
|
||||||
)
|
|
||||||
|
|
||||||
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
|
|
||||||
if scope["type"] != "http":
|
|
||||||
await self.app(scope, receive, send)
|
|
||||||
return
|
|
||||||
|
|
||||||
async def wrapped_send(message: Message) -> None:
|
|
||||||
if message["type"] == "http.response.start":
|
|
||||||
headers = list(message.get("headers", []))
|
|
||||||
existing = {k for k, _ in headers}
|
|
||||||
for k, v in self.headers:
|
|
||||||
if k not in existing:
|
|
||||||
headers.append((k, v))
|
|
||||||
message["headers"] = headers
|
|
||||||
await send(message)
|
|
||||||
|
|
||||||
await self.app(scope, receive, wrapped_send)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Reject helper (also writes an audit event when DB is available)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
async def _send_rejection(
|
|
||||||
scope: Scope,
|
|
||||||
send: Send,
|
|
||||||
*,
|
|
||||||
code: int,
|
|
||||||
reason: str,
|
|
||||||
detail: str,
|
|
||||||
) -> None:
|
|
||||||
"""Build a 413/429 JSON response, send it, and emit a log + audit event."""
|
|
||||||
method = scope.get("method", "GET")
|
|
||||||
path = scope.get("path", "/")
|
|
||||||
ip = _client_ip_from_scope(scope)
|
|
||||||
log.warning(
|
|
||||||
"api.request_rejected",
|
|
||||||
extra={
|
|
||||||
"status": code,
|
|
||||||
"reason": reason,
|
|
||||||
"path": path,
|
|
||||||
"method": method,
|
|
||||||
"ip": ip,
|
|
||||||
"detail": detail,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
payload = {"error": reason, "detail": detail, "status": code}
|
|
||||||
body = json.dumps(payload).encode("utf-8")
|
|
||||||
await send({
|
|
||||||
"type": "http.response.start",
|
|
||||||
"status": code,
|
|
||||||
"headers": [
|
|
||||||
(b"content-type", b"application/json"),
|
|
||||||
(b"content-length", str(len(body)).encode("latin-1")),
|
|
||||||
],
|
|
||||||
})
|
|
||||||
await send({"type": "http.response.body", "body": body, "more_body": False})
|
|
||||||
# Best-effort audit-log append. Don't block the response on a DB
|
|
||||||
# outage (the rejection is the more important signal anyway).
|
|
||||||
try:
|
|
||||||
from cyclone import db
|
|
||||||
from cyclone.audit_log import AuditEvent, append_event
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
append_event(
|
|
||||||
session,
|
|
||||||
AuditEvent(
|
|
||||||
event_type="api.request_rejected",
|
|
||||||
entity_type="http_request",
|
|
||||||
entity_id=f"{method} {path}",
|
|
||||||
payload={
|
|
||||||
"status": code,
|
|
||||||
"reason": reason,
|
|
||||||
"path": path,
|
|
||||||
"method": method,
|
|
||||||
"ip": ip,
|
|
||||||
},
|
|
||||||
actor=f"api:{ip}",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
session.commit()
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
log.debug("SP19: audit-log append failed for rejection: %s", exc)
|
|
||||||
|
|
||||||
|
|
||||||
def _reject(
|
|
||||||
request: Request,
|
|
||||||
*,
|
|
||||||
code: int,
|
|
||||||
reason: str,
|
|
||||||
detail: str,
|
|
||||||
) -> JSONResponse:
|
|
||||||
"""Sync helper kept for back-compat (the ``audit_log`` payload path)."""
|
|
||||||
return JSONResponse(
|
|
||||||
{"error": reason, "detail": detail, "status": code},
|
|
||||||
status_code=code,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Health snapshot
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class HealthSnapshot:
|
|
||||||
status: str
|
|
||||||
version: str
|
|
||||||
db: dict[str, Any] = field(default_factory=dict)
|
|
||||||
scheduler: dict[str, Any] = field(default_factory=dict)
|
|
||||||
pubsub: dict[str, Any] = field(default_factory=dict)
|
|
||||||
batch: dict[str, Any] = field(default_factory=dict)
|
|
||||||
|
|
||||||
def to_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"status": self.status,
|
|
||||||
"version": self.version,
|
|
||||||
"db": self.db,
|
|
||||||
"scheduler": self.scheduler,
|
|
||||||
"pubsub": self.pubsub,
|
|
||||||
"batch": self.batch,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def get_health_snapshot() -> HealthSnapshot:
|
|
||||||
"""Gather a best-effort snapshot of every Cyclone subsystem.
|
|
||||||
|
|
||||||
Returns ``HealthSnapshot`` with ``status="ok"`` only if every
|
|
||||||
subsystem check passes. ``"degraded"`` if any subsystem is
|
|
||||||
unhappy but the API itself is responsive. Each subsystem reports
|
|
||||||
independently so one outage doesn't blank out the rest.
|
|
||||||
"""
|
|
||||||
from cyclone import __version__, db
|
|
||||||
|
|
||||||
snap = HealthSnapshot(status="ok", version=__version__)
|
|
||||||
|
|
||||||
# DB connectivity.
|
|
||||||
try:
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
session.execute(db.text("SELECT 1"))
|
|
||||||
snap.db = {"ok": True}
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
snap.db = {"ok": False, "error": str(exc)}
|
|
||||||
snap.status = "degraded"
|
|
||||||
|
|
||||||
# Scheduler state.
|
|
||||||
try:
|
|
||||||
from cyclone import scheduler as scheduler_mod
|
|
||||||
sched = scheduler_mod.get_scheduler()
|
|
||||||
snap.scheduler = {
|
|
||||||
"running": sched.is_running(),
|
|
||||||
"interval_s": sched._poll_interval, # noqa: SLF001
|
|
||||||
"sftp_block": sched._sftp_block_name, # noqa: SLF001
|
|
||||||
}
|
|
||||||
except RuntimeError:
|
|
||||||
snap.scheduler = {"running": False, "configured": False}
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
snap.scheduler = {"ok": False, "error": str(exc)}
|
|
||||||
snap.status = "degraded"
|
|
||||||
|
|
||||||
# Backup scheduler.
|
|
||||||
try:
|
|
||||||
from cyclone import backup_scheduler as bks_mod
|
|
||||||
bks = bks_mod.get_backup_scheduler()
|
|
||||||
snap.scheduler["backup_scheduler_running"] = bks.is_running()
|
|
||||||
snap.scheduler["backup_interval_hours"] = bks.interval_hours
|
|
||||||
except (RuntimeError, ImportError):
|
|
||||||
snap.scheduler["backup_scheduler_running"] = False
|
|
||||||
except Exception: # noqa: BLE001
|
|
||||||
pass # secondary subsystem; don't degrade the overall status
|
|
||||||
|
|
||||||
# Pubsub bus stats — placeholder. The /api/health handler fills
|
|
||||||
# in the real subscriber counts using request.app.state.event_bus.
|
|
||||||
snap.pubsub = {"note": "filled in by health router"}
|
|
||||||
|
|
||||||
# Last batch timestamp + count.
|
|
||||||
try:
|
|
||||||
from cyclone import db as db_mod
|
|
||||||
from cyclone.db import Batch
|
|
||||||
with db_mod.SessionLocal()() as session:
|
|
||||||
row = (
|
|
||||||
session.query(Batch)
|
|
||||||
.order_by(Batch.parsed_at.desc())
|
|
||||||
.first()
|
|
||||||
)
|
|
||||||
if row is not None:
|
|
||||||
snap.batch = {
|
|
||||||
"last_batch_id": row.id,
|
|
||||||
"last_batch_kind": row.kind,
|
|
||||||
"last_batch_at": row.parsed_at.isoformat() if row.parsed_at else None,
|
|
||||||
"last_batch_filename": row.input_filename,
|
|
||||||
}
|
|
||||||
else:
|
|
||||||
snap.batch = {"last_batch_id": None, "note": "no batches yet"}
|
|
||||||
except Exception as exc: # noqa: BLE001
|
|
||||||
snap.batch = {"ok": False, "error": str(exc)}
|
|
||||||
snap.status = "degraded"
|
|
||||||
|
|
||||||
return snap
|
|
||||||
@@ -1,439 +0,0 @@
|
|||||||
"""CLI subcommand: ``python -m cyclone seed``.
|
|
||||||
|
|
||||||
Populates the local DB with a deterministic batch of sample claims
|
|
||||||
plus matching activity events, so the Dashboard / Claims / Activity
|
|
||||||
Log pages have something to render in a fresh dev environment.
|
|
||||||
|
|
||||||
This is a dev-only convenience — the production DB never sees this
|
|
||||||
command (no ``[env: dev]`` gate, but it's never wired into any
|
|
||||||
container image). It writes rows with a recognizable batch id prefix
|
|
||||||
(``SEED-``) so ``--reset`` can clean them up without touching real
|
|
||||||
ingested data.
|
|
||||||
|
|
||||||
Why the data shape is hand-rolled here
|
|
||||||
--------------------------------------
|
|
||||||
|
|
||||||
The ``Claim`` ORM table stores its billing_provider / payer /
|
|
||||||
subscriber / service_lines payloads in a ``raw_json`` blob that the
|
|
||||||
read path (``store.to_ui_claim_from_orm``) parses back into the UI
|
|
||||||
shape. Mirroring the 837 parser's structure keeps the UI rendering
|
|
||||||
identical to a real ingestion — wire format parity, not shortcuts.
|
|
||||||
|
|
||||||
Activity rows are similar: ``payload_json`` carries the message
|
|
||||||
string the Dashboard's activity card shows.
|
|
||||||
|
|
||||||
Subcommands
|
|
||||||
-----------
|
|
||||||
|
|
||||||
``python -m cyclone seed`` — insert the default batch.
|
|
||||||
``python -m cyclone seed --count N`` — insert N claims (default 96).
|
|
||||||
``python -m cyclone seed --reset`` — wipe previously-seeded rows
|
|
||||||
first, then insert a fresh batch.
|
|
||||||
``python -m cyclone seed --status`` — print row counts without
|
|
||||||
inserting anything.
|
|
||||||
|
|
||||||
Exit codes
|
|
||||||
----------
|
|
||||||
|
|
||||||
* 0 — success (or already seeded and not asked to reset).
|
|
||||||
* 1 — DB error during insert.
|
|
||||||
* 2 — usage error (bad flag value).
|
|
||||||
|
|
||||||
The command is idempotent: re-running without ``--reset`` is a no-op
|
|
||||||
once a seed batch exists. This keeps ``cyclone``-driven boot scripts
|
|
||||||
safe to re-run.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import random
|
|
||||||
import sys
|
|
||||||
from datetime import datetime, timedelta, timezone
|
|
||||||
|
|
||||||
import click
|
|
||||||
|
|
||||||
from cyclone.db import ActivityEvent, Batch, Claim, ClaimState, Remittance, SessionLocal
|
|
||||||
|
|
||||||
SEED_BATCH_PREFIX = "SEED-"
|
|
||||||
SEED_CLAIM_PREFIX = "CLM-S"
|
|
||||||
SEED_REMIT_PREFIX = "REM-S"
|
|
||||||
SEED_DEFAULT_COUNT = 96
|
|
||||||
SEED_ACTIVITY_LIMIT = 28
|
|
||||||
|
|
||||||
# Mirror src/data/sampleData.ts on the frontend so the Dashboard looks
|
|
||||||
# the same in dev mode as it did with the in-memory fixtures.
|
|
||||||
SAMPLE_PROVIDERS = [
|
|
||||||
{
|
|
||||||
"npi": "1730187395",
|
|
||||||
"name": "Cedar Park Family Medicine",
|
|
||||||
"tax_id": "47-3829104",
|
|
||||||
"address": "1401 Medical Pkwy",
|
|
||||||
"city": "Cedar Park",
|
|
||||||
"state": "TX",
|
|
||||||
"zip": "78613",
|
|
||||||
"phone": "(512) 555-0142",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"npi": "1528471902",
|
|
||||||
"name": "Lakeside Orthopedics",
|
|
||||||
"tax_id": "83-1172654",
|
|
||||||
"address": "900 W Lake Dr",
|
|
||||||
"city": "Austin",
|
|
||||||
"state": "TX",
|
|
||||||
"zip": "78746",
|
|
||||||
"phone": "(512) 555-0188",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"npi": "1982036471",
|
|
||||||
"name": "Hill Country Pediatrics",
|
|
||||||
"tax_id": "74-5520183",
|
|
||||||
"address": "205 State Hwy 27",
|
|
||||||
"city": "Marble Falls",
|
|
||||||
"state": "TX",
|
|
||||||
"zip": "78654",
|
|
||||||
"phone": "(830) 555-0117",
|
|
||||||
},
|
|
||||||
]
|
|
||||||
|
|
||||||
SAMPLE_PAYERS = [
|
|
||||||
"Blue Cross Blue Shield",
|
|
||||||
"United Healthcare",
|
|
||||||
"Aetna",
|
|
||||||
"Cigna",
|
|
||||||
"Humana",
|
|
||||||
"Medicare",
|
|
||||||
"Medicaid TX",
|
|
||||||
]
|
|
||||||
|
|
||||||
SAMPLE_CPTS = ["99213", "99214", "99203", "93000", "85025", "80053", "73721", "20610"]
|
|
||||||
|
|
||||||
SAMPLE_FIRST_NAMES = [
|
|
||||||
"Avery", "Jordan", "Riley", "Casey", "Morgan",
|
|
||||||
"Quinn", "Reese", "Sasha", "Drew", "Hayden",
|
|
||||||
]
|
|
||||||
SAMPLE_LAST_NAMES = [
|
|
||||||
"Nguyen", "Patel", "Garcia", "Cohen", "Okafor",
|
|
||||||
"Martinez", "Hwang", "Brooks", "Singh", "Tanaka",
|
|
||||||
]
|
|
||||||
|
|
||||||
# Frontend uses "accepted"/"pending" which the backend ClaimState
|
|
||||||
# enum doesn't carry. Map them to the closest real states so the
|
|
||||||
# Dashboard's filters (e.g. "status === 'submitted' || 'pending'")
|
|
||||||
# still find a match for in-flight work.
|
|
||||||
STATUS_WEIGHTS: list[tuple[ClaimState, int]] = [
|
|
||||||
(ClaimState.SUBMITTED, 1),
|
|
||||||
(ClaimState.RECEIVED, 1),
|
|
||||||
(ClaimState.DENIED, 1),
|
|
||||||
(ClaimState.PAID, 3),
|
|
||||||
(ClaimState.PAID, 3),
|
|
||||||
(ClaimState.PARTIAL, 1),
|
|
||||||
]
|
|
||||||
|
|
||||||
DENIAL_REASONS = [
|
|
||||||
"CO-97: Service included in another service",
|
|
||||||
"CO-16: Claim lacks information",
|
|
||||||
"CO-50: Non-covered service",
|
|
||||||
"PR-1: Deductible amount",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def _now_utc() -> datetime:
|
|
||||||
return datetime.now(timezone.utc).replace(tzinfo=None)
|
|
||||||
|
|
||||||
|
|
||||||
def _pick(rng: random.Random, items):
|
|
||||||
return items[rng.randint(0, len(items) - 1)]
|
|
||||||
|
|
||||||
|
|
||||||
def _build_seed_rows(
|
|
||||||
count: int, *, seed: int = 42,
|
|
||||||
) -> tuple[Batch, list[Claim], list[ActivityEvent], list[Remittance]]:
|
|
||||||
"""Build a deterministic Batch + N Claims + matching activity events.
|
|
||||||
|
|
||||||
PAID and PARTIAL claims get a paired ``Remittance`` row (with a
|
|
||||||
realistic ``total_paid`` derived from the billed amount) so the
|
|
||||||
Dashboard's "Received" KPI lights up; the wire shape mirrors what
|
|
||||||
``CycloneStore.iter_claims`` expects to find via
|
|
||||||
``Claim.matched_remittance_id`` → ``Remittance.total_paid``.
|
|
||||||
|
|
||||||
The seed is fixed so re-running produces identical data — keeps
|
|
||||||
screenshots and dev environments stable.
|
|
||||||
"""
|
|
||||||
rng = random.Random(seed)
|
|
||||||
now = _now_utc()
|
|
||||||
parsed_at = now - timedelta(minutes=5)
|
|
||||||
|
|
||||||
batch = Batch(
|
|
||||||
id=f"{SEED_BATCH_PREFIX}{now.strftime('%Y%m%d-%H%M%S')}",
|
|
||||||
kind="837",
|
|
||||||
input_filename="seed/sample-837.edi",
|
|
||||||
parsed_at=parsed_at,
|
|
||||||
totals_json={"claim_count": count},
|
|
||||||
validation_json={"errors": [], "warnings": []},
|
|
||||||
raw_result_json={},
|
|
||||||
)
|
|
||||||
|
|
||||||
claims: list[Claim] = []
|
|
||||||
activity: list[ActivityEvent] = []
|
|
||||||
remittances: list[Remittance] = []
|
|
||||||
seq = 10428 # Match the frontend fixture's id range
|
|
||||||
|
|
||||||
for i in range(count):
|
|
||||||
provider = _pick(rng, SAMPLE_PROVIDERS)
|
|
||||||
payer = _pick(rng, SAMPLE_PAYERS)
|
|
||||||
cpt = _pick(rng, SAMPLE_CPTS)
|
|
||||||
first = _pick(rng, SAMPLE_FIRST_NAMES)
|
|
||||||
last = _pick(rng, SAMPLE_LAST_NAMES)
|
|
||||||
state, _ = _pick(rng, STATUS_WEIGHTS)
|
|
||||||
|
|
||||||
days_back = rng.randint(0, 200)
|
|
||||||
submitted = now - timedelta(days=days_back, hours=rng.randint(0, 23))
|
|
||||||
|
|
||||||
billed = 80 + rng.randint(0, 1400)
|
|
||||||
if state == ClaimState.PAID:
|
|
||||||
received = int(billed * (0.6 + rng.random() * 0.4))
|
|
||||||
elif state == ClaimState.PARTIAL:
|
|
||||||
received = int(billed * (0.2 + rng.random() * 0.3))
|
|
||||||
elif state == ClaimState.DENIED:
|
|
||||||
received = 0
|
|
||||||
else:
|
|
||||||
received = 0
|
|
||||||
|
|
||||||
claim_id = f"{SEED_CLAIM_PREFIX}{(seq + i):05d}"
|
|
||||||
denial_reason = _pick(rng, DENIAL_REASONS) if state == ClaimState.DENIED else None
|
|
||||||
|
|
||||||
# Build a paired Remittance for any claim with received > 0. Status
|
|
||||||
# code 1 = "Primary payer forward" — the 835 CAS code that the
|
|
||||||
# remittance mapper turns into "received". Mirror the 835 parser's
|
|
||||||
# raw_json shape (a stripped provider/payer/service_lines block)
|
|
||||||
# so downstream debug views still render something useful.
|
|
||||||
matched_remit_id: str | None = None
|
|
||||||
if received > 0:
|
|
||||||
matched_remit_id = f"{SEED_REMIT_PREFIX}{(seq + i):05d}"
|
|
||||||
remittances.append(Remittance(
|
|
||||||
id=matched_remit_id,
|
|
||||||
batch_id=batch.id,
|
|
||||||
payer_claim_control_number=f"PCN-{(seq + i):05d}",
|
|
||||||
claim_id=claim_id,
|
|
||||||
status_code="1",
|
|
||||||
status_label="Primary payer forward",
|
|
||||||
total_charge=float(billed),
|
|
||||||
total_paid=float(received),
|
|
||||||
adjustment_amount=float(billed - received),
|
|
||||||
received_at=submitted + timedelta(days=rng.randint(2, 14)),
|
|
||||||
raw_json={
|
|
||||||
"payer": {"name": payer},
|
|
||||||
"provider": {
|
|
||||||
"npi": provider["npi"],
|
|
||||||
"name": provider["name"],
|
|
||||||
},
|
|
||||||
"service_lines": [
|
|
||||||
{
|
|
||||||
"procedure": {"code": cpt},
|
|
||||||
"charge_amount": float(billed),
|
|
||||||
"paid_amount": float(received),
|
|
||||||
}
|
|
||||||
],
|
|
||||||
},
|
|
||||||
))
|
|
||||||
|
|
||||||
raw_json = {
|
|
||||||
"billing_provider": {
|
|
||||||
"npi": provider["npi"],
|
|
||||||
"name": provider["name"],
|
|
||||||
"tax_id": provider["tax_id"],
|
|
||||||
# 837 parser produces ``address`` as a structured dict so
|
|
||||||
# the claim-detail drawer can render line1/line2/city/state/zip.
|
|
||||||
# A flat string here crashes ``_address_to_ui`` with
|
|
||||||
# ``AttributeError: 'str' object has no attribute 'get'``.
|
|
||||||
"address": {
|
|
||||||
"line1": provider["address"],
|
|
||||||
"line2": None,
|
|
||||||
"city": provider["city"],
|
|
||||||
"state": provider["state"],
|
|
||||||
"zip": provider["zip"],
|
|
||||||
},
|
|
||||||
"phone": provider["phone"],
|
|
||||||
},
|
|
||||||
"payer": {"name": payer},
|
|
||||||
"subscriber": {"first_name": first, "last_name": last},
|
|
||||||
"service_lines": [
|
|
||||||
{
|
|
||||||
"procedure": {"code": cpt},
|
|
||||||
"charge_amount": float(billed),
|
|
||||||
"service_date": submitted.date().isoformat(),
|
|
||||||
}
|
|
||||||
],
|
|
||||||
}
|
|
||||||
|
|
||||||
claims.append(Claim(
|
|
||||||
id=claim_id,
|
|
||||||
batch_id=batch.id,
|
|
||||||
patient_control_number=claim_id.replace(SEED_CLAIM_PREFIX, "PCN-"),
|
|
||||||
service_date_from=submitted.date(),
|
|
||||||
service_date_to=submitted.date(),
|
|
||||||
charge_amount=billed,
|
|
||||||
provider_npi=provider["npi"],
|
|
||||||
payer_id=None,
|
|
||||||
state=state,
|
|
||||||
state_changed_at=submitted,
|
|
||||||
rejection_reason=denial_reason,
|
|
||||||
resubmit_count=0,
|
|
||||||
matched_remittance_id=matched_remit_id,
|
|
||||||
raw_json=raw_json,
|
|
||||||
))
|
|
||||||
|
|
||||||
# First SEED_ACTIVITY_LIMIT claims get an activity event. Mirrors
|
|
||||||
# the frontend buildActivity() shape (claim_paid / claim_denied /
|
|
||||||
# claim_accepted / claim_submitted) but maps frontend statuses
|
|
||||||
# onto the backend's wire enum.
|
|
||||||
if i < SEED_ACTIVITY_LIMIT:
|
|
||||||
if state == ClaimState.PAID:
|
|
||||||
kind = "claim_paid"
|
|
||||||
verb = "Paid"
|
|
||||||
elif state == ClaimState.DENIED:
|
|
||||||
kind = "claim_denied"
|
|
||||||
verb = "Denied"
|
|
||||||
elif state in (ClaimState.RECEIVED, ClaimState.PARTIAL):
|
|
||||||
kind = "claim_accepted"
|
|
||||||
verb = "Accepted"
|
|
||||||
else:
|
|
||||||
kind = "claim_submitted"
|
|
||||||
verb = "Submitted"
|
|
||||||
|
|
||||||
payload = {
|
|
||||||
"message": f"{verb} {claim_id} · {first} {last}",
|
|
||||||
"npi": provider["npi"],
|
|
||||||
"amount": float(billed),
|
|
||||||
}
|
|
||||||
activity.append(ActivityEvent(
|
|
||||||
ts=submitted,
|
|
||||||
kind=kind,
|
|
||||||
batch_id=batch.id,
|
|
||||||
claim_id=claim_id,
|
|
||||||
remittance_id=None,
|
|
||||||
payload_json=payload,
|
|
||||||
))
|
|
||||||
|
|
||||||
return batch, claims, activity, remittances
|
|
||||||
|
|
||||||
|
|
||||||
def _existing_seed_batch_ids(s) -> list[str]:
|
|
||||||
rows = s.query(Batch.id).filter(Batch.id.like(f"{SEED_BATCH_PREFIX}%")).all()
|
|
||||||
return [r[0] for r in rows]
|
|
||||||
|
|
||||||
|
|
||||||
def _delete_seed_rows(s) -> int:
|
|
||||||
"""Delete every row that was inserted by the seed.
|
|
||||||
|
|
||||||
The seed writes rows whose ids begin with ``SEED-`` (batches),
|
|
||||||
``CLM-S`` (claims), or ``REM-S`` (remittances). Activity events
|
|
||||||
are joined to seeded batches when the batch is still around, but
|
|
||||||
we also clean up orphan activity rows (no FK from
|
|
||||||
``activity_events.claim_id`` to ``claims.id``) by ``claim_id``
|
|
||||||
prefix. Returns the number of batch rows deleted.
|
|
||||||
"""
|
|
||||||
# Activity first — its rows reference both claims and batches, so
|
|
||||||
# delete the events tied to seed claims first, then any that were
|
|
||||||
# only tied to a now-orphaned seed batch.
|
|
||||||
activity = s.query(ActivityEvent).filter(
|
|
||||||
ActivityEvent.claim_id.like(f"{SEED_CLAIM_PREFIX}%")
|
|
||||||
).delete(synchronize_session=False)
|
|
||||||
activity2 = s.query(ActivityEvent).filter(
|
|
||||||
ActivityEvent.batch_id.like(f"{SEED_BATCH_PREFIX}%")
|
|
||||||
).delete(synchronize_session=False)
|
|
||||||
# Remittances (FK to claims; cascade may not fire without FK pragma,
|
|
||||||
# so we delete them explicitly to clear the symmetric FK first).
|
|
||||||
remits = s.query(Remittance).filter(Remittance.id.like(f"{SEED_REMIT_PREFIX}%")).delete(synchronize_session=False)
|
|
||||||
claims = s.query(Claim).filter(Claim.id.like(f"{SEED_CLAIM_PREFIX}%")).delete(synchronize_session=False)
|
|
||||||
batches = s.query(Batch).filter(Batch.id.like(f"{SEED_BATCH_PREFIX}%")).delete(synchronize_session=False)
|
|
||||||
s.commit()
|
|
||||||
click.echo(f" Removed {batches} seeded batch(es), {claims} claim(s), "
|
|
||||||
f"{remits} remittance(s), {activity + activity2} activity event(s).")
|
|
||||||
return batches
|
|
||||||
|
|
||||||
|
|
||||||
def _insert(
|
|
||||||
batch: Batch,
|
|
||||||
claims: list[Claim],
|
|
||||||
activity: list[ActivityEvent],
|
|
||||||
remittances: list[Remittance],
|
|
||||||
) -> None:
|
|
||||||
with SessionLocal()() as s:
|
|
||||||
s.add(batch)
|
|
||||||
s.add_all(claims)
|
|
||||||
s.add_all(activity)
|
|
||||||
s.add_all(remittances)
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def _print_status(s) -> None:
|
|
||||||
from sqlalchemy import func
|
|
||||||
seed_batches = s.query(func.count(Batch.id)).filter(Batch.id.like(f"{SEED_BATCH_PREFIX}%")).scalar() or 0
|
|
||||||
seed_claims = s.query(func.count(Claim.id)).filter(Claim.id.like(f"{SEED_CLAIM_PREFIX}%")).scalar() or 0
|
|
||||||
seed_remits = s.query(func.count(Remittance.id)).filter(Remittance.id.like(f"{SEED_REMIT_PREFIX}%")).scalar() or 0
|
|
||||||
total_claims = s.query(func.count(Claim.id)).scalar() or 0
|
|
||||||
total_remits = s.query(func.count(Remittance.id)).scalar() or 0
|
|
||||||
total_activity = s.query(func.count(ActivityEvent.id)).scalar() or 0
|
|
||||||
click.echo(f" Seed batches: {seed_batches}")
|
|
||||||
click.echo(f" Seed claims: {seed_claims}")
|
|
||||||
click.echo(f" Seed remits: {seed_remits}")
|
|
||||||
click.echo(f" Total claims: {total_claims}")
|
|
||||||
click.echo(f" Total remits: {total_remits}")
|
|
||||||
click.echo(f" Total activity: {total_activity}")
|
|
||||||
|
|
||||||
|
|
||||||
@click.command("seed")
|
|
||||||
@click.option(
|
|
||||||
"--count",
|
|
||||||
default=SEED_DEFAULT_COUNT,
|
|
||||||
show_default=True,
|
|
||||||
type=click.IntRange(min=1, max=10_000),
|
|
||||||
help="Number of claims to insert in the new batch.",
|
|
||||||
)
|
|
||||||
@click.option(
|
|
||||||
"--reset",
|
|
||||||
is_flag=True,
|
|
||||||
help="Delete any existing seeded batch (and its claims/activity) before inserting.",
|
|
||||||
)
|
|
||||||
@click.option(
|
|
||||||
"--status",
|
|
||||||
"show_status",
|
|
||||||
is_flag=True,
|
|
||||||
help="Print current seeded row counts and exit.",
|
|
||||||
)
|
|
||||||
def seed_cli(count: int, reset: bool, show_status: bool) -> None:
|
|
||||||
"""Populate the local DB with a deterministic batch of sample claims."""
|
|
||||||
with SessionLocal()() as s:
|
|
||||||
if show_status:
|
|
||||||
_print_status(s)
|
|
||||||
return
|
|
||||||
|
|
||||||
existing = _existing_seed_batch_ids(s)
|
|
||||||
if existing and not reset:
|
|
||||||
click.echo(
|
|
||||||
f"Seed already present (batch ids: {', '.join(existing)}). "
|
|
||||||
f"Re-run with --reset to replace, or --status to inspect.",
|
|
||||||
err=True,
|
|
||||||
)
|
|
||||||
sys.exit(0)
|
|
||||||
|
|
||||||
if reset and existing:
|
|
||||||
deleted = _delete_seed_rows(s)
|
|
||||||
click.echo(f" Removed {deleted} seeded batch(es).")
|
|
||||||
|
|
||||||
try:
|
|
||||||
batch, claims, activity, remittances = _build_seed_rows(count)
|
|
||||||
except Exception as exc:
|
|
||||||
click.echo(f"Failed to build seed rows: {exc}", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
try:
|
|
||||||
_insert(batch, claims, activity, remittances)
|
|
||||||
except Exception as exc:
|
|
||||||
click.echo(f"Failed to insert seed rows: {exc}", err=True)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
click.echo(f" Inserted batch {batch.id} with {len(claims)} claims, "
|
|
||||||
f"{len(remittances)} remittances, {len(activity)} activity events.")
|
|
||||||
_print_status(s)
|
|
||||||
@@ -431,7 +431,6 @@ def to_ui_claim_from_orm(
|
|||||||
*,
|
*,
|
||||||
batch_id: str,
|
batch_id: str,
|
||||||
parsed_at: datetime,
|
parsed_at: datetime,
|
||||||
received_total: float = 0.0,
|
|
||||||
) -> dict:
|
) -> dict:
|
||||||
"""Map an ORM ``Claim`` row to the UI's claim shape.
|
"""Map an ORM ``Claim`` row to the UI's claim shape.
|
||||||
|
|
||||||
@@ -477,10 +476,7 @@ def to_ui_claim_from_orm(
|
|||||||
"batchId": batch_id,
|
"batchId": batch_id,
|
||||||
# Parity with ``to_ui_claim``'s shape — the UI tolerates extra keys
|
# Parity with ``to_ui_claim``'s shape — the UI tolerates extra keys
|
||||||
# but expects these on freshly-loaded rows from /api/claims too.
|
# but expects these on freshly-loaded rows from /api/claims too.
|
||||||
# ``received_total`` comes from the matched Remittance row when one
|
"receivedAmount": 0.0,
|
||||||
# exists; callers that don't pre-compute it (write path, unmatched
|
|
||||||
# list) get the default of 0.0 — which matches the unmapped state.
|
|
||||||
"receivedAmount": float(received_total),
|
|
||||||
"denialReason": None,
|
"denialReason": None,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1072,9 +1068,6 @@ class CycloneStore:
|
|||||||
ui = to_ui_claim_from_orm(
|
ui = to_ui_claim_from_orm(
|
||||||
row, batch_id=row.batch_id or record.id,
|
row, batch_id=row.batch_id or record.id,
|
||||||
parsed_at=record.parsed_at,
|
parsed_at=record.parsed_at,
|
||||||
# Fresh ingest — no remittance has been paired yet,
|
|
||||||
# so ``Received`` is necessarily 0.
|
|
||||||
received_total=0.0,
|
|
||||||
)
|
)
|
||||||
self._sync_publish(event_bus, "claim_written", ui)
|
self._sync_publish(event_bus, "claim_written", ui)
|
||||||
for rid in remit_ids:
|
for rid in remit_ids:
|
||||||
@@ -1495,24 +1488,6 @@ class CycloneStore:
|
|||||||
q = q.filter(Claim.provider_npi == provider_npi)
|
q = q.filter(Claim.provider_npi == provider_npi)
|
||||||
|
|
||||||
rows = q.all()
|
rows = q.all()
|
||||||
# Bulk-load matched-remittance totals so the UI's "Received"
|
|
||||||
# KPI + per-claim received_amount reflect real paid amounts
|
|
||||||
# rather than always-0. One SQL roundtrip for the whole page
|
|
||||||
# rather than per-claim lookups.
|
|
||||||
matched_ids = [
|
|
||||||
r.matched_remittance_id
|
|
||||||
for r in rows
|
|
||||||
if r.matched_remittance_id
|
|
||||||
]
|
|
||||||
received_by_remit: dict[str, float] = {}
|
|
||||||
if matched_ids:
|
|
||||||
for rid, total_paid in (
|
|
||||||
s.query(Remittance.id, Remittance.total_paid)
|
|
||||||
.filter(Remittance.id.in_(matched_ids))
|
|
||||||
.all()
|
|
||||||
):
|
|
||||||
received_by_remit[rid] = float(total_paid or 0)
|
|
||||||
|
|
||||||
out: list[dict] = []
|
out: list[dict] = []
|
||||||
for r in rows:
|
for r in rows:
|
||||||
raw = r.raw_json or {}
|
raw = r.raw_json or {}
|
||||||
@@ -1541,9 +1516,7 @@ class CycloneStore:
|
|||||||
"payerName": payer_obj.get("name") or "",
|
"payerName": payer_obj.get("name") or "",
|
||||||
"cptCode": cpt,
|
"cptCode": cpt,
|
||||||
"billedAmount": float(r.charge_amount or 0),
|
"billedAmount": float(r.charge_amount or 0),
|
||||||
"receivedAmount": received_by_remit.get(
|
"receivedAmount": 0.0,
|
||||||
r.matched_remittance_id, 0.0
|
|
||||||
),
|
|
||||||
"status": r.state.value if hasattr(r.state, "value") else str(r.state),
|
"status": r.state.value if hasattr(r.state, "value") else str(r.state),
|
||||||
"state": r.state.value if hasattr(r.state, "value") else str(r.state),
|
"state": r.state.value if hasattr(r.state, "value") else str(r.state),
|
||||||
"denialReason": None,
|
"denialReason": None,
|
||||||
@@ -1695,16 +1668,7 @@ class CycloneStore:
|
|||||||
return list(by_npi.values())
|
return list(by_npi.values())
|
||||||
|
|
||||||
def recent_activity(self, *, limit: int = 200) -> list[dict]:
|
def recent_activity(self, *, limit: int = 200) -> list[dict]:
|
||||||
"""Return recent activity events from the DB, newest first.
|
"""Return recent activity events from the DB, newest first."""
|
||||||
|
|
||||||
SP21 Task 2.5: each row also carries ``claimId`` and
|
|
||||||
``remittanceId`` (read from the ORM columns) so the Dashboard's
|
|
||||||
Recent-activity card can route clicks to the right entity
|
|
||||||
drawer via ``src/lib/event-routing.ts``. Both are nullable
|
|
||||||
strings; the wire shape uses camelCase keys to match the
|
|
||||||
existing ``npi`` / ``amount`` fields and the frontend
|
|
||||||
``Activity`` interface.
|
|
||||||
"""
|
|
||||||
with db.SessionLocal()() as s:
|
with db.SessionLocal()() as s:
|
||||||
rows = (
|
rows = (
|
||||||
s.query(ActivityEvent)
|
s.query(ActivityEvent)
|
||||||
@@ -1720,8 +1684,6 @@ class CycloneStore:
|
|||||||
"timestamp": r.ts.isoformat().replace("+00:00", "Z"),
|
"timestamp": r.ts.isoformat().replace("+00:00", "Z"),
|
||||||
"npi": (r.payload_json or {}).get("npi"),
|
"npi": (r.payload_json or {}).get("npi"),
|
||||||
"amount": (r.payload_json or {}).get("amount"),
|
"amount": (r.payload_json or {}).get("amount"),
|
||||||
"claimId": r.claim_id,
|
|
||||||
"remittanceId": r.remittance_id,
|
|
||||||
}
|
}
|
||||||
for r in rows
|
for r in rows
|
||||||
]
|
]
|
||||||
@@ -1887,32 +1849,6 @@ class CycloneStore:
|
|||||||
with db.SessionLocal()() as s:
|
with db.SessionLocal()() as s:
|
||||||
return s.get(db.Two77caAck, ack_id)
|
return s.get(db.Two77caAck, ack_id)
|
||||||
|
|
||||||
# -- SP17: encrypted DB backups -------------------------------------
|
|
||||||
|
|
||||||
def add_backup_pending(self, *, filename: str, backup_dir: str) -> db.DbBackup:
|
|
||||||
"""Insert a ``pending`` row for a backup that is about to start.
|
|
||||||
|
|
||||||
The BackupService fills in ``status`` / ``size_bytes`` /
|
|
||||||
``db_fingerprint`` / ``table_count`` / ``completed_at`` after
|
|
||||||
the encrypted blob lands on disk.
|
|
||||||
"""
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
row = db.DbBackup(
|
|
||||||
filename=filename,
|
|
||||||
backup_dir=backup_dir,
|
|
||||||
size_bytes=0,
|
|
||||||
db_fingerprint=None,
|
|
||||||
table_count=0,
|
|
||||||
created_at=utcnow(),
|
|
||||||
completed_at=None,
|
|
||||||
status="pending",
|
|
||||||
error_message=None,
|
|
||||||
)
|
|
||||||
s.add(row)
|
|
||||||
s.commit()
|
|
||||||
s.refresh(row)
|
|
||||||
return row
|
|
||||||
|
|
||||||
# -- manual reconciliation (T12) -----------------------------------
|
# -- manual reconciliation (T12) -----------------------------------
|
||||||
|
|
||||||
def list_unmatched(self, *, kind: str = "both") -> dict:
|
def list_unmatched(self, *, kind: str = "both") -> dict:
|
||||||
@@ -1963,9 +1899,6 @@ class CycloneStore:
|
|||||||
result["claims"].append(
|
result["claims"].append(
|
||||||
to_ui_claim_from_orm(
|
to_ui_claim_from_orm(
|
||||||
r, batch_id=r.batch_id, parsed_at=parsed_at,
|
r, batch_id=r.batch_id, parsed_at=parsed_at,
|
||||||
# list_unmatched filters matched_remittance_id IS NULL,
|
|
||||||
# so every row has no remittance yet.
|
|
||||||
received_total=0.0,
|
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -2093,7 +2026,6 @@ class CycloneStore:
|
|||||||
)
|
)
|
||||||
claim_dict = to_ui_claim_from_orm(
|
claim_dict = to_ui_claim_from_orm(
|
||||||
claim, batch_id=claim.batch_id, parsed_at=parsed_at,
|
claim, batch_id=claim.batch_id, parsed_at=parsed_at,
|
||||||
received_total=float(remit.total_paid or 0),
|
|
||||||
)
|
)
|
||||||
matched_at_iso = now.isoformat().replace("+00:00", "Z")
|
matched_at_iso = now.isoformat().replace("+00:00", "Z")
|
||||||
return {
|
return {
|
||||||
@@ -2150,11 +2082,9 @@ class CycloneStore:
|
|||||||
# rows exist. Shouldn't happen, but if it does, fall back
|
# rows exist. Shouldn't happen, but if it does, fall back
|
||||||
# to clearing the FK and starting fresh.
|
# to clearing the FK and starting fresh.
|
||||||
latest = None
|
latest = None
|
||||||
paired_remit = None
|
|
||||||
restored_state = ClaimState.SUBMITTED
|
restored_state = ClaimState.SUBMITTED
|
||||||
else:
|
else:
|
||||||
latest = matches[0]
|
latest = matches[0]
|
||||||
paired_remit = s.get(Remittance, latest.remittance_id)
|
|
||||||
restored_state = (
|
restored_state = (
|
||||||
latest.prior_claim_state
|
latest.prior_claim_state
|
||||||
if latest.prior_claim_state is not None
|
if latest.prior_claim_state is not None
|
||||||
@@ -2170,7 +2100,9 @@ class CycloneStore:
|
|||||||
# Clear the symmetric FK on the remittance so list_unmatched
|
# Clear the symmetric FK on the remittance so list_unmatched
|
||||||
# surfaces the pair again. The remittance may have been
|
# surfaces the pair again. The remittance may have been
|
||||||
# deleted between the match and this call — guard with a
|
# deleted between the match and this call — guard with a
|
||||||
# None check so we don't blow up on a stale FK.
|
# get() so we don't blow up on a stale FK.
|
||||||
|
if latest is not None:
|
||||||
|
paired_remit = s.get(Remittance, latest.remittance_id)
|
||||||
if paired_remit is not None:
|
if paired_remit is not None:
|
||||||
paired_remit.claim_id = None
|
paired_remit.claim_id = None
|
||||||
|
|
||||||
@@ -2192,19 +2124,8 @@ class CycloneStore:
|
|||||||
if claim.batch is not None
|
if claim.batch is not None
|
||||||
else now
|
else now
|
||||||
)
|
)
|
||||||
# ``paired_remit`` is the matched remittance we cleared in
|
|
||||||
# the unmatch; use its ``total_paid`` for the response shape
|
|
||||||
# so the UI sees what was paid before the unpair. May be
|
|
||||||
# ``None`` if the remittance was deleted since the match —
|
|
||||||
# default to 0.0 in that case.
|
|
||||||
received_total = (
|
|
||||||
float(paired_remit.total_paid or 0)
|
|
||||||
if paired_remit is not None
|
|
||||||
else 0.0
|
|
||||||
)
|
|
||||||
claim_dict = to_ui_claim_from_orm(
|
claim_dict = to_ui_claim_from_orm(
|
||||||
claim, batch_id=claim.batch_id, parsed_at=parsed_at,
|
claim, batch_id=claim.batch_id, parsed_at=parsed_at,
|
||||||
received_total=received_total,
|
|
||||||
)
|
)
|
||||||
return {
|
return {
|
||||||
"claim": claim_dict,
|
"claim": claim_dict,
|
||||||
@@ -2319,7 +2240,7 @@ class CycloneStore:
|
|||||||
submitter_contact_email="tyler@dzinesco.com",
|
submitter_contact_email="tyler@dzinesco.com",
|
||||||
filename_block={
|
filename_block={
|
||||||
"tz": "America/Denver",
|
"tz": "America/Denver",
|
||||||
"outbound_template": "tp{tpid}-{tx}-{ts_mt}-1of1.{ext}",
|
"outbound_template": "{tpid}-{tx}-{ts_mt}-1of1.{ext}",
|
||||||
"inbound_template": "TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12",
|
"inbound_template": "TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12",
|
||||||
},
|
},
|
||||||
sftp_block={
|
sftp_block={
|
||||||
|
|||||||
@@ -24,34 +24,17 @@ def _auto_init_db(tmp_path, monkeypatch):
|
|||||||
Also wires a fresh ``EventBus`` onto ``app.state`` because ``TestClient``
|
Also wires a fresh ``EventBus`` onto ``app.state`` because ``TestClient``
|
||||||
does not invoke the FastAPI lifespan handler unless used as a context
|
does not invoke the FastAPI lifespan handler unless used as a context
|
||||||
manager. The bus is reset between tests so subscribers don't leak.
|
manager. The bus is reset between tests so subscribers don't leak.
|
||||||
|
|
||||||
Auth gating is enabled at the router/endpoint level via the
|
|
||||||
``Depends(matrix_gate)`` wiring in ``cyclone.api``. The auth tests
|
|
||||||
(``test_auth_*``) explicitly flip ``AUTH_DISABLED = False`` and
|
|
||||||
authenticate via the public login route to exercise the real
|
|
||||||
auth path. Every other test — the original test suite predates
|
|
||||||
auth — gets ``AUTH_DISABLED = True`` here so the existing tests
|
|
||||||
keep working without each one having to login first.
|
|
||||||
"""
|
"""
|
||||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
|
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
|
||||||
from cyclone import db
|
from cyclone import db
|
||||||
|
from cyclone.api import app
|
||||||
from cyclone.pubsub import EventBus
|
from cyclone.pubsub import EventBus
|
||||||
from cyclone.auth import deps
|
|
||||||
|
|
||||||
db._reset_for_tests()
|
db._reset_for_tests()
|
||||||
db.init_db()
|
db.init_db()
|
||||||
# Re-resolve `app` each fixture invocation because some tests
|
app.state.event_bus = EventBus()
|
||||||
# (test_cors_extra_origins_via_env) call ``importlib.reload`` on
|
|
||||||
# ``cyclone.api`` to mutate the CORS allow-list. If we cached
|
|
||||||
# the app reference at conftest module load, we'd be setting
|
|
||||||
# ``event_bus`` on a stale instance that no test client is
|
|
||||||
# actually using.
|
|
||||||
from cyclone import api as _api_mod
|
|
||||||
_api_mod.app.state.event_bus = EventBus()
|
|
||||||
deps.AUTH_DISABLED = True
|
|
||||||
try:
|
try:
|
||||||
yield
|
yield
|
||||||
finally:
|
finally:
|
||||||
deps.AUTH_DISABLED = False
|
app.state.event_bus = None
|
||||||
_api_mod.app.state.event_bus = None
|
|
||||||
db._reset_for_tests()
|
db._reset_for_tests()
|
||||||
+1
-1
@@ -7,7 +7,7 @@ PER*IC*Test Contact*EM*test@example.com~
|
|||||||
NM1*40*2*COLORADO MEDICAL ASSISTANCE PROGRAM*****46*COMEDASSISTPROG~
|
NM1*40*2*COLORADO MEDICAL ASSISTANCE PROGRAM*****46*COMEDASSISTPROG~
|
||||||
HL*1**20*1~
|
HL*1**20*1~
|
||||||
PRV*BI*PXC*251E00000X~
|
PRV*BI*PXC*251E00000X~
|
||||||
NM1*85*2*Test Provider Inc*****XX*1993999998~
|
NM1*85*2*Test Provider Inc*****XX*1234567890~
|
||||||
N3*123 Test St~
|
N3*123 Test St~
|
||||||
N4*Denver*CO*80202~
|
N4*Denver*CO*80202~
|
||||||
REF*EI*123456789~
|
REF*EI*123456789~
|
||||||
|
|||||||
-1
@@ -1 +0,0 @@
|
|||||||
ISA*00* *00* *ZZ*SENDER *ZZ*RECEIVER *260520*1750*^*00501*000000001*0*P*:~TA1*000000001*20260520*1750*A*000*20260520~IEA*1*000000001~
|
|
||||||
@@ -51,21 +51,18 @@ def test_migration_0002_creates_acks_table():
|
|||||||
|
|
||||||
def test_migration_latest_idempotent_on_fresh_db():
|
def test_migration_latest_idempotent_on_fresh_db():
|
||||||
"""Re-running the migration on the same DB must be a no-op (PRAGMA
|
"""Re-running the migration on the same DB must be a no-op (PRAGMA
|
||||||
user_version already at the latest version — currently 15 after
|
user_version already at the latest version — currently 10 after
|
||||||
0004-0006 line_reconciliation, 0005 ta1_acks, SP9's 0007
|
0004-0006 line_reconciliation, 0005 ta1_acks, SP9's 0007
|
||||||
providers/payers/clearhouse, SP10's 0008 payer_rejected,
|
providers/payers/clearhouse, SP10's 0008 payer_rejected,
|
||||||
SP11's 0009 audit_log, SP14's 0010 payer_rejected_acknowledged,
|
SP11's 0009 audit_log, and SP14's 0010 payer_rejected_acknowledged)."""
|
||||||
SP16's 0011 processed_inbound_files, SP17's 0012 db_backups,
|
|
||||||
SP-auth's 0013 users + sessions, SP-audit's 0014 audit_log.user_id,
|
|
||||||
SP22's 0015 drop_claims_unique_constraint)."""
|
|
||||||
with db.engine().begin() as c:
|
with db.engine().begin() as c:
|
||||||
v1 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
|
v1 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
|
||||||
assert v1 == 15
|
assert v1 == 10
|
||||||
# A second run should not raise and should not bump the version.
|
# A second run should not raise and should not bump the version.
|
||||||
db_migrate.run(db.engine())
|
db_migrate.run(db.engine())
|
||||||
with db.engine().begin() as c:
|
with db.engine().begin() as c:
|
||||||
v2 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
|
v2 = c.exec_driver_sql("PRAGMA user_version").scalar() or 0
|
||||||
assert v2 == 15
|
assert v2 == 10
|
||||||
|
|
||||||
|
|
||||||
def test_add_ack_persists_row():
|
def test_add_ack_persists_row():
|
||||||
|
|||||||
@@ -31,18 +31,10 @@ def client() -> TestClient:
|
|||||||
|
|
||||||
|
|
||||||
def test_health_endpoint(client: TestClient):
|
def test_health_endpoint(client: TestClient):
|
||||||
"""SP19: health endpoint now returns a subsystem snapshot."""
|
|
||||||
resp = client.get("/api/health")
|
resp = client.get("/api/health")
|
||||||
assert resp.status_code == 200
|
assert resp.status_code == 200
|
||||||
body = resp.json()
|
body = resp.json()
|
||||||
# Old contract (status + version) is preserved.
|
assert body == {"status": "ok", "version": __version__}
|
||||||
assert body["status"] == "ok"
|
|
||||||
assert body["version"] == __version__
|
|
||||||
# SP19 additions.
|
|
||||||
assert "db" in body and body["db"].get("ok") is True
|
|
||||||
assert "scheduler" in body
|
|
||||||
assert "pubsub" in body
|
|
||||||
assert "batch" in body
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
@@ -101,14 +93,6 @@ def test_parse_837_endpoint_streams_ndjson(client: TestClient):
|
|||||||
# Summary numbers match the JSON path.
|
# Summary numbers match the JSON path.
|
||||||
assert parsed[3]["data"]["total_claims"] == 2
|
assert parsed[3]["data"]["total_claims"] == 2
|
||||||
assert parsed[3]["data"]["passed"] == 2
|
assert parsed[3]["data"]["passed"] == 2
|
||||||
# The streaming summary carries the server-side batch id so the
|
|
||||||
# frontend can call /api/batches/{id}/export-837 without a separate
|
|
||||||
# GET /api/batches round-trip (regression: it used to be missing
|
|
||||||
# on the stream path, only present on the JSON path, which made the
|
|
||||||
# Upload page's Export button say "no batch to export").
|
|
||||||
assert parsed[3]["type"] == "summary"
|
|
||||||
assert isinstance(parsed[3]["data"]["batch_id"], str)
|
|
||||||
assert len(parsed[3]["data"]["batch_id"]) == 32 # uuid4().hex
|
|
||||||
|
|
||||||
|
|
||||||
def test_parse_837_endpoint_streams_ndjson_without_raw_segments(client: TestClient):
|
def test_parse_837_endpoint_streams_ndjson_without_raw_segments(client: TestClient):
|
||||||
@@ -168,51 +152,3 @@ def test_cors_headers_present(client: TestClient):
|
|||||||
)
|
)
|
||||||
assert resp.headers.get("access-control-allow-origin") == "http://localhost:5173"
|
assert resp.headers.get("access-control-allow-origin") == "http://localhost:5173"
|
||||||
assert "POST" in resp.headers.get("access-control-allow-methods", "").upper()
|
assert "POST" in resp.headers.get("access-control-allow-methods", "").upper()
|
||||||
|
|
||||||
|
|
||||||
def test_cors_headers_present_for_loopback_ip(client: TestClient):
|
|
||||||
# ``http://127.0.0.1:5173`` is a distinct origin from
|
|
||||||
# ``http://localhost:5173`` per the CORS spec, even though both resolve
|
|
||||||
# to the same Vite dev server. Both must be allow-listed or tabs opened
|
|
||||||
# via the IP form silently break.
|
|
||||||
resp = client.options(
|
|
||||||
"/api/parse-837",
|
|
||||||
headers={
|
|
||||||
"Origin": "http://127.0.0.1:5173",
|
|
||||||
"Access-Control-Request-Method": "POST",
|
|
||||||
"Access-Control-Request-Headers": "content-type",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
assert resp.headers.get("access-control-allow-origin") == "http://127.0.0.1:5173"
|
|
||||||
|
|
||||||
|
|
||||||
def test_cors_extra_origins_via_env(client: TestClient, monkeypatch):
|
|
||||||
# LAN / staging hosts opt in via CYCLONE_ALLOWED_ORIGINS. The env var
|
|
||||||
# is a comma-separated list; the middleware must reflect each entry.
|
|
||||||
# The allow-list is built at module import, so we re-execute the
|
|
||||||
# module under the env var and build a TestClient against the
|
|
||||||
# reloaded app.
|
|
||||||
monkeypatch.setenv(
|
|
||||||
"CYCLONE_ALLOWED_ORIGINS", "http://192.168.1.42:5173,https://staging.example.com"
|
|
||||||
)
|
|
||||||
import importlib
|
|
||||||
from cyclone import api as api_module
|
|
||||||
from fastapi.testclient import TestClient as _TC
|
|
||||||
importlib.reload(api_module)
|
|
||||||
try:
|
|
||||||
with _TC(api_module.app) as tc:
|
|
||||||
for origin in ("http://192.168.1.42:5173", "https://staging.example.com"):
|
|
||||||
resp = tc.options(
|
|
||||||
"/api/parse-837",
|
|
||||||
headers={
|
|
||||||
"Origin": origin,
|
|
||||||
"Access-Control-Request-Method": "POST",
|
|
||||||
"Access-Control-Request-Headers": "content-type",
|
|
||||||
},
|
|
||||||
)
|
|
||||||
assert resp.headers.get("access-control-allow-origin") == origin
|
|
||||||
finally:
|
|
||||||
monkeypatch.delenv("CYCLONE_ALLOWED_ORIGINS", raising=False)
|
|
||||||
# Reload once more so the module-level allow-list returns to its
|
|
||||||
# default for any test that imports `cyclone.api` after this one.
|
|
||||||
importlib.reload(api_module)
|
|
||||||
|
|||||||
@@ -82,13 +82,6 @@ def test_parse_835_endpoint_streams_ndjson(client: TestClient):
|
|||||||
# Summary numbers match the JSON path.
|
# Summary numbers match the JSON path.
|
||||||
assert parsed[7]["data"]["total_claims"] == 2
|
assert parsed[7]["data"]["total_claims"] == 2
|
||||||
assert parsed[7]["data"]["passed"] == 2
|
assert parsed[7]["data"]["passed"] == 2
|
||||||
# The streaming summary carries the server-side batch id so the
|
|
||||||
# frontend can call /api/batches/{id}/export-837 without a separate
|
|
||||||
# GET /api/batches round-trip (matches the parallel fix on
|
|
||||||
# /api/parse-837).
|
|
||||||
assert parsed[7]["type"] == "summary"
|
|
||||||
assert isinstance(parsed[7]["data"]["batch_id"], str)
|
|
||||||
assert len(parsed[7]["data"]["batch_id"]) == 32 # uuid4().hex
|
|
||||||
|
|
||||||
|
|
||||||
def test_parse_835_endpoint_streams_ndjson_without_raw_segments(client: TestClient):
|
def test_parse_835_endpoint_streams_ndjson_without_raw_segments(client: TestClient):
|
||||||
|
|||||||
@@ -1,313 +0,0 @@
|
|||||||
"""SP17 — Admin backup API endpoint tests.
|
|
||||||
|
|
||||||
Covers:
|
|
||||||
- POST /api/admin/backup/create
|
|
||||||
- GET /api/admin/backup/list
|
|
||||||
- GET /api/admin/backup/status
|
|
||||||
- POST /api/admin/backup/{id}/verify
|
|
||||||
- POST /api/admin/backup/{id}/restore/initiate
|
|
||||||
- POST /api/admin/backup/{id}/restore/confirm
|
|
||||||
- POST /api/admin/backup/prune
|
|
||||||
- POST /api/admin/backup/scheduler/{start,stop,tick}
|
|
||||||
|
|
||||||
Each fixture starts a clean DB + BackupService configured with a
|
|
||||||
known passphrase. We deliberately do NOT enable SQLCipher here —
|
|
||||||
the backup layer is independent of SQLCipher encryption at rest.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from datetime import datetime, timedelta, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def _backup_env(tmp_path, monkeypatch):
|
|
||||||
"""Fresh sqlite DB + BackupService with passphrase. Reset module singletons."""
|
|
||||||
from cyclone import db
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
from cyclone import backup_scheduler as sched_mod
|
|
||||||
from cyclone.db import Batch
|
|
||||||
|
|
||||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
|
|
||||||
db._reset_for_tests()
|
|
||||||
db.init_db()
|
|
||||||
|
|
||||||
# Make sure there's at least one row so the backup isn't a no-op.
|
|
||||||
import uuid
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
s.add(Batch(
|
|
||||||
id=str(uuid.uuid4()),
|
|
||||||
kind="837P",
|
|
||||||
input_filename="seed.x12",
|
|
||||||
parsed_at=datetime.now(timezone.utc),
|
|
||||||
totals_json=None,
|
|
||||||
validation_json=None,
|
|
||||||
raw_result_json={"envelope": {"control_number": "1"}, "claims": [], "summary": {"passed": 0, "failed": 0, "failed_claim_ids": []}},
|
|
||||||
))
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
backup_dir = tmp_path / "backups"
|
|
||||||
svc_mod.reset_backup_service_for_tests()
|
|
||||||
sched_mod.reset_backup_scheduler_for_tests()
|
|
||||||
svc = svc_mod.configure_backup_service(
|
|
||||||
backup_dir=backup_dir, passphrase="api-test-pass", retention_days=7,
|
|
||||||
)
|
|
||||||
yield svc, backup_dir
|
|
||||||
sched_mod.reset_backup_scheduler_for_tests()
|
|
||||||
svc_mod.reset_backup_service_for_tests()
|
|
||||||
db._reset_for_tests()
|
|
||||||
|
|
||||||
|
|
||||||
def _client():
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
from cyclone.api import app
|
|
||||||
return TestClient(app)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# /backup/create
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_create_returns_metadata_and_persists_row(_backup_env):
|
|
||||||
svc, backup_dir = _backup_env
|
|
||||||
r = _client().post("/api/admin/backup/create")
|
|
||||||
assert r.status_code == 200, r.text
|
|
||||||
body = r.json()
|
|
||||||
assert body["ok"] is True
|
|
||||||
b = body["backup"]
|
|
||||||
assert b["size_bytes"] > 0
|
|
||||||
assert b["db_fingerprint"].startswith("sha256:")
|
|
||||||
assert b["table_count"] >= 1
|
|
||||||
assert b["created_at"]
|
|
||||||
# File actually exists on disk.
|
|
||||||
assert (backup_dir / b["filename"]).exists()
|
|
||||||
# Sidecar metadata echoed.
|
|
||||||
sc = body["sidecar"]
|
|
||||||
assert sc["kdf"] == "PBKDF2-HMAC-SHA256"
|
|
||||||
assert sc["kdf_iterations"] == 200_000
|
|
||||||
assert sc["cipher"] == "AES-256-GCM"
|
|
||||||
|
|
||||||
|
|
||||||
def test_create_503_when_service_unconfigured(tmp_path, monkeypatch):
|
|
||||||
"""If BackupService was never configured, create returns 503."""
|
|
||||||
from cyclone import db
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
|
|
||||||
db._reset_for_tests()
|
|
||||||
db.init_db()
|
|
||||||
svc_mod.reset_backup_service_for_tests()
|
|
||||||
try:
|
|
||||||
r = _client().post("/api/admin/backup/create")
|
|
||||||
assert r.status_code == 503
|
|
||||||
assert "not configured" in r.json()["detail"].lower()
|
|
||||||
finally:
|
|
||||||
db._reset_for_tests()
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# /backup/list
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_list_returns_newest_first(_backup_env):
|
|
||||||
svc, _ = _backup_env
|
|
||||||
client = _client()
|
|
||||||
client.post("/api/admin/backup/create")
|
|
||||||
client.post("/api/admin/backup/create")
|
|
||||||
r = client.get("/api/admin/backup/list")
|
|
||||||
assert r.status_code == 200
|
|
||||||
body = r.json()
|
|
||||||
assert body["count"] == 2
|
|
||||||
assert body["files"][0]["id"] > body["files"][1]["id"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_list_filter_by_status(_backup_env):
|
|
||||||
svc, _ = _backup_env
|
|
||||||
client = _client()
|
|
||||||
client.post("/api/admin/backup/create")
|
|
||||||
r = client.get("/api/admin/backup/list?status=ok")
|
|
||||||
assert r.json()["count"] == 1
|
|
||||||
r = client.get("/api/admin/backup/list?status=error")
|
|
||||||
assert r.json()["count"] == 0
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# /backup/status
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_status_returns_counts_and_dirs(_backup_env):
|
|
||||||
svc, backup_dir = _backup_env
|
|
||||||
client = _client()
|
|
||||||
client.post("/api/admin/backup/create")
|
|
||||||
r = client.get("/api/admin/backup/status")
|
|
||||||
assert r.status_code == 200
|
|
||||||
body = r.json()
|
|
||||||
assert body["totals"]["ok"] == 1
|
|
||||||
assert body["backup_dir"] == str(backup_dir)
|
|
||||||
assert body["retention_days"] == 7
|
|
||||||
assert body["last_backup_at"] is not None
|
|
||||||
assert body["last_ok_backup_at"] is not None
|
|
||||||
# The scheduler may or may not be configured depending on lifespan.
|
|
||||||
assert "scheduler" in body
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# /backup/{id}/verify
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_verify_ok_after_create(_backup_env):
|
|
||||||
svc, _ = _backup_env
|
|
||||||
client = _client()
|
|
||||||
cid = client.post("/api/admin/backup/create").json()["backup"]["id"]
|
|
||||||
r = client.post(f"/api/admin/backup/{cid}/verify")
|
|
||||||
assert r.status_code == 200
|
|
||||||
body = r.json()
|
|
||||||
assert body["ok"] is True
|
|
||||||
assert body["expected_fingerprint"] == body["actual_fingerprint"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_verify_detects_tampered_ciphertext(_backup_env):
|
|
||||||
from cyclone import backup as backup_mod
|
|
||||||
svc, backup_dir = _backup_env
|
|
||||||
client = _client()
|
|
||||||
cid = client.post("/api/admin/backup/create").json()["backup"]["id"]
|
|
||||||
fname = svc.list_backups()[0].filename
|
|
||||||
# Flip a bit in the ciphertext.
|
|
||||||
bin_path = backup_dir / fname
|
|
||||||
data = bytearray(bin_path.read_bytes())
|
|
||||||
data[backup_mod.NONCE_LEN + 5] ^= 0x01
|
|
||||||
bin_path.write_bytes(bytes(data))
|
|
||||||
r = client.post(f"/api/admin/backup/{cid}/verify")
|
|
||||||
assert r.status_code == 200
|
|
||||||
assert r.json()["ok"] is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_verify_404_when_unknown_backup(_backup_env):
|
|
||||||
r = _client().post("/api/admin/backup/99999/verify")
|
|
||||||
# The service raises BackupError; the endpoint should return 503 (no svc) or 400
|
|
||||||
# depending on flow. Let's see what happens.
|
|
||||||
assert r.status_code in (400, 404, 503)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# /backup/{id}/restore/{initiate,confirm}
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_restore_two_step_via_api(_backup_env):
|
|
||||||
svc, _ = _backup_env
|
|
||||||
from cyclone.db import Batch
|
|
||||||
import uuid
|
|
||||||
client = _client()
|
|
||||||
cid = client.post("/api/admin/backup/create").json()["backup"]["id"]
|
|
||||||
|
|
||||||
# Mutate the live DB (add another Batch row).
|
|
||||||
with __import__("cyclone").db.SessionLocal()() as s:
|
|
||||||
s.add(Batch(
|
|
||||||
id=str(uuid.uuid4()),
|
|
||||||
kind="837P",
|
|
||||||
input_filename="mutated.x12",
|
|
||||||
parsed_at=datetime.now(timezone.utc),
|
|
||||||
totals_json=None,
|
|
||||||
validation_json=None,
|
|
||||||
raw_result_json={"envelope": {"control_number": "2"}, "claims": [], "summary": {"passed": 0, "failed": 0, "failed_claim_ids": []}},
|
|
||||||
))
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
# Step 1: initiate.
|
|
||||||
r1 = client.post(f"/api/admin/backup/{cid}/restore/initiate")
|
|
||||||
assert r1.status_code == 200, r1.text
|
|
||||||
body1 = r1.json()
|
|
||||||
assert body1["restore_token"]
|
|
||||||
assert body1["preview"]["backup_table_count"] >= 1
|
|
||||||
assert body1["preview"]["backup_db_fingerprint"] != body1["preview"]["current_db_fingerprint"]
|
|
||||||
|
|
||||||
# Step 2: confirm.
|
|
||||||
r2 = client.post(
|
|
||||||
f"/api/admin/backup/{cid}/restore/confirm",
|
|
||||||
json={"restore_token": body1["restore_token"], "actor": "test"},
|
|
||||||
)
|
|
||||||
assert r2.status_code == 200, r2.text
|
|
||||||
body2 = r2.json()
|
|
||||||
assert body2["ok"] is True
|
|
||||||
assert body2["new_db_fingerprint"] == body1["preview"]["backup_db_fingerprint"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_restore_confirm_requires_token(_backup_env):
|
|
||||||
svc, _ = _backup_env
|
|
||||||
client = _client()
|
|
||||||
cid = client.post("/api/admin/backup/create").json()["backup"]["id"]
|
|
||||||
r = client.post(f"/api/admin/backup/{cid}/restore/confirm", json={})
|
|
||||||
assert r.status_code == 400
|
|
||||||
|
|
||||||
|
|
||||||
def test_restore_confirm_rejects_wrong_token(_backup_env):
|
|
||||||
svc, _ = _backup_env
|
|
||||||
client = _client()
|
|
||||||
cid = client.post("/api/admin/backup/create").json()["backup"]["id"]
|
|
||||||
r = client.post(
|
|
||||||
f"/api/admin/backup/{cid}/restore/confirm",
|
|
||||||
json={"restore_token": "0" * 64},
|
|
||||||
)
|
|
||||||
assert r.status_code == 400
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# /backup/prune
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_prune_deletes_old_backups(_backup_env):
|
|
||||||
svc, _ = _backup_env
|
|
||||||
from cyclone.db import DbBackup
|
|
||||||
client = _client()
|
|
||||||
cid = client.post("/api/admin/backup/create").json()["backup"]["id"]
|
|
||||||
# Age the backup past the retention cutoff.
|
|
||||||
with __import__("cyclone").db.SessionLocal()() as s:
|
|
||||||
row = s.get(DbBackup, cid)
|
|
||||||
row.created_at = datetime.now(timezone.utc) - timedelta(days=30)
|
|
||||||
s.commit()
|
|
||||||
r = client.post("/api/admin/backup/prune")
|
|
||||||
assert r.status_code == 200
|
|
||||||
body = r.json()
|
|
||||||
assert body["ok"] is True
|
|
||||||
assert body["deleted_count"] == 2 # .bin + .meta.json
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# /backup/scheduler/{start,stop,tick}
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_scheduler_endpoints_require_configured_scheduler(_backup_env, monkeypatch):
|
|
||||||
"""Without calling configure_backup_scheduler, the endpoints 503."""
|
|
||||||
svc, _ = _backup_env
|
|
||||||
# We did NOT call configure_backup_scheduler; the lifespan
|
|
||||||
# *might* have called it as a side effect of the TestClient
|
|
||||||
# entering its context. Either way, the scheduler endpoints
|
|
||||||
# need it to be present.
|
|
||||||
client = _client()
|
|
||||||
r = client.post("/api/admin/backup/scheduler/tick")
|
|
||||||
assert r.status_code in (200, 503)
|
|
||||||
|
|
||||||
|
|
||||||
def test_scheduler_tick_when_configured(_backup_env):
|
|
||||||
"""With a configured scheduler, tick runs and returns a result."""
|
|
||||||
from cyclone import backup_scheduler as sched_mod
|
|
||||||
svc, _ = _backup_env
|
|
||||||
sched_mod.configure_backup_scheduler(svc, interval_hours=24.0)
|
|
||||||
try:
|
|
||||||
client = _client()
|
|
||||||
r = client.post("/api/admin/backup/scheduler/tick")
|
|
||||||
assert r.status_code == 200
|
|
||||||
body = r.json()
|
|
||||||
assert body["ok"] is True
|
|
||||||
assert body["tick"]["created"] is not None
|
|
||||||
assert body["tick"]["created"]["id"] >= 1
|
|
||||||
finally:
|
|
||||||
sched_mod.reset_backup_scheduler_for_tests()
|
|
||||||
@@ -1,317 +0,0 @@
|
|||||||
"""Tests for POST /api/batches/{batch_id}/export-837.
|
|
||||||
|
|
||||||
Reads Claim.raw_json for each requested claim_id and returns a ZIP of
|
|
||||||
regenerated X12 837 files. No DB state mutation. Mirrors the
|
|
||||||
X-Cyclone-Serialize-Errors convention from /api/inbox/rejected/resubmit?download=true.
|
|
||||||
"""
|
|
||||||
import io
|
|
||||||
import json
|
|
||||||
import zipfile
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
|
|
||||||
from cyclone.api import app
|
|
||||||
|
|
||||||
|
|
||||||
def _seed_batch(client: TestClient, filename: str = "claim.txt") -> dict:
|
|
||||||
"""Parse a single 837P file and return a dict with ``batch_id`` and
|
|
||||||
``claims``.
|
|
||||||
|
|
||||||
The parse-837 happy-path response does not currently surface
|
|
||||||
``batch_id`` at the top level (it's a server-side UUID, surfaced in
|
|
||||||
409 errors but not in 200s). We look it up from the DB — the Batch
|
|
||||||
row is already persisted by the time the parse endpoint returns.
|
|
||||||
"""
|
|
||||||
from cyclone import db
|
|
||||||
from cyclone.db import Batch
|
|
||||||
|
|
||||||
fixture = Path("tests/fixtures/co_medicaid_837p.txt").read_text()
|
|
||||||
r = client.post(
|
|
||||||
"/api/parse-837",
|
|
||||||
files={"file": (filename, io.BytesIO(fixture.encode()), "text/plain")},
|
|
||||||
headers={"Accept": "application/json"},
|
|
||||||
)
|
|
||||||
assert r.status_code == 200, r.text
|
|
||||||
body = r.json()
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
most_recent = s.query(Batch).order_by(Batch.parsed_at.desc()).first()
|
|
||||||
assert most_recent is not None, "expected at least one Batch row after parse"
|
|
||||||
batch_id = most_recent.id
|
|
||||||
return {"batch_id": batch_id, "claims": body["claims"]}
|
|
||||||
|
|
||||||
|
|
||||||
def _claim_ids_from_seed(seeded: dict) -> list[str]:
|
|
||||||
return [c["claim_id"] for c in seeded["claims"]]
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Happy path
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_happy_path_returns_zip_with_one_x12_per_claim():
|
|
||||||
with TestClient(app) as client:
|
|
||||||
seeded = _seed_batch(client)
|
|
||||||
batch_id = seeded["batch_id"]
|
|
||||||
claim_ids = _claim_ids_from_seed(seeded)
|
|
||||||
|
|
||||||
r = client.post(
|
|
||||||
f"/api/batches/{batch_id}/export-837",
|
|
||||||
json={"claim_ids": claim_ids},
|
|
||||||
)
|
|
||||||
|
|
||||||
assert r.status_code == 200, r.text
|
|
||||||
assert r.headers["content-type"].startswith("application/zip")
|
|
||||||
assert "attachment" in r.headers["content-disposition"]
|
|
||||||
assert (
|
|
||||||
f"batch-{batch_id}-{len(claim_ids)}-claims.zip"
|
|
||||||
in r.headers["content-disposition"]
|
|
||||||
)
|
|
||||||
# Per-claim failures header absent on full success.
|
|
||||||
assert "x-cyclone-serialize-errors" not in {k.lower() for k in r.headers.keys()}
|
|
||||||
|
|
||||||
with zipfile.ZipFile(io.BytesIO(r.content)) as zf:
|
|
||||||
names = zf.namelist()
|
|
||||||
assert len(names) == len(claim_ids)
|
|
||||||
# Each entry follows the HCPF outbound naming template
|
|
||||||
# "{tpid}-837P-{yyyymmddhhmmssSSS}-1of1.x12" (the seeded
|
|
||||||
# clearhouse TPID is "11525703"). Every name must be unique.
|
|
||||||
from cyclone.edi.filenames import is_outbound_filename
|
|
||||||
seen = set()
|
|
||||||
for name in names:
|
|
||||||
assert is_outbound_filename(name), (
|
|
||||||
f"expected HCPF outbound filename, got {name!r}"
|
|
||||||
)
|
|
||||||
assert name not in seen, f"duplicate filename: {name}"
|
|
||||||
seen.add(name)
|
|
||||||
with zf.open(name) as f:
|
|
||||||
first_line = f.readline().decode("ascii", errors="replace")
|
|
||||||
assert first_line.startswith("ISA*"), f"{name} didn't start with ISA"
|
|
||||||
|
|
||||||
|
|
||||||
def test_each_x12_in_zip_uses_clearhouse_submit_and_payer_receiver():
|
|
||||||
"""Regression: regenerated 837s used to emit 'CYCLONE' / 'RECEIVER'
|
|
||||||
placeholders. The export endpoint must thread the clearhouse
|
|
||||||
submitter (dzinesco's TPID 11525703) and payer receiver
|
|
||||||
(COMEDASSISTPROG) through to the serializer."""
|
|
||||||
with TestClient(app) as client:
|
|
||||||
seeded = _seed_batch(client)
|
|
||||||
batch_id = seeded["batch_id"]
|
|
||||||
claim_ids = _claim_ids_from_seed(seeded)
|
|
||||||
r = client.post(
|
|
||||||
f"/api/batches/{batch_id}/export-837",
|
|
||||||
json={"claim_ids": claim_ids},
|
|
||||||
)
|
|
||||||
assert r.status_code == 200, r.text
|
|
||||||
|
|
||||||
with zipfile.ZipFile(io.BytesIO(r.content)) as zf:
|
|
||||||
for name in zf.namelist():
|
|
||||||
with zf.open(name) as f:
|
|
||||||
text = f.read().decode("ascii")
|
|
||||||
# Submitter block must use the clearhouse TPID + name, not
|
|
||||||
# the 'CYCLONE' placeholder.
|
|
||||||
assert "CYCLONE" not in text, f"{name} still emits CYCLONE placeholder"
|
|
||||||
assert "11525703" in text, f"{name} missing clearhouse TPID"
|
|
||||||
assert "Dzinesco" in text, f"{name} missing clearhouse name"
|
|
||||||
# Receiver block must use the CO_TXIX payer config
|
|
||||||
# (COMEDASSISTPROG), not the 'RECEIVER' placeholder.
|
|
||||||
assert "RECEIVER" not in text, f"{name} still emits RECEIVER placeholder"
|
|
||||||
assert "COMEDASSISTPROG" in text, f"{name} missing receiver id"
|
|
||||||
# Loop 1000A requires PER — must be present, not omitted.
|
|
||||||
assert "PER*IC*" in text, f"{name} missing required PER segment"
|
|
||||||
# SBR09 should be 'MC' (Medicaid claim filing indicator),
|
|
||||||
# not the member id.
|
|
||||||
sbr_line = next(seg for seg in text.split("~") if seg.startswith("SBR*"))
|
|
||||||
sbr09 = sbr_line.rstrip("~").split("*")[9]
|
|
||||||
assert sbr09 == "MC", f"{name} SBR09 expected 'MC', got {sbr09!r}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_each_x12_in_zip_round_trips_through_parser():
|
|
||||||
"""Fidelity check: each .x12 must parse back to a ClaimOutput deep-equal
|
|
||||||
to the source row's raw_json (modulo recomputed validation)."""
|
|
||||||
from cyclone.parsers.parse_837 import parse
|
|
||||||
from cyclone.parsers.payer import PayerConfig
|
|
||||||
|
|
||||||
with TestClient(app) as client:
|
|
||||||
seeded = _seed_batch(client)
|
|
||||||
batch_id = seeded["batch_id"]
|
|
||||||
claim_ids = _claim_ids_from_seed(seeded)
|
|
||||||
|
|
||||||
r = client.post(
|
|
||||||
f"/api/batches/{batch_id}/export-837",
|
|
||||||
json={"claim_ids": claim_ids},
|
|
||||||
)
|
|
||||||
assert r.status_code == 200, r.text
|
|
||||||
|
|
||||||
by_id = {c["claim_id"]: c for c in seeded["claims"]}
|
|
||||||
with zipfile.ZipFile(io.BytesIO(r.content)) as zf:
|
|
||||||
for name in zf.namelist():
|
|
||||||
with zf.open(name) as f:
|
|
||||||
text = f.read().decode("ascii")
|
|
||||||
result = parse(text, PayerConfig(name="CO_MEDICAID"))
|
|
||||||
assert result.claims, f"{name} didn't parse back to any claims"
|
|
||||||
# Claim ids must round-trip (proves the serializer didn't drop
|
|
||||||
# or rewrite the id).
|
|
||||||
assert result.claims[0].claim.claim_id in by_id, (
|
|
||||||
f"{name} parsed to an unknown claim_id"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Partial-failure surface
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_partial_failure_one_claim_with_no_raw_json_returns_zip_and_errors_header():
|
|
||||||
"""If one claim has raw_json=None, the ZIP still returns for the others
|
|
||||||
and the failure is surfaced via X-Cyclone-Serialize-Errors."""
|
|
||||||
from cyclone import db
|
|
||||||
from cyclone.edi.filenames import is_outbound_filename
|
|
||||||
|
|
||||||
with TestClient(app) as client:
|
|
||||||
seeded = _seed_batch(client)
|
|
||||||
batch_id = seeded["batch_id"]
|
|
||||||
claim_ids = _claim_ids_from_seed(seeded)
|
|
||||||
assert len(claim_ids) >= 2, "fixture must produce at least 2 claims"
|
|
||||||
|
|
||||||
# Wipe raw_json on one claim to simulate a corrupted row.
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
from cyclone.db import Claim
|
|
||||||
target = s.get(Claim, claim_ids[0])
|
|
||||||
target.raw_json = None
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
r = client.post(
|
|
||||||
f"/api/batches/{batch_id}/export-837",
|
|
||||||
json={"claim_ids": claim_ids},
|
|
||||||
)
|
|
||||||
|
|
||||||
assert r.status_code == 200, r.text
|
|
||||||
# Filename uses SUCCESS count, not requested count.
|
|
||||||
expected_success = len(claim_ids) - 1
|
|
||||||
assert (
|
|
||||||
f"batch-{batch_id}-{expected_success}-claims.zip"
|
|
||||||
in r.headers["content-disposition"]
|
|
||||||
)
|
|
||||||
err_header = r.headers.get("x-cyclone-serialize-errors")
|
|
||||||
assert err_header, "expected X-Cyclone-Serialize-Errors header"
|
|
||||||
errs = json.loads(err_header)
|
|
||||||
assert len(errs) == 1
|
|
||||||
assert errs[0]["claim_id"] == claim_ids[0]
|
|
||||||
assert "raw_json" in errs[0]["reason"].lower()
|
|
||||||
|
|
||||||
with zipfile.ZipFile(io.BytesIO(r.content)) as zf:
|
|
||||||
names = zf.namelist()
|
|
||||||
assert len(names) == expected_success
|
|
||||||
# Every entry follows HCPF outbound template; none is the
|
|
||||||
# 'claim-CLM-X.x12' placeholder from the old endpoint.
|
|
||||||
for name in names:
|
|
||||||
assert is_outbound_filename(name), f"non-HCPF name: {name!r}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_all_claims_fail_to_serialize_returns_422():
|
|
||||||
from cyclone import db
|
|
||||||
|
|
||||||
with TestClient(app) as client:
|
|
||||||
seeded = _seed_batch(client)
|
|
||||||
batch_id = seeded["batch_id"]
|
|
||||||
claim_ids = _claim_ids_from_seed(seeded)
|
|
||||||
assert len(claim_ids) >= 1
|
|
||||||
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
from cyclone.db import Claim
|
|
||||||
for cid in claim_ids:
|
|
||||||
c = s.get(Claim, cid)
|
|
||||||
c.raw_json = None
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
r = client.post(
|
|
||||||
f"/api/batches/{batch_id}/export-837",
|
|
||||||
json={"claim_ids": claim_ids},
|
|
||||||
)
|
|
||||||
|
|
||||||
assert r.status_code == 422, r.text
|
|
||||||
body = r.json()
|
|
||||||
# The 422 body surfaces the failure list so the UI can show details
|
|
||||||
# without parsing a header.
|
|
||||||
errs = body.get("detail", {}).get("serialize_errors") or body.get("serialize_errors")
|
|
||||||
assert errs is not None
|
|
||||||
assert len(errs) == len(claim_ids)
|
|
||||||
for entry in errs:
|
|
||||||
assert entry["claim_id"] in claim_ids
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Error cases
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_empty_claim_ids_returns_400():
|
|
||||||
with TestClient(app) as client:
|
|
||||||
seeded = _seed_batch(client)
|
|
||||||
batch_id = seeded["batch_id"]
|
|
||||||
r = client.post(
|
|
||||||
f"/api/batches/{batch_id}/export-837",
|
|
||||||
json={"claim_ids": []},
|
|
||||||
)
|
|
||||||
assert r.status_code == 400, r.text
|
|
||||||
|
|
||||||
|
|
||||||
def test_missing_claim_ids_key_returns_400():
|
|
||||||
with TestClient(app) as client:
|
|
||||||
seeded = _seed_batch(client)
|
|
||||||
batch_id = seeded["batch_id"]
|
|
||||||
r = client.post(
|
|
||||||
f"/api/batches/{batch_id}/export-837",
|
|
||||||
json={},
|
|
||||||
)
|
|
||||||
assert r.status_code == 400, r.text
|
|
||||||
|
|
||||||
|
|
||||||
def test_unknown_batch_id_returns_404():
|
|
||||||
with TestClient(app) as client:
|
|
||||||
r = client.post(
|
|
||||||
"/api/batches/BATCH-DOES-NOT-EXIST/export-837",
|
|
||||||
json={"claim_ids": ["CLM-1"]},
|
|
||||||
)
|
|
||||||
assert r.status_code == 404, r.text
|
|
||||||
body = r.json()
|
|
||||||
assert "BATCH-DOES-NOT-EXIST" in (body.get("detail") or "")
|
|
||||||
|
|
||||||
|
|
||||||
def test_unknown_claim_id_silently_omitted():
|
|
||||||
"""A claim_id that doesn't exist is omitted from the ZIP and surfaced
|
|
||||||
in the errors header — same convention as the resubmit endpoint."""
|
|
||||||
with TestClient(app) as client:
|
|
||||||
seeded = _seed_batch(client)
|
|
||||||
batch_id = seeded["batch_id"]
|
|
||||||
real_ids = _claim_ids_from_seed(seeded)
|
|
||||||
assert real_ids, "fixture must produce at least 1 claim"
|
|
||||||
|
|
||||||
requested = real_ids + ["CLM-GHOST"]
|
|
||||||
r = client.post(
|
|
||||||
f"/api/batches/{batch_id}/export-837",
|
|
||||||
json={"claim_ids": requested},
|
|
||||||
)
|
|
||||||
|
|
||||||
assert r.status_code == 200, r.text
|
|
||||||
# Filename uses success count = len(real_ids), not len(requested).
|
|
||||||
assert (
|
|
||||||
f"batch-{batch_id}-{len(real_ids)}-claims.zip"
|
|
||||||
in r.headers["content-disposition"]
|
|
||||||
)
|
|
||||||
err_header = r.headers.get("x-cyclone-serialize-errors")
|
|
||||||
assert err_header
|
|
||||||
errs = json.loads(err_header)
|
|
||||||
assert any(e["claim_id"] == "CLM-GHOST" for e in errs)
|
|
||||||
|
|
||||||
with zipfile.ZipFile(io.BytesIO(r.content)) as zf:
|
|
||||||
names = zf.namelist()
|
|
||||||
# HCPF outbound filenames, one per real claim. We don't pin the
|
|
||||||
# exact ts (it depends on the wall clock at test time), but the
|
|
||||||
# count and the HCPF template must match.
|
|
||||||
from cyclone.edi.filenames import is_outbound_filename
|
|
||||||
assert len(names) == len(real_ids)
|
|
||||||
for name in names:
|
|
||||||
assert is_outbound_filename(name), f"non-HCPF name: {name!r}"
|
|
||||||
@@ -65,42 +65,6 @@ def test_batches_returns_summary_after_parse(seeded_store):
|
|||||||
assert body["items"][0]["inputFilename"] == "x.txt"
|
assert body["items"][0]["inputFilename"] == "x.txt"
|
||||||
|
|
||||||
|
|
||||||
def test_batches_includes_claim_ids_for_837p(seeded_store):
|
|
||||||
"""The Upload page's History tab renders a Re-export ZIP button per
|
|
||||||
row that fires ``POST /api/batches/{id}/export-837`` with the row's
|
|
||||||
claim ids. Carry those ids in the list response so the UI doesn't
|
|
||||||
need an extra round-trip per row to fetch them."""
|
|
||||||
batches = seeded_store.get("/api/batches", headers=JSON).json()["items"]
|
|
||||||
assert len(batches) == 1
|
|
||||||
item = batches[0]
|
|
||||||
assert item["kind"] == "837p"
|
|
||||||
# claimIds is a non-empty list of the same claim ids that live
|
|
||||||
# inside the parsed result — see `seeded_store` in conftest.py.
|
|
||||||
assert isinstance(item["claimIds"], list)
|
|
||||||
assert len(item["claimIds"]) == 2
|
|
||||||
full = seeded_store.get(f"/api/batches/{item['id']}").json()
|
|
||||||
assert sorted(item["claimIds"]) == sorted(c["claim_id"] for c in full["claims"])
|
|
||||||
|
|
||||||
|
|
||||||
def test_batches_claim_ids_empty_for_835(client: TestClient, tmp_path):
|
|
||||||
"""835 has no re-export endpoint — the field is always ``[]`` so the
|
|
||||||
History tab can use it as the signal to hide the Re-export button."""
|
|
||||||
src = Path(__file__).parent / "fixtures" / "minimal_835.txt"
|
|
||||||
files = {"file": ("minimal_835.txt", src.read_bytes(), "text/plain")}
|
|
||||||
r = client.post(
|
|
||||||
"/api/parse-835",
|
|
||||||
params={"payer": "co_medicaid_835"},
|
|
||||||
files=files,
|
|
||||||
headers=JSON,
|
|
||||||
)
|
|
||||||
assert r.status_code == 200, r.text
|
|
||||||
body = client.get("/api/batches", headers=JSON).json()
|
|
||||||
assert body["total"] == 1
|
|
||||||
item = body["items"][0]
|
|
||||||
assert item["kind"] == "835"
|
|
||||||
assert item["claimIds"] == []
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# /api/batches/{id}
|
# /api/batches/{id}
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
|
|||||||
@@ -1,29 +0,0 @@
|
|||||||
"""Tests that /api/parse-837 includes the persisted batch_id in its
|
|
||||||
JSON response, so the frontend can correlate its in-memory batch with
|
|
||||||
the server's row (and later call /api/batches/{id}/export-837).
|
|
||||||
|
|
||||||
The 409 (duplicate) path has always included batch_id; the 200 path
|
|
||||||
did not. This brings the 200 path in line so the frontend doesn't have
|
|
||||||
to query listBatches after every parse just to learn the server's id.
|
|
||||||
"""
|
|
||||||
import io
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
|
|
||||||
from cyclone.api import app
|
|
||||||
|
|
||||||
|
|
||||||
def test_parse_837_happy_path_includes_batch_id_in_response():
|
|
||||||
with TestClient(app) as client:
|
|
||||||
fixture = Path("tests/fixtures/co_medicaid_837p.txt").read_text()
|
|
||||||
r = client.post(
|
|
||||||
"/api/parse-837",
|
|
||||||
files={"file": ("claim.txt", io.BytesIO(fixture.encode()), "text/plain")},
|
|
||||||
headers={"Accept": "application/json"},
|
|
||||||
)
|
|
||||||
assert r.status_code == 200, r.text
|
|
||||||
body = r.json()
|
|
||||||
assert "batch_id" in body, "parse-837 response must include batch_id so the frontend can call /api/batches/{id}/export-837"
|
|
||||||
# Sanity: the batch_id should be a non-empty string (UUID-shaped).
|
|
||||||
assert isinstance(body["batch_id"], str) and body["batch_id"]
|
|
||||||
@@ -1,152 +0,0 @@
|
|||||||
"""SP16 — Admin scheduler API endpoint tests.
|
|
||||||
|
|
||||||
The endpoints under /api/admin/scheduler/* are thin wrappers around
|
|
||||||
:class:`cyclone.scheduler.Scheduler`. These tests exercise them via
|
|
||||||
the FastAPI TestClient to confirm wiring (auth-free admin endpoints
|
|
||||||
work, response shapes match, idempotency holds).
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import json
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def _stub_scheduler_env(tmp_path, monkeypatch):
|
|
||||||
"""Set up: a stub-mode SFTP block, scheduler configured.
|
|
||||||
|
|
||||||
Yields (staging_dir, scheduler_singleton). We deliberately do
|
|
||||||
NOT enable SQLCipher encryption in this fixture — the scheduler
|
|
||||||
doesn't care about encryption, and patching ``db_crypto.get_secret``
|
|
||||||
here would cause the lifespan handler to rebuild the engine with
|
|
||||||
SQLCipher on a plain-SQLite test file (which raises "file is not
|
|
||||||
a database"). The encryption-at-rest tests live in
|
|
||||||
``test_db_crypto.py``.
|
|
||||||
"""
|
|
||||||
from cyclone import db
|
|
||||||
from cyclone import scheduler as sched_mod
|
|
||||||
from cyclone.providers import SftpBlock
|
|
||||||
|
|
||||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
|
|
||||||
db._reset_for_tests()
|
|
||||||
|
|
||||||
staging = tmp_path / "staging"
|
|
||||||
inbound = staging / "ToHPE"
|
|
||||||
inbound.mkdir(parents=True)
|
|
||||||
sftp_block = SftpBlock(
|
|
||||||
host="mft.example.com",
|
|
||||||
port=22,
|
|
||||||
username="test",
|
|
||||||
paths={"outbound": "/FromHPE", "inbound": "/ToHPE"},
|
|
||||||
stub=True,
|
|
||||||
staging_dir=str(staging),
|
|
||||||
poll_seconds=60,
|
|
||||||
auth={"method": "keychain", "secret_ref": "test.password"},
|
|
||||||
)
|
|
||||||
sched_mod.reset_scheduler_for_tests()
|
|
||||||
sched = sched_mod.configure_scheduler(sftp_block, sftp_block_name="t")
|
|
||||||
yield staging, sched
|
|
||||||
sched_mod.reset_scheduler_for_tests()
|
|
||||||
db._reset_for_tests()
|
|
||||||
|
|
||||||
|
|
||||||
def _drop_file(staging: Path, name: str, body: bytes) -> Path:
|
|
||||||
p = staging / "ToHPE" / name
|
|
||||||
p.write_bytes(body)
|
|
||||||
return p
|
|
||||||
|
|
||||||
|
|
||||||
def test_scheduler_status_starts_not_running(_stub_scheduler_env):
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
from cyclone.api import app
|
|
||||||
_, sched = _stub_scheduler_env
|
|
||||||
with TestClient(app) as client:
|
|
||||||
r = client.get("/api/admin/scheduler/status")
|
|
||||||
assert r.status_code == 200
|
|
||||||
body = r.json()
|
|
||||||
assert body["running"] is False
|
|
||||||
assert body["poll_interval_seconds"] == 60
|
|
||||||
assert body["sftp_block_name"] == "t"
|
|
||||||
|
|
||||||
|
|
||||||
def test_scheduler_start_then_status_then_stop(_stub_scheduler_env):
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
from cyclone.api import app
|
|
||||||
with TestClient(app) as client:
|
|
||||||
r1 = client.post("/api/admin/scheduler/start")
|
|
||||||
assert r1.status_code == 200
|
|
||||||
assert r1.json()["status"]["running"] is True
|
|
||||||
|
|
||||||
r2 = client.get("/api/admin/scheduler/status")
|
|
||||||
assert r2.json()["running"] is True
|
|
||||||
|
|
||||||
r3 = client.post("/api/admin/scheduler/stop")
|
|
||||||
assert r3.status_code == 200
|
|
||||||
assert r3.json()["status"]["running"] is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_scheduler_tick_processes_one_file(_stub_scheduler_env):
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
from cyclone.api import app
|
|
||||||
staging, _ = _stub_scheduler_env
|
|
||||||
_drop_file(
|
|
||||||
staging,
|
|
||||||
"TP11525703-837P_M019048402-20260618130000000-1of1_TA1.x12",
|
|
||||||
(Path(__file__).parent / "fixtures" / "minimal_ta1.txt").read_bytes(),
|
|
||||||
)
|
|
||||||
with TestClient(app) as client:
|
|
||||||
r = client.post("/api/admin/scheduler/tick")
|
|
||||||
assert r.status_code == 200
|
|
||||||
body = r.json()
|
|
||||||
assert body["ok"] is True
|
|
||||||
assert body["tick"]["files_seen"] == 1
|
|
||||||
assert body["tick"]["files_processed"] == 1
|
|
||||||
|
|
||||||
|
|
||||||
def test_scheduler_processed_files_lists_history(_stub_scheduler_env):
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
from cyclone.api import app
|
|
||||||
staging, _ = _stub_scheduler_env
|
|
||||||
_drop_file(
|
|
||||||
staging,
|
|
||||||
"TP11525703-837P_M019048402-20260618130000000-1of1_TA1.x12",
|
|
||||||
(Path(__file__).parent / "fixtures" / "minimal_ta1.txt").read_bytes(),
|
|
||||||
)
|
|
||||||
with TestClient(app) as client:
|
|
||||||
client.post("/api/admin/scheduler/tick")
|
|
||||||
r = client.get("/api/admin/scheduler/processed-files")
|
|
||||||
assert r.status_code == 200
|
|
||||||
body = r.json()
|
|
||||||
assert body["count"] == 1
|
|
||||||
f = body["files"][0]
|
|
||||||
assert f["status"] == "ok"
|
|
||||||
assert f["parser_used"] == "parse_ta1"
|
|
||||||
assert "TP11525703" in f["name"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_scheduler_processed_files_filters_by_status(_stub_scheduler_env):
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
from cyclone.api import app
|
|
||||||
staging, _ = _stub_scheduler_env
|
|
||||||
# Drop a file with a type Cyclone doesn't parse — gets recorded as
|
|
||||||
# "skipped".
|
|
||||||
_drop_file(
|
|
||||||
staging,
|
|
||||||
"TP11525703-837P_M019048402-20260618130000000-1of1_270.x12",
|
|
||||||
b"some bytes",
|
|
||||||
)
|
|
||||||
with TestClient(app) as client:
|
|
||||||
client.post("/api/admin/scheduler/tick")
|
|
||||||
r_all = client.get("/api/admin/scheduler/processed-files")
|
|
||||||
r_skipped = client.get(
|
|
||||||
"/api/admin/scheduler/processed-files?status=skipped",
|
|
||||||
)
|
|
||||||
r_ok = client.get(
|
|
||||||
"/api/admin/scheduler/processed-files?status=ok",
|
|
||||||
)
|
|
||||||
assert r_all.json()["count"] == 1
|
|
||||||
assert r_skipped.json()["count"] == 1
|
|
||||||
assert r_ok.json()["count"] == 0
|
|
||||||
@@ -1,69 +0,0 @@
|
|||||||
"""Tests for ``GET /api/admin/validate-provider`` (SP20).
|
|
||||||
|
|
||||||
Pure read-only endpoint — runs the local NPI Luhn + EIN format checks.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
|
|
||||||
from cyclone.api import app
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def client() -> TestClient:
|
|
||||||
return TestClient(app)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Both fields populated
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_validate_provider_both_valid(client: TestClient):
|
|
||||||
resp = client.get("/api/admin/validate-provider", params={
|
|
||||||
"npi": "1234567893", # CMS-published valid NPI
|
|
||||||
"tax_id": "72-1587149", # Touch of Care EIN
|
|
||||||
})
|
|
||||||
assert resp.status_code == 200
|
|
||||||
body = resp.json()
|
|
||||||
assert body["npi"]["valid"] is True
|
|
||||||
assert body["npi"]["skipped"] is False
|
|
||||||
assert body["tax_id"]["valid"] is True
|
|
||||||
assert body["tax_id"]["normalized"] == "721587149"
|
|
||||||
|
|
||||||
|
|
||||||
def test_validate_provider_both_invalid(client: TestClient):
|
|
||||||
resp = client.get("/api/admin/validate-provider", params={
|
|
||||||
"npi": "1234567890", # format OK but Luhn fails
|
|
||||||
"tax_id": "00-1234567", # reserved prefix
|
|
||||||
})
|
|
||||||
assert resp.status_code == 200
|
|
||||||
body = resp.json()
|
|
||||||
assert body["npi"]["valid"] is False
|
|
||||||
assert body["tax_id"]["valid"] is False
|
|
||||||
assert body["tax_id"]["normalized"] == "001234567"
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Param omission → skipped
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_validate_provider_skips_missing_npi(client: TestClient):
|
|
||||||
resp = client.get("/api/admin/validate-provider", params={"tax_id": "721587149"})
|
|
||||||
assert resp.status_code == 200
|
|
||||||
body = resp.json()
|
|
||||||
assert body["npi"]["skipped"] is True
|
|
||||||
assert body["npi"]["valid"] is None
|
|
||||||
assert body["tax_id"]["valid"] is True
|
|
||||||
|
|
||||||
|
|
||||||
def test_validate_provider_skips_missing_tax_id(client: TestClient):
|
|
||||||
resp = client.get("/api/admin/validate-provider", params={"npi": "1234567893"})
|
|
||||||
assert resp.status_code == 200
|
|
||||||
body = resp.json()
|
|
||||||
assert body["tax_id"]["skipped"] is True
|
|
||||||
assert body["tax_id"]["valid"] is None
|
|
||||||
assert body["tax_id"]["normalized"] is None
|
|
||||||
assert body["npi"]["valid"] is True
|
|
||||||
@@ -1,61 +0,0 @@
|
|||||||
"""Audit log entries record the acting user_id.
|
|
||||||
|
|
||||||
Schema: the ``audit_log`` table has a ``user_id INTEGER`` column added
|
|
||||||
by migration 0011; the SQLAlchemy ``AuditLog`` model itself does not
|
|
||||||
declare it yet, so ``append_event`` cannot pass it through. These tests
|
|
||||||
fail before the model + dataclass are updated, and pass after.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from sqlalchemy import delete, select
|
|
||||||
|
|
||||||
from cyclone.audit_log import AuditEvent, append_event
|
|
||||||
from cyclone.db import AuditLog, SessionLocal, User
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _clear():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(AuditLog))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
yield
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(AuditLog))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def test_append_event_accepts_user_id_kwarg():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
append_event(
|
|
||||||
db,
|
|
||||||
AuditEvent(
|
|
||||||
event_type="test",
|
|
||||||
entity_type="x",
|
|
||||||
entity_id="y",
|
|
||||||
user_id=42,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
db.commit()
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
row = db.execute(select(AuditLog)).scalars().one()
|
|
||||||
assert row.user_id == 42
|
|
||||||
|
|
||||||
|
|
||||||
def test_append_event_without_user_id_is_null():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
append_event(
|
|
||||||
db,
|
|
||||||
AuditEvent(
|
|
||||||
event_type="test",
|
|
||||||
entity_type="x",
|
|
||||||
entity_id="y",
|
|
||||||
),
|
|
||||||
)
|
|
||||||
db.commit()
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
row = db.execute(select(AuditLog)).scalars().one()
|
|
||||||
assert row.user_id is None
|
|
||||||
@@ -1,107 +0,0 @@
|
|||||||
"""Admin-only user management endpoints."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
from sqlalchemy import delete
|
|
||||||
|
|
||||||
from cyclone.api import app
|
|
||||||
from cyclone.auth import users
|
|
||||||
from cyclone.db import Session as DbSession
|
|
||||||
from cyclone.db import SessionLocal, User
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _clear(monkeypatch):
|
|
||||||
# conftest sets AUTH_DISABLED=True so pre-auth tests keep passing
|
|
||||||
# without a login. The auth-admin tests need the real auth path
|
|
||||||
# exercised, so flip it back off here.
|
|
||||||
monkeypatch.setattr("cyclone.auth.deps.AUTH_DISABLED", False)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
yield
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def admin_client():
|
|
||||||
client = TestClient(app)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="root", password="rootpassword1", role="admin")
|
|
||||||
login = client.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "root", "password": "rootpassword1"},
|
|
||||||
)
|
|
||||||
assert login.status_code == 200
|
|
||||||
return client
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def user_client():
|
|
||||||
client = TestClient(app)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="plain", password="plainpassword1", role="user")
|
|
||||||
login = client.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "plain", "password": "plainpassword1"},
|
|
||||||
)
|
|
||||||
return client
|
|
||||||
|
|
||||||
|
|
||||||
def test_list_users_as_admin(admin_client):
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="alice", password="hunter2hunter2", role="user")
|
|
||||||
resp = admin_client.get("/api/admin/users")
|
|
||||||
assert resp.status_code == 200
|
|
||||||
body = resp.json()
|
|
||||||
usernames = {u["username"] for u in body}
|
|
||||||
assert {"root", "alice"} <= usernames
|
|
||||||
|
|
||||||
|
|
||||||
def test_list_users_as_nonadmin_returns_403(user_client):
|
|
||||||
resp = user_client.get("/api/admin/users")
|
|
||||||
assert resp.status_code == 403
|
|
||||||
|
|
||||||
|
|
||||||
def test_create_user_as_admin(admin_client):
|
|
||||||
resp = admin_client.post(
|
|
||||||
"/api/admin/users",
|
|
||||||
json={"username": "newbie", "password": "newbiepassword1", "role": "viewer"},
|
|
||||||
)
|
|
||||||
assert resp.status_code == 201
|
|
||||||
assert resp.json()["username"] == "newbie"
|
|
||||||
assert resp.json()["role"] == "viewer"
|
|
||||||
|
|
||||||
|
|
||||||
def test_create_user_rejects_invalid_role(admin_client):
|
|
||||||
resp = admin_client.post(
|
|
||||||
"/api/admin/users",
|
|
||||||
json={"username": "badrole", "password": "hunter2hunter2", "role": "owner"},
|
|
||||||
)
|
|
||||||
assert resp.status_code == 422
|
|
||||||
|
|
||||||
|
|
||||||
def test_patch_user_role_and_password(admin_client):
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.create(db, username="subject", password="hunter2hunter2", role="viewer")
|
|
||||||
resp = admin_client.patch(
|
|
||||||
f"/api/admin/users/{u.id}",
|
|
||||||
json={"role": "user", "password": "newpassword1"},
|
|
||||||
)
|
|
||||||
assert resp.status_code == 200
|
|
||||||
assert resp.json()["role"] == "user"
|
|
||||||
|
|
||||||
|
|
||||||
def test_admin_cannot_demote_self(admin_client):
|
|
||||||
me = admin_client.get("/api/auth/me").json()
|
|
||||||
resp = admin_client.patch(
|
|
||||||
f"/api/admin/users/{me['id']}",
|
|
||||||
json={"role": "viewer"},
|
|
||||||
)
|
|
||||||
assert resp.status_code == 409
|
|
||||||
@@ -1,79 +0,0 @@
|
|||||||
"""Bootstrap admin user on backend startup."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from sqlalchemy import delete, select
|
|
||||||
|
|
||||||
from cyclone.auth import bootstrap, users
|
|
||||||
from cyclone.auth.deps import AUTH_DISABLED
|
|
||||||
from cyclone.auth.permissions import Role
|
|
||||||
from cyclone.db import Session as DbSession
|
|
||||||
from cyclone.db import SessionLocal, User
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _clear(monkeypatch):
|
|
||||||
# Reset the bootstrap-side AUTH_DISABLED flag so tests don't leak state
|
|
||||||
# into each other. The conftest fixture flips this back to True at the
|
|
||||||
# start of every test, but bootstrap.run() mutates this module-level
|
|
||||||
# value, so we restore it here too.
|
|
||||||
monkeypatch.setattr("cyclone.auth.deps.AUTH_DISABLED", False)
|
|
||||||
# conftest sets CYCLONE_AUTH_DISABLED=1 at module import. Drop it
|
|
||||||
# by default so tests exercise the real bootstrap path; the
|
|
||||||
# AUTH_DISABLED-specific test re-sets it explicitly.
|
|
||||||
monkeypatch.delenv("CYCLONE_AUTH_DISABLED", raising=False)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
yield
|
|
||||||
monkeypatch.setattr("cyclone.auth.deps.AUTH_DISABLED", False)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def test_bootstrap_creates_admin_when_users_empty_and_env_set(monkeypatch):
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_USERNAME", "firstadmin")
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_PASSWORD", "firstadminpw1")
|
|
||||||
bootstrap.run()
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = db.execute(select(User).where(User.username == "firstadmin")).scalar_one()
|
|
||||||
assert u.role == Role.ADMIN.value
|
|
||||||
|
|
||||||
|
|
||||||
def test_bootstrap_noop_when_users_exist(monkeypatch):
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_USERNAME", "ignored")
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_PASSWORD", "ignoredignored1")
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="existing", password="hunter2hunter2", role="admin")
|
|
||||||
bootstrap.run()
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
all_users = db.execute(select(User)).scalars().all()
|
|
||||||
usernames = {u.username for u in all_users}
|
|
||||||
assert usernames == {"existing"}
|
|
||||||
|
|
||||||
|
|
||||||
def test_bootstrap_refuses_to_run_without_env(monkeypatch):
|
|
||||||
monkeypatch.delenv("CYCLONE_ADMIN_USERNAME", raising=False)
|
|
||||||
monkeypatch.delenv("CYCLONE_ADMIN_PASSWORD", raising=False)
|
|
||||||
with pytest.raises(RuntimeError, match="CYCLONE_ADMIN_USERNAME"):
|
|
||||||
bootstrap.run()
|
|
||||||
|
|
||||||
|
|
||||||
def test_bootstrap_rejects_short_password(monkeypatch):
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_USERNAME", "weak")
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_PASSWORD", "short")
|
|
||||||
with pytest.raises(RuntimeError, match="12 characters"):
|
|
||||||
bootstrap.run()
|
|
||||||
|
|
||||||
|
|
||||||
def test_bootstrap_skips_when_auth_disabled(monkeypatch):
|
|
||||||
monkeypatch.setenv("CYCLONE_AUTH_DISABLED", "1")
|
|
||||||
# Even without env vars, bootstrap should NOT raise when AUTH_DISABLED=1.
|
|
||||||
bootstrap.run()
|
|
||||||
# And it must flip the deps flag so the API skips auth checks.
|
|
||||||
from cyclone.auth import deps as _deps
|
|
||||||
assert _deps.AUTH_DISABLED is True
|
|
||||||
@@ -1,131 +0,0 @@
|
|||||||
"""Auth bootstrap reads CYCLONE_ADMIN_*_FILE env vars when set.
|
|
||||||
|
|
||||||
Mirrors the Docker-secret posture in ``docker-compose.yml``: secrets
|
|
||||||
mounted at ``/run/secrets/<name>`` with the ``*_FILE`` env var pointing
|
|
||||||
at the path. The bootstrap should prefer the file over the bare env var
|
|
||||||
when both are present (file = Docker secret wins).
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from cyclone import db
|
|
||||||
from cyclone.auth import bootstrap, users
|
|
||||||
from cyclone.auth.permissions import Role
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def tmp_db(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
|
|
||||||
db_path = tmp_path / "test.db"
|
|
||||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{db_path}")
|
|
||||||
db.init_db()
|
|
||||||
return db_path
|
|
||||||
|
|
||||||
|
|
||||||
def _write_secrets(tmp_path: Path, *, username: str, password: str) -> tuple[Path, Path]:
|
|
||||||
user_file = tmp_path / "admin_username"
|
|
||||||
pw_file = tmp_path / "admin_pw"
|
|
||||||
user_file.write_text(f"{username}\n")
|
|
||||||
pw_file.write_text(f"{password}\n")
|
|
||||||
return user_file, pw_file
|
|
||||||
|
|
||||||
|
|
||||||
def test_bootstrap_creates_admin_from_file_env_vars(
|
|
||||||
tmp_path: Path, tmp_db: Path, monkeypatch: pytest.MonkeyPatch
|
|
||||||
) -> None:
|
|
||||||
user_file, pw_file = _write_secrets(
|
|
||||||
tmp_path, username="deploy-admin", password="super-secret-password-123"
|
|
||||||
)
|
|
||||||
monkeypatch.delenv("CYCLONE_ADMIN_USERNAME", raising=False)
|
|
||||||
monkeypatch.delenv("CYCLONE_ADMIN_PASSWORD", raising=False)
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_USERNAME_FILE", str(user_file))
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_PASSWORD_FILE", str(pw_file))
|
|
||||||
|
|
||||||
bootstrap.run()
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
user = users.get_by_username(session, "deploy-admin")
|
|
||||||
assert user is not None
|
|
||||||
assert user.role == Role.ADMIN.value
|
|
||||||
assert users.verify_password("super-secret-password-123", user.password_hash)
|
|
||||||
|
|
||||||
|
|
||||||
def test_file_env_var_overrides_plain_env_var(
|
|
||||||
tmp_path: Path, tmp_db: Path, monkeypatch: pytest.MonkeyPatch
|
|
||||||
) -> None:
|
|
||||||
"""When both _FILE and bare env vars are set, _FILE wins."""
|
|
||||||
user_file, pw_file = _write_secrets(
|
|
||||||
tmp_path, username="file-user", password="file-password-12345"
|
|
||||||
)
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_USERNAME", "env-user")
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_PASSWORD", "env-password-12345")
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_USERNAME_FILE", str(user_file))
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_PASSWORD_FILE", str(pw_file))
|
|
||||||
|
|
||||||
bootstrap.run()
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
file_user = users.get_by_username(session, "file-user")
|
|
||||||
env_user = users.get_by_username(session, "env-user")
|
|
||||||
assert file_user is not None
|
|
||||||
assert env_user is None, "bare env var should be ignored when _FILE is set"
|
|
||||||
|
|
||||||
|
|
||||||
def test_bootstrap_falls_back_to_plain_env_var(
|
|
||||||
tmp_path: Path, tmp_db: Path, monkeypatch: pytest.MonkeyPatch
|
|
||||||
) -> None:
|
|
||||||
monkeypatch.delenv("CYCLONE_ADMIN_USERNAME_FILE", raising=False)
|
|
||||||
monkeypatch.delenv("CYCLONE_ADMIN_PASSWORD_FILE", raising=False)
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_USERNAME", "env-user")
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_PASSWORD", "env-password-12345")
|
|
||||||
|
|
||||||
bootstrap.run()
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
user = users.get_by_username(session, "env-user")
|
|
||||||
assert user is not None
|
|
||||||
assert user.role == Role.ADMIN.value
|
|
||||||
|
|
||||||
|
|
||||||
def test_bootstrap_strips_trailing_whitespace_from_file(
|
|
||||||
tmp_path: Path, tmp_db: Path, monkeypatch: pytest.MonkeyPatch
|
|
||||||
) -> None:
|
|
||||||
"""Secret files often have a trailing newline from `printf`/`echo`.
|
|
||||||
The bootstrap must strip it so bcrypt verify doesn't see whitespace."""
|
|
||||||
user_file = tmp_path / "admin_username"
|
|
||||||
pw_file = tmp_path / "admin_pw"
|
|
||||||
user_file.write_text("deploy-admin\n\n")
|
|
||||||
pw_file.write_text("super-secret-password-123\n")
|
|
||||||
monkeypatch.delenv("CYCLONE_ADMIN_USERNAME", raising=False)
|
|
||||||
monkeypatch.delenv("CYCLONE_ADMIN_PASSWORD", raising=False)
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_USERNAME_FILE", str(user_file))
|
|
||||||
monkeypatch.setenv("CYCLONE_ADMIN_PASSWORD_FILE", str(pw_file))
|
|
||||||
|
|
||||||
bootstrap.run()
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
user = users.get_by_username(session, "deploy-admin")
|
|
||||||
assert user is not None
|
|
||||||
# Should verify cleanly with no trailing whitespace.
|
|
||||||
assert users.verify_password("super-secret-password-123", user.password_hash)
|
|
||||||
assert not users.verify_password(
|
|
||||||
"super-secret-password-123\n", user.password_hash
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_bootstrap_raises_when_file_path_missing(
|
|
||||||
tmp_path: Path, tmp_db: Path, monkeypatch: pytest.MonkeyPatch
|
|
||||||
) -> None:
|
|
||||||
monkeypatch.delenv("CYCLONE_ADMIN_USERNAME", raising=False)
|
|
||||||
monkeypatch.delenv("CYCLONE_ADMIN_PASSWORD", raising=False)
|
|
||||||
monkeypatch.setenv(
|
|
||||||
"CYCLONE_ADMIN_USERNAME_FILE", str(tmp_path / "does-not-exist")
|
|
||||||
)
|
|
||||||
monkeypatch.setenv(
|
|
||||||
"CYCLONE_ADMIN_PASSWORD_FILE", str(tmp_path / "also-missing")
|
|
||||||
)
|
|
||||||
|
|
||||||
with pytest.raises(RuntimeError, match="failed to read"):
|
|
||||||
bootstrap.run()
|
|
||||||
@@ -1,86 +0,0 @@
|
|||||||
"""Login rate limit: 5 fails / 5 min per username."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
from sqlalchemy import delete
|
|
||||||
|
|
||||||
from cyclone.api import app
|
|
||||||
from cyclone.auth import rate_limit, users
|
|
||||||
from cyclone.db import Session as DbSession
|
|
||||||
from cyclone.db import SessionLocal, User
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _clear(monkeypatch):
|
|
||||||
# conftest sets AUTH_DISABLED=True so pre-auth tests keep passing
|
|
||||||
# without a login. The auth-login-rate-limit tests need the real
|
|
||||||
# auth path exercised (the rate limiter sits in front of /login),
|
|
||||||
# so flip it back off here.
|
|
||||||
monkeypatch.setattr("cyclone.auth.deps.AUTH_DISABLED", False)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
# Reset the in-memory rate-limit counter so a previous test's 5
|
|
||||||
# failures don't poison this test. The plan recipe calls this out.
|
|
||||||
rate_limit.reset("victim")
|
|
||||||
yield
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
rate_limit.reset("victim")
|
|
||||||
|
|
||||||
|
|
||||||
def test_5_fails_then_429():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="victim", password="hunter2hunter2", role="user")
|
|
||||||
client = TestClient(app)
|
|
||||||
for _ in range(5):
|
|
||||||
r = client.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "victim", "password": "WRONG"},
|
|
||||||
)
|
|
||||||
assert r.status_code == 401
|
|
||||||
# 6th should be 429.
|
|
||||||
r = client.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "victim", "password": "WRONG"},
|
|
||||||
)
|
|
||||||
assert r.status_code == 429
|
|
||||||
assert "Retry-After" in r.headers
|
|
||||||
# And the module-level helper confirms we're now over the threshold.
|
|
||||||
assert rate_limit.check("victim") > 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_successful_login_resets_counter():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="victim", password="hunter2hunter2", role="user")
|
|
||||||
client = TestClient(app)
|
|
||||||
for _ in range(4):
|
|
||||||
r = client.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "victim", "password": "WRONG"},
|
|
||||||
)
|
|
||||||
assert r.status_code == 401
|
|
||||||
# Successful login — should reset the counter.
|
|
||||||
r = client.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "victim", "password": "hunter2hunter2"},
|
|
||||||
)
|
|
||||||
assert r.status_code == 200
|
|
||||||
# Counter reset — 5 more fails allowed.
|
|
||||||
for _ in range(5):
|
|
||||||
r = client.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "victim", "password": "WRONG"},
|
|
||||||
)
|
|
||||||
assert r.status_code == 401
|
|
||||||
# 6th is now throttled.
|
|
||||||
r = client.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "victim", "password": "WRONG"},
|
|
||||||
)
|
|
||||||
assert r.status_code == 429
|
|
||||||
@@ -1,115 +0,0 @@
|
|||||||
"""API tests for /api/auth/login, /api/auth/logout, /api/auth/me."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
from sqlalchemy import delete
|
|
||||||
|
|
||||||
from cyclone.api import app
|
|
||||||
from cyclone.auth import users
|
|
||||||
from cyclone.db import Session as DbSession
|
|
||||||
from cyclone.db import SessionLocal, User
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _clear(monkeypatch):
|
|
||||||
# conftest sets AUTH_DISABLED=True so pre-auth tests keep passing
|
|
||||||
# without a login. The auth-route tests need the real auth path
|
|
||||||
# exercised, so flip it back off here.
|
|
||||||
monkeypatch.setattr("cyclone.auth.deps.AUTH_DISABLED", False)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
yield
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def client():
|
|
||||||
return TestClient(app)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def seeded_admin(client):
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="admin", password="adminpassword1", role="admin")
|
|
||||||
return client
|
|
||||||
|
|
||||||
|
|
||||||
def test_login_success_returns_user_and_cookie(client):
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="alice", password="hunter2hunter2", role="user")
|
|
||||||
resp = client.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "alice", "password": "hunter2hunter2"},
|
|
||||||
)
|
|
||||||
assert resp.status_code == 200
|
|
||||||
body = resp.json()
|
|
||||||
assert body["username"] == "alice"
|
|
||||||
assert body["role"] == "user"
|
|
||||||
assert "password_hash" not in body
|
|
||||||
assert "cyclone_session" in resp.cookies
|
|
||||||
|
|
||||||
|
|
||||||
def test_login_bad_password_returns_401(client):
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="bob", password="hunter2hunter2", role="user")
|
|
||||||
resp = client.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "bob", "password": "WRONG"},
|
|
||||||
)
|
|
||||||
assert resp.status_code == 401
|
|
||||||
assert resp.json()["error"] == "invalid_credentials"
|
|
||||||
|
|
||||||
|
|
||||||
def test_login_unknown_user_returns_401(client):
|
|
||||||
resp = client.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "ghost", "password": "whatever"},
|
|
||||||
)
|
|
||||||
assert resp.status_code == 401
|
|
||||||
assert resp.json()["error"] == "invalid_credentials"
|
|
||||||
|
|
||||||
|
|
||||||
def test_login_disabled_user_returns_403(client):
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.create(db, username="carol", password="hunter2hunter2", role="user")
|
|
||||||
users.disable(db, u.id)
|
|
||||||
resp = client.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "carol", "password": "hunter2hunter2"},
|
|
||||||
)
|
|
||||||
assert resp.status_code == 403
|
|
||||||
assert resp.json()["error"] == "account_disabled"
|
|
||||||
|
|
||||||
|
|
||||||
def test_logout_clears_session_and_cookie(seeded_admin):
|
|
||||||
login = seeded_admin.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "admin", "password": "adminpassword1"},
|
|
||||||
)
|
|
||||||
cookie = login.cookies.get("cyclone_session")
|
|
||||||
assert cookie
|
|
||||||
resp = seeded_admin.post("/api/auth/logout", cookies={"cyclone_session": cookie})
|
|
||||||
assert resp.status_code == 204
|
|
||||||
|
|
||||||
|
|
||||||
def test_me_returns_current_user(seeded_admin):
|
|
||||||
login = seeded_admin.post(
|
|
||||||
"/api/auth/login",
|
|
||||||
json={"username": "admin", "password": "adminpassword1"},
|
|
||||||
)
|
|
||||||
cookie = login.cookies.get("cyclone_session")
|
|
||||||
resp = seeded_admin.get("/api/auth/me", cookies={"cyclone_session": cookie})
|
|
||||||
assert resp.status_code == 200
|
|
||||||
assert resp.json()["username"] == "admin"
|
|
||||||
|
|
||||||
|
|
||||||
def test_me_without_cookie_returns_401(seeded_admin):
|
|
||||||
resp = seeded_admin.get("/api/auth/me")
|
|
||||||
assert resp.status_code == 401
|
|
||||||
@@ -1,90 +0,0 @@
|
|||||||
"""Unit tests for cyclone.auth.sessions — Session create/validate/expire/touch."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from datetime import datetime, timedelta, timezone
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from sqlalchemy import delete
|
|
||||||
|
|
||||||
from cyclone.auth import sessions, users
|
|
||||||
from cyclone.db import Session as DbSession
|
|
||||||
from cyclone.db import SessionLocal, User
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _clear(monkeypatch):
|
|
||||||
# conftest sets AUTH_DISABLED=True so pre-auth tests keep passing
|
|
||||||
# without a login. The auth-sessions tests need the real auth path
|
|
||||||
# exercised, so flip it back off here.
|
|
||||||
monkeypatch.setattr("cyclone.auth.deps.AUTH_DISABLED", False)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
yield
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def _make_user():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
return users.create(db, username="sessuser", password="hunter2hunter2", role="user")
|
|
||||||
|
|
||||||
|
|
||||||
def test_create_session_returns_id_and_session():
|
|
||||||
user = _make_user()
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
sid, sess = sessions.create(db, user_id=user.id)
|
|
||||||
assert len(sid) >= 32
|
|
||||||
assert sess.user_id == user.id
|
|
||||||
assert sess.expires_at > datetime.now(timezone.utc)
|
|
||||||
|
|
||||||
|
|
||||||
def test_get_valid_returns_session_for_active():
|
|
||||||
user = _make_user()
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
sid, _ = sessions.create(db, user_id=user.id)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
got = sessions.get_valid(db, sid)
|
|
||||||
assert got is not None
|
|
||||||
assert got.user_id == user.id
|
|
||||||
|
|
||||||
|
|
||||||
def test_get_valid_returns_none_for_missing():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
assert sessions.get_valid(db, "does-not-exist") is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_get_valid_returns_none_for_expired():
|
|
||||||
user = _make_user()
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
sid, _ = sessions.create(db, user_id=user.id)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
sess = sessions.get_valid(db, sid)
|
|
||||||
sess.expires_at = datetime.now(timezone.utc) - timedelta(seconds=1)
|
|
||||||
db.commit()
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
assert sessions.get_valid(db, sid) is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_delete_removes_session():
|
|
||||||
user = _make_user()
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
sid, _ = sessions.create(db, user_id=user.id)
|
|
||||||
sessions.delete(db, sid)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
assert sessions.get_valid(db, sid) is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_touch_extends_expiry():
|
|
||||||
user = _make_user()
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
sid, sess = sessions.create(db, user_id=user.id)
|
|
||||||
original_expiry = sess.expires_at
|
|
||||||
sessions.touch(db, sid)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
refreshed = sessions.get_valid(db, sid)
|
|
||||||
assert refreshed.expires_at >= original_expiry
|
|
||||||
@@ -1,110 +0,0 @@
|
|||||||
"""Unit tests for cyclone.auth.users — User CRUD + bcrypt hashing."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from sqlalchemy import delete
|
|
||||||
from sqlalchemy.exc import IntegrityError
|
|
||||||
|
|
||||||
from cyclone.auth import users
|
|
||||||
from cyclone.db import SessionLocal, User
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _clear_users(monkeypatch):
|
|
||||||
# conftest sets AUTH_DISABLED=True so pre-auth tests keep passing
|
|
||||||
# without a login. The auth-users tests need the real auth path
|
|
||||||
# exercised, so flip it back off here.
|
|
||||||
monkeypatch.setattr("cyclone.auth.deps.AUTH_DISABLED", False)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
yield
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def test_hash_password_returns_bcrypt():
|
|
||||||
h = users.hash_password("hunter2hunter2")
|
|
||||||
assert h.startswith("$2")
|
|
||||||
|
|
||||||
|
|
||||||
def test_hash_password_produces_unique_salts():
|
|
||||||
a = users.hash_password("same-password")
|
|
||||||
b = users.hash_password("same-password")
|
|
||||||
assert a != b
|
|
||||||
|
|
||||||
|
|
||||||
def test_verify_password_correct():
|
|
||||||
h = users.hash_password("hunter2hunter2")
|
|
||||||
assert users.verify_password("hunter2hunter2", h) is True
|
|
||||||
|
|
||||||
|
|
||||||
def test_verify_password_incorrect():
|
|
||||||
h = users.hash_password("hunter2hunter2")
|
|
||||||
assert users.verify_password("WRONG", h) is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_create_user_persists_with_hashed_password():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.create(db, username="alice", password="hunter2hunter2", role="admin")
|
|
||||||
assert u.id is not None
|
|
||||||
assert u.username == "alice"
|
|
||||||
assert u.role == "admin"
|
|
||||||
assert u.disabled_at is None
|
|
||||||
assert u.password_hash != "hunter2hunter2"
|
|
||||||
assert u.password_hash.startswith("$2")
|
|
||||||
|
|
||||||
|
|
||||||
def test_create_user_rejects_duplicate_username():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="bob", password="hunter2hunter2", role="user")
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
with pytest.raises(IntegrityError):
|
|
||||||
users.create(db, username="bob", password="anotherone", role="user")
|
|
||||||
|
|
||||||
|
|
||||||
def test_get_by_username_returns_user():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="carol", password="hunter2hunter2", role="viewer")
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.get_by_username(db, "carol")
|
|
||||||
assert u is not None
|
|
||||||
assert u.username == "carol"
|
|
||||||
|
|
||||||
|
|
||||||
def test_get_by_username_returns_none_for_missing():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
assert users.get_by_username(db, "ghost") is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_disable_user_sets_disabled_at():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.create(db, username="dave", password="hunter2hunter2", role="user")
|
|
||||||
users.disable(db, u.id)
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
refreshed = users.get_by_username(db, "dave")
|
|
||||||
assert refreshed.disabled_at is not None
|
|
||||||
|
|
||||||
|
|
||||||
def test_to_public_shape_omits_password_hash():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.create(db, username="eve", password="hunter2hunter2", role="admin")
|
|
||||||
shape = users.to_public(u)
|
|
||||||
assert "password_hash" not in shape
|
|
||||||
assert shape["username"] == "eve"
|
|
||||||
assert shape["role"] == "admin"
|
|
||||||
assert "id" in shape and "createdAt" in shape
|
|
||||||
|
|
||||||
|
|
||||||
def test_update_password_actually_rehashes_and_verifies():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.create(db, username="frank", password="hunter2hunter2", role="user")
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.update_password(db, u.id, "newpassword1")
|
|
||||||
# Old password no longer verifies.
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
refreshed = users.get_by_username(db, "frank")
|
|
||||||
assert not users.verify_password("hunter2hunter2", refreshed.password_hash)
|
|
||||||
assert users.verify_password("newpassword1", refreshed.password_hash)
|
|
||||||
@@ -1,165 +0,0 @@
|
|||||||
"""SP17 — low-level backup crypto tests.
|
|
||||||
|
|
||||||
Pure-Python, no DB. Covers key derivation determinism, encrypt /
|
|
||||||
decrypt round-trip, tampered-ciphertext failure, wrong-passphrase
|
|
||||||
failure, and the sidecar JSON format.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from cyclone import backup as backup_mod
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Key derivation
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_derive_key_is_deterministic():
|
|
||||||
salt = os.urandom(16)
|
|
||||||
k1 = backup_mod.derive_key("correct horse battery staple", salt)
|
|
||||||
k2 = backup_mod.derive_key("correct horse battery staple", salt)
|
|
||||||
assert k1 == k2
|
|
||||||
assert len(k1) == backup_mod.KEY_LEN == 32
|
|
||||||
|
|
||||||
|
|
||||||
def test_derive_key_different_salts_produce_different_keys():
|
|
||||||
"""Salt is what makes the same passphrase produce different keys."""
|
|
||||||
k1 = backup_mod.derive_key("hunter2", os.urandom(16))
|
|
||||||
k2 = backup_mod.derive_key("hunter2", os.urandom(16))
|
|
||||||
assert k1 != k2
|
|
||||||
|
|
||||||
|
|
||||||
def test_derive_key_different_passphrases_produce_different_keys():
|
|
||||||
salt = os.urandom(16)
|
|
||||||
k1 = backup_mod.derive_key("a", salt)
|
|
||||||
k2 = backup_mod.derive_key("b", salt)
|
|
||||||
assert k1 != k2
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Encrypt / decrypt round-trip
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_encrypt_decrypt_roundtrip():
|
|
||||||
key = os.urandom(32)
|
|
||||||
plaintext = b"hello cyclone backup " * 1000
|
|
||||||
blob = backup_mod.encrypt(plaintext, key)
|
|
||||||
assert len(blob) == backup_mod.NONCE_LEN + len(plaintext) + 16 # tag
|
|
||||||
out = backup_mod.decrypt(blob, key)
|
|
||||||
assert out == plaintext
|
|
||||||
|
|
||||||
|
|
||||||
def test_encrypt_decrypt_empty_plaintext():
|
|
||||||
"""Edge case: zero-byte payload still produces nonce + tag."""
|
|
||||||
key = os.urandom(32)
|
|
||||||
blob = backup_mod.encrypt(b"", key)
|
|
||||||
out = backup_mod.decrypt(blob, key)
|
|
||||||
assert out == b""
|
|
||||||
|
|
||||||
|
|
||||||
def test_decrypt_with_wrong_key_raises():
|
|
||||||
plaintext = b"some bytes"
|
|
||||||
key1 = os.urandom(32)
|
|
||||||
key2 = os.urandom(32)
|
|
||||||
blob = backup_mod.encrypt(plaintext, key1)
|
|
||||||
with pytest.raises(backup_mod.BackupDecryptError):
|
|
||||||
backup_mod.decrypt(blob, key2)
|
|
||||||
|
|
||||||
|
|
||||||
def test_decrypt_tampered_ciphertext_raises():
|
|
||||||
"""Flipping a single ciphertext byte must fail GCM auth."""
|
|
||||||
key = os.urandom(32)
|
|
||||||
blob = backup_mod.encrypt(b"a" * 200, key)
|
|
||||||
tampered = bytearray(blob)
|
|
||||||
# Flip a bit somewhere in the ciphertext region (past the nonce).
|
|
||||||
tampered[backup_mod.NONCE_LEN + 5] ^= 0x01
|
|
||||||
with pytest.raises(backup_mod.BackupDecryptError):
|
|
||||||
backup_mod.decrypt(bytes(tampered), key)
|
|
||||||
|
|
||||||
|
|
||||||
def test_decrypt_truncated_blob_raises():
|
|
||||||
key = os.urandom(32)
|
|
||||||
blob = backup_mod.encrypt(b"x" * 100, key)
|
|
||||||
with pytest.raises(backup_mod.BackupDecryptError):
|
|
||||||
# Strip the GCM tag.
|
|
||||||
backup_mod.decrypt(blob[: -16], key)
|
|
||||||
|
|
||||||
|
|
||||||
def test_encrypt_with_wrong_key_length_raises():
|
|
||||||
with pytest.raises(backup_mod.BackupError):
|
|
||||||
backup_mod.encrypt(b"data", b"short") # not 32 bytes
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Fingerprint
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_fingerprint_format_and_stability():
|
|
||||||
fp = backup_mod.fingerprint(b"hello")
|
|
||||||
assert fp.startswith("sha256:")
|
|
||||||
assert len(fp) == len("sha256:") + 64
|
|
||||||
assert fp == backup_mod.fingerprint(b"hello")
|
|
||||||
assert fp != backup_mod.fingerprint(b"hellp")
|
|
||||||
|
|
||||||
|
|
||||||
def test_fingerprint_file_matches_fingerprint_bytes(tmp_path):
|
|
||||||
p = tmp_path / "data.bin"
|
|
||||||
p.write_bytes(b"\x00\x01\x02" * 100)
|
|
||||||
assert backup_mod.fingerprint_file(p) == backup_mod.fingerprint(p.read_bytes())
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Sidecar
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_sidecar_round_trip_json():
|
|
||||||
sc = backup_mod.Sidecar(
|
|
||||||
format_version="v1",
|
|
||||||
created_at="2026-06-21T15:30:00+00:00",
|
|
||||||
db_fingerprint="sha256:" + "a" * 64,
|
|
||||||
table_count=11,
|
|
||||||
size_bytes=1024,
|
|
||||||
kdf="PBKDF2-HMAC-SHA256",
|
|
||||||
kdf_iterations=200_000,
|
|
||||||
cipher="AES-256-GCM",
|
|
||||||
key_fingerprint="sha256:" + "b" * 64,
|
|
||||||
)
|
|
||||||
text = sc.to_json()
|
|
||||||
parsed = json.loads(text)
|
|
||||||
assert parsed["format_version"] == "v1"
|
|
||||||
assert parsed["encryption"]["kdf_iterations"] == 200_000
|
|
||||||
sc2 = backup_mod.Sidecar.from_json(text)
|
|
||||||
assert sc2 == sc
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Filenames
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_backup_filename_format():
|
|
||||||
"""The timestamp prefix is fixed; the suffix is random per call."""
|
|
||||||
import re
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
ts = datetime(2026, 6, 21, 15, 30, 0, tzinfo=timezone.utc)
|
|
||||||
name = backup_mod.backup_filename(ts)
|
|
||||||
assert re.match(r"^cyclone-backup-20260621T153000Z-[0-9a-f]{8}\.bin$", name), name
|
|
||||||
|
|
||||||
|
|
||||||
def test_backup_filename_random_suffix_avoids_collisions():
|
|
||||||
"""Two calls in the same second get different filenames."""
|
|
||||||
a = backup_mod.backup_filename()
|
|
||||||
b = backup_mod.backup_filename()
|
|
||||||
assert a != b
|
|
||||||
|
|
||||||
|
|
||||||
def test_sidecar_filename_appends_meta_json():
|
|
||||||
assert backup_mod.sidecar_filename("foo.bin") == "foo.bin.meta.json"
|
|
||||||
@@ -1,207 +0,0 @@
|
|||||||
"""SP17 — BackupScheduler unit tests.
|
|
||||||
|
|
||||||
Exercises the asyncio tick / start / stop loop without spinning up
|
|
||||||
the FastAPI app. The scheduler wraps a real BackupService against
|
|
||||||
a real on-disk sqlite DB.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
from datetime import datetime, timedelta, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
from cyclone import backup_scheduler as sched_mod
|
|
||||||
from cyclone import db
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def fresh_db(tmp_path, monkeypatch):
|
|
||||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
|
|
||||||
db._reset_for_tests()
|
|
||||||
db.init_db()
|
|
||||||
from cyclone.db import Batch
|
|
||||||
import uuid
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
s.add(Batch(
|
|
||||||
id=str(uuid.uuid4()),
|
|
||||||
kind="837P",
|
|
||||||
input_filename="seed.x12",
|
|
||||||
parsed_at=datetime.now(timezone.utc),
|
|
||||||
totals_json=None,
|
|
||||||
validation_json=None,
|
|
||||||
raw_result_json={"envelope": {"control_number": "1"}, "claims": [], "summary": {"passed": 0, "failed": 0, "failed_claim_ids": []}},
|
|
||||||
))
|
|
||||||
s.commit()
|
|
||||||
yield
|
|
||||||
db._reset_for_tests()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def backup_svc(tmp_path):
|
|
||||||
return svc_mod.BackupService(
|
|
||||||
backup_dir=tmp_path / "backups",
|
|
||||||
passphrase="test-pass",
|
|
||||||
retention_days=7,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# tick
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_tick_creates_backup_and_audits_it(fresh_db, backup_svc):
|
|
||||||
sched = sched_mod.BackupScheduler(backup_svc, interval_hours=24.0)
|
|
||||||
result = await sched.tick()
|
|
||||||
assert result.ok
|
|
||||||
assert result.created is not None
|
|
||||||
assert result.error is None
|
|
||||||
assert len(backup_svc.list_backups()) == 1
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_tick_creates_audit_event(fresh_db, backup_svc):
|
|
||||||
"""db.backup_created audit event is written (SP11 hash chain)."""
|
|
||||||
sched = sched_mod.BackupScheduler(backup_svc, interval_hours=24.0)
|
|
||||||
await sched.tick()
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
from cyclone.db import AuditLog
|
|
||||||
rows = (
|
|
||||||
s.query(AuditLog)
|
|
||||||
.filter(AuditLog.event_type == "db.backup_created")
|
|
||||||
.all()
|
|
||||||
)
|
|
||||||
assert len(rows) == 1
|
|
||||||
assert "backup_id" in rows[0].payload_json
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_tick_handles_create_failure_without_crashing(fresh_db, backup_svc, monkeypatch):
|
|
||||||
"""If create_now raises, tick records the error and continues."""
|
|
||||||
def boom():
|
|
||||||
raise RuntimeError("simulated failure")
|
|
||||||
monkeypatch.setattr(backup_svc, "create_now", boom)
|
|
||||||
sched = sched_mod.BackupScheduler(backup_svc, interval_hours=24.0)
|
|
||||||
result = await sched.tick()
|
|
||||||
assert result.error is not None
|
|
||||||
assert "simulated failure" in result.error
|
|
||||||
# Audit event written for the failure.
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
from cyclone.db import AuditLog
|
|
||||||
rows = (
|
|
||||||
s.query(AuditLog)
|
|
||||||
.filter(AuditLog.event_type == "db.backup_failed")
|
|
||||||
.all()
|
|
||||||
)
|
|
||||||
assert len(rows) == 1
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_tick_prunes_old_backups_and_audits(fresh_db, backup_svc):
|
|
||||||
"""A tick prunes backups past retention and writes a db.backup_pruned event."""
|
|
||||||
from cyclone.db import DbBackup
|
|
||||||
|
|
||||||
# Take an initial backup.
|
|
||||||
initial = backup_svc.create_now()
|
|
||||||
# Age it past retention.
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
row = s.get(DbBackup, initial.backup.id)
|
|
||||||
row.created_at = datetime.now(timezone.utc) - timedelta(days=30)
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
sched = sched_mod.BackupScheduler(backup_svc, interval_hours=24.0)
|
|
||||||
result = await sched.tick()
|
|
||||||
assert result.ok # create_now succeeded even though prune removed old
|
|
||||||
assert len(result.pruned_paths) == 2 # .bin + .meta.json
|
|
||||||
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
from cyclone.db import AuditLog
|
|
||||||
pruned_events = (
|
|
||||||
s.query(AuditLog)
|
|
||||||
.filter(AuditLog.event_type == "db.backup_pruned")
|
|
||||||
.all()
|
|
||||||
)
|
|
||||||
assert len(pruned_events) == 1
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# start / stop / is_running
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_start_then_stop(fresh_db, backup_svc):
|
|
||||||
sched = sched_mod.BackupScheduler(backup_svc, interval_hours=24.0)
|
|
||||||
assert not sched.is_running()
|
|
||||||
await sched.start()
|
|
||||||
assert sched.is_running()
|
|
||||||
# Don't wait for the staggered first tick; just stop.
|
|
||||||
await sched.stop()
|
|
||||||
assert not sched.is_running()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_double_start_is_idempotent(fresh_db, backup_svc):
|
|
||||||
sched = sched_mod.BackupScheduler(backup_svc, interval_hours=24.0)
|
|
||||||
await sched.start()
|
|
||||||
await sched.start() # no-op
|
|
||||||
assert sched.is_running()
|
|
||||||
await sched.stop()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_concurrent_ticks_are_coalesced(fresh_db, backup_svc):
|
|
||||||
"""Two tick() calls in flight — second waits for first."""
|
|
||||||
sched = sched_mod.BackupScheduler(backup_svc, interval_hours=24.0)
|
|
||||||
r1, r2 = await asyncio.gather(sched.tick(), sched.tick())
|
|
||||||
# Both should succeed and produce a single backup (the second
|
|
||||||
# call returned the first call's result, or ran back-to-back
|
|
||||||
# and produced a second backup — both are valid coalescings).
|
|
||||||
assert r1 is not None
|
|
||||||
assert r2 is not None
|
|
||||||
# No matter the order, exactly 1 backup should exist OR 2 if they
|
|
||||||
# ran sequentially. The point of coalescing is no-overlap, so
|
|
||||||
# both should be ok=True.
|
|
||||||
assert r1.ok
|
|
||||||
assert r2.ok
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# status
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_status_snapshot(fresh_db, backup_svc):
|
|
||||||
sched = sched_mod.BackupScheduler(backup_svc, interval_hours=12.0)
|
|
||||||
snap = sched.status()
|
|
||||||
assert snap.running is False
|
|
||||||
assert snap.interval_hours == 12.0
|
|
||||||
assert snap.backup_dir == str(backup_svc.backup_dir)
|
|
||||||
assert snap.retention_days == 7
|
|
||||||
assert snap.tick_count == 0
|
|
||||||
assert snap.last_tick is None
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Module-level singleton
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_module_singleton_round_trip(fresh_db, tmp_path):
|
|
||||||
sched_mod.reset_backup_scheduler_for_tests()
|
|
||||||
svc = svc_mod.BackupService(tmp_path / "b", passphrase="x", retention_days=1)
|
|
||||||
sched = sched_mod.configure_backup_scheduler(svc, interval_hours=1)
|
|
||||||
assert sched_mod.get_backup_scheduler() is sched
|
|
||||||
# Second configure is a no-op.
|
|
||||||
assert sched_mod.configure_backup_scheduler(svc) is sched
|
|
||||||
sched_mod.reset_backup_scheduler_for_tests()
|
|
||||||
|
|
||||||
|
|
||||||
def test_module_singleton_get_raises_when_unset():
|
|
||||||
sched_mod.reset_backup_scheduler_for_tests()
|
|
||||||
with pytest.raises(RuntimeError):
|
|
||||||
sched_mod.get_backup_scheduler()
|
|
||||||
@@ -1,400 +0,0 @@
|
|||||||
"""SP17 — BackupService integration tests.
|
|
||||||
|
|
||||||
Exercises the full create / list / verify / restore / prune flow
|
|
||||||
against a real on-disk SQLite file (no SQLCipher, no Keychain). We
|
|
||||||
inject the passphrase directly into the BackupService constructor.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from cyclone import backup as backup_mod
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
from cyclone import db
|
|
||||||
from cyclone.backup import BackupError
|
|
||||||
from cyclone.backup_service import (
|
|
||||||
BackupService,
|
|
||||||
STATUS_ERROR,
|
|
||||||
STATUS_OK,
|
|
||||||
STATUS_PENDING,
|
|
||||||
STATUS_PRUNED,
|
|
||||||
configure_backup_service,
|
|
||||||
get_backup_service,
|
|
||||||
reset_backup_service_for_tests,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Fixtures
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def fresh_db(tmp_path, monkeypatch):
|
|
||||||
"""Fresh sqlite DB; init_db + create tables; yield the path."""
|
|
||||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
|
|
||||||
db._reset_for_tests()
|
|
||||||
db.init_db()
|
|
||||||
yield tmp_path / "test.db"
|
|
||||||
db._reset_for_tests()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def backup_svc(fresh_db, tmp_path):
|
|
||||||
"""A BackupService rooted in a temp backup directory."""
|
|
||||||
backup_dir = tmp_path / "backups"
|
|
||||||
return BackupService(
|
|
||||||
backup_dir=backup_dir,
|
|
||||||
passphrase="test-passphrase-123",
|
|
||||||
retention_days=7,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _make_a_row(s: "sa.orm.Session") -> None:
|
|
||||||
"""Insert one minimal Batch row so the DB has a real schema + content.
|
|
||||||
|
|
||||||
Bypasses the Claim model (which has many NOT NULL columns tied to
|
|
||||||
BatchRecord lifecycle) and just writes a Batch directly — the
|
|
||||||
backup flow doesn't care which tables exist, only that there
|
|
||||||
are some.
|
|
||||||
"""
|
|
||||||
from cyclone.db import Batch
|
|
||||||
import uuid
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from decimal import Decimal
|
|
||||||
s.add(Batch(
|
|
||||||
id=str(uuid.uuid4()),
|
|
||||||
kind="837P",
|
|
||||||
input_filename="test.x12",
|
|
||||||
parsed_at=datetime.now(timezone.utc),
|
|
||||||
totals_json=None,
|
|
||||||
validation_json=None,
|
|
||||||
raw_result_json={
|
|
||||||
"envelope": {"control_number": "1"},
|
|
||||||
"claims": [],
|
|
||||||
"summary": {"passed": 0, "failed": 0, "failed_claim_ids": []},
|
|
||||||
},
|
|
||||||
))
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# create_now
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_create_now_writes_encrypted_blob_and_sidecar(fresh_db, backup_svc):
|
|
||||||
from cyclone.db import DbBackup
|
|
||||||
# Add a claim so the DB has content + table_count > 0.
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_a_row(s)
|
|
||||||
|
|
||||||
result = backup_svc.create_now()
|
|
||||||
record = result.backup
|
|
||||||
sidecar = result.sidecar
|
|
||||||
|
|
||||||
assert record.status == STATUS_OK
|
|
||||||
assert record.size_bytes > 0
|
|
||||||
assert record.db_fingerprint.startswith("sha256:")
|
|
||||||
assert record.table_count >= 1
|
|
||||||
assert record.completed_at is not None
|
|
||||||
|
|
||||||
# The .bin file exists, is non-trivial size, and does NOT look
|
|
||||||
# like a SQLite header (which is the whole point of encryption).
|
|
||||||
bin_path = backup_svc.backup_dir / record.filename
|
|
||||||
assert bin_path.exists()
|
|
||||||
blob = bin_path.read_bytes()
|
|
||||||
assert blob[:6] != b"SQLite" # not a plaintext SQLite file
|
|
||||||
|
|
||||||
# Sidecar exists and round-trips.
|
|
||||||
meta_path = backup_svc.backup_dir / backup_mod.sidecar_filename(record.filename)
|
|
||||||
assert meta_path.exists()
|
|
||||||
parsed = backup_mod.Sidecar.from_json(meta_path.read_text())
|
|
||||||
assert parsed.db_fingerprint == record.db_fingerprint
|
|
||||||
assert parsed.table_count == record.table_count
|
|
||||||
|
|
||||||
|
|
||||||
def test_create_now_marks_error_on_db_failure(fresh_db, tmp_path, monkeypatch):
|
|
||||||
"""If SQLite .backup() raises, the row is marked error + files cleaned."""
|
|
||||||
backup_dir = tmp_path / "backups"
|
|
||||||
svc = BackupService(backup_dir=backup_dir, passphrase="x", retention_days=7)
|
|
||||||
|
|
||||||
# Force the .backup() call to fail by patching sqlite3.connect to raise.
|
|
||||||
import sqlite3 as _sqlite3
|
|
||||||
real_connect = _sqlite3.connect
|
|
||||||
def boom(path):
|
|
||||||
raise RuntimeError("simulated disk failure")
|
|
||||||
monkeypatch.setattr(_sqlite3, "connect", boom)
|
|
||||||
# But we also need to make sure engine.raw_connection().driver_connection
|
|
||||||
# is reachable — it's still using real_connect via the engine's
|
|
||||||
# internals. So patch at the higher level: the BackupService's
|
|
||||||
# _sqlite_backup_to.
|
|
||||||
monkeypatch.setattr(svc, "_sqlite_backup_to",
|
|
||||||
lambda p: (_ for _ in ()).throw(RuntimeError("boom")))
|
|
||||||
|
|
||||||
with pytest.raises(RuntimeError, match="boom"):
|
|
||||||
svc.create_now()
|
|
||||||
|
|
||||||
rows = svc.list_backups()
|
|
||||||
assert len(rows) == 1
|
|
||||||
assert rows[0].status == STATUS_ERROR
|
|
||||||
assert "boom" in rows[0].error_message
|
|
||||||
# No files left in the backup dir.
|
|
||||||
assert list(backup_dir.iterdir()) == []
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# list_backups
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_list_backups_orders_newest_first(fresh_db, backup_svc):
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_a_row(s)
|
|
||||||
r1 = backup_svc.create_now()
|
|
||||||
r2 = backup_svc.create_now()
|
|
||||||
rows = backup_svc.list_backups()
|
|
||||||
assert [r.id for r in rows] == [r2.backup.id, r1.backup.id]
|
|
||||||
|
|
||||||
|
|
||||||
def test_list_backups_filter_by_status(fresh_db, backup_svc):
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_a_row(s)
|
|
||||||
backup_svc.create_now()
|
|
||||||
rows = backup_svc.list_backups(status=STATUS_OK)
|
|
||||||
assert all(r.status == STATUS_OK for r in rows)
|
|
||||||
rows = backup_svc.list_backups(status=STATUS_PENDING)
|
|
||||||
assert rows == []
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# verify
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_verify_ok_after_create(fresh_db, backup_svc):
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_a_row(s)
|
|
||||||
r = backup_svc.create_now()
|
|
||||||
v = backup_svc.verify(r.backup.id)
|
|
||||||
assert v.ok
|
|
||||||
assert v.expected_fingerprint == v.actual_fingerprint
|
|
||||||
assert v.table_count >= 1
|
|
||||||
|
|
||||||
|
|
||||||
def test_verify_detects_tampered_ciphertext(fresh_db, backup_svc):
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_a_row(s)
|
|
||||||
r = backup_svc.create_now()
|
|
||||||
bin_path = backup_svc.backup_dir / r.backup.filename
|
|
||||||
# Flip a bit in the middle of the encrypted blob.
|
|
||||||
data = bytearray(bin_path.read_bytes())
|
|
||||||
idx = backup_mod.NONCE_LEN + 5
|
|
||||||
data[idx] ^= 0x01
|
|
||||||
bin_path.write_bytes(bytes(data))
|
|
||||||
|
|
||||||
v = backup_svc.verify(r.backup.id)
|
|
||||||
assert not v.ok
|
|
||||||
assert "decryption failed" in (v.reason or "")
|
|
||||||
|
|
||||||
|
|
||||||
def test_verify_handles_missing_file(fresh_db, backup_svc):
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_a_row(s)
|
|
||||||
r = backup_svc.create_now()
|
|
||||||
(backup_svc.backup_dir / r.backup.filename).unlink()
|
|
||||||
v = backup_svc.verify(r.backup.id)
|
|
||||||
assert not v.ok
|
|
||||||
assert "missing" in (v.reason or "")
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# restore — two-step
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_restore_two_step_round_trip(fresh_db, backup_svc, tmp_path):
|
|
||||||
"""Create a backup, mutate the live DB, restore, confirm mutation gone."""
|
|
||||||
from cyclone.db import Batch
|
|
||||||
import uuid
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
|
|
||||||
# 1. Backup a DB with one Batch row.
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_a_row(s)
|
|
||||||
snap = backup_svc.create_now()
|
|
||||||
|
|
||||||
# 2. Mutate the live DB (add another Batch row).
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
s.add(Batch(
|
|
||||||
id=str(uuid.uuid4()),
|
|
||||||
kind="837P",
|
|
||||||
input_filename="mutated.x12",
|
|
||||||
parsed_at=datetime.now(timezone.utc),
|
|
||||||
totals_json=None,
|
|
||||||
validation_json=None,
|
|
||||||
raw_result_json={"envelope": {"control_number": "2"}, "claims": [], "summary": {"passed": 0, "failed": 0, "failed_claim_ids": []}},
|
|
||||||
))
|
|
||||||
s.commit()
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
assert s.query(Batch).count() == 2
|
|
||||||
|
|
||||||
# 3. Initiate restore.
|
|
||||||
init = backup_svc.restore_initiate(snap.backup.id)
|
|
||||||
assert init.table_count >= 1
|
|
||||||
assert init.current_db_fingerprint != init.db_fingerprint # live != backup now
|
|
||||||
assert init.restore_token and len(init.restore_token) == 64
|
|
||||||
|
|
||||||
# 4. Confirm restore.
|
|
||||||
result = backup_svc.restore_confirm(snap.backup.id, init.restore_token)
|
|
||||||
assert result.new_db_fingerprint == init.db_fingerprint
|
|
||||||
|
|
||||||
# 5. The live DB now reflects the snapshot (1 row, not 2).
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
assert s.query(Batch).count() == 1
|
|
||||||
|
|
||||||
|
|
||||||
def test_restore_initiate_rejects_non_ok_backup(fresh_db, backup_svc, tmp_path, monkeypatch):
|
|
||||||
"""A backup row with status='error' cannot be restored."""
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_a_row(s)
|
|
||||||
r = backup_svc.create_now()
|
|
||||||
# Force the row to error.
|
|
||||||
from cyclone.db import DbBackup
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
row = session.get(DbBackup, r.backup.id)
|
|
||||||
row.status = STATUS_ERROR
|
|
||||||
row.error_message = "simulated"
|
|
||||||
session.commit()
|
|
||||||
with pytest.raises(BackupError, match="only 'ok' backups"):
|
|
||||||
backup_svc.restore_initiate(r.backup.id)
|
|
||||||
|
|
||||||
|
|
||||||
def test_restore_confirm_rejects_wrong_token(fresh_db, backup_svc):
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_a_row(s)
|
|
||||||
r = backup_svc.create_now()
|
|
||||||
init = backup_svc.restore_initiate(r.backup.id)
|
|
||||||
with pytest.raises(BackupError, match="not found"):
|
|
||||||
backup_svc.restore_confirm(r.backup.id, "0" * 64)
|
|
||||||
|
|
||||||
|
|
||||||
def test_restore_confirm_rejects_expired_token(fresh_db, backup_svc, monkeypatch):
|
|
||||||
"""A token whose expires_at is in the past is rejected."""
|
|
||||||
from datetime import datetime, timedelta, timezone
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_a_row(s)
|
|
||||||
r = backup_svc.create_now()
|
|
||||||
init = backup_svc.restore_initiate(r.backup.id)
|
|
||||||
# Manually age the token past its expiry.
|
|
||||||
with backup_svc._lock:
|
|
||||||
backup_svc._pending_restores[init.restore_token] = (
|
|
||||||
init.backup_id,
|
|
||||||
datetime.now(timezone.utc) - timedelta(seconds=1),
|
|
||||||
)
|
|
||||||
with pytest.raises(BackupError, match="expired"):
|
|
||||||
backup_svc.restore_confirm(r.backup.id, init.restore_token)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# prune
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_prune_deletes_files_and_marks_status(fresh_db, backup_svc):
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_a_row(s)
|
|
||||||
r1 = backup_svc.create_now()
|
|
||||||
# The retention cutoff is 7 days from now. Move the row's created_at
|
|
||||||
# back 30 days so it's definitely past retention.
|
|
||||||
from datetime import datetime, timedelta, timezone
|
|
||||||
from cyclone.db import DbBackup
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
row = session.get(DbBackup, r1.backup.id)
|
|
||||||
row.created_at = datetime.now(timezone.utc) - timedelta(days=30)
|
|
||||||
session.commit()
|
|
||||||
|
|
||||||
deleted = backup_svc.prune()
|
|
||||||
assert len(deleted) == 2 # .bin + .meta.json
|
|
||||||
rows = backup_svc.list_backups()
|
|
||||||
assert rows[0].status == STATUS_PRUNED
|
|
||||||
|
|
||||||
|
|
||||||
def test_prune_keeps_recent_backups(fresh_db, backup_svc):
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_a_row(s)
|
|
||||||
backup_svc.create_now()
|
|
||||||
deleted = backup_svc.prune()
|
|
||||||
assert deleted == []
|
|
||||||
rows = backup_svc.list_backups()
|
|
||||||
assert rows[0].status == STATUS_OK
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# status
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_status_reports_counts(fresh_db, backup_svc):
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_a_row(s)
|
|
||||||
backup_svc.create_now()
|
|
||||||
snap = backup_svc.status()
|
|
||||||
assert snap["totals"]["ok"] == 1
|
|
||||||
assert snap["totals"]["all"] == 1
|
|
||||||
assert snap["backup_dir"] == str(backup_svc.backup_dir)
|
|
||||||
assert snap["retention_days"] == 7
|
|
||||||
assert snap["used_fallback_key"] is False
|
|
||||||
assert snap["last_backup_at"] is not None
|
|
||||||
assert snap["last_ok_backup_at"] is not None
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Fallback key
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_fallback_key_used_when_no_passphrase(fresh_db, tmp_path):
|
|
||||||
"""If no passphrase AND no SQLCipher, refuse. Otherwise fallback + warn."""
|
|
||||||
backup_dir = tmp_path / "backups"
|
|
||||||
svc = BackupService(backup_dir=backup_dir, passphrase=None, retention_days=7)
|
|
||||||
# No SQLCipher key either → BackupError.
|
|
||||||
with pytest.raises(BackupError, match="no backup passphrase"):
|
|
||||||
svc._ensure_key()
|
|
||||||
|
|
||||||
|
|
||||||
def test_key_fingerprint_changes_per_passphrase(fresh_db, tmp_path):
|
|
||||||
"""Two services with different passphrases have different key fingerprints."""
|
|
||||||
s1 = BackupService(tmp_path / "b1", passphrase="alpha", retention_days=1)
|
|
||||||
s2 = BackupService(tmp_path / "b2", passphrase="beta", retention_days=1)
|
|
||||||
# Force key derivation.
|
|
||||||
s1._ensure_key()
|
|
||||||
s2._ensure_key()
|
|
||||||
assert s1.key_fingerprint != s2.key_fingerprint
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Module-level singleton
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_module_singleton_round_trip(fresh_db, tmp_path):
|
|
||||||
reset_backup_service_for_tests()
|
|
||||||
svc = configure_backup_service(
|
|
||||||
tmp_path / "backups", passphrase="x", retention_days=1,
|
|
||||||
)
|
|
||||||
assert get_backup_service() is svc
|
|
||||||
# Second configure is a no-op (returns existing).
|
|
||||||
assert configure_backup_service(
|
|
||||||
tmp_path / "backups2", passphrase="y", retention_days=2,
|
|
||||||
) is svc
|
|
||||||
reset_backup_service_for_tests()
|
|
||||||
|
|
||||||
|
|
||||||
def test_module_singleton_get_raises_when_unset():
|
|
||||||
reset_backup_service_for_tests()
|
|
||||||
with pytest.raises(RuntimeError):
|
|
||||||
get_backup_service()
|
|
||||||
@@ -1,144 +0,0 @@
|
|||||||
"""SP17 — `cyclone backup` CLI subcommand tests.
|
|
||||||
|
|
||||||
Uses Click's CliRunner + monkeypatching of Keychain + DB env so the
|
|
||||||
subcommands can run without the operator's machine state.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from click.testing import CliRunner
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def _cli_env(tmp_path, monkeypatch):
|
|
||||||
"""Fresh sqlite DB + in-memory Keychain stub."""
|
|
||||||
from cyclone import db
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
from cyclone import secrets as secrets_mod
|
|
||||||
from cyclone.db import Batch
|
|
||||||
import uuid
|
|
||||||
|
|
||||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
|
|
||||||
db._reset_for_tests()
|
|
||||||
db.init_db()
|
|
||||||
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
s.add(Batch(
|
|
||||||
id=str(uuid.uuid4()),
|
|
||||||
kind="837P",
|
|
||||||
input_filename="seed.x12",
|
|
||||||
parsed_at=datetime.now(timezone.utc),
|
|
||||||
totals_json=None,
|
|
||||||
validation_json=None,
|
|
||||||
raw_result_json={"envelope": {"control_number": "1"}, "claims": [], "summary": {"passed": 0, "failed": 0, "failed_claim_ids": []}},
|
|
||||||
))
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
# In-memory Keychain so passphrase + salt persist across
|
|
||||||
# separate CliRunner invocations within one test (each
|
|
||||||
# subprocess-like invocation would otherwise generate a fresh
|
|
||||||
# random salt and fail to decrypt).
|
|
||||||
store: dict[str, str] = {}
|
|
||||||
store[svc_mod.KEYCHAIN_BACKUP_PASSPHRASE_ACCOUNT] = "cli-test-passphrase"
|
|
||||||
# Pre-populate a stable salt so the very first invocation
|
|
||||||
# doesn't generate a new random one (which the next invocation
|
|
||||||
# would then fail to reproduce).
|
|
||||||
store[svc_mod.KEYCHAIN_BACKUP_SALT_ACCOUNT] = "0123456789abcdef0123456789abcdef"
|
|
||||||
|
|
||||||
def _get(name):
|
|
||||||
return store.get(name)
|
|
||||||
def _set(name, value):
|
|
||||||
store[name] = value
|
|
||||||
return True
|
|
||||||
|
|
||||||
monkeypatch.setattr(secrets_mod, "get_secret", _get)
|
|
||||||
monkeypatch.setattr(secrets_mod, "set_secret", _set)
|
|
||||||
|
|
||||||
backup_dir = tmp_path / "backups"
|
|
||||||
monkeypatch.setenv("CYCLONE_BACKUP_DIR", str(backup_dir))
|
|
||||||
monkeypatch.setenv("CYCLONE_BACKUP_RETENTION_DAYS", "7")
|
|
||||||
|
|
||||||
yield backup_dir
|
|
||||||
db._reset_for_tests()
|
|
||||||
|
|
||||||
|
|
||||||
def _run(args, env):
|
|
||||||
from cyclone.cli import main
|
|
||||||
runner = CliRunner()
|
|
||||||
return runner.invoke(main, args, catch_exceptions=False)
|
|
||||||
|
|
||||||
|
|
||||||
def test_backup_create_list_verify_status(_cli_env):
|
|
||||||
"""Happy path: create → list → verify → status."""
|
|
||||||
backup_dir = _cli_env
|
|
||||||
|
|
||||||
# create
|
|
||||||
r = _run(["backup", "create"], _cli_env)
|
|
||||||
assert r.exit_code == 0, r.output
|
|
||||||
assert "created backup id=" in r.output
|
|
||||||
|
|
||||||
# list
|
|
||||||
r = _run(["backup", "list"], _cli_env)
|
|
||||||
assert r.exit_code == 0, r.output
|
|
||||||
assert ".bin" in r.output
|
|
||||||
|
|
||||||
# verify (we don't know the id, parse it from the list output)
|
|
||||||
import re
|
|
||||||
m = re.search(r"^\s*(\d+)\s+ok\s+", r.output, re.MULTILINE)
|
|
||||||
assert m, r.output
|
|
||||||
backup_id = int(m.group(1))
|
|
||||||
r = _run(["backup", "verify", str(backup_id)], _cli_env)
|
|
||||||
assert r.exit_code == 0, r.output
|
|
||||||
assert r.output.startswith("OK:")
|
|
||||||
|
|
||||||
# status
|
|
||||||
r = _run(["backup", "status"], _cli_env)
|
|
||||||
assert r.exit_code == 0, r.output
|
|
||||||
assert '"totals"' in r.output
|
|
||||||
assert '"ok": 1' in r.output
|
|
||||||
|
|
||||||
|
|
||||||
def test_backup_verify_fails_on_tampered_ciphertext(_cli_env):
|
|
||||||
from cyclone import backup as backup_mod
|
|
||||||
from cyclone import backup_service as svc_mod
|
|
||||||
from cyclone import secrets as secrets_mod
|
|
||||||
|
|
||||||
# Create a backup.
|
|
||||||
r = _run(["backup", "create"], _cli_env)
|
|
||||||
assert r.exit_code == 0
|
|
||||||
# Tamper.
|
|
||||||
bin_path = next(_cli_env.glob("*.bin"))
|
|
||||||
data = bytearray(bin_path.read_bytes())
|
|
||||||
data[backup_mod.NONCE_LEN + 5] ^= 0x01
|
|
||||||
bin_path.write_bytes(bytes(data))
|
|
||||||
|
|
||||||
# Verify should fail.
|
|
||||||
r = _run(["backup", "verify", "1"], _cli_env)
|
|
||||||
assert r.exit_code == 1
|
|
||||||
assert "FAIL" in r.output
|
|
||||||
|
|
||||||
|
|
||||||
def test_backup_restore_requires_yes_flag(_cli_env):
|
|
||||||
"""Without --yes, an interactive confirm blocks and the command aborts."""
|
|
||||||
r = _run(["backup", "create"], _cli_env)
|
|
||||||
assert r.exit_code == 0
|
|
||||||
# Click's runner auto-declines the confirm prompt; expect abort.
|
|
||||||
r = _run(["backup", "restore", "1"], _cli_env, )
|
|
||||||
# CliRunner auto-aborts confirm prompts by default → exit code != 0.
|
|
||||||
assert r.exit_code != 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_backup_prune_aborts_without_yes(_cli_env):
|
|
||||||
r = _run(["backup", "create"], _cli_env)
|
|
||||||
assert r.exit_code == 0
|
|
||||||
# Same auto-abort for the prune confirm.
|
|
||||||
r = _run(["backup", "prune"], _cli_env)
|
|
||||||
assert r.exit_code != 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_backup_init_passphrase_rejects_short(_cli_env):
|
|
||||||
"""init-passphrase enforces a 12-char minimum."""
|
|
||||||
r = _run(["backup", "init-passphrase", "--passphrase", "short"], _cli_env)
|
|
||||||
assert r.exit_code != 0
|
|
||||||
assert "12 characters" in r.output
|
|
||||||
@@ -1,128 +0,0 @@
|
|||||||
"""CLI: python -m cyclone users {create,list,disable,reset-password,set-role}.
|
|
||||||
|
|
||||||
Uses Click's CliRunner (matches the existing parse-837/parse-835 CLI tests).
|
|
||||||
The full CLI group lives in ``cyclone.cli.main`` — we exercise the
|
|
||||||
``users`` subgroup through the top-level ``main`` so the wiring is real.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from sqlalchemy import delete, select
|
|
||||||
|
|
||||||
from cyclone.auth import users
|
|
||||||
from cyclone.cli import main as cli_main
|
|
||||||
from cyclone.db import Session as DbSession
|
|
||||||
from cyclone.db import SessionLocal, User
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _clear():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
yield
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def test_create_user_via_cli():
|
|
||||||
from click.testing import CliRunner
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(
|
|
||||||
cli_main,
|
|
||||||
[
|
|
||||||
"users", "create", "cli-user",
|
|
||||||
"--role", "viewer",
|
|
||||||
"--password", "clipassword1",
|
|
||||||
],
|
|
||||||
input="", # don't prompt
|
|
||||||
)
|
|
||||||
assert result.exit_code == 0, result.output
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.get_by_username(db, "cli-user")
|
|
||||||
assert u is not None
|
|
||||||
assert u.role == "viewer"
|
|
||||||
|
|
||||||
|
|
||||||
def test_list_users_via_cli():
|
|
||||||
from click.testing import CliRunner
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="listed", password="hunter2hunter2", role="user")
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(cli_main, ["users", "list"])
|
|
||||||
assert result.exit_code == 0, result.output
|
|
||||||
assert "listed" in result.output
|
|
||||||
|
|
||||||
|
|
||||||
def test_disable_user_via_cli():
|
|
||||||
from click.testing import CliRunner
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="todie", password="hunter2hunter2", role="user")
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(cli_main, ["users", "disable", "todie"])
|
|
||||||
assert result.exit_code == 0, result.output
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.get_by_username(db, "todie")
|
|
||||||
assert u.disabled_at is not None
|
|
||||||
|
|
||||||
|
|
||||||
def test_reset_password_via_cli():
|
|
||||||
from click.testing import CliRunner
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="pwchange", password="oldpassword1", role="user")
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(
|
|
||||||
cli_main,
|
|
||||||
[
|
|
||||||
"users", "reset-password", "pwchange",
|
|
||||||
"--password", "newpassword1",
|
|
||||||
],
|
|
||||||
)
|
|
||||||
assert result.exit_code == 0, result.output
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.get_by_username(db, "pwchange")
|
|
||||||
assert users.verify_password("newpassword1", u.password_hash)
|
|
||||||
|
|
||||||
|
|
||||||
def test_set_role_via_cli():
|
|
||||||
from click.testing import CliRunner
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
users.create(db, username="promote", password="hunter2hunter2", role="viewer")
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(
|
|
||||||
cli_main,
|
|
||||||
["users", "set-role", "promote", "--role", "admin"],
|
|
||||||
)
|
|
||||||
assert result.exit_code == 0, result.output
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
u = users.get_by_username(db, "promote")
|
|
||||||
assert u.role == "admin"
|
|
||||||
|
|
||||||
|
|
||||||
def test_create_rejects_short_password():
|
|
||||||
from click.testing import CliRunner
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(
|
|
||||||
cli_main,
|
|
||||||
[
|
|
||||||
"users", "create", "weak",
|
|
||||||
"--role", "viewer",
|
|
||||||
"--password", "short",
|
|
||||||
],
|
|
||||||
)
|
|
||||||
# Click surfaces validation failures with a non-zero exit code.
|
|
||||||
assert result.exit_code != 0, result.output
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
rows = db.execute(select(User).where(User.username == "weak")).scalars().all()
|
|
||||||
assert rows == []
|
|
||||||
|
|
||||||
|
|
||||||
def test_disable_unknown_user_exits_nonzero():
|
|
||||||
from click.testing import CliRunner
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(cli_main, ["users", "disable", "ghost"])
|
|
||||||
assert result.exit_code != 0, result.output
|
|
||||||
@@ -1,80 +0,0 @@
|
|||||||
"""Tests for ``cyclone validate-npi`` + ``cyclone validate-tax-id`` (SP20).
|
|
||||||
|
|
||||||
CLI smoke tests — verify exit codes (0 = valid, 1 = invalid) and that
|
|
||||||
the help text references the new subcommands. We don't pipe the value
|
|
||||||
into shared logs (NPI / EIN are PHI / PII); the CliRunner captures it.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from click.testing import CliRunner
|
|
||||||
|
|
||||||
from cyclone.cli import main
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# validate-npi
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_cli_validate_npi_valid_exits_zero():
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(main, ["validate-npi", "1234567893"])
|
|
||||||
assert result.exit_code == 0, result.output
|
|
||||||
assert "OK" in result.output
|
|
||||||
|
|
||||||
|
|
||||||
def test_cli_validate_npi_bad_checksum_exits_one():
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(main, ["validate-npi", "1234567890"])
|
|
||||||
assert result.exit_code == 1
|
|
||||||
assert "INVALID" in result.output
|
|
||||||
|
|
||||||
|
|
||||||
def test_cli_validate_npi_wrong_length_exits_one():
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(main, ["validate-npi", "12345"])
|
|
||||||
assert result.exit_code == 1
|
|
||||||
assert "INVALID" in result.output
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# validate-tax-id
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_cli_validate_tax_id_formatted_exits_zero():
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(main, ["validate-tax-id", "72-1587149"])
|
|
||||||
assert result.exit_code == 0, result.output
|
|
||||||
assert "721587149" in result.output # normalized form echoed
|
|
||||||
|
|
||||||
|
|
||||||
def test_cli_validate_tax_id_unformatted_exits_zero():
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(main, ["validate-tax-id", "721587149"])
|
|
||||||
assert result.exit_code == 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_cli_validate_tax_id_reserved_prefix_exits_one():
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(main, ["validate-tax-id", "00-1234567"])
|
|
||||||
assert result.exit_code == 1
|
|
||||||
assert "reserved" in result.output.lower()
|
|
||||||
|
|
||||||
|
|
||||||
def test_cli_validate_tax_id_malformed_exits_one():
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(main, ["validate-tax-id", "not-an-ein"])
|
|
||||||
assert result.exit_code == 1
|
|
||||||
assert "9-digit" in result.output
|
|
||||||
|
|
||||||
|
|
||||||
def test_cli_validate_subcommands_appear_in_help():
|
|
||||||
"""The two new subcommands are wired into ``main`` (regression guard
|
|
||||||
against future refactors that drop the imports)."""
|
|
||||||
runner = CliRunner()
|
|
||||||
result = runner.invoke(main, ["--help"])
|
|
||||||
assert result.exit_code == 0
|
|
||||||
assert "validate-npi" in result.output
|
|
||||||
assert "validate-tax-id" in result.output
|
|
||||||
@@ -113,69 +113,3 @@ def test_run_ignores_non_sql_files(
|
|||||||
).all()
|
).all()
|
||||||
assert len(rows) == 0
|
assert len(rows) == 0
|
||||||
assert _user_version(engine) == 1
|
assert _user_version(engine) == 1
|
||||||
|
|
||||||
|
|
||||||
def test_migration_latest_idempotent_on_fresh_db(tmp_path: Path) -> None:
|
|
||||||
"""All migrations up to the current head run cleanly on a fresh DB,
|
|
||||||
and a second run is a no-op (no version bump). SP22 bumped the
|
|
||||||
expected head from 14 to 15 with the new UNIQUE-drop migration.
|
|
||||||
"""
|
|
||||||
engine = _fresh_engine(tmp_path)
|
|
||||||
db_migrate.run(engine)
|
|
||||||
v_after_first = _user_version(engine)
|
|
||||||
assert v_after_first == 15, f"expected head=15, got {v_after_first}"
|
|
||||||
|
|
||||||
db_migrate.run(engine)
|
|
||||||
assert _user_version(engine) == 15, "second run should not bump version"
|
|
||||||
|
|
||||||
|
|
||||||
def test_drop_claims_unique_constraint_migration(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
||||||
"""SP22: migration 0015 recreates the `claims` table without the inline
|
|
||||||
`UNIQUE(batch_id, patient_control_number)` constraint, so two claims in
|
|
||||||
one batch can share a patient_control_number (real 837P multi-claim
|
|
||||||
subscriber loops do this).
|
|
||||||
|
|
||||||
Discovery (2026-06-23): the inline UNIQUE does not exist in the current
|
|
||||||
production DB or in main's fresh-DB schema, so this migration is a
|
|
||||||
defensive no-op against the current state. The test still proves
|
|
||||||
migration correctness: (a) all migrations up to v15 run cleanly,
|
|
||||||
(b) two rows with the same (batch_id, patient_control_number) can
|
|
||||||
be inserted after the migration (proving no UNIQUE was re-introduced
|
|
||||||
by the table recreation).
|
|
||||||
"""
|
|
||||||
# Real migrations dir so the test exercises the actual 0015 file.
|
|
||||||
monkeypatch.setattr(
|
|
||||||
db_migrate,
|
|
||||||
"MIGRATIONS_DIR",
|
|
||||||
Path(__file__).parent.parent / "src" / "cyclone" / "migrations",
|
|
||||||
)
|
|
||||||
engine = _fresh_engine(tmp_path)
|
|
||||||
|
|
||||||
db_migrate.run(engine)
|
|
||||||
assert _user_version(engine) == 15, f"expected head=15, got {_user_version(engine)}"
|
|
||||||
|
|
||||||
# Two claims in one batch with the same patient_control_number
|
|
||||||
# must be insertable. If 0015's table recreation re-introduced a
|
|
||||||
# UNIQUE(batch_id, patient_control_number), this would raise
|
|
||||||
# IntegrityError. (The test also implicitly asserts the FK from
|
|
||||||
# claims to batches still works after the recreation.)
|
|
||||||
with engine.begin() as conn:
|
|
||||||
conn.exec_driver_sql(
|
|
||||||
"INSERT INTO batches (id, kind, input_filename, parsed_at) "
|
|
||||||
"VALUES ('B1', '837p', 'test.txt', '2026-01-01 00:00:00')"
|
|
||||||
)
|
|
||||||
conn.exec_driver_sql(
|
|
||||||
"INSERT INTO claims (id, batch_id, patient_control_number, charge_amount) "
|
|
||||||
"VALUES ('CLM-1', 'B1', 'SAME-PCN', 100)"
|
|
||||||
)
|
|
||||||
conn.exec_driver_sql(
|
|
||||||
"INSERT INTO claims (id, batch_id, patient_control_number, charge_amount) "
|
|
||||||
"VALUES ('CLM-2', 'B1', 'SAME-PCN', 200)"
|
|
||||||
)
|
|
||||||
rows = conn.exec_driver_sql(
|
|
||||||
"SELECT id, charge_amount FROM claims "
|
|
||||||
"WHERE patient_control_number='SAME-PCN' ORDER BY id"
|
|
||||||
).all()
|
|
||||||
assert [r[0] for r in rows] == ["CLM-1", "CLM-2"]
|
|
||||||
assert [float(r[1]) for r in rows] == [100.0, 200.0]
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,236 +0,0 @@
|
|||||||
"""Dockerfile + compose-config smoke tests for SP23.
|
|
||||||
|
|
||||||
These tests do NOT require a running Docker daemon (the compose-up test
|
|
||||||
is gated on ``DOCKER_TESTS=1`` so it can be skipped on bare CI without
|
|
||||||
Docker). They validate that:
|
|
||||||
|
|
||||||
* ``docker-compose.yml`` at the repo root is syntactically valid and that
|
|
||||||
the shape we expect (services, secrets, volumes, networks) is present.
|
|
||||||
* Both Dockerfiles parse with ``docker build --check`` if Docker is on PATH.
|
|
||||||
* The named-volume mount paths match what ``cyclone.db`` + the BackupService
|
|
||||||
expect at runtime.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import shutil
|
|
||||||
import subprocess
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
import yaml
|
|
||||||
|
|
||||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
|
||||||
COMPOSE_FILE = REPO_ROOT / "docker-compose.yml"
|
|
||||||
BACKEND_DOCKERFILE = REPO_ROOT / "backend" / "Dockerfile"
|
|
||||||
FRONTEND_DOCKERFILE = REPO_ROOT / "Dockerfile.frontend"
|
|
||||||
FRONTEND_NGINX_CONF = REPO_ROOT / "nginx.conf"
|
|
||||||
|
|
||||||
|
|
||||||
def _has_docker() -> bool:
|
|
||||||
return shutil.which("docker") is not None
|
|
||||||
|
|
||||||
|
|
||||||
def _has_docker_compose() -> bool:
|
|
||||||
if shutil.which("docker") is None:
|
|
||||||
return False
|
|
||||||
return (
|
|
||||||
subprocess.run(
|
|
||||||
["docker", "compose", "version"],
|
|
||||||
capture_output=True,
|
|
||||||
check=False,
|
|
||||||
).returncode
|
|
||||||
== 0
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _volume_source(v) -> str:
|
|
||||||
"""Normalize a compose volume entry to the source name (str form or 'source' key)."""
|
|
||||||
if isinstance(v, str):
|
|
||||||
# Long form: "named_volume:/container/path" — split on ':'.
|
|
||||||
return v.split(":", 1)[0]
|
|
||||||
if isinstance(v, dict):
|
|
||||||
return v.get("source", "")
|
|
||||||
return ""
|
|
||||||
|
|
||||||
|
|
||||||
def test_compose_file_exists():
|
|
||||||
assert COMPOSE_FILE.exists(), f"missing {COMPOSE_FILE}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_compose_config_validates():
|
|
||||||
"""``docker compose config`` should exit 0 with no stderr."""
|
|
||||||
if not _has_docker_compose():
|
|
||||||
pytest.skip("docker compose not on PATH")
|
|
||||||
result = subprocess.run(
|
|
||||||
[
|
|
||||||
"docker",
|
|
||||||
"compose",
|
|
||||||
"-f",
|
|
||||||
str(COMPOSE_FILE),
|
|
||||||
"config",
|
|
||||||
"--quiet",
|
|
||||||
],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
cwd=REPO_ROOT,
|
|
||||||
)
|
|
||||||
assert result.returncode == 0, (
|
|
||||||
f"compose config failed: stderr={result.stderr!r}"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_compose_declares_required_services():
|
|
||||||
compose = yaml.safe_load(COMPOSE_FILE.read_text())
|
|
||||||
services = compose.get("services", {})
|
|
||||||
assert "backend" in services, "compose must declare a 'backend' service"
|
|
||||||
assert "frontend" in services, "compose must declare a 'frontend' service"
|
|
||||||
|
|
||||||
backend = services["backend"]
|
|
||||||
backend_volume_sources = {_volume_source(v) for v in backend.get("volumes", [])}
|
|
||||||
assert "cyclone_db" in backend_volume_sources, (
|
|
||||||
"backend must mount the cyclone_db volume"
|
|
||||||
)
|
|
||||||
assert backend.get("restart") == "unless-stopped", (
|
|
||||||
"backend must restart: unless-stopped so healthcheck failures recover"
|
|
||||||
)
|
|
||||||
assert "healthcheck" in backend, "backend must declare a healthcheck"
|
|
||||||
|
|
||||||
frontend = services["frontend"]
|
|
||||||
assert "8080:8080" in frontend.get("ports", []), (
|
|
||||||
"frontend must publish 8080:8080 for LAN access"
|
|
||||||
)
|
|
||||||
depends_on = frontend.get("depends_on") or {}
|
|
||||||
if isinstance(depends_on, dict):
|
|
||||||
backend_dep = depends_on.get("backend") or {}
|
|
||||||
assert backend_dep.get("condition") == "service_healthy", (
|
|
||||||
"frontend must wait for backend healthy before starting"
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
# Short-form `depends_on: [backend]` is acceptable too — it implies
|
|
||||||
# service_started, not service_healthy. Flag a soft warning.
|
|
||||||
pytest.skip(
|
|
||||||
"frontend uses short-form depends_on; switch to long-form for service_healthy"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def test_compose_declares_required_secrets_and_volumes():
|
|
||||||
compose = yaml.safe_load(COMPOSE_FILE.read_text())
|
|
||||||
secrets = compose.get("secrets", {})
|
|
||||||
for required in ("cyclone_db_key", "cyclone_admin_password"):
|
|
||||||
assert required in secrets, f"compose must declare secret {required!r}"
|
|
||||||
volumes = compose.get("volumes", {})
|
|
||||||
for required in (
|
|
||||||
"cyclone_db",
|
|
||||||
"cyclone_backups",
|
|
||||||
"cyclone_prodfiles",
|
|
||||||
"cyclone_sftp_staging",
|
|
||||||
"cyclone_logs",
|
|
||||||
):
|
|
||||||
assert required in volumes, f"compose must declare volume {required!r}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_compose_backend_wires_backup_autostart():
|
|
||||||
"""The existing BackupService (SP17) needs CYCLONE_BACKUP_AUTOSTART=1
|
|
||||||
on container boot. The compose env block must include it."""
|
|
||||||
compose = yaml.safe_load(COMPOSE_FILE.read_text())
|
|
||||||
env = compose["services"]["backend"].get("environment", {})
|
|
||||||
assert str(env.get("CYCLONE_BACKUP_AUTOSTART")) == "1", (
|
|
||||||
"backend must autostart the backup scheduler (CYCLONE_BACKUP_AUTOSTART=1)"
|
|
||||||
)
|
|
||||||
assert "CYCLONE_BACKUP_INTERVAL_HOURS" in env
|
|
||||||
assert "CYCLONE_BACKUP_RETENTION_DAYS" in env
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.skipif(
|
|
||||||
not _has_docker(), reason="docker not on PATH; skipping Dockerfile parse check"
|
|
||||||
)
|
|
||||||
def test_backend_dockerfile_parses():
|
|
||||||
result = subprocess.run(
|
|
||||||
[
|
|
||||||
"docker",
|
|
||||||
"build",
|
|
||||||
"--check",
|
|
||||||
"-f",
|
|
||||||
str(BACKEND_DOCKERFILE),
|
|
||||||
str(REPO_ROOT / "backend"),
|
|
||||||
],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
)
|
|
||||||
assert result.returncode == 0, (
|
|
||||||
f"backend Dockerfile failed to parse: stderr={result.stderr!r}"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.skipif(
|
|
||||||
not _has_docker(), reason="docker not on PATH; skipping Dockerfile parse check"
|
|
||||||
)
|
|
||||||
def test_frontend_dockerfile_parses():
|
|
||||||
result = subprocess.run(
|
|
||||||
[
|
|
||||||
"docker",
|
|
||||||
"build",
|
|
||||||
"--check",
|
|
||||||
"-f",
|
|
||||||
str(FRONTEND_DOCKERFILE),
|
|
||||||
str(REPO_ROOT),
|
|
||||||
],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
)
|
|
||||||
assert result.returncode == 0, (
|
|
||||||
f"frontend Dockerfile failed to parse: stderr={result.stderr!r}"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.skipif(
|
|
||||||
not _has_docker_compose()
|
|
||||||
or not os.environ.get("DOCKER_TESTS"),
|
|
||||||
reason="DOCKER_TESTS=1 + docker compose required for live bring-up",
|
|
||||||
)
|
|
||||||
def test_compose_up_brings_up_healthy_stack():
|
|
||||||
"""Gated live test — only runs when DOCKER_TESTS=1 and docker compose
|
|
||||||
is available. Builds + brings up the full stack and waits up to 120s
|
|
||||||
for the backend healthcheck to come up healthy. Uses the override
|
|
||||||
file (docker-compose.override.yml) when present so the test doesn't
|
|
||||||
require sudo to create /etc/cyclone/secrets/."""
|
|
||||||
# The stack publishes host port 8080. If something else on this host
|
|
||||||
# is already using it (e.g. nocodb on a dev box) the test can't run
|
|
||||||
# here but will run cleanly on a fresh CI worker. Skip with a clear
|
|
||||||
# message rather than failing with a confusing port-bind error.
|
|
||||||
import socket
|
|
||||||
|
|
||||||
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
|
|
||||||
if s.connect_ex(("127.0.0.1", 8080)) == 0:
|
|
||||||
pytest.skip("host port 8080 already bound — rerun on a fresh host")
|
|
||||||
|
|
||||||
override = REPO_ROOT / "docker-compose.override.yml"
|
|
||||||
cmd_base = ["docker", "compose", "-f", str(COMPOSE_FILE)]
|
|
||||||
if override.exists():
|
|
||||||
cmd_base.extend(["-f", str(override)])
|
|
||||||
subprocess.run(
|
|
||||||
cmd_base + ["up", "-d", "--build"],
|
|
||||||
check=True,
|
|
||||||
cwd=REPO_ROOT,
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
import time
|
|
||||||
|
|
||||||
for _ in range(60):
|
|
||||||
ps = subprocess.run(
|
|
||||||
cmd_base + ["ps", "--format", "json"],
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
cwd=REPO_ROOT,
|
|
||||||
)
|
|
||||||
if "healthy" in ps.stdout:
|
|
||||||
return
|
|
||||||
time.sleep(2)
|
|
||||||
pytest.fail("compose stack did not become healthy within 120s")
|
|
||||||
finally:
|
|
||||||
subprocess.run(
|
|
||||||
cmd_base + ["down", "-v"],
|
|
||||||
check=False,
|
|
||||||
cwd=REPO_ROOT,
|
|
||||||
)
|
|
||||||
@@ -1,70 +0,0 @@
|
|||||||
"""Spot-check that existing endpoints now require auth (when AUTH_DISABLED is not set).
|
|
||||||
|
|
||||||
The conftest in ``tests/conftest.py`` flips ``AUTH_DISABLED=True`` for
|
|
||||||
every test module that does NOT start with ``test_auth`` — this module
|
|
||||||
deliberately is NOT named ``test_auth_*`` so it inherits the default
|
|
||||||
disabled posture... wait, that's the opposite of what we want.
|
|
||||||
|
|
||||||
This module is named ``test_existing_endpoints_require_auth`` so the
|
|
||||||
conftest will treat it as a legacy test and set ``AUTH_DISABLED=True``,
|
|
||||||
bypassing the auth check entirely. To actually verify the gate, this
|
|
||||||
test's ``client`` fixture flips ``AUTH_DISABLED`` to ``False``
|
|
||||||
*just for this test*, then restores it afterwards. This way the
|
|
||||||
rest of the suite still sees the disabled posture and existing
|
|
||||||
tests keep passing.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
from sqlalchemy import delete
|
|
||||||
|
|
||||||
from cyclone.api import app
|
|
||||||
from cyclone.auth import deps as _auth_deps
|
|
||||||
from cyclone.auth.deps import AUTH_DISABLED
|
|
||||||
from cyclone.db import Session as DbSession
|
|
||||||
from cyclone.db import SessionLocal, User
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _clear():
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
yield
|
|
||||||
with SessionLocal()() as db:
|
|
||||||
db.execute(delete(DbSession))
|
|
||||||
db.execute(delete(User))
|
|
||||||
db.commit()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def client():
|
|
||||||
# Force AUTH_DISABLED=False for these tests so the dep actually checks the cookie.
|
|
||||||
original = AUTH_DISABLED
|
|
||||||
_auth_deps.AUTH_DISABLED = False
|
|
||||||
try:
|
|
||||||
yield TestClient(app)
|
|
||||||
finally:
|
|
||||||
_auth_deps.AUTH_DISABLED = original
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("method,path", [
|
|
||||||
("GET", "/api/claims"),
|
|
||||||
("GET", "/api/remittances"),
|
|
||||||
("GET", "/api/providers"),
|
|
||||||
("GET", "/api/batches"),
|
|
||||||
("GET", "/api/payers/p1/summary"),
|
|
||||||
("GET", "/api/activity"),
|
|
||||||
])
|
|
||||||
def test_existing_get_endpoints_require_auth(client, method, path):
|
|
||||||
resp = client.request(method, path)
|
|
||||||
assert resp.status_code == 401, f"{method} {path} returned {resp.status_code}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_health_is_public(client):
|
|
||||||
"""``/api/health`` is the public healthcheck and must remain reachable."""
|
|
||||||
resp = client.get("/api/health")
|
|
||||||
assert resp.status_code != 401, f"/api/health returned {resp.status_code}"
|
|
||||||
@@ -27,8 +27,7 @@ MT = ZoneInfo("America/Denver")
|
|||||||
def test_build_outbound_with_explicit_mt():
|
def test_build_outbound_with_explicit_mt():
|
||||||
now = datetime(2026, 6, 20, 13, 22, 43, 505_000, tzinfo=MT)
|
now = datetime(2026, 6, 20, 13, 22, 43, 505_000, tzinfo=MT)
|
||||||
name = build_outbound_filename("11525703", "837P", now_mt=now)
|
name = build_outbound_filename("11525703", "837P", now_mt=now)
|
||||||
# HCPF outbound format: tp prefix on the tpid
|
assert name == "11525703-837P-20260620132243505-1of1.x12"
|
||||||
assert name == "tp11525703-837P-20260620132243505-1of1.x12"
|
|
||||||
|
|
||||||
|
|
||||||
def test_build_outbound_default_extension():
|
def test_build_outbound_default_extension():
|
||||||
@@ -40,17 +39,15 @@ def test_build_outbound_default_extension():
|
|||||||
def test_build_outbound_custom_extension():
|
def test_build_outbound_custom_extension():
|
||||||
now = datetime(2026, 6, 20, 13, 22, 43, 505_000, tzinfo=MT)
|
now = datetime(2026, 6, 20, 13, 22, 43, 505_000, tzinfo=MT)
|
||||||
name = build_outbound_filename("11525703", "837P", ext="txt", now_mt=now)
|
name = build_outbound_filename("11525703", "837P", ext="txt", now_mt=now)
|
||||||
assert name == "tp11525703-837P-20260620132243505-1of1.txt"
|
assert name == "11525703-837P-20260620132243505-1of1.txt"
|
||||||
|
|
||||||
|
|
||||||
def test_build_outbound_uses_mt_when_no_arg():
|
def test_build_outbound_uses_mt_when_no_arg():
|
||||||
# Snapshot test — the timestamp will be very recent; check format only
|
# Snapshot test — the timestamp will be very recent; check format only
|
||||||
name = build_outbound_filename("11525703", "837P")
|
name = build_outbound_filename("11525703", "837P")
|
||||||
assert OUTBOUND_RE.match(name), name
|
assert OUTBOUND_RE.match(name), name
|
||||||
# tp11525703-837P-YYYYMMDDhhmmssSSS-1of1.x12 — 4 dash-separated parts
|
|
||||||
parts = name.split("-")
|
parts = name.split("-")
|
||||||
assert len(parts) == 4
|
assert len(parts) == 4
|
||||||
assert parts[0] == "tp11525703"
|
|
||||||
assert len(parts[2]) == 17 # yyyymmddhhmmssSSS
|
assert len(parts[2]) == 17 # yyyymmddhhmmssSSS
|
||||||
|
|
||||||
|
|
||||||
@@ -131,9 +128,8 @@ def test_parse_inbound_rejects_non_x12_ext():
|
|||||||
|
|
||||||
|
|
||||||
def test_roundtrip_outbound_to_inbound():
|
def test_roundtrip_outbound_to_inbound():
|
||||||
# Outbound uses tp{...}, inbound uses TP{...} (case differs but both
|
# Outbound tpid is bare (no TP); inbound tpid is bare inside TP{...}
|
||||||
# prefixes are required). The two regexes use different shapes —
|
# The two regexes use different shapes — round-trip via tpid only.
|
||||||
# round-trip via tpid only.
|
|
||||||
now = datetime(2026, 6, 20, 13, 22, 43, 505_000, tzinfo=MT)
|
now = datetime(2026, 6, 20, 13, 22, 43, 505_000, tzinfo=MT)
|
||||||
out = build_outbound_filename("11525703", "837P", now_mt=now)
|
out = build_outbound_filename("11525703", "837P", now_mt=now)
|
||||||
assert OUTBOUND_RE.match(out)
|
assert OUTBOUND_RE.match(out)
|
||||||
@@ -146,19 +142,13 @@ def test_roundtrip_outbound_to_inbound():
|
|||||||
|
|
||||||
|
|
||||||
def test_is_outbound_filename():
|
def test_is_outbound_filename():
|
||||||
# HCPF outbound always has the lowercase "tp" prefix
|
assert is_outbound_filename("11525703-837P-20260620132243505-1of1.x12")
|
||||||
assert is_outbound_filename("tp11525703-837P-20260620132243505-1of1.x12")
|
|
||||||
# Bare tpid (no tp prefix) is no longer a valid outbound filename
|
|
||||||
assert not is_outbound_filename("11525703-837P-20260620132243505-1of1.x12")
|
|
||||||
# Uppercase TP prefix is the inbound shape, not outbound
|
|
||||||
assert not is_outbound_filename("TP11525703-837P-20260620132243505-1of1.x12")
|
assert not is_outbound_filename("TP11525703-837P-20260620132243505-1of1.x12")
|
||||||
assert not is_outbound_filename("not-a-filename")
|
assert not is_outbound_filename("not-a-filename")
|
||||||
|
|
||||||
|
|
||||||
def test_is_inbound_filename():
|
def test_is_inbound_filename():
|
||||||
assert is_inbound_filename("TP11525703-837P_M019048402-20260520231513488-1of1_999.x12")
|
assert is_inbound_filename("TP11525703-837P_M019048402-20260520231513488-1of1_999.x12")
|
||||||
# Lowercase tp prefix is the outbound shape, not inbound
|
|
||||||
assert not is_inbound_filename("tp11525703-837P_M019048402-20260520231513488-1of1_999.x12")
|
|
||||||
assert not is_inbound_filename("11525703-837P-20260620132243505-1of1.x12")
|
assert not is_inbound_filename("11525703-837P-20260620132243505-1of1.x12")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,134 +0,0 @@
|
|||||||
"""Tests that ``CycloneStore.iter_claims`` populates ``receivedAmount``
|
|
||||||
from the matched ``Remittance.total_paid``.
|
|
||||||
|
|
||||||
Before SP_Auth follow-up: every claim came back with
|
|
||||||
``receivedAmount: 0.0`` regardless of whether it had been paired with
|
|
||||||
a paid remittance, so the Dashboard's "Received" KPI was always $0
|
|
||||||
even when claims had been paid and reconciled.
|
|
||||||
|
|
||||||
The fix: ``iter_claims`` bulk-loads ``Remittance.total_paid`` for every
|
|
||||||
matched claim id in the result set (single SQL, no N+1) and stamps
|
|
||||||
the sum onto each claim dict.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
from datetime import date, datetime, timezone
|
|
||||||
from decimal import Decimal
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from cyclone import db
|
|
||||||
from cyclone.db import ActivityEvent, Batch, Claim, ClaimState, Remittance
|
|
||||||
from cyclone.store import store as global_store
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _setup(tmp_path, monkeypatch):
|
|
||||||
monkeypatch.setenv("CYCLONE_DB_URL", f"sqlite:///{tmp_path}/test.db")
|
|
||||||
db._reset_for_tests()
|
|
||||||
db.init_db()
|
|
||||||
|
|
||||||
|
|
||||||
def _make_batch(s, batch_id: str) -> None:
|
|
||||||
s.add(Batch(
|
|
||||||
id=batch_id,
|
|
||||||
kind="837p",
|
|
||||||
input_filename="seed.edi",
|
|
||||||
parsed_at=datetime(2026, 6, 19, 12, 0, tzinfo=timezone.utc),
|
|
||||||
totals_json={"total_claims": 3},
|
|
||||||
validation_json={"passed": True, "warnings": [], "errors": []},
|
|
||||||
raw_result_json={"_": "stub"},
|
|
||||||
))
|
|
||||||
|
|
||||||
|
|
||||||
def _make_claim(s, claim_id: str, batch_id: str, *, matched_remit_id: str | None = None) -> None:
|
|
||||||
s.add(Claim(
|
|
||||||
id=claim_id,
|
|
||||||
batch_id=batch_id,
|
|
||||||
patient_control_number=claim_id,
|
|
||||||
service_date_from=date(2026, 6, 1),
|
|
||||||
service_date_to=date(2026, 6, 1),
|
|
||||||
charge_amount=Decimal("200.00"),
|
|
||||||
provider_npi="1234567890",
|
|
||||||
payer_id="SKCO0",
|
|
||||||
state=ClaimState.PAID,
|
|
||||||
matched_remittance_id=matched_remit_id,
|
|
||||||
))
|
|
||||||
|
|
||||||
|
|
||||||
def _make_remit(s, remit_id: str, batch_id: str, *, total_paid: Decimal) -> None:
|
|
||||||
s.add(Remittance(
|
|
||||||
id=remit_id,
|
|
||||||
batch_id=batch_id,
|
|
||||||
payer_claim_control_number=remit_id,
|
|
||||||
claim_id=None,
|
|
||||||
status_code="1",
|
|
||||||
total_charge=Decimal("200.00"),
|
|
||||||
total_paid=total_paid,
|
|
||||||
adjustment_amount=Decimal("0"),
|
|
||||||
received_at=datetime(2026, 6, 20, 12, 0, tzinfo=timezone.utc),
|
|
||||||
))
|
|
||||||
|
|
||||||
|
|
||||||
def test_iter_claims_populates_received_amount_from_matched_remittance():
|
|
||||||
"""A matched claim should reflect its remittance's ``total_paid``."""
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_batch(s, "b1")
|
|
||||||
_make_remit(s, "r1", "b1", total_paid=Decimal("180.00"))
|
|
||||||
_make_claim(s, "CLM-A", "b1", matched_remit_id="r1")
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
items = global_store.iter_claims(limit=10)
|
|
||||||
by_id = {c["id"]: c for c in items}
|
|
||||||
assert by_id["CLM-A"]["receivedAmount"] == pytest.approx(180.00)
|
|
||||||
|
|
||||||
|
|
||||||
def test_iter_claims_unmatched_claim_has_zero_received():
|
|
||||||
"""Claims with no matched remittance still get 0.0, not stale data."""
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_batch(s, "b1")
|
|
||||||
_make_claim(s, "CLM-U", "b1", matched_remit_id=None)
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
items = global_store.iter_claims(limit=10)
|
|
||||||
by_id = {c["id"]: c for c in items}
|
|
||||||
assert by_id["CLM-U"]["receivedAmount"] == 0.0
|
|
||||||
|
|
||||||
|
|
||||||
def test_iter_claims_handles_orphan_match_fk():
|
|
||||||
"""A claim with a stale ``matched_remittance_id`` whose remittance row
|
|
||||||
was deleted should default to 0.0 rather than blow up."""
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_batch(s, "b1")
|
|
||||||
_make_claim(s, "CLM-ORPHAN", "b1", matched_remit_id="r-deleted")
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
# No remittance row exists — the FK is dangling, which can happen
|
|
||||||
# if the remittance was deleted between match and now.
|
|
||||||
items = global_store.iter_claims(limit=10)
|
|
||||||
by_id = {c["id"]: c for c in items}
|
|
||||||
assert by_id["CLM-ORPHAN"]["receivedAmount"] == 0.0
|
|
||||||
|
|
||||||
|
|
||||||
def test_iter_claims_bulk_loads_multiple_matches_in_one_pass():
|
|
||||||
"""All matched claims in a page reflect their distinct remittance
|
|
||||||
totals — the bulk load must aggregate per remittance id."""
|
|
||||||
with db.SessionLocal()() as s:
|
|
||||||
_make_batch(s, "b1")
|
|
||||||
_make_remit(s, "r1", "b1", total_paid=Decimal("120.00"))
|
|
||||||
_make_remit(s, "r2", "b1", total_paid=Decimal("175.50"))
|
|
||||||
_make_remit(s, "r3", "b1", total_paid=Decimal("0"))
|
|
||||||
_make_claim(s, "CLM-1", "b1", matched_remit_id="r1")
|
|
||||||
_make_claim(s, "CLM-2", "b1", matched_remit_id="r2")
|
|
||||||
_make_claim(s, "CLM-3", "b1", matched_remit_id="r3")
|
|
||||||
_make_claim(s, "CLM-4", "b1", matched_remit_id=None)
|
|
||||||
s.commit()
|
|
||||||
|
|
||||||
items = global_store.iter_claims(limit=10)
|
|
||||||
by_id = {c["id"]: c for c in items}
|
|
||||||
assert by_id["CLM-1"]["receivedAmount"] == pytest.approx(120.00)
|
|
||||||
assert by_id["CLM-2"]["receivedAmount"] == pytest.approx(175.50)
|
|
||||||
assert by_id["CLM-3"]["receivedAmount"] == pytest.approx(0.0)
|
|
||||||
assert by_id["CLM-4"]["receivedAmount"] == 0.0
|
|
||||||
@@ -1,165 +0,0 @@
|
|||||||
"""SP18 — JsonFormatter + CycloneDevFormatter tests.
|
|
||||||
|
|
||||||
Covers the structural shape of log records, exception handling,
|
|
||||||
and the ``extra`` kwarg passthrough.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import io
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from cyclone.logging_config import (
|
|
||||||
CycloneDevFormatter,
|
|
||||||
JsonFormatter,
|
|
||||||
setup_logging,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _make_record(
|
|
||||||
msg: str = "hello",
|
|
||||||
args: tuple = (),
|
|
||||||
level: int = logging.INFO,
|
|
||||||
name: str = "test.logger",
|
|
||||||
extras: dict | None = None,
|
|
||||||
exc_info=None,
|
|
||||||
) -> logging.LogRecord:
|
|
||||||
record = logging.getLogger(name).makeRecord(
|
|
||||||
name=name,
|
|
||||||
level=level,
|
|
||||||
fn="t.py",
|
|
||||||
lno=1,
|
|
||||||
msg=msg,
|
|
||||||
args=args,
|
|
||||||
exc_info=exc_info,
|
|
||||||
)
|
|
||||||
if extras:
|
|
||||||
for k, v in extras.items():
|
|
||||||
setattr(record, k, v)
|
|
||||||
return record
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# JsonFormatter
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_json_formatter_basic_shape():
|
|
||||||
f = JsonFormatter()
|
|
||||||
line = f.format(_make_record(msg="hello %s", args=("cyclone",)))
|
|
||||||
parsed = json.loads(line)
|
|
||||||
assert parsed["level"] == "INFO"
|
|
||||||
assert parsed["logger"] == "test.logger"
|
|
||||||
assert parsed["msg"] == "hello cyclone"
|
|
||||||
assert "ts" in parsed
|
|
||||||
# ts must be ISO 8601 with milliseconds + Z suffix.
|
|
||||||
assert parsed["ts"].endswith("Z") or "+" in parsed["ts"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_json_formatter_includes_extras():
|
|
||||||
f = JsonFormatter()
|
|
||||||
line = f.format(_make_record(
|
|
||||||
msg="processed",
|
|
||||||
extras={"input_filename": "foo.x12", "parser_kind": "parse_999", "claims": 3},
|
|
||||||
))
|
|
||||||
parsed = json.loads(line)
|
|
||||||
assert parsed["extra"] == {
|
|
||||||
"input_filename": "foo.x12", "parser_kind": "parse_999", "claims": 3,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def test_json_formatter_handles_exception_info():
|
|
||||||
f = JsonFormatter()
|
|
||||||
try:
|
|
||||||
raise ValueError("boom")
|
|
||||||
except ValueError:
|
|
||||||
import sys
|
|
||||||
rec = _make_record(msg="oops", exc_info=sys.exc_info())
|
|
||||||
line = f.format(rec)
|
|
||||||
parsed = json.loads(line)
|
|
||||||
assert "traceback" in parsed
|
|
||||||
assert "ValueError: boom" in parsed["traceback"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_json_formatter_no_extras_key_when_none():
|
|
||||||
f = JsonFormatter()
|
|
||||||
line = f.format(_make_record(msg="plain"))
|
|
||||||
parsed = json.loads(line)
|
|
||||||
assert "extra" not in parsed
|
|
||||||
|
|
||||||
|
|
||||||
def test_json_formatter_handles_non_serializable_extras():
|
|
||||||
"""Non-JSON-serializable extras go through ``default=str``."""
|
|
||||||
class Opaque:
|
|
||||||
def __str__(self):
|
|
||||||
return "opaque-string"
|
|
||||||
f = JsonFormatter()
|
|
||||||
line = f.format(_make_record(msg="x", extras={"thing": Opaque()}))
|
|
||||||
parsed = json.loads(line)
|
|
||||||
assert parsed["extra"]["thing"] == "opaque-string"
|
|
||||||
|
|
||||||
|
|
||||||
def test_json_formatter_preserves_warning_level():
|
|
||||||
f = JsonFormatter()
|
|
||||||
line = f.format(_make_record(msg="careful", level=logging.WARNING))
|
|
||||||
parsed = json.loads(line)
|
|
||||||
assert parsed["level"] == "WARNING"
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# CycloneDevFormatter
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_dev_formatter_basic_shape():
|
|
||||||
f = CycloneDevFormatter()
|
|
||||||
line = f.format(_make_record(msg="hello %s", args=("cyclone",)))
|
|
||||||
assert "INFO" in line
|
|
||||||
assert "test.logger" in line
|
|
||||||
assert "hello cyclone" in line
|
|
||||||
|
|
||||||
|
|
||||||
def test_dev_formatter_includes_extras():
|
|
||||||
f = CycloneDevFormatter()
|
|
||||||
line = f.format(_make_record(
|
|
||||||
msg="processed",
|
|
||||||
extras={"input_filename": "foo.x12", "claims": 3},
|
|
||||||
))
|
|
||||||
assert "input_filename='foo.x12'" in line
|
|
||||||
assert "claims=3" in line
|
|
||||||
|
|
||||||
|
|
||||||
def test_dev_formatter_handles_exception():
|
|
||||||
f = CycloneDevFormatter()
|
|
||||||
try:
|
|
||||||
raise RuntimeError("nope")
|
|
||||||
except RuntimeError:
|
|
||||||
import sys
|
|
||||||
rec = _make_record(msg="oops", exc_info=sys.exc_info())
|
|
||||||
line = f.format(rec)
|
|
||||||
assert "RuntimeError: nope" in line
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# setup_logging (light — deeper coverage in test_logging_setup.py)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_setup_logging_attaches_handler_to_root():
|
|
||||||
setup_logging(level="DEBUG", json_format=True)
|
|
||||||
root = logging.getLogger()
|
|
||||||
assert len(root.handlers) >= 1
|
|
||||||
assert isinstance(root.handlers[0].formatter, JsonFormatter)
|
|
||||||
|
|
||||||
|
|
||||||
def test_setup_logging_is_idempotent():
|
|
||||||
"""Re-calling clears handlers; the formatter toggle takes effect."""
|
|
||||||
setup_logging(level="INFO", json_format=True)
|
|
||||||
setup_logging(level="INFO", json_format=False)
|
|
||||||
root = logging.getLogger()
|
|
||||||
assert len(root.handlers) == 1
|
|
||||||
assert isinstance(root.handlers[0].formatter, CycloneDevFormatter)
|
|
||||||
# Reset back to JSON for the rest of the test suite.
|
|
||||||
setup_logging(level="INFO", json_format=True)
|
|
||||||
@@ -1,157 +0,0 @@
|
|||||||
"""SP18 — PII scrubber tests.
|
|
||||||
|
|
||||||
Covers each PHI pattern, the false-positive guard, and the
|
|
||||||
disable toggle.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from cyclone.logging_config import (
|
|
||||||
PiiScrubber,
|
|
||||||
get_scrubber,
|
|
||||||
setup_logging,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _make_record(msg: str, extras: dict | None = None) -> logging.LogRecord:
|
|
||||||
record = logging.getLogger("test.scrub").makeRecord(
|
|
||||||
name="test.scrub", level=logging.INFO, fn="t.py", lno=1,
|
|
||||||
msg=msg, args=(), exc_info=None,
|
|
||||||
)
|
|
||||||
if extras:
|
|
||||||
for k, v in extras.items():
|
|
||||||
setattr(record, k, v)
|
|
||||||
return record
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _reset_scrubber():
|
|
||||||
"""Make sure the scrubber is enabled + on the root after each test."""
|
|
||||||
yield
|
|
||||||
get_scrubber().enable()
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# NPI
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_scrubs_ten_digit_npi_in_message():
|
|
||||||
scrubber = PiiScrubber()
|
|
||||||
rec = _make_record("processed claim with npi 1881068062 ok")
|
|
||||||
assert scrubber.filter(rec) is True
|
|
||||||
assert rec.getMessage() == "processed claim with npi <redacted:npi> ok"
|
|
||||||
|
|
||||||
|
|
||||||
def test_scrubs_npi_in_extras():
|
|
||||||
scrubber = PiiScrubber()
|
|
||||||
rec = _make_record("ok", extras={"provider_npi": "1881068062"})
|
|
||||||
scrubber.filter(rec)
|
|
||||||
assert rec.provider_npi == "<redacted:npi>"
|
|
||||||
|
|
||||||
|
|
||||||
def test_does_not_scrub_short_numbers():
|
|
||||||
scrubber = PiiScrubber()
|
|
||||||
rec = _make_record("processed 5 claims in 2 batches")
|
|
||||||
scrubber.filter(rec)
|
|
||||||
assert rec.getMessage() == "processed 5 claims in 2 batches"
|
|
||||||
|
|
||||||
|
|
||||||
def test_does_not_scrub_eleven_digit_numbers():
|
|
||||||
"""11+ digit numbers aren't NPIs — leave them alone."""
|
|
||||||
scrubber = PiiScrubber()
|
|
||||||
rec = _make_record("control number 12345678901")
|
|
||||||
scrubber.filter(rec)
|
|
||||||
assert rec.getMessage() == "control number 12345678901"
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# SSN
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_scrubs_dashed_ssn_in_message():
|
|
||||||
scrubber = PiiScrubber()
|
|
||||||
rec = _make_record("ssn=123-45-6789 detected")
|
|
||||||
scrubber.filter(rec)
|
|
||||||
assert rec.getMessage() == "ssn=<redacted:ssn> detected"
|
|
||||||
|
|
||||||
|
|
||||||
def test_scrubs_undashed_ssn_at_phrase_boundary():
|
|
||||||
"""A bare 9-digit number is ambiguous — we scrub it when followed
|
|
||||||
by whitespace/punctuation/closing paren/brace/comma (so we don't
|
|
||||||
hit claim control numbers or zip codes mid-sentence)."""
|
|
||||||
scrubber = PiiScrubber()
|
|
||||||
rec = _make_record("ssn 123456789 on file")
|
|
||||||
scrubber.filter(rec)
|
|
||||||
assert "<redacted:ssn>" in rec.getMessage()
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# DOB
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_scrubs_dob_field():
|
|
||||||
scrubber = PiiScrubber()
|
|
||||||
rec = _make_record("dob=1980-04-12 verified")
|
|
||||||
scrubber.filter(rec)
|
|
||||||
assert rec.getMessage() == "dob=<redacted:dob> verified"
|
|
||||||
|
|
||||||
|
|
||||||
def test_scrubs_dob_in_extras():
|
|
||||||
scrubber = PiiScrubber()
|
|
||||||
rec = _make_record("ok", extras={"date_of_birth": "1980-04-12"})
|
|
||||||
scrubber.filter(rec)
|
|
||||||
assert rec.date_of_birth == "<redacted:dob>"
|
|
||||||
|
|
||||||
|
|
||||||
def test_does_not_scrub_bare_iso_date():
|
|
||||||
"""A YYYY-MM-DD without a dob= prefix isn't necessarily PHI."""
|
|
||||||
scrubber = PiiScrubber()
|
|
||||||
rec = _make_record("parsed at 2026-06-21")
|
|
||||||
scrubber.filter(rec)
|
|
||||||
assert rec.getMessage() == "parsed at 2026-06-21"
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Patient name
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_scrubs_patient_name_field():
|
|
||||||
scrubber = PiiScrubber()
|
|
||||||
rec = _make_record('patient_name="John Doe" verified')
|
|
||||||
scrubber.filter(rec)
|
|
||||||
assert "John Doe" not in rec.getMessage()
|
|
||||||
assert "<redacted:patient_name>" in rec.getMessage()
|
|
||||||
|
|
||||||
|
|
||||||
def test_does_not_scrub_random_words():
|
|
||||||
scrubber = PiiScrubber()
|
|
||||||
rec = _make_record("the parser ran successfully")
|
|
||||||
scrubber.filter(rec)
|
|
||||||
assert rec.getMessage() == "the parser ran successfully"
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Disable toggle
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_scrubber_disabled_leaves_message_intact():
|
|
||||||
scrubber = PiiScrubber()
|
|
||||||
scrubber.disable()
|
|
||||||
rec = _make_record("npi 1881068062 ok")
|
|
||||||
scrubber.filter(rec)
|
|
||||||
assert rec.getMessage() == "npi 1881068062 ok"
|
|
||||||
|
|
||||||
|
|
||||||
def test_scrubber_does_not_crash_on_unusual_records():
|
|
||||||
"""Even with weird attribute combinations the filter returns True."""
|
|
||||||
scrubber = PiiScrubber()
|
|
||||||
rec = _make_record("ok", extras={"weird": object()})
|
|
||||||
assert scrubber.filter(rec) is True
|
|
||||||
@@ -1,127 +0,0 @@
|
|||||||
"""SP18 — ``setup_logging`` entry-point tests.
|
|
||||||
|
|
||||||
Covers level resolution, handler attachment, env-var overrides, and
|
|
||||||
the idempotent re-setup behavior used by the FastAPI lifespan and
|
|
||||||
the CLI's ``main()``.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import logging
|
|
||||||
import os
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from cyclone.logging_config import (
|
|
||||||
CycloneDevFormatter,
|
|
||||||
JsonFormatter,
|
|
||||||
PiiScrubber,
|
|
||||||
setup_logging,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _reset_root_logger():
|
|
||||||
"""Strip our handlers + filters before each test so setup runs clean."""
|
|
||||||
root = logging.getLogger()
|
|
||||||
for h in list(root.handlers):
|
|
||||||
root.removeHandler(h)
|
|
||||||
for flt in list(root.filters):
|
|
||||||
if isinstance(flt, PiiScrubber):
|
|
||||||
root.removeFilter(flt)
|
|
||||||
yield
|
|
||||||
for h in list(root.handlers):
|
|
||||||
root.removeHandler(h)
|
|
||||||
for flt in list(root.filters):
|
|
||||||
if isinstance(flt, PiiScrubber):
|
|
||||||
root.removeFilter(flt)
|
|
||||||
|
|
||||||
|
|
||||||
def test_setup_respects_level_string():
|
|
||||||
setup_logging(level="DEBUG", json_format=True)
|
|
||||||
assert logging.getLogger().level == logging.DEBUG
|
|
||||||
setup_logging(level="WARNING", json_format=True)
|
|
||||||
assert logging.getLogger().level == logging.WARNING
|
|
||||||
|
|
||||||
|
|
||||||
def test_setup_attaches_rotating_file_handler(tmp_path: Path):
|
|
||||||
log_file = tmp_path / "cyclone.log"
|
|
||||||
setup_logging(level="INFO", log_file=str(log_file), json_format=True)
|
|
||||||
root = logging.getLogger()
|
|
||||||
assert len(root.handlers) == 1
|
|
||||||
h = root.handlers[0]
|
|
||||||
# RotatingFileHandler has ``baseFilename`` attr.
|
|
||||||
assert hasattr(h, "baseFilename")
|
|
||||||
assert Path(h.baseFilename).name == "cyclone.log"
|
|
||||||
|
|
||||||
|
|
||||||
def test_setup_defaults_to_json_formatter():
|
|
||||||
setup_logging(level="INFO")
|
|
||||||
root = logging.getLogger()
|
|
||||||
assert isinstance(root.handlers[0].formatter, JsonFormatter)
|
|
||||||
|
|
||||||
|
|
||||||
def test_setup_dev_toggle_uses_dev_formatter():
|
|
||||||
setup_logging(level="INFO", json_format=False)
|
|
||||||
root = logging.getLogger()
|
|
||||||
assert isinstance(root.handlers[0].formatter, CycloneDevFormatter)
|
|
||||||
|
|
||||||
|
|
||||||
def test_setup_idempotent_re_setup_replaces_handlers():
|
|
||||||
"""Re-calling setup_logging clears the previous handler(s)."""
|
|
||||||
setup_logging(level="INFO", json_format=True)
|
|
||||||
setup_logging(level="DEBUG", json_format=False)
|
|
||||||
root = logging.getLogger()
|
|
||||||
assert len(root.handlers) == 1
|
|
||||||
assert isinstance(root.handlers[0].formatter, CycloneDevFormatter)
|
|
||||||
assert root.level == logging.DEBUG
|
|
||||||
|
|
||||||
|
|
||||||
def test_setup_quietens_noisy_third_party_loggers():
|
|
||||||
setup_logging(level="DEBUG", json_format=True)
|
|
||||||
for noisy in ("urllib3", "paramiko", "sqlalchemy.engine"):
|
|
||||||
assert logging.getLogger(noisy).level >= logging.WARNING
|
|
||||||
|
|
||||||
|
|
||||||
def test_setup_attaches_pii_scrubber_by_default():
|
|
||||||
setup_logging(level="INFO", json_format=True)
|
|
||||||
root = logging.getLogger()
|
|
||||||
assert any(isinstance(f, PiiScrubber) for f in root.filters)
|
|
||||||
|
|
||||||
|
|
||||||
def test_setup_honors_scrub_pii_false():
|
|
||||||
setup_logging(level="INFO", json_format=True, scrub_pii=False)
|
|
||||||
root = logging.getLogger()
|
|
||||||
# Scrubber is still attached but disabled.
|
|
||||||
scrubbers = [f for f in root.filters if isinstance(f, PiiScrubber)]
|
|
||||||
assert len(scrubbers) == 1
|
|
||||||
assert scrubbers[0]._enabled is False # noqa: SLF001
|
|
||||||
|
|
||||||
|
|
||||||
def test_setup_honors_env_var_no_pii_scrub(monkeypatch):
|
|
||||||
monkeypatch.setenv("CYCLONE_LOG_NO_PII_SCRUB", "1")
|
|
||||||
setup_logging(level="INFO", json_format=True)
|
|
||||||
scrubbers = [f for f in logging.getLogger().filters if isinstance(f, PiiScrubber)]
|
|
||||||
assert scrubbers and scrubbers[0]._enabled is False # noqa: SLF001
|
|
||||||
|
|
||||||
|
|
||||||
def test_setup_emits_json_to_stderr_by_default(caplog):
|
|
||||||
"""Records emitted after setup flow through JsonFormatter."""
|
|
||||||
setup_logging(level="INFO", json_format=True)
|
|
||||||
logger = logging.getLogger("cyclone.test_setup")
|
|
||||||
logger.info("hello %s", "world", extra={"x": 1})
|
|
||||||
# Cyclone attaches the handler to root, not the named logger; caplog
|
|
||||||
# won't capture unless we propagate (which is the default).
|
|
||||||
# Just assert the handler is on root and would format correctly.
|
|
||||||
root = logging.getLogger()
|
|
||||||
h = root.handlers[0]
|
|
||||||
record = logger.makeRecord(
|
|
||||||
name="cyclone.test_setup", level=logging.INFO, fn="t.py", lno=1,
|
|
||||||
msg="hello %s", args=("world",), exc_info=None,
|
|
||||||
)
|
|
||||||
record.x = 1
|
|
||||||
formatted = h.formatter.format(record)
|
|
||||||
import json as _json
|
|
||||||
parsed = _json.loads(formatted)
|
|
||||||
assert parsed["msg"] == "hello world"
|
|
||||||
assert parsed["extra"]["x"] == 1
|
|
||||||
@@ -1,199 +0,0 @@
|
|||||||
"""Tests for ``cyclone.npi`` — NPI checksum + Tax ID (EIN) format validation.
|
|
||||||
|
|
||||||
Pure local validation, no NPPES calls. Covers the NPPES-published Luhn
|
|
||||||
checksum (with prefix ``80840``) and a small set of obvious EIN typo cases.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from cyclone.npi import (
|
|
||||||
_luhn_check_digit,
|
|
||||||
is_valid_npi,
|
|
||||||
is_valid_tax_id,
|
|
||||||
npi_checksum,
|
|
||||||
normalize_tax_id,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Luhn internals
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_luhn_check_digit_known_sequence():
|
|
||||||
"""Standard Luhn for the empty body returns 0.
|
|
||||||
|
|
||||||
With no digits the sum is 0, so (10 - 0 % 10) % 10 = 0.
|
|
||||||
"""
|
|
||||||
assert _luhn_check_digit("") == 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_luhn_check_digit_single_digit():
|
|
||||||
"""For a single body digit the check digit is the standard Luhn value.
|
|
||||||
|
|
||||||
With the corrected "double at rightmost" pattern: "0" doubles to 0
|
|
||||||
(total=0, check=0); "1" doubles to 2 (total=2, check=(10-2)%10=8).
|
|
||||||
"""
|
|
||||||
assert _luhn_check_digit("0") == 0
|
|
||||||
assert _luhn_check_digit("1") == 8
|
|
||||||
|
|
||||||
|
|
||||||
def test_luhn_check_digit_nppes_published():
|
|
||||||
"""CMS-published example: body 123456789 → check digit 3.
|
|
||||||
|
|
||||||
Per https://www.cms.gov/.../NPIcheckdigit.pdf the Luhn sum of the
|
|
||||||
prefixed body ``80840123456789`` is 67, so check digit = (10-7)%10 = 3,
|
|
||||||
giving the full NPI ``1234567893``.
|
|
||||||
"""
|
|
||||||
assert _luhn_check_digit("80840123456789") == 3
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# npi_checksum
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_npi_checksum_cms_published_body():
|
|
||||||
"""For NPI body 123456789 the check digit is 3 → NPI 1234567893.
|
|
||||||
|
|
||||||
Confirmed against the CMS-published NPI Luhn example.
|
|
||||||
"""
|
|
||||||
assert npi_checksum("123456789") == 3
|
|
||||||
|
|
||||||
|
|
||||||
def test_npi_checksum_rejects_non_digits():
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
npi_checksum("12345abc6") # letter in body
|
|
||||||
|
|
||||||
|
|
||||||
def test_npi_checksum_rejects_wrong_length():
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
npi_checksum("12345") # only 5 digits
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
npi_checksum("1234567890") # 10 digits — includes the check digit
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# is_valid_npi
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_npi_nppes_sample_is_true():
|
|
||||||
"""The CMS-published example NPI 1234567893 must validate."""
|
|
||||||
assert is_valid_npi("1234567893") is True
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_npi_off_by_one_is_false():
|
|
||||||
assert is_valid_npi("1234567894") is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_npi_all_zeros_is_false():
|
|
||||||
"""All zeros fails the Luhn check."""
|
|
||||||
assert is_valid_npi("0000000000") is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_npi_rejects_empty():
|
|
||||||
assert is_valid_npi("") is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_npi_rejects_none():
|
|
||||||
assert is_valid_npi(None) is False # type: ignore[arg-type]
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_npi_rejects_non_string():
|
|
||||||
assert is_valid_npi(1234567890) is False # type: ignore[arg-type]
|
|
||||||
assert is_valid_npi(["1881068062"]) is False # type: ignore[arg-type]
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_npi_rejects_short():
|
|
||||||
assert is_valid_npi("123456789") is False # 9 digits
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_npi_rejects_long():
|
|
||||||
assert is_valid_npi("12345678901") is False # 11 digits
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_npi_rejects_non_digits():
|
|
||||||
assert is_valid_npi("188106806X") is False
|
|
||||||
assert is_valid_npi("18810 68062") is False # space
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_npi_all_ones_fails_luhn():
|
|
||||||
"""1111111111 has all-1 sum: alternating double = 1,2,1,2,..., 1+2=3 then
|
|
||||||
collapse. Total for 10 digits body (body=9 of all 1's, sum doubled):
|
|
||||||
Position from right (i=0..8): 1, 1, 1, 1, 1, 1, 1, 1, 1
|
|
||||||
i even (not doubled): 1, 1, 1, 1, 1 → 5
|
|
||||||
i odd (doubled): 1*2=2, 1*2=2, 1*2=2, 1*2=2 → 8
|
|
||||||
Total body = 13. Body alone has check digit (10 - 13 % 10) % 10 = 7.
|
|
||||||
So full NPI 1111111111's body (9 ones) check = 7, and 1 != 7 → invalid.
|
|
||||||
"""
|
|
||||||
assert is_valid_npi("1111111111") is False
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# is_valid_tax_id / normalize_tax_id
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_tax_id_touch_of_care_true():
|
|
||||||
"""The operator's reference EIN — Touch of Care Family Practice."""
|
|
||||||
assert is_valid_tax_id("72-1587149") is True
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_tax_id_unformatted_true():
|
|
||||||
assert is_valid_tax_id("721587149") is True
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_tax_id_00_prefix_rejected():
|
|
||||||
"""``00`` is reserved / never assigned by the IRS."""
|
|
||||||
assert is_valid_tax_id("00-1234567") is False
|
|
||||||
assert is_valid_tax_id("001234567") is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_tax_id_07_prefix_rejected():
|
|
||||||
"""``07`` is a campus prefix reserved for future use."""
|
|
||||||
assert is_valid_tax_id("07-1234567") is False
|
|
||||||
assert is_valid_tax_id("071234567") is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_tax_id_8x_prefix_rejected():
|
|
||||||
"""``80``–``89`` is the IRS Pension Plan Branch — never assigned otherwise."""
|
|
||||||
assert is_valid_tax_id("80-1234567") is False
|
|
||||||
assert is_valid_tax_id("89-1234567") is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_tax_id_rejects_malformed():
|
|
||||||
assert is_valid_tax_id("not-an-ein") is False
|
|
||||||
assert is_valid_tax_id("12345") is False
|
|
||||||
assert is_valid_tax_id("1234567890") is False # 10 digits
|
|
||||||
assert is_valid_tax_id("12-345678") is False # 8 digits after hyphen
|
|
||||||
|
|
||||||
|
|
||||||
def test_is_valid_tax_id_rejects_none_and_non_string():
|
|
||||||
assert is_valid_tax_id(None) is False
|
|
||||||
assert is_valid_tax_id(721587149) is False # type: ignore[arg-type]
|
|
||||||
|
|
||||||
|
|
||||||
def test_normalize_tax_id_returns_plain_form():
|
|
||||||
assert normalize_tax_id("72-1587149") == "721587149"
|
|
||||||
assert normalize_tax_id("721587149") == "721587149"
|
|
||||||
|
|
||||||
|
|
||||||
def test_normalize_tax_id_strips_whitespace():
|
|
||||||
assert normalize_tax_id(" 72-1587149 ") == "721587149"
|
|
||||||
|
|
||||||
|
|
||||||
def test_normalize_tax_id_returns_none_for_invalid():
|
|
||||||
assert normalize_tax_id("not-an-ein") is None
|
|
||||||
assert normalize_tax_id(None) is None
|
|
||||||
assert normalize_tax_id("") is None
|
|
||||||
assert normalize_tax_id("12-345678") is None # wrong digit count
|
|
||||||
|
|
||||||
|
|
||||||
def test_normalize_tax_id_keeps_reserved_prefix():
|
|
||||||
"""normalize_tax_id is a *structural* normalizer — it doesn't reject
|
|
||||||
reserved prefixes. That's ``is_valid_tax_id``'s job. Operators who
|
|
||||||
want to store 00-prefixed EINs as placeholders still get a clean
|
|
||||||
9-digit string."""
|
|
||||||
assert normalize_tax_id("00-1234567") == "001234567"
|
|
||||||
@@ -14,7 +14,7 @@ def test_parse_minimal_fixture_returns_one_claim():
|
|||||||
assert len(result.claims) == 1
|
assert len(result.claims) == 1
|
||||||
claim = result.claims[0]
|
claim = result.claims[0]
|
||||||
assert claim.claim_id == "CLM001"
|
assert claim.claim_id == "CLM001"
|
||||||
assert claim.billing_provider.npi == "1993999998"
|
assert claim.billing_provider.npi == "1234567890"
|
||||||
assert claim.subscriber.last_name == "Doe"
|
assert claim.subscriber.last_name == "Doe"
|
||||||
assert claim.subscriber.first_name == "John"
|
assert claim.subscriber.first_name == "John"
|
||||||
assert claim.subscriber.member_id == "ABC123"
|
assert claim.subscriber.member_id == "ABC123"
|
||||||
|
|||||||
@@ -1,106 +0,0 @@
|
|||||||
"""Tests for GET /api/payers/{payer_id}/summary (SP21 Task 1.5).
|
|
||||||
|
|
||||||
The endpoint is the payer-level aggregate that the drill-down UI's
|
|
||||||
"Payer → Claims" panel hangs off. It returns billed/received totals,
|
|
||||||
denial rate, and the top 5 NPIs by claim volume for one payer_id,
|
|
||||||
cached in-process for 60s.
|
|
||||||
|
|
||||||
The minimal 837P fixture ships one CLM with ``payer_id="SKCO0"``,
|
|
||||||
charge_amount=100.00; the minimal 835 carries one CLP for the same
|
|
||||||
claim with total_paid=85.00. So ``/api/payers/SKCO0/summary`` returns
|
|
||||||
``claim_count >= 1`` after both files are ingested.
|
|
||||||
|
|
||||||
Note: the spec calls this ``payer_id`` (the X12 NM1*PR*PI qualifier,
|
|
||||||
e.g. ``SKCO0``). It is NOT the configured payer name from
|
|
||||||
``config/payers.yaml``. The filter key in the store layer is
|
|
||||||
``Claim.payer_id`` — not the ``payer=`` substring filter used by
|
|
||||||
``/api/claims?payer=...``.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
|
|
||||||
from cyclone import api as api_mod
|
|
||||||
from cyclone.api import app
|
|
||||||
|
|
||||||
FIXTURE_837 = Path(__file__).parent / "fixtures" / "minimal_837p.txt"
|
|
||||||
FIXTURE_835 = Path(__file__).parent / "fixtures" / "minimal_835.txt"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(autouse=True)
|
|
||||||
def _clear_summary_cache():
|
|
||||||
"""Wipe the in-process payer-summary cache between tests.
|
|
||||||
|
|
||||||
conftest resets the DB per test but the cache is module-level
|
|
||||||
state on ``cyclone.api``. Without this clear, a stale payload
|
|
||||||
from a previous test's seed would leak into a later test's first
|
|
||||||
call — masking recompute behavior. The 60s TTL is the only
|
|
||||||
invalidation story today (see api.py docstring on the endpoint).
|
|
||||||
"""
|
|
||||||
api_mod._clear_summary_cache()
|
|
||||||
yield
|
|
||||||
api_mod._clear_summary_cache()
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def client() -> TestClient:
|
|
||||||
return TestClient(app)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def seeded_db(client: TestClient):
|
|
||||||
"""Ingest one minimal 837P + one minimal 835.
|
|
||||||
|
|
||||||
Both fixtures carry ``payer_id="SKCO0"`` so the summary endpoint
|
|
||||||
has something to aggregate. ``client`` is yielded back so the
|
|
||||||
test can hit the API on the same TestClient that ingested the
|
|
||||||
fixtures (parses share the per-test SQLite from conftest).
|
|
||||||
"""
|
|
||||||
text_837 = FIXTURE_837.read_text()
|
|
||||||
text_835 = FIXTURE_835.read_text()
|
|
||||||
r837 = client.post(
|
|
||||||
"/api/parse-837",
|
|
||||||
files={"file": ("x.txt", text_837, "text/plain")},
|
|
||||||
headers={"Accept": "application/json"},
|
|
||||||
)
|
|
||||||
assert r837.status_code == 200, r837.text
|
|
||||||
r835 = client.post(
|
|
||||||
"/api/parse-835",
|
|
||||||
files={"file": ("era.txt", text_835, "text/plain")},
|
|
||||||
headers={"Accept": "application/json"},
|
|
||||||
)
|
|
||||||
assert r835.status_code == 200, r835.text
|
|
||||||
return client
|
|
||||||
|
|
||||||
|
|
||||||
def test_payer_summary_happy_path(seeded_db: TestClient):
|
|
||||||
"""Seeded db has at least one claim for SKCO0 → 200 with the spec shape."""
|
|
||||||
resp = seeded_db.get("/api/payers/SKCO0/summary")
|
|
||||||
assert resp.status_code == 200, resp.text
|
|
||||||
data = resp.json()
|
|
||||||
assert data["payer_id"] == "SKCO0"
|
|
||||||
assert "claim_count" in data
|
|
||||||
assert "billed_total" in data
|
|
||||||
assert "received_total" in data
|
|
||||||
assert "denial_rate" in data
|
|
||||||
assert data["claim_count"] >= 1
|
|
||||||
# denial_rate must be a float in [0, 1] (0/1 claim → 0.0).
|
|
||||||
assert isinstance(data["denial_rate"], (int, float))
|
|
||||||
assert 0.0 <= data["denial_rate"] <= 1.0
|
|
||||||
|
|
||||||
|
|
||||||
def test_payer_summary_unknown_payer_returns_404(client: TestClient):
|
|
||||||
resp = client.get("/api/payers/DOES_NOT_EXIST/summary")
|
|
||||||
assert resp.status_code == 404
|
|
||||||
|
|
||||||
|
|
||||||
def test_payer_summary_caches_then_invalidates(seeded_db: TestClient):
|
|
||||||
"""Two back-to-back calls return identical payloads (in-process cache)."""
|
|
||||||
resp1 = seeded_db.get("/api/payers/SKCO0/summary")
|
|
||||||
resp2 = seeded_db.get("/api/payers/SKCO0/summary")
|
|
||||||
assert resp1.status_code == 200
|
|
||||||
assert resp2.status_code == 200
|
|
||||||
assert resp1.json() == resp2.json()
|
|
||||||
@@ -1,177 +0,0 @@
|
|||||||
"""Tests for the extended GET /api/config/providers/{npi} (SP21 Task 1.6).
|
|
||||||
|
|
||||||
The endpoint gains two new top-level arrays for the drill-down panel:
|
|
||||||
``recent_claims`` (top 10 by submission date desc) and ``recent_activity``
|
|
||||||
(top 10 by ``ts`` desc, joined to claims by ``claim_id`` because
|
|
||||||
``ActivityEvent`` has no direct ``provider_npi`` column).
|
|
||||||
|
|
||||||
Existing SP9 fields (``label``, ``legal_name``, ``tax_id``,
|
|
||||||
``address_line1``, ``city``, ``state``, ``zip``, etc.) must remain
|
|
||||||
present — the new arrays are additive only.
|
|
||||||
|
|
||||||
The fixtures in ``fixtures/minimal_837p.txt`` and ``fixtures/minimal_835.txt``
|
|
||||||
pair up to a single claim with ``provider_npi='1993999998'``. That NPI is
|
|
||||||
NOT in the seeded provider set (Montrose/Delta/Salida → 1881068062/1851446637/
|
|
||||||
1467507269). For the tests we hit the seeded Montrose NPI, which is the
|
|
||||||
canonical SP9 fixture NPI. The arrays come back as empty lists — the
|
|
||||||
contract under test is the *shape* (array, ≤10) and the *backwards compat*
|
|
||||||
of the existing fields; the data-path itself is exercised by the existing
|
|
||||||
ingestion path that backs ``/api/claims``.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
|
|
||||||
from cyclone.api import app
|
|
||||||
from cyclone.providers import Provider
|
|
||||||
from cyclone.store import store
|
|
||||||
|
|
||||||
FIXTURE_837 = Path(__file__).parent / "fixtures" / "minimal_837p.txt"
|
|
||||||
FIXTURE_835 = Path(__file__).parent / "fixtures" / "minimal_835.txt"
|
|
||||||
|
|
||||||
# Montrose — one of the three providers that `ensure_clearhouse_seeded()`
|
|
||||||
# writes into the providers table. Using a seeded NPI means the endpoint
|
|
||||||
# won't 404; the seeded claims (provider_npi='1993999998') won't appear
|
|
||||||
# under this NPI, so recent_claims/activity are expected empty lists.
|
|
||||||
MONTROSE_NPI = "1881068062"
|
|
||||||
|
|
||||||
# NPI the minimal 837P fixture bills under. NOT in the default seed —
|
|
||||||
# registering it before ingest is required to pass R204 (NPI must exist
|
|
||||||
# in the providers table).
|
|
||||||
TEST_837_NPI = "1993999998"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def client() -> TestClient:
|
|
||||||
return TestClient(app)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def seeded_db(client: TestClient):
|
|
||||||
"""Seed the clearhouse + ingest the minimal 837P/835 fixtures.
|
|
||||||
|
|
||||||
Mirrors the Task 1.5 ``seeded_db`` pattern: seed → ingest → hand the
|
|
||||||
client back so the test hits the same TestClient.
|
|
||||||
|
|
||||||
The 837 fixture bills under ``TEST_837_NPI``; without first registering
|
|
||||||
that provider the parser's R204 rule rejects the claim with HTTP 422.
|
|
||||||
"""
|
|
||||||
store.ensure_clearhouse_seeded()
|
|
||||||
test_provider = Provider(
|
|
||||||
npi=TEST_837_NPI,
|
|
||||||
label="Test Provider",
|
|
||||||
legal_name="Test Provider Inc",
|
|
||||||
tax_id="123456789",
|
|
||||||
taxonomy_code="207R00000X",
|
|
||||||
address_line1="123 Test St",
|
|
||||||
city="Denver",
|
|
||||||
state="CO",
|
|
||||||
zip="80202",
|
|
||||||
created_at=datetime.now(timezone.utc),
|
|
||||||
updated_at=datetime.now(timezone.utc),
|
|
||||||
)
|
|
||||||
store.upsert_provider(test_provider)
|
|
||||||
text_837 = FIXTURE_837.read_text()
|
|
||||||
text_835 = FIXTURE_835.read_text()
|
|
||||||
r837 = client.post(
|
|
||||||
"/api/parse-837",
|
|
||||||
files={"file": ("x.txt", text_837, "text/plain")},
|
|
||||||
headers={"Accept": "application/json"},
|
|
||||||
)
|
|
||||||
assert r837.status_code == 200, r837.text
|
|
||||||
r835 = client.post(
|
|
||||||
"/api/parse-835",
|
|
||||||
files={"file": ("era.txt", text_835, "text/plain")},
|
|
||||||
headers={"Accept": "application/json"},
|
|
||||||
)
|
|
||||||
assert r835.status_code == 200, r835.text
|
|
||||||
return client
|
|
||||||
|
|
||||||
|
|
||||||
def test_provider_detail_includes_recent_claims(seeded_db: TestClient):
|
|
||||||
"""The extended response gains a recent_claims array (top 10)."""
|
|
||||||
resp = seeded_db.get(f"/api/config/providers/{MONTROSE_NPI}")
|
|
||||||
assert resp.status_code == 200, resp.text
|
|
||||||
data = resp.json()
|
|
||||||
assert "recent_claims" in data
|
|
||||||
assert isinstance(data["recent_claims"], list)
|
|
||||||
assert len(data["recent_claims"]) <= 10
|
|
||||||
|
|
||||||
|
|
||||||
def test_provider_detail_includes_recent_activity(seeded_db: TestClient):
|
|
||||||
"""The extended response gains a recent_activity array (top 10)."""
|
|
||||||
resp = seeded_db.get(f"/api/config/providers/{MONTROSE_NPI}")
|
|
||||||
assert resp.status_code == 200, resp.text
|
|
||||||
data = resp.json()
|
|
||||||
assert "recent_activity" in data
|
|
||||||
assert isinstance(data["recent_activity"], list)
|
|
||||||
assert len(data["recent_activity"]) <= 10
|
|
||||||
|
|
||||||
|
|
||||||
def test_provider_detail_backwards_compat(seeded_db: TestClient):
|
|
||||||
"""All SP9 fields still present; new arrays don't break the contract.
|
|
||||||
|
|
||||||
The Provider Pydantic model (backend/src/cyclone/providers.py)
|
|
||||||
serializes snake_case fields — that's what the wire carries. The
|
|
||||||
TS ``Provider`` interface in ``src/types/index.ts`` is the
|
|
||||||
in-memory sample shape and intentionally diverges; the contract
|
|
||||||
being verified here is the API's actual payload.
|
|
||||||
"""
|
|
||||||
resp = seeded_db.get(f"/api/config/providers/{MONTROSE_NPI}")
|
|
||||||
assert resp.status_code == 200, resp.text
|
|
||||||
data = resp.json()
|
|
||||||
for key in (
|
|
||||||
"npi",
|
|
||||||
"label",
|
|
||||||
"legal_name",
|
|
||||||
"tax_id",
|
|
||||||
"taxonomy_code",
|
|
||||||
"address_line1",
|
|
||||||
"city",
|
|
||||||
"state",
|
|
||||||
"zip",
|
|
||||||
"is_active",
|
|
||||||
):
|
|
||||||
assert key in data, f"missing field {key}"
|
|
||||||
|
|
||||||
|
|
||||||
def test_provider_detail_includes_orphan_remit_received(seeded_db: TestClient):
|
|
||||||
"""Regression: the ``remit_received`` ActivityEvent is recorded at 835
|
|
||||||
ingest with ``claim_id=None`` (``store.add`` lines 999-1003) — the
|
|
||||||
remittance hasn't been matched to a claim yet. The original
|
|
||||||
``ActivityEvent.claim_id IN (claim_ids)`` filter misses it because
|
|
||||||
the orphan's claim_id is NULL.
|
|
||||||
|
|
||||||
Once reconciliation (auto or manual) populates
|
|
||||||
``Remittance.claim_id``, the activity filter must surface the event
|
|
||||||
via the ``Remittance.claim_id IN (claim_ids)`` branch of the OR.
|
|
||||||
Without that branch, a provider's activity feed appears to freeze
|
|
||||||
the moment an 835 lands — the most common activity, invisible.
|
|
||||||
|
|
||||||
Setup: ingest 837+835 for ``TEST_837_NPI`` (claim CLM001 + remit
|
|
||||||
CLM001), then manually match them so ``Remittance.claim_id`` is
|
|
||||||
populated. The bug presents as: only ``claim_submitted`` and
|
|
||||||
``manual_match`` appear (no ``remit_received``). The fix surfaces
|
|
||||||
all three.
|
|
||||||
"""
|
|
||||||
# Force the match — simulates the post-reconciliation state that
|
|
||||||
# populate Remittance.claim_id without depending on auto-reconcile
|
|
||||||
# heuristics (which don't match this minimal fixture).
|
|
||||||
match_resp = seeded_db.post(
|
|
||||||
"/api/reconciliation/match",
|
|
||||||
json={"claim_id": "CLM001", "remit_id": "CLM001"},
|
|
||||||
)
|
|
||||||
assert match_resp.status_code == 200, match_resp.text
|
|
||||||
|
|
||||||
resp = seeded_db.get(f"/api/config/providers/{TEST_837_NPI}")
|
|
||||||
assert resp.status_code == 200, resp.text
|
|
||||||
data = resp.json()
|
|
||||||
kinds = {event["kind"] for event in data["recent_activity"]}
|
|
||||||
assert "remit_received" in kinds, (
|
|
||||||
f"expected remit_received in recent_activity (orphan remits "
|
|
||||||
f"must surface via the Remittance join), got kinds={sorted(kinds)}"
|
|
||||||
)
|
|
||||||
@@ -1,287 +0,0 @@
|
|||||||
"""SP16 — Inbound MFT polling scheduler tests.
|
|
||||||
|
|
||||||
We test the Scheduler class with a fake ``SftpClient`` factory that
|
|
||||||
returns files we drop on disk (the SFTP stub already does this; we
|
|
||||||
just need to control which files appear between ticks). The handlers
|
|
||||||
themselves (999/835/277CA/TA1) are exercised through real parsers
|
|
||||||
using the fixtures in ``tests/fixtures/``.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Iterable
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from cyclone import db, scheduler as sched_mod
|
|
||||||
from cyclone.db import ProcessedInboundFile
|
|
||||||
from cyclone.providers import SftpBlock
|
|
||||||
from cyclone.scheduler import (
|
|
||||||
HANDLERS,
|
|
||||||
ROUTED_FILE_TYPES,
|
|
||||||
Scheduler,
|
|
||||||
STATUS_ERROR,
|
|
||||||
STATUS_OK,
|
|
||||||
STATUS_SKIPPED,
|
|
||||||
TickResult,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ----- fixtures -----------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def sftp_block(tmp_path):
|
|
||||||
staging = tmp_path / "staging"
|
|
||||||
inbound_dir = staging / "ToHPE"
|
|
||||||
inbound_dir.mkdir(parents=True)
|
|
||||||
return SftpBlock(
|
|
||||||
host="mft.example.com",
|
|
||||||
port=22,
|
|
||||||
username="test",
|
|
||||||
paths={
|
|
||||||
"outbound": "/FromHPE",
|
|
||||||
"inbound": "/ToHPE",
|
|
||||||
},
|
|
||||||
stub=True,
|
|
||||||
staging_dir=str(staging),
|
|
||||||
poll_seconds=60,
|
|
||||||
auth={"method": "keychain", "secret_ref": "test.password"},
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
|
||||||
def _drop_file(sftp_block):
|
|
||||||
"""Helper: drop a named file in the inbound dir. Returns the path."""
|
|
||||||
inbound_dir = Path(sftp_block.staging_dir) / "ToHPE"
|
|
||||||
|
|
||||||
def _drop(name: str, body: bytes) -> Path:
|
|
||||||
inbound_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
p = inbound_dir / name
|
|
||||||
p.write_bytes(body)
|
|
||||||
return p
|
|
||||||
|
|
||||||
return _drop
|
|
||||||
|
|
||||||
|
|
||||||
def _make_scheduler(sftp_block, tmp_path) -> Scheduler:
|
|
||||||
"""Build a Scheduler wired to the real (stub) SftpClient."""
|
|
||||||
sched = Scheduler(
|
|
||||||
sftp_block,
|
|
||||||
poll_interval_seconds=60,
|
|
||||||
sftp_block_name="test-block",
|
|
||||||
# Use the real SftpClient — it reads from the stub staging dir.
|
|
||||||
sftp_client_factory=None,
|
|
||||||
)
|
|
||||||
return sched
|
|
||||||
|
|
||||||
|
|
||||||
def _load_999_text() -> str:
|
|
||||||
return (Path(__file__).parent / "fixtures" / "minimal_999.txt").read_text()
|
|
||||||
|
|
||||||
|
|
||||||
def _load_835_text() -> str:
|
|
||||||
return (Path(__file__).parent / "fixtures" / "minimal_835.txt").read_text()
|
|
||||||
|
|
||||||
|
|
||||||
def _load_277ca_text() -> str:
|
|
||||||
return (Path(__file__).parent / "fixtures" / "minimal_277ca.txt").read_text()
|
|
||||||
|
|
||||||
|
|
||||||
def _load_ta1_text() -> str:
|
|
||||||
return (Path(__file__).parent / "fixtures" / "minimal_ta1.txt").read_text()
|
|
||||||
|
|
||||||
|
|
||||||
# ----- tests --------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
class TestSchedulerStatus:
|
|
||||||
def test_not_running_by_default(self, sftp_block):
|
|
||||||
sched = _make_scheduler(sftp_block, Path("/tmp"))
|
|
||||||
st = sched.status()
|
|
||||||
assert st.running is False
|
|
||||||
assert st.poll_count == 0
|
|
||||||
assert st.last_poll_at is None
|
|
||||||
|
|
||||||
def test_running_after_start(self, sftp_block):
|
|
||||||
async def _go():
|
|
||||||
sched = _make_scheduler(sftp_block, Path("/tmp"))
|
|
||||||
await sched.start()
|
|
||||||
try:
|
|
||||||
assert sched.is_running() is True
|
|
||||||
finally:
|
|
||||||
await sched.stop()
|
|
||||||
asyncio.run(_go())
|
|
||||||
|
|
||||||
|
|
||||||
class TestTickOnEmptyInbox:
|
|
||||||
def test_tick_with_no_files_records_zero(self, sftp_block):
|
|
||||||
async def _go():
|
|
||||||
sched = _make_scheduler(sftp_block, Path("/tmp"))
|
|
||||||
result = await sched.tick()
|
|
||||||
assert isinstance(result, TickResult)
|
|
||||||
assert result.files_seen == 0
|
|
||||||
assert result.files_processed == 0
|
|
||||||
assert result.files_skipped == 0
|
|
||||||
assert result.files_errored == 0
|
|
||||||
assert result.finished_at is not None
|
|
||||||
assert result.errors == []
|
|
||||||
asyncio.run(_go())
|
|
||||||
|
|
||||||
|
|
||||||
class TestTickRoutesFiles:
|
|
||||||
def test_999_file_processed(self, sftp_block, _drop_file):
|
|
||||||
async def _go():
|
|
||||||
_drop_file("TP11525703-837P_M019048402-20260618130000000-1of1_999.x12",
|
|
||||||
_load_999_text().encode("utf-8"))
|
|
||||||
sched = _make_scheduler(sftp_block, Path("/tmp"))
|
|
||||||
result = await sched.tick()
|
|
||||||
assert result.files_seen == 1
|
|
||||||
assert result.files_processed == 1
|
|
||||||
assert result.files_errored == 0
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
rows = (
|
|
||||||
session.query(ProcessedInboundFile)
|
|
||||||
.filter_by(name="TP11525703-837P_M019048402-20260618130000000-1of1_999.x12")
|
|
||||||
.all()
|
|
||||||
)
|
|
||||||
assert len(rows) == 1
|
|
||||||
assert rows[0].status == STATUS_OK
|
|
||||||
assert rows[0].parser_used == "parse_999"
|
|
||||||
asyncio.run(_go())
|
|
||||||
|
|
||||||
def test_ta1_file_processed(self, sftp_block, _drop_file):
|
|
||||||
async def _go():
|
|
||||||
_drop_file("TP11525703-837P_M019048402-20260618130000000-1of1_TA1.x12",
|
|
||||||
_load_ta1_text().encode("utf-8"))
|
|
||||||
sched = _make_scheduler(sftp_block, Path("/tmp"))
|
|
||||||
result = await sched.tick()
|
|
||||||
assert result.files_processed == 1, result.errors
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
row = (
|
|
||||||
session.query(ProcessedInboundFile)
|
|
||||||
.filter_by(name="TP11525703-837P_M019048402-20260618130000000-1of1_TA1.x12")
|
|
||||||
.first()
|
|
||||||
)
|
|
||||||
assert row is not None
|
|
||||||
assert row.status == STATUS_OK
|
|
||||||
assert row.parser_used == "parse_ta1"
|
|
||||||
asyncio.run(_go())
|
|
||||||
|
|
||||||
def test_unknown_file_type_marked_skipped(self, sftp_block, _drop_file):
|
|
||||||
async def _go():
|
|
||||||
# 270 (eligibility request) is in the HCPF allowed set but
|
|
||||||
# NOT in ROUTED_FILE_TYPES — Cyclone doesn't have a 270 parser.
|
|
||||||
_drop_file("TP11525703-837P_M019048402-20260618130000000-1of1_270.x12",
|
|
||||||
b"some bytes")
|
|
||||||
sched = _make_scheduler(sftp_block, Path("/tmp"))
|
|
||||||
result = await sched.tick()
|
|
||||||
assert result.files_skipped == 1
|
|
||||||
assert result.files_processed == 0
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
row = (
|
|
||||||
session.query(ProcessedInboundFile)
|
|
||||||
.filter_by(name="TP11525703-837P_M019048402-20260618130000000-1of1_270.x12")
|
|
||||||
.first()
|
|
||||||
)
|
|
||||||
assert row is not None
|
|
||||||
assert row.status == STATUS_SKIPPED
|
|
||||||
assert row.error_message and "270" in row.error_message
|
|
||||||
asyncio.run(_go())
|
|
||||||
|
|
||||||
def test_filename_not_matching_hcpf_marked_skipped(self, sftp_block, _drop_file):
|
|
||||||
async def _go():
|
|
||||||
_drop_file("random.txt", b"garbage")
|
|
||||||
sched = _make_scheduler(sftp_block, Path("/tmp"))
|
|
||||||
result = await sched.tick()
|
|
||||||
assert result.files_skipped == 1
|
|
||||||
asyncio.run(_go())
|
|
||||||
|
|
||||||
def test_parse_error_marked_error(self, sftp_block, _drop_file):
|
|
||||||
async def _go():
|
|
||||||
# Valid filename but malformed body — parser raises.
|
|
||||||
_drop_file("TP11525703-837P_M019048402-20260618130000000-1of1_999.x12",
|
|
||||||
b"this is not a 999 file at all")
|
|
||||||
sched = _make_scheduler(sftp_block, Path("/tmp"))
|
|
||||||
result = await sched.tick()
|
|
||||||
assert result.files_errored == 1
|
|
||||||
assert result.files_processed == 0
|
|
||||||
|
|
||||||
with db.SessionLocal()() as session:
|
|
||||||
row = (
|
|
||||||
session.query(ProcessedInboundFile)
|
|
||||||
.filter_by(name="TP11525703-837P_M019048402-20260618130000000-1of1_999.x12")
|
|
||||||
.first()
|
|
||||||
)
|
|
||||||
assert row.status == STATUS_ERROR
|
|
||||||
assert row.error_message
|
|
||||||
asyncio.run(_go())
|
|
||||||
|
|
||||||
|
|
||||||
class TestTickIdempotent:
|
|
||||||
def test_second_tick_does_not_reprocess(self, sftp_block, _drop_file):
|
|
||||||
async def _go():
|
|
||||||
_drop_file("TP11525703-837P_M019048402-20260618130000000-1of1_TA1.x12",
|
|
||||||
_load_ta1_text().encode("utf-8"))
|
|
||||||
sched = _make_scheduler(sftp_block, Path("/tmp"))
|
|
||||||
r1 = await sched.tick()
|
|
||||||
r2 = await sched.tick()
|
|
||||||
assert r1.files_processed == 1
|
|
||||||
assert r2.files_seen == 1 # still lists it
|
|
||||||
assert r2.files_processed == 0 # but skips — already done
|
|
||||||
asyncio.run(_go())
|
|
||||||
|
|
||||||
|
|
||||||
class TestSchedulerStartStop:
|
|
||||||
def test_start_then_stop_returns_to_not_running(self, sftp_block):
|
|
||||||
async def _go():
|
|
||||||
sched = _make_scheduler(sftp_block, Path("/tmp"))
|
|
||||||
await sched.start()
|
|
||||||
assert sched.is_running()
|
|
||||||
await sched.stop()
|
|
||||||
assert not sched.is_running()
|
|
||||||
asyncio.run(_go())
|
|
||||||
|
|
||||||
def test_double_start_is_idempotent(self, sftp_block):
|
|
||||||
async def _go():
|
|
||||||
sched = _make_scheduler(sftp_block, Path("/tmp"))
|
|
||||||
await sched.start()
|
|
||||||
await sched.start() # no-op
|
|
||||||
assert sched.is_running()
|
|
||||||
await sched.stop()
|
|
||||||
asyncio.run(_go())
|
|
||||||
|
|
||||||
def test_stop_when_not_running_is_safe(self, sftp_block):
|
|
||||||
async def _go():
|
|
||||||
sched = _make_scheduler(sftp_block, Path("/tmp"))
|
|
||||||
await sched.stop() # no-op
|
|
||||||
asyncio.run(_go())
|
|
||||||
|
|
||||||
|
|
||||||
class TestModuleSingleton:
|
|
||||||
def test_get_scheduler_raises_if_not_configured(self):
|
|
||||||
sched_mod.reset_scheduler_for_tests()
|
|
||||||
with pytest.raises(RuntimeError, match="not configured"):
|
|
||||||
sched_mod.get_scheduler()
|
|
||||||
|
|
||||||
def test_configure_then_get_returns_same_instance(self, sftp_block):
|
|
||||||
sched_mod.reset_scheduler_for_tests()
|
|
||||||
s = sched_mod.configure_scheduler(sftp_block, sftp_block_name="t")
|
|
||||||
try:
|
|
||||||
assert sched_mod.get_scheduler() is s
|
|
||||||
finally:
|
|
||||||
sched_mod.reset_scheduler_for_tests()
|
|
||||||
|
|
||||||
|
|
||||||
class TestRoutedFileTypes:
|
|
||||||
"""Frozen-set guards — adding a new routed type without updating
|
|
||||||
the dispatch table would silently skip every inbound file of that
|
|
||||||
type. These tests are the regression net."""
|
|
||||||
|
|
||||||
def test_handlers_cover_all_routed_types(self):
|
|
||||||
assert set(HANDLERS.keys()) == ROUTED_FILE_TYPES
|
|
||||||
@@ -1,225 +0,0 @@
|
|||||||
"""SP19 — Security middleware + health probe tests.
|
|
||||||
|
|
||||||
Covers each middleware in isolation (using FastAPI's TestClient with a
|
|
||||||
minimal app) and the integration into the real ``cyclone.api`` app
|
|
||||||
(headers present on every response, /api/health returns the rich
|
|
||||||
snapshot).
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
from typing import Any, Callable
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
from fastapi import FastAPI, Request
|
|
||||||
from fastapi.testclient import TestClient
|
|
||||||
|
|
||||||
from cyclone.security import (
|
|
||||||
DEFAULT_MAX_BODY_BYTES,
|
|
||||||
DEFAULT_RATE_LIMIT_PER_MIN,
|
|
||||||
BodySizeLimitMiddleware,
|
|
||||||
RateLimitMiddleware,
|
|
||||||
SecurityHeadersMiddleware,
|
|
||||||
get_health_snapshot,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Test apps — built inline per-test to avoid state pollution between tests.
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _echo_app_with(
|
|
||||||
*middlewares: Callable[..., Any],
|
|
||||||
) -> FastAPI:
|
|
||||||
"""Build a tiny FastAPI app with the given middleware chain.
|
|
||||||
|
|
||||||
``middlewares`` are listed outermost-first (the first element
|
|
||||||
runs first on the request). Starlette's ``add_middleware``
|
|
||||||
*prepends*, so we add in reverse to preserve "outermost first"
|
|
||||||
in the public API.
|
|
||||||
"""
|
|
||||||
app = FastAPI()
|
|
||||||
|
|
||||||
@app.post("/echo")
|
|
||||||
async def echo(request: Request):
|
|
||||||
body = await request.body()
|
|
||||||
return {"received_bytes": len(body)}
|
|
||||||
|
|
||||||
@app.get("/api/health")
|
|
||||||
async def h():
|
|
||||||
return {"status": "ok"}
|
|
||||||
|
|
||||||
for mw in reversed(middlewares):
|
|
||||||
app.add_middleware(mw)
|
|
||||||
return app
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# BodySizeLimitMiddleware
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_body_size_accepts_under_limit():
|
|
||||||
"""500 bytes is under the 1024-byte limit; body passes through."""
|
|
||||||
app = _echo_app_with(
|
|
||||||
SecurityHeadersMiddleware,
|
|
||||||
lambda a: BodySizeLimitMiddleware(a, max_bytes=1024),
|
|
||||||
)
|
|
||||||
client = TestClient(app)
|
|
||||||
resp = client.post("/echo", content=b"x" * 500)
|
|
||||||
assert resp.status_code == 200, resp.text
|
|
||||||
assert resp.json() == {"received_bytes": 500}
|
|
||||||
|
|
||||||
|
|
||||||
def test_body_size_rejects_over_content_length():
|
|
||||||
"""A 200-byte body against a 100-byte cap returns 413."""
|
|
||||||
app = _echo_app_with(
|
|
||||||
SecurityHeadersMiddleware,
|
|
||||||
lambda a: BodySizeLimitMiddleware(a, max_bytes=100),
|
|
||||||
)
|
|
||||||
client = TestClient(app)
|
|
||||||
resp = client.post("/echo", content=b"x" * 200)
|
|
||||||
assert resp.status_code == 413
|
|
||||||
body = resp.json()
|
|
||||||
assert body["error"] == "body_too_large"
|
|
||||||
assert "100" in body["detail"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_body_size_default_is_50mb():
|
|
||||||
"""The default cap is 50 MB so even large prodfiles fit."""
|
|
||||||
assert DEFAULT_MAX_BODY_BYTES == 50 * 1024 * 1024
|
|
||||||
|
|
||||||
|
|
||||||
def test_body_size_rejects_bad_content_length():
|
|
||||||
"""A non-integer Content-Length is rejected as 400, not 500."""
|
|
||||||
sent: list = []
|
|
||||||
inner_app = FastAPI()
|
|
||||||
|
|
||||||
@inner_app.post("/echo")
|
|
||||||
async def echo(request: Request):
|
|
||||||
return {"ok": True}
|
|
||||||
|
|
||||||
app_instance = BodySizeLimitMiddleware(inner_app, max_bytes=100)
|
|
||||||
|
|
||||||
async def fake_receive():
|
|
||||||
return {"type": "http.request", "body": b"", "more_body": False}
|
|
||||||
|
|
||||||
async def fake_send(msg):
|
|
||||||
sent.append(msg)
|
|
||||||
|
|
||||||
scope = {
|
|
||||||
"type": "http",
|
|
||||||
"method": "POST",
|
|
||||||
"path": "/echo",
|
|
||||||
"headers": [(b"content-length", b"not-a-number")],
|
|
||||||
"query_string": b"",
|
|
||||||
}
|
|
||||||
asyncio.run(app_instance(scope, fake_receive, fake_send))
|
|
||||||
start = next(m for m in sent if m["type"] == "http.response.start")
|
|
||||||
assert start["status"] == 400
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# RateLimitMiddleware
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_rate_limit_default_is_300_per_min():
|
|
||||||
assert DEFAULT_RATE_LIMIT_PER_MIN == 300
|
|
||||||
|
|
||||||
|
|
||||||
def test_rate_limit_allows_under_threshold():
|
|
||||||
"""A few requests under the per-minute limit are allowed."""
|
|
||||||
app = _echo_app_with(
|
|
||||||
SecurityHeadersMiddleware,
|
|
||||||
lambda a: RateLimitMiddleware(a, per_minute=5),
|
|
||||||
)
|
|
||||||
client = TestClient(app)
|
|
||||||
for _ in range(5):
|
|
||||||
resp = client.post("/echo", content=b"x")
|
|
||||||
assert resp.status_code == 200, resp.text
|
|
||||||
|
|
||||||
|
|
||||||
def test_rate_limit_blocks_over_threshold():
|
|
||||||
"""The 6th request within a 60s window is rate-limited."""
|
|
||||||
app = _echo_app_with(
|
|
||||||
SecurityHeadersMiddleware,
|
|
||||||
lambda a: RateLimitMiddleware(a, per_minute=3),
|
|
||||||
)
|
|
||||||
client = TestClient(app)
|
|
||||||
for _ in range(3):
|
|
||||||
assert client.post("/echo", content=b"x").status_code == 200
|
|
||||||
resp = client.post("/echo", content=b"x")
|
|
||||||
assert resp.status_code == 429
|
|
||||||
assert resp.json()["error"] == "rate_limited"
|
|
||||||
|
|
||||||
|
|
||||||
def test_rate_limit_exempts_health_probes():
|
|
||||||
"""A load balancer hammering /api/health should not trip the limiter."""
|
|
||||||
app = _echo_app_with(
|
|
||||||
SecurityHeadersMiddleware,
|
|
||||||
lambda a: RateLimitMiddleware(a, per_minute=2),
|
|
||||||
)
|
|
||||||
client = TestClient(app)
|
|
||||||
for _ in range(20):
|
|
||||||
assert client.get("/api/health").status_code == 200
|
|
||||||
assert client.post("/echo", content=b"x").status_code == 200
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# SecurityHeadersMiddleware
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_security_headers_present_on_200():
|
|
||||||
app = _echo_app_with(SecurityHeadersMiddleware)
|
|
||||||
client = TestClient(app)
|
|
||||||
resp = client.get("/api/health")
|
|
||||||
assert resp.headers["X-Content-Type-Options"] == "nosniff"
|
|
||||||
assert resp.headers["X-Frame-Options"] == "DENY"
|
|
||||||
assert resp.headers["Referrer-Policy"] == "same-origin"
|
|
||||||
assert "default-src 'none'" in resp.headers["Content-Security-Policy"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_security_headers_present_on_error_response():
|
|
||||||
"""413/429 still carry the security headers."""
|
|
||||||
# Define a tiny factory so Starlette can introspect it as a class.
|
|
||||||
class _BoundBodySize(BodySizeLimitMiddleware):
|
|
||||||
def __init__(self, app): # type: ignore[no-untyped-def]
|
|
||||||
super().__init__(app, max_bytes=10)
|
|
||||||
|
|
||||||
app = _echo_app_with(SecurityHeadersMiddleware, _BoundBodySize)
|
|
||||||
client = TestClient(app)
|
|
||||||
resp = client.post("/echo", content=b"x" * 100)
|
|
||||||
assert resp.status_code == 413, resp.text
|
|
||||||
assert resp.headers["X-Content-Type-Options"] == "nosniff"
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# get_health_snapshot
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def test_health_snapshot_basic_shape():
|
|
||||||
snap = get_health_snapshot()
|
|
||||||
d = snap.to_dict()
|
|
||||||
assert "status" in d
|
|
||||||
assert "version" in d
|
|
||||||
assert "db" in d
|
|
||||||
assert "scheduler" in d
|
|
||||||
assert "pubsub" in d
|
|
||||||
assert "batch" in d
|
|
||||||
|
|
||||||
|
|
||||||
def test_health_snapshot_db_ok_in_tests():
|
|
||||||
"""The conftest DB fixture is live; ``SELECT 1`` works."""
|
|
||||||
snap = get_health_snapshot()
|
|
||||||
assert snap.db.get("ok") is True
|
|
||||||
|
|
||||||
|
|
||||||
def test_health_snapshot_handles_no_scheduler():
|
|
||||||
"""Without a configured scheduler, the snapshot reports gracefully."""
|
|
||||||
snap = get_health_snapshot()
|
|
||||||
sched = snap.scheduler
|
|
||||||
assert "running" in sched or "configured" in sched
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user