tyler/cyclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sp27): server-aggregate Dashboard KPIs so 100-row sample doesn't lie

Browse Source 
The Dashboard was hardcoded to useClaims({ limit: 100 }) and reduce
KPIs client-side. With 60k+ claims in production, every tile
(Billed $940K, Received $59K, Denial rate, Pending AR, monthly
sparkline, top providers, recent denials) was computed from a
0.16% sample of the dataset — silently wrong numbers on the
operator's primary view.

Fix: add GET /api/dashboard/kpis that aggregates server-side in
one read over the entire claim population. The new useDashboardKpis
hook consumes it and polls every 60s. Dashboard.tsx drops
useClaims({limit:100}) + useProviders() + the client-side
buildMonthly reduce.

Backend:
- store.py: dashboard_kpis() — one Claim query (selectinload on
  batch to avoid N+1) + one bulk Remittance lookup, Python reduce
  over the full population. Zero-filled response for empty DB.
- api.py: GET /api/dashboard/kpis behind matrix_gate, query-param
  clamps (1..24 months, 0..50 top_n_*).

Frontend:
- api.ts: DashboardKpis types + getDashboardKpis() wrapper.
- useDashboardKpis.ts: TanStack Query hook, 60s refetchInterval,
  bypass to data:undefined when not configured.
- Dashboard.tsx: switched to useDashboardKpis, extracted
  ZERO_TOTALS constant, dropped the buildMonthly helper.

Tests:
- backend/tests/test_dashboard_kpis.py: 12 tests covering empty DB,
  matched-remit math, pending-state semantics, monthly binning,
  top-providers/top-denials sort + cap, orphan-claim defensive
  guard, HTTP wiring + param validation.
- src/hooks/useDashboardKpis.test.ts: 3 tests for the hook
  contract (configured path, unconfigured fallback, param
  passthrough).
- src/pages/Dashboard.test.tsx: wrapped renders in
  QueryClientProvider + stubbed useAuth + isConfigured=false. This
  fixes 3 pre-existing Dashboard test failures (the page never had
  a QueryClient set up because useClaims/useProviders were the
  first useQuery hooks in the page).

Reviewer fixes (same commit):
1. topDenials sort placed empty submissionDate claims first under
   reverse-lex. Drop them at append time.
2. r.batch lazy-load → N+1 on 60k rows. selectinload(Claim.batch).
3. pending_states rebuilt per call as a mutable set — moved to
   module-level _DASHBOARD_PENDING_STATES frozenset.
4. Module-level ProviderORM import inconsistent with the
   "local import inside the function" pattern — moved inline.
This commit is contained in:
Nora
2026-06-29 12:51:28 -06:00
parent f5d119fbe7
commit 59c3275adf
8 changed files with 1173 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/cyclone/api.py
+29
View File
@@ -88,6 +88,7 @@ from cyclone.store import (
AlreadyMatchedError,
BatchRecord,
InvalidStateError,
dashboard_kpis,
store,
utcnow,
)
@@ -2291,6 +2292,34 @@ def get_remittance(remittance_id: str) -> dict:
return body
@app.get("/api/dashboard/kpis", dependencies=[Depends(matrix_gate)])
def get_dashboard_kpis(
months: int = Query(6, ge=1, le=24),
top_n_providers: int = Query(4, ge=0, le=50),
top_n_denials: int = Query(5, ge=0, le=50),
) -> dict:
"""Server-aggregated Dashboard KPIs over the whole claim population.
Backs the Dashboard's "Claims / Billed / Received / Pending AR /
Denial rate" tiles + the monthly sparkline series + the
top-providers and top-denials lists.
Why this exists instead of ``GET /api/claims?limit=N``:
The Dashboard's KPIs are aggregates over *every* claim — billed,
received, denial rate, pending count, monthly billed/received. With
60k+ claims in production, paginating ``/api/claims`` and reducing
client-side silently produces wrong numbers (denial rate sampled,
billed summed from the first 100 rows). This endpoint does the
aggregation server-side in a single read so the Dashboard's numbers
are always correct regardless of dataset size.
"""
return dashboard_kpis(
months=months,
top_n_providers=top_n_providers,
top_n_denials=top_n_denials,
)
@app.get("/api/providers", dependencies=[Depends(matrix_gate)])
def list_providers(
request: Request,
backend/src/cyclone/store.py
+291
View File
@@ -2578,6 +2578,297 @@ def _provider_orm_to_dict(row) -> dict:
}
# ---------------------------------------------------------------------------
# SP27 Task 13: Dashboard aggregate KPIs.
#
# The Dashboard's "Billed / Received / Denial rate / Pending AR" tiles are
# computed from the *whole* claim population, not a sample. With 60k+ claims
# in production, fetching ``/api/claims?limit=100`` and reducing in JS would
# silently produce wrong numbers (denial rate sampled, billed summed from
# 100 rows). This module-level function does the aggregation server-side in
# a single session and returns a small structured payload the Dashboard
# can render directly.
#
# Performance: one ``SELECT * FROM claims`` (no pagination) + one
# ``SELECT id, total_paid FROM remittances WHERE id IN (...)`` for matched
# remits. SQLite returns 60k claim rows in ~30ms on the development
# machine; the Python reduce is microseconds. If the dataset grows past
# ~500k claims we'd want a SQL-side ``GROUP BY month`` instead — but at
# that volume the dashboard should probably be backed by a materialized
# view, not a live query.
# ---------------------------------------------------------------------------
def _claim_state_str(claim: Claim) -> str:
"""Stringify a Claim's ``state`` regardless of enum vs raw str storage."""
st = claim.state
return st.value if hasattr(st, "value") else str(st)
# Claim states counted toward the Dashboard's "pending" tile. SUBMITTED
# is the initial post-parse state; REJECTED is the 999 envelope-level
# rejection that means the payer never saw the claim. Both are
# "outstanding adjudication" from the operator's POV; ``denied`` is
# payer adjudication and lives in its own tile.
_DASHBOARD_PENDING_STATES: frozenset[str] = frozenset({"submitted", "rejected"})
def dashboard_kpis(
*,
months: int = 6,
top_n_providers: int = 4,
top_n_denials: int = 5,
) -> dict:
"""Compute Dashboard KPIs over the entire claim/remittance population.
Parameters
----------
months
Number of trailing calendar months to include in the ``monthly``
sparkline series (default 6 — matches the existing frontend
``MONTHS_BACK`` constant).
top_n_providers
How many providers to include in the ``topProviders`` array
(default 4 — matches the existing Dashboard layout).
top_n_denials
How many most-recent denied claims to include in the
``topDenials`` array (default 5).
Returns
-------
dict with keys:
- ``totals``: aggregate counts + dollar sums + rates for the whole DB.
- ``monthly``: list of ``{month, label, count, billed, received,
denied, denialRate, ar}`` dicts, oldest-first, length = ``months``.
``ar`` is the running outstanding accounts-receivable (billed -
received) carried forward across months, clamped at zero.
- ``topProviders``: list of ``{npi, label, claimCount, billed,
denied}`` dicts, sorted by claimCount desc.
- ``topDenials``: list of ``{id, patientName, billedAmount,
denialReason, submissionDate}`` dicts, sorted by submissionDate
desc, capped at ``top_n_denials``.
Notes
-----
- Empty DB returns zero-filled aggregates, an empty ``monthly`` array
(one entry per requested month), an empty ``topProviders`` array,
and an empty ``topDenials`` array.
- ``received`` for a claim is sourced from the matched Remittance's
``total_paid`` column. Claims without a matched remittance
contribute 0 to the received sum (and thus inflate the
outstanding-AR figure, which is the correct operator-visible
semantic — "money we haven't been told was paid yet").
"""
# Pre-build the trailing-N-months skeleton so the response always has
# exactly ``months`` entries even when the DB is empty or sparse.
now = datetime.now(timezone.utc)
skeleton: list[dict] = []
for i in range(months - 1, -1, -1):
d = now.replace(day=1)
# Walk backwards N months without ``relativedelta``.
for _ in range(i):
prev_month = d.month - 1
if prev_month == 0:
d = d.replace(year=d.year - 1, month=12)
else:
d = d.replace(month=prev_month)
skeleton.append({
"month": f"{d.year:04d}-{d.month:02d}",
"label": d.strftime("%b"),
"count": 0,
"billed": 0.0,
"received": 0.0,
"denied": 0,
"ar": 0.0,
})
skeleton_index = {entry["month"]: entry for entry in skeleton}
with db.SessionLocal()() as s:
# ``Claim.batch`` is a lazy ``relationship`` (default
# ``lazy="select"``); without ``selectinload`` each access to
# ``r.batch`` in the reduce loop below issues a fresh
# ``SELECT ... FROM batches WHERE id=?``. ``selectinload``
# pulls every distinct batch in one round-trip instead of N+1
# — critical for the 60k-claim dataset.
from sqlalchemy.orm import selectinload
claims: list[Claim] = (
s.query(Claim)
.options(selectinload(Claim.batch))
.all()
)
# Bulk-load matched-remit total_paid so a 60k-claim DB doesn't
# produce a 60k-query N+1.
matched_ids = [
r.matched_remittance_id
for r in claims
if r.matched_remittance_id
]
received_by_remit: dict[str, float] = {}
if matched_ids:
for rid, total_paid in (
s.query(Remittance.id, Remittance.total_paid)
.filter(Remittance.id.in_(matched_ids))
.all()
):
received_by_remit[rid] = float(total_paid or 0)
# Per-provider accumulator for the topProviders leaderboard.
provider_counts: dict[str, int] = {}
provider_billed: dict[str, float] = {}
provider_denied: dict[str, int] = {}
# Per-month accumulator + totals in a single pass.
total_count = 0
total_billed = 0.0
total_received = 0.0
denied_count = 0
pending_count = 0
# Collect denied candidates as we walk so we don't issue a
# second pass for the topDenials array.
denied_candidates: list[dict] = []
for r in claims:
billed = float(r.charge_amount or 0)
received = received_by_remit.get(r.matched_remittance_id, 0.0)
state_str = _claim_state_str(r)
total_count += 1
total_billed += billed
total_received += received
if state_str == "denied":
denied_count += 1
# Drop denied claims with no ``submissionDate`` — they'd
# sort first under reverse-lex (empty string < ISO) and
# render as "Invalid Date" in the Dashboard. A denial
# without a batch is exceptional and operator-irrelevant
# for the "recent denials" widget.
if r.batch is None or r.batch.parsed_at is None:
pass
else:
raw = r.raw_json or {}
sub = raw.get("subscriber", {})
denied_candidates.append({
"id": r.id,
"patientName": (
f"{sub.get('first_name', '')} "
f"{sub.get('last_name', '')}".strip()
),
"billedAmount": billed,
"denialReason": r.rejection_reason,
"submissionDate": (
r.batch.parsed_at
.isoformat()
.replace("+00:00", "Z")
),
})
if state_str in _DASHBOARD_PENDING_STATES:
pending_count += 1
# Monthly bin. ``submissionDate`` lives on the parent Batch
# (all claims in a batch share a parsed_at). Use UTC year-month
# to match the skeleton.
if r.batch is not None and r.batch.parsed_at is not None:
pa = r.batch.parsed_at
key = f"{pa.year:04d}-{pa.month:02d}"
bucket = skeleton_index.get(key)
if bucket is not None:
bucket["count"] += 1
bucket["billed"] += billed
bucket["received"] += received
if state_str == "denied":
bucket["denied"] += 1
# Provider bin (keyed on NPI).
npi = r.provider_npi or ""
if npi:
provider_counts[npi] = provider_counts.get(npi, 0) + 1
provider_billed[npi] = provider_billed.get(npi, 0.0) + billed
if state_str == "denied":
provider_denied[npi] = provider_denied.get(npi, 0) + 1
# Compute denial rate per month + running AR.
running_ar = 0.0
for entry in skeleton:
if entry["count"] > 0:
entry["denialRate"] = (entry["denied"] / entry["count"]) * 100.0
else:
entry["denialRate"] = 0.0
running_ar = max(0.0, running_ar + entry["billed"] - entry["received"])
entry["ar"] = running_ar
# Resolve top provider labels from the Provider table in one
# round-trip. NPIs without a Provider row still appear in the
# leaderboard with an empty label so operators can see
# "unknown-NPI" claims are coming from somewhere.
# Local import keeps the module-level ``cyclone.providers.Provider``
# Pydantic DTO and the SQLAlchemy ``cyclone.db.Provider`` ORM
# separate (same pattern as ``list_providers`` / ``upsert_provider``).
provider_labels: dict[str, str] = {}
if provider_counts:
from cyclone.db import Provider as ProviderORM
for npi, label in (
s.query(ProviderORM.npi, ProviderORM.label).filter(
ProviderORM.npi.in_(provider_counts.keys())
).all()
):
provider_labels[npi] = label or ""
top_providers = sorted(
provider_counts.items(),
key=lambda kv: kv[1],
reverse=True,
)[: max(0, top_n_providers)]
top_providers_out = [
{
"npi": npi,
"label": provider_labels.get(npi, ""),
"claimCount": count,
"billed": round(provider_billed.get(npi, 0.0), 2),
"denied": provider_denied.get(npi, 0),
}
for npi, count in top_providers
]
# Top denials = most recently submitted denied claims, capped at
# ``top_n_denials``. submissionDate is ISO-8601 so lex sort ==
# chronological sort when timestamps share a tz.
denied_candidates.sort(key=lambda d: d["submissionDate"], reverse=True)
top_denials = denied_candidates[: max(0, top_n_denials)]
total_denial_rate = (
(denied_count / total_count) * 100.0 if total_count > 0 else 0.0
)
return {
"totals": {
"count": total_count,
"billed": round(total_billed, 2),
"received": round(total_received, 2),
"outstandingAr": round(max(0.0, total_billed - total_received), 2),
"denied": denied_count,
"denialRate": round(total_denial_rate, 4),
"pending": pending_count,
},
"monthly": [
{
"month": e["month"],
"label": e["label"],
"count": e["count"],
"billed": round(e["billed"], 2),
"received": round(e["received"], 2),
"denied": e["denied"],
"denialRate": round(e["denialRate"], 4),
"ar": round(e["ar"], 2),
}
for e in skeleton
],
"topProviders": top_providers_out,
"topDenials": top_denials,
}
def _payer_orm_to_dict(row) -> dict:
return {
"payer_id": row.payer_id,